filebeat banner

Monitoring on AWS with filebeat

Monday, September 25, 2017

Generally you will have an ElasticSearch instance setup where your Kibana will find its information for monitoring your applications. When running your applications on-premise you will use Logstash or equivalent to write log messages to your ElasticSearch. On AWS you can use filebeat to read the log-files and send the information to ElasticSearch.

Install and setup filebeat

Only for documentation, this is installed automatically as documented under Elastic Beanstalk configuration.

We have to install filebeat. To do this use the following commands.

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.5.1-x86_64.rpm
​sudo yum install filebeat-5.5.1-x86_64.rpm

Filebeat configuration is setup under /etc/filebeat/filebeat.yml. This configuration will determine which .log files will be read and what ElasticSearch hosts to send them to. You can find an example here: https://s3-eu-west-1.amazonaws.com/juvo-aws-filebeat/filebeat.yml

After installing filebeat and setting up the configuration execute the following command:

sudo service filebeat restart

This will (re)start the filebeat service. Running filebeat as a service produces no output on sysout while filebeat is running. However we can run filebeat itself with output using following command:

sudo filebeat.sh -e -d "*"

 

Filebeat pattern as default in Kibana

Specific patterns for filebeat come pre-installed in Kibana. You can find more info on the Elastic site.  https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-index-pa...

Elastic Beanstalk configuration

When deploying on Elastic Beanstalk you can include an ebextension file in your application (which is also checked in to your git!). Under this file you can list files and commands which must be initialized / performed when the EBS instance is booted up or (automatically) restarted.

This filebeat.config file must be placed under resources/.ebextensions in your application. It will create the filebeat.yml configuration file and perform the commands necessary to install and run filebeat. Note that spacing and enters are important, otherwise you will get an yml-exception when deploying your EBS.

You can find an example of a ebextension file at https://s3-eu-west-1.amazonaws.com/juvo-aws-filebeat/filebeat.config.

References

https://www.elastic.co/blog/using-beats-with-ElasticSearch-on-aws
https://fauie.com/2016/02/02/installing-filebeat-to-ship-data-to-Elastic...
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configur...
https://jtran21.gitbooks.io/ElasticSearch/content/install_beats_agent.html
https://www.elastic.co/guide/en/beats/filebeat/master
https://www.elastic.co/guide/en/ElasticSearch/reference/current/docs.html
https://www.elastic.co/guide/en/kibana/current/console-kibana.html
https://discuss.elastic.co/t/deploying-filebeat-on-aws-beanstalk/72246/2
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html