Generally you will have an ElasticSearch instance setup where your Kibana will find its information for monitoring your applications. When running your applications on-premise you will use Logstash or equivalent to write log messages to your ElasticSearch. On AWS you can use filebeat to read the log-files and send the information to ElasticSearch.
Only for documentation, this is installed automatically as documented under Elastic Beanstalk configuration.
We have to install filebeat. To do this use the following commands.
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.5.1-x86_64.rpm
sudo yum install filebeat-5.5.1-x86_64.rpm
Filebeat configuration is setup under /etc/filebeat/filebeat.yml. This configuration will determine which .log files will be read and what ElasticSearch hosts to send them to. You can find an example here: https://s3-eu-west-1.amazonaws.com/juvo-aws-filebeat/filebeat.yml
After installing filebeat and setting up the configuration execute the following command:
sudo service filebeat restart
This will (re)start the filebeat service. Running filebeat as a service produces no output on sysout while filebeat is running. However we can run filebeat itself with output using following command:
sudo filebeat.sh -e -d "*"
Specific patterns for filebeat come pre-installed in Kibana. You can find more info on the Elastic site. https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-index-pa...
When deploying on Elastic Beanstalk you can include an ebextension file in your application (which is also checked in to your git!). Under this file you can list files and commands which must be initialized / performed when the EBS instance is booted up or (automatically) restarted.
This filebeat.config file must be placed under resources/.ebextensions in your application. It will create the filebeat.yml configuration file and perform the commands necessary to install and run filebeat. Note that spacing and enters are important, otherwise you will get an yml-exception when deploying your EBS.
You can find an example of a ebextension file at https://s3-eu-west-1.amazonaws.com/juvo-aws-filebeat/filebeat.config.
https://www.elastic.co/blog/using-beats-with-ElasticSearch-on-aws
https://fauie.com/2016/02/02/installing-filebeat-to-ship-data-to-Elastic...
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configur...
https://jtran21.gitbooks.io/ElasticSearch/content/install_beats_agent.html
https://www.elastic.co/guide/en/beats/filebeat/master
https://www.elastic.co/guide/en/ElasticSearch/reference/current/docs.html
https://www.elastic.co/guide/en/kibana/current/console-kibana.html
https://discuss.elastic.co/t/deploying-filebeat-on-aws-beanstalk/72246/2
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html