Machine Learning & Big Data Blog

How To Monitor NGINX Using Kibana and Elasticsearch on Docker

3 minute read
Walker Rowe

In this article, I’ll show how to use Kibana to monitor the nginx web server. We will use the nginx Filebeat module and, of course, Elasticsearch.

Kibana is the graphical front-end for Elasticsearch. Filebeat is one of several Elasticsearch data shippers; others are Logstash, Metricbeat, and Packetbeat, plus a couple of specialized ones.

Install Kibana and ElasticSearch

We use Docker since it’s the simplest way to install Kibana.

First, install ElasticSearch:

sudo docker pull docker.elastic.co/elasticsearch/elasticsearch:7.6.2
sudo nohup docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node"
docker.elastic.co/elasticsearch/elasticsearch:7.6.2&

Then you need the docker container ID to install and start Kibana. Run this command to get the container ID:

sudo docker ps

Now get a root shell. Then pass a routable IP address to Kibana using the SERVER_HOST environment variable. Otherwise Kibana will bind to localhost, which does no good since you can’t reach it across the network.

(Important note: To put Kibana on a public IP address, do not follow these steps. Instead, you’ll need to turn on security, which I don’t cover in this article. If you don’t turn on security, people running port scans against your service will find your Kibana install and be able to use it without logging in. Instead, I leave off security, set my firewall rules to allow access only from my designated IP address, and I follow these instructions.)

sudo -i
export SERVER_HOST="172.31.46.15"
nohup docker run --link(container id):elasticsearch -p 5601:5601 kibana:7.6.2&

Now open the console:

http://(ip address):5601/app/kibana#/home

Install Filebeat

You don’t need to install Filebeat in a container as the installation is simple and requires no configuration.

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.6.2-amd64.deb
sudo dpkg -i filebeat-7.6.2-amd64.deb

There is no configuration necessary, unless you have modified the log location in nginx, in which case you would need to put the paths in /etc/filebeat/modules.d/nginx.yml. Assuming you are using the regular nginx layout, just enable the nginx module.

sudo filebeat modules enable nginx

Now run this command to push the filebeat dashboards to Kibana:

sudo filebeat setup --dashboards
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
sudo filebeat setup -e

After a while it will stop, once it has installed the dashboards. So, start Filebeat like this:

sudo service filebeat start

Open the Kibana nginx Dashboard

The Kibana nginx dashboards are basic. For now, we’ll use the dashboards that nginx has created, although you can make your own dashboards for specific use cases. (Tutorial coming soon.)

Go to the dashboards display:

Then type the letters nginx to find the dashboard:

Search

Now you will see charts on the dashboard. Here is the world map showing where your traffic comes from:

Map

This shows the top pages:

Top Pages

This shows page counts by status codes. One observation you can make here as this web server, as any public web server, is constantly being hit by hackers looking to see what vulnerabilities you have left exposed. That’s why there are so many HTTP error 400s (bad request) on this server.

Bad request

Set the time filter as it will default to the last 15 minutes:

Filter

This view gives log details:

Log details

Learn ML with our free downloadable guide

This e-book teaches machine learning in the simplest way possible. This book is for managers, programmers, directors – and anyone else who wants to learn machine learning. We start with very basic stats and algebra and build upon that.


These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.

See an error or have a suggestion? Please let us know by emailing blogs@bmc.com.

Business, Faster than Humanly Possible

BMC empowers 86% of the Forbes Global 50 to accelerate business value faster than humanly possible. Our industry-leading portfolio unlocks human and machine potential to drive business growth, innovation, and sustainable success. BMC does this in a simple and optimized way by connecting people, systems, and data that power the world’s largest organizations so they can seize a competitive advantage.
Learn more about BMC ›

About the author

Walker Rowe

Walker Rowe is an American freelancer tech writer and programmer living in Cyprus. He writes tutorials on analytics and big data and specializes in documenting SDKs and APIs. He is the founder of the Hypatia Academy Cyprus, an online school to teach secondary school children programming. You can find Walker here and here.