Gen AI in Business - Global Trends Report 2024.pdf
Monitoring the ELK stack using Zabbix and Grafana (Dennis Kanbier / 26-11-2015)
1. Xifeo ICT | Dennis Kanbier
Monitoring the ELK stack
using Zabbix (and
Grafana)
Dennis Kanbier
26 Nov 2015 - NLZGG
2. Xifeo ICT | Dennis Kanbier
Overview
• Introduction
• What is ELK and why do I want it?
• How to monitor ELK with Zabbix
• Impress people with pretty graphs
• Demo time ( please tweet using #nlzgg! )
3. Xifeo ICT | Dennis Kanbier
Who am I?
• Dennis Kanbier
• dennis.kanbier@xifeo.nl
• Linux Consultant with Xifeo ICT
• “dkanbier” on Zabbix forums and IRC
4. Xifeo ICT | Dennis Kanbier
What is the ELK stack
• Elasticsearch
• Search server
• Based on Apache Lucene
• Logstash
• Data pipeline
• Processes logs and other data
• Plugins
• Kibana
• Web frontend for Elasticsearch
5. Xifeo ICT | Dennis Kanbier
Basic Logstash Pipeline
6. Xifeo ICT | Dennis Kanbier
Apache log example
• 83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/
logstash-monitorama-2013/images/ahiruyaki.png HTTP/1.1" 200 203023
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36”
• A few interesting fields:
• client_ip: 83.149.9.216
• status code: 200
• timestamp: 04/Jan/2015:05:13:42 +0000
• and much more
8. Xifeo ICT | Dennis Kanbier
Now what?
• Use the processed data to do fun things like:
• plot all entries using the [timestamp] field to see how
busy the web server is during the day
• which part of your website gets requested the most
using the [request] field
• monitor the status of your web server using the
[response] field
• Or do even more fun things using plugins…
9. Xifeo ICT | Dennis Kanbier
logstash-filter-geoip
• Use the geo-ip filter plugin while logstash is processing the
message
• Looks up the geographical location of an IP address and adds
this information to the message
• Use Kibana to create a heat map based on the geographical
information in each message to see from which part in the world
your website gets visited
• Very nice to have in case of DDOS attacks originating from
places you don’t expect visitors from
• Or to determine where to up your advertising game
12. Xifeo ICT | Dennis Kanbier
Not always the developers fault
13. Xifeo ICT | Dennis Kanbier
Why developers love ELK
• Logs from all components and servers are available in one central place, easily searchable using
the Kibana web interface
• No more begging the scary Unix administrator for log files
• Log message are enriched with meaningful fields which enables users to:
• query on stack traces
• query on specific thread id’s
• query on log_level
• query on customer_id
• list faults per server/application/cluster/log_type/etc..
• You can easily plot logs, see how fast requests are handled
• Timestamps are normalised, even log entries from different time zones are searchable
chronologically
14. Xifeo ICT | Dennis Kanbier
Example Kibana screen
15. Xifeo ICT | Dennis Kanbier
Monitoring Logstash with
Zabbix
• Used 3 components to generate data and send it
to the Zabbix server:
• logstash-input-heartbeat
• logstash-filter-metrics
• logstash-output-zabbix
16. Xifeo ICT | Dennis Kanbier
logstash-input-heartbeat
• Plugin to generate a heartbeat message at a
configurable interval
• The message contains a field called “clock” with the
epoch time of the moment it was send
• Assign a type of “heartbeat” to the message so we can
use that to filter out the message further in the pipeline
and send it to the Zabbix server
18. Xifeo ICT | Dennis Kanbier
logstash-output-zabbix
• Used to send data to Zabbix
• Requires a configured Zabbix host and trapper
item to receive the data
• These fields must be added to the [@metadata]
field of the message you want to send to the
Zabbix server
20. Xifeo ICT | Dennis Kanbier
Results in Zabbix
• Zabbix host logstash-indexer-1 should already have the item
with key “logstash_heartbeat” configured as a Zabbix
trapper item.
• Logstash now sends the epoch value every 20 seconds to
the logstash_heartbeat trapper item
• You can use this to further process the data in Zabbix:
• trigger when there is too much time between epoch values
• trigger when you do not receive any values for a period of
time
21. Xifeo ICT | Dennis Kanbier
Example item in Zabbix
22. Xifeo ICT | Dennis Kanbier
logstash-filter-metrics
• Plugin to count messages and flush the result on a
specific interval
• Used to monitor logstash performance by counting
messages and store their count value in Zabbix as
Delta (speed per second)
• which gives us the messages this logstash
server handles per second
24. Xifeo ICT | Dennis Kanbier
More results!
• We now have the item “logstash_events” which
shows how many messages we process per
second:
25. Xifeo ICT | Dennis Kanbier
Now about that last slide….
• Presenting reports and graphs are not exactly
Zabbix’ strongest points
• Works all right to view single values like on the
previous slide
• But try to align multiple graphs nicely on one
screen to put on a wall and impress people
26. Xifeo ICT | Dennis Kanbier
Grafana
• Build to present metric data in a simple and nice
manner
• Supports multiple data sources like Graphite and
InfluxdB
• Alexander Zobnin created grafana-zabbix, which
enables us to use the Zabbix database as a
Grafana backend through the Zabbix API
• https://github.com/alexanderzobnin/grafana-zabbix
28. Xifeo ICT | Dennis Kanbier
Things to consider
• Grafana queries the Zabbix database through the
API
• If you have a lot of graphs with a lot of items
these queries are quite expensive
• Especially if you have multiple dashboards
refreshing automatically on a high interval
• Still learning on how to best counter these
challenges
29. Xifeo ICT | Dennis Kanbier
Live demo time!
Did you remember to tweet things with hashtag #nlzgg?