Centralized Log Management using the ELK stack: ElasticSearch, Logstash, Kibana

1. Log Management with ELK
ELASTICSEARCH, LOGSTASH, KIBANA FOR CENTRALIZED LOGS

2. Purpose
Centralized Log Management
◦ Collect, Parse and Filter using Logstash
◦ Store, Index and Search using Elasticsearch
◦ Visualize using Kibana
Full open source stack
◦ Use for free
◦ Support plan from Elasticsearch company

3. Elasticsearch
Real-time search engine
◦ Based on Apache Solr/Lucene
◦ Pure Java
◦ Document database
◦ Advanced text indexing
◦ Fuzzy search
◦ Replication/Sharding for true scalability

4. Logstash
JRuby Based log processor
Pluggable event pipeline
◦ Input plugins
◦ Filter plugins
◦ Codec plugins
◦ Output plugins
DevOps Comunity
◦ Mix of developers, operations and system administrators

5. Kibana
Browser based dashboard for ElasticSearch
Visualization of query results
◦ Time Charts
◦ Filter any field
◦ Compare subsets

6. Logstash pipeline
Define input, filters and outputs
Simple configuration file
Ruby syntax

7. Logstash not just for logs
Interpretes different log formats
◦ Syslog messages
◦ Log4j with full details
◦ Apache log files
Other event types too
◦ Ganglia server monitoring events
◦ SNMP events
◦ Windows EventLog
Pre-proces before sending
◦ lumberjack

8. Logstash Inputs
Rsyslog via TCP/UDP
Log4j appender
JMX Listener
Logstash-forwarder
File tails
SNMP
Ganglia
…

9. Logstash Filters

10. Grok Filter
Readable regex
Predefined patterns for common log data
Extract to properties
◦ Indexed properties

11. Metrics Filter
Aggregate metrics
◦ Event rate using sliding windows
◦ 1 min
◦ 5 min
◦ 15 min
◦ Min/max/stddev/percentiles

12. Logstash Outputs

13. Statsd output
Node.js based
◦ Counters
◦ Timers
◦ Graphite frontend

14. Alert outputs
Send alerts
◦ Email
◦ Pagerduty
◦ XMPP/Jabber
◦ Hipchat
◦ Nagios
Use treshold from metrics filter

15. ElasticSearch output
Auto-creates new index per day
Index all recognized fields
Full text index, customizable indexer, mapper

16. Kibana dashboard

17. Scalability
Easy deployment using chef/puppet/docker