Recommended
More Related Content
What's hot
What's hot (20)
Similar to Logging logs with Logstash - Devops MK 10-02-2016
Similar to Logging logs with Logstash - Devops MK 10-02-2016 (20)
Recently uploaded
Recently uploaded (20)
Logging logs with Logstash - Devops MK 10-02-2016
- 1. Logging logs with Logstash Steve Howe – RetailMeNot UK Ltd. showe@rmn.com
- 2. About me.. Devops engineer at RetailMeNot Obsessed with making things better Husband, father, frequently busy
- 4. Why Logstash? Jordan Sissel Noticed a problem
- 5. Why Logstash? Jordan Sissel Noticed a problem
- 6. Why Logstash? Jordan Sissel Noticed a problem
- 7. Why Logstash? Jordan Sissel Noticed a problem
- 8. Why Logstash? Jordan Sissel Noticed a problem
- 9. What does Logstash provide? Empowerment
- 10. What does Logstash provide? Empowerment Aggregation
- 11. What does Logstash provide? Empowerment Aggregation Search
- 12. What does Logstash provide? Empowerment Aggregation Search
- 13. What is Logstash (ELK stack)? Centralization
- 14. What is Logstash (ELK stack)? Centralization Visualization
- 15. What is Logstash (ELK stack)? Centralization Visualization Segregation
- 17. Logstash-forwarder shipper Small package (java) (used to be “lumberjack”)
- 18. Logstash-forwarder shipper Small package (java) (used to be “lumberjack”) Grok processing can happen on client
- 19. Logstash-forwarder shipper Small package (java) (used to be “lumberjack”) Grok processing can happen on client One shipper, multiple logfiles (perms)
- 20. Logstash-forwarder shipper Small package (java) (used to be “lumberjack”) Grok processing can happen on client One shipper, multiple logfiles (perms) Many log formats, one output - JSON
- 22. Redis Message broker SSL encryption for non-VPN networks
- 23. Redis Message broker SSL encryption for non-VPN networks Scalable
- 25. Logstash Indexer Java app Grok processing of logs off app-servers
- 26. Logstash Indexer Java app Grok processing of logs off app-servers Outputs to multiple endpoints, if required
- 27. Logstash Indexer Java app Grok processing of logs off app-servers Outputs to multiple endpoints, if required Scalable
- 29. Elasticsearch Mature, scalable db Document-based index, Query DSL API
- 30. Elasticsearch Mature, scalable db Document-based index, Query DSL API Sharded, clustered
- 31. Elasticsearch Mature, scalable db Document-based index, Query DSL API Sharded, clustered Indices are stored in date format
- 32. Kibana Standalone (used to be simple website)
- 33. Kibana Standalone (used to be simple website) Points directly at the Elasticsearch db
- 34. Kibana Standalone (used to be simple website) Points directly at the Elasticsearch db Solr Lucene syntax
- 35. Kibana Standalone (used to be simple website) Points directly at the Elasticsearch db Solr Lucene syntax Scalable
- 36. Kibana Standalone (used to be simple website) Points directly at the Elasticsearch db Solr Lucene syntax Scalable
- 37. Demo
- 38. Tricks Elasticsearch Curator, single indices
- 39. Tricks Elasticsearch Curator, single indices Outputs to Graphite, Nagios, Cloudwatch, Datadog, GELF, Slack, Hipchat, PagerDuty etc.
- 40. Tricks Elasticsearch Curator, single indices Outputs to Graphite, Nagios, Cloudwatch, Datadog, GELF, Slack, Hipchat, PagerDuty etc. Kibana dashboards
- 41. Gotchas Logstash Elasticsearch client (versions, AWS)
- 42. Gotchas Logstash Elasticsearch client (versions, AWS) Enterprise ready? (utter lack of users/security, streams, alerting)
- 43. Gotchas Logstash Elasticsearch client (versions, AWS) Enterprise ready? (utter lack of users/security, streams, alerting) Graylog – Alternative log engine
- 44. Links Jordan Sissels Github - https://github.com/jordansissel Jordan Sissel pres - http://semicomplete.com/presentations/logstash- scale11x/#/33 Logstash book - http://www.logstashbook.com/ Enterprise ready? (users, streams, alerting)
- 45. Cheers!
Editor's Notes
- Jordan Sissel, dreamhost, a sysadmin who's annoyingly intelligent and imaginative
- Jordan saw a problem in his capacity as sysadmin, he had immense problems reading logs.
- Not hard to look at single server apache logs
- Gets more interesting when you have an nginx server AND apache server
- Gets really interesting with apache, nginx AND mysql
- End up with hero syndrome, single guy that knows regex well drafted in repeatedly
- Empowers developers to look at logs in a coherent, searchable manner
- Aggregates logs from multiple sources into one interface, as long as NTP is setup
- Allows developers to use the flexible Solr Lucene search syntax, can also focus on specific fields of the logfile.
- Allows developers to use the flexible Solr Lucene search syntax, can also focus on specific fields of the logfile.
- Logstash is now owned by Elastic.co, who also own Kibana and Elasticsearch. Centralizes looking at logs, analysis and diagnosis in one place
- ELK stack allows for clear visualization in whatever medium, bar charts, pie charts, etc
- Segregates developers from production environments, keeps them from having to log in or sysadmins to get logs for them.