What problems with logs may we encounter? How to deal with them? This is how we do it at Divante.

1. Logs aggregation
and analysis

2. Agenda
● Problems with logs
● How do we do it at Divante - ELK Stack
o ElasticSearch
o Logstash
o Kibana
o Architecture
o Additional tools
● Summary
● Questions

3. What problems with logs
may we encounter?

4. Problems with logs
No consistent log format
http://blog.tersmitten.nl/how-to-colorize-your-log-files-with-ccze.html

5. Problems with logs
• cat
• grep
• awk
• sed
• tail
• regular expressions
Hampered log analysis, increased response time.
Log search, analysis – old school

6. Problems with logs
Heavy server load, low application performance
• Synchronous, blocking writing
• IO operations burdening the server
• Limited amount of inodes in the file system
• Relatively slow write speed
• Logs cleanup
http://wiki.processmaker.com/index.php/Advanced_Performance_Monitor_Dashboards

7. Problems with logs
Complex architecture - read and write problem
• Read/write on servers after the load balancer
- NFS application?
https://www.digitalocean.com/community/tutorials/5-common-server-setups-for-your-web-application

8. Problems with logs
Continuous monitoring
• Continuous monitoring of all application parameters is not easy
• Technical knowledge is required

9. How do we do it at Divante?
ELK Stack

10. ELK Stack - what is it?
ELK Stack is a set of tools:
It provides a centralized log management in distributed, high-availability systems.

11. E as in ElasticSearch
ElasticSearch
● + NoSQL database
● + Full-text search
● + REST Api (json)
● + Based on Apache Lucene
● + Replication, snapshots
● + Official PHP and JavaScript libraries available
● - No transactions
● Requirements: Java

12. E as in ElasticSearch
Installation and setup
● Java Oracle installation
o sudo apt-get install python-software-properties
o sudo add-apt-repository -y ppa:webupd8team/java
o sudo apt-get update
o sudo apt-get -y install oracle-java8-installer
● ElasticSearch installation
o wget and unzip of the latest distribution available at
https://www.elastic.co/downloads
o Setup in the elasticsearch.yml file
● Running ElasticSearch:
o ./bin/elasticsearch -d

13. L as in Logstash
● Aggregation of logs from multiple sources
● Normalization (parsing, filtering) of the collected data
● Sending the normalized data to various sources
● Requirements: Java
INPUT
FILTER
PARSE
OUTPUT

14. L as in Logstash
Input: (41)
● elasticsearch, imap
● rabbitmq, redis
● sqlite, syslog
● tcp, twitter
● varnishlog, websocket
Output: (55)
● elasticsearch, mongodb,
● redis, email, file, csv
● hipchat, http
● jira, redmine, rabbitmq
● tcp, websocket, zabbix
More at: http://logstash.net/docs/1.4.2/
INPUT
FILTER
PARSE
OUTPUT

15. L as in Logstash
Filters: (50)
● checksum
● csv, date
● elasticsearch
● fingerprint
● geoip, grep, grok
● json, json_encode
● ruby, split
● translate
● urldecode
● useragent, xml
More at: http://logstash.net/docs/1.4.2/
INPUT
FILTER
PARSE
OUTPUT

16. L as in Logstash
GROK filter
● Parsing and analysis of any text
● Grok is the best way to process even the most unstructured data
● Over 120 samples and possibility to create new ones
[ERROR] - 2015/13/03-11:01:31 - 192.168.0.1 - Some error message
[%{GREEDYDATA:message}] - %{DATESTAMP} - %{IP:client} - %{GREEDYDATA:message}

17. L as in Logstash
Installation and setup
● Java Oracle and Logstash installation
o wget and unzip of the latest distribution available at
https://www.elastic.co/downloads
● Setup:
input {
file {
type => "syslog"
path => ["/var/log/auth.log", "/var/log/syslog"]
}}
● Running Logstash:
o ./bin/logstash

18. K as in Kibana
● Data visualization in the form of a web app
● Data search, filters and analysis
● Intuitive interface, not only for the programmers
● Instant sharing and embedding of multiple dashboards
● Easy dashboard adjustment - JSON
● Export of the results

19. K as in Kibana
https://www.elastic.co/blog/kibana-4-literally

20. K as in Kibana
Bar chart:
https://www.elastic.co/blog/kibana-4-beta-2-get-now

21. K as in Kibana
Pie chart:
https://www.elastic.co/blog/kibana-4-for-investigating-pacs-super-pacs-and-your-neighbors

22. K as in Kibana
Histogram:
http://blog.qbox.io/kibana-4-and-elasticsearch-v-1-4-4-and-1-3-9

23. K as in Kibana
Data table:
https://www.elastic.co/blog/kibana-4-for-investigating-pacs-super-pacs-and-your-neighbors

24. K as in Kibana
Geolocation:
https://www.elastic.co/blog/kibana-4-literally

25. K as in Kibana
Relation chart:
http://demo.packetbeat.com/#/dashboard/elasticsearch/Packetbeat%2520Statistics

26. K as in Kibana
Installation and setup
● Kibana installation:
o wget and unzip of the latest distribution available at
https://www.elastic.co/downloads
● Setup:
o Kibana is set up on the default ElasticSearch port - 9300
● Running Kibana:
o ./bin/kibana
● Start in a browser:
o http://YOURDOMAIN.com:5601

27. ELK Stack
ARCHITECTURE

28. ELK - Architecture
Basic:
What is the potential problem?
ElasticSearch REST
API

29. ELK - Architecture
Performance + scalability
Setup with Redis
Providers
Indexer

30. ELK - Architecture
Performance + scalability
Setup with the use of a queuing mechanism
Indexer
Providers

31. Additional
TOOLS

32. Tools
ElasticSearch Plugin: Head
ElasticSearch monitoring and management panel
Installation:
● elasticsearch/bin/plugin -install mobz/elasticsearch-head
● http://localhost:9200/_plugin/head/

33. Tools
ElasticSearch Plugin: Morfologik
Plugin enabling the use of Polish characters
in ElasticSearch queries
Installation:
● cd elasticsearch
● bin/plugin -install com.github.chytreg/elasticsearch-analysis-
morfologik/2.3.1

34. Tools
Marvel
Application for the continuous monitoring of ElasticSearch
http://blog.qbox.io/elasticsearch-marvel-released

35. ELK Stack
Summary
● Consistent log structure
● Centralized log system
● Dashboard supporting the search and analysis processes
● Continuous monitoring
● Reduced response time
● Happy customer receives new charts :)

36. VS
http://blog.rootshell.be/tag/unix/
http://imgbuddy.com/kibana-demo.asp

37. Thank you!
Bartosz Picho
sales@divante.co
divante.co/blog