2. Agenda
● Problems with logs
● How do we do it at Divante - ELK Stack
o ElasticSearch
o Logstash
o Kibana
o Architecture
o Additional tools
● Summary
● Questions
4. Problems with logs
No consistent log format
http://blog.tersmitten.nl/how-to-colorize-your-log-files-with-ccze.html
5. Problems with logs
• cat
• grep
• awk
• sed
• tail
• regular expressions
Hampered log analysis, increased response time.
Log search, analysis – old school
6. Problems with logs
Heavy server load, low application performance
• Synchronous, blocking writing
• IO operations burdening the server
• Limited amount of inodes in the file system
• Relatively slow write speed
• Logs cleanup
http://wiki.processmaker.com/index.php/Advanced_Performance_Monitor_Dashboards
7. Problems with logs
Complex architecture - read and write problem
• Read/write on servers after the load balancer
- NFS application?
https://www.digitalocean.com/community/tutorials/5-common-server-setups-for-your-web-application
8. Problems with logs
Continuous monitoring
• Continuous monitoring of all application parameters is not easy
• Technical knowledge is required
10. ELK Stack - what is it?
ELK Stack is a set of tools:
It provides a centralized log management in distributed, high-availability systems.
11. E as in ElasticSearch
ElasticSearch
● + NoSQL database
● + Full-text search
● + REST Api (json)
● + Based on Apache Lucene
● + Replication, snapshots
● + Official PHP and JavaScript libraries available
● - No transactions
● Requirements: Java
12. E as in ElasticSearch
Installation and setup
● Java Oracle installation
o sudo apt-get install python-software-properties
o sudo add-apt-repository -y ppa:webupd8team/java
o sudo apt-get update
o sudo apt-get -y install oracle-java8-installer
● ElasticSearch installation
o wget and unzip of the latest distribution available at
https://www.elastic.co/downloads
o Setup in the elasticsearch.yml file
● Running ElasticSearch:
o ./bin/elasticsearch -d
13. L as in Logstash
● Aggregation of logs from multiple sources
● Normalization (parsing, filtering) of the collected data
● Sending the normalized data to various sources
● Requirements: Java
INPUT
FILTER
PARSE
OUTPUT
15. L as in Logstash
Filters: (50)
● checksum
● csv, date
● elasticsearch
● fingerprint
● geoip, grep, grok
● json, json_encode
● ruby, split
● translate
● urldecode
● useragent, xml
More at: http://logstash.net/docs/1.4.2/
INPUT
FILTER
PARSE
OUTPUT
16. L as in Logstash
GROK filter
● Parsing and analysis of any text
● Grok is the best way to process even the most unstructured data
● Over 120 samples and possibility to create new ones
[ERROR] - 2015/13/03-11:01:31 - 192.168.0.1 - Some error message
[%{GREEDYDATA:message}] - %{DATESTAMP} - %{IP:client} - %{GREEDYDATA:message}
17. L as in Logstash
Installation and setup
● Java Oracle and Logstash installation
o wget and unzip of the latest distribution available at
https://www.elastic.co/downloads
● Setup:
input {
file {
type => "syslog"
path => ["/var/log/auth.log", "/var/log/syslog"]
}}
● Running Logstash:
o ./bin/logstash
18. K as in Kibana
● Data visualization in the form of a web app
● Data search, filters and analysis
● Intuitive interface, not only for the programmers
● Instant sharing and embedding of multiple dashboards
● Easy dashboard adjustment - JSON
● Export of the results
19. K as in Kibana
https://www.elastic.co/blog/kibana-4-literally
20. K as in Kibana
Bar chart:
https://www.elastic.co/blog/kibana-4-beta-2-get-now
21. K as in Kibana
Pie chart:
https://www.elastic.co/blog/kibana-4-for-investigating-pacs-super-pacs-and-your-neighbors
22. K as in Kibana
Histogram:
http://blog.qbox.io/kibana-4-and-elasticsearch-v-1-4-4-and-1-3-9
23. K as in Kibana
Data table:
https://www.elastic.co/blog/kibana-4-for-investigating-pacs-super-pacs-and-your-neighbors
24. K as in Kibana
Geolocation:
https://www.elastic.co/blog/kibana-4-literally
25. K as in Kibana
Relation chart:
http://demo.packetbeat.com/#/dashboard/elasticsearch/Packetbeat%2520Statistics
26. K as in Kibana
Installation and setup
● Kibana installation:
o wget and unzip of the latest distribution available at
https://www.elastic.co/downloads
● Setup:
o Kibana is set up on the default ElasticSearch port - 9300
● Running Kibana:
o ./bin/kibana
● Start in a browser:
o http://YOURDOMAIN.com:5601
32. Tools
ElasticSearch Plugin: Head
ElasticSearch monitoring and management panel
Installation:
● elasticsearch/bin/plugin -install mobz/elasticsearch-head
● http://localhost:9200/_plugin/head/
33. Tools
ElasticSearch Plugin: Morfologik
Plugin enabling the use of Polish characters
in ElasticSearch queries
Installation:
● cd elasticsearch
● bin/plugin -install com.github.chytreg/elasticsearch-analysis-
morfologik/2.3.1