1. Introduction to ELK
Proof of Concept & Demo

2. What is ELK
•Elastic Search
–Lucene based search engine (Java Stack)
–Distributed capability
–REST API over Http
–Data share using JSON format
•Logstash
–Ruby agent application
–Agent to collect log data in numerous input
formats
–Filters can be applied
–Many Output formats supported
•Kibana

3. Setup Elastic Search
•Download and extract to a local directory
•JRE 7 is required
•Default configuration is good
•Start
–C:Programselasticsearch-1.3.2binelasticsearch.bat

4. Setup Logstash
•Download and un-archive to a local directory
•Create a configuration file
–Needs to have input{}, filter{optional}, output{}
-Sample
input {
file {
path => ["C:/TEMP/server-logs/*.*"]
codec => "json"
}
}
filter{
date {
match => ["eventTime","yyyy-MM-dd HH:mm:ss,SSS"]
add_tag => ["date_matched"]
}
}
output {
stdout{}
elasticsearch {
port => 9200
}
}

5. Setup Logstash Contd..
•Groking is most common method for parsing
log contents
•Grok patterns can be configured and passed as
an input file
•Sample grok pattern
–KMMLOG4J %{TIMESTAMP_LOG4J:timestamp} %{THREAD:thread} u:%{LOG4JUSER:user}/d: %{LOG4JCATEGORY:category} %{GREEDYDATA:logmessage}
•Master file available here
–logstash-1.4.2patternsgrok-patterns

6. Setup Logstash Contd..
•Log 4j parsing is tricky when there is multiline parsing
required for stack traces.
•PatternLayout for log4j that generates logstash
json_event formatted data is available
–Refer https://github.com/logstash/log4j-jsonevent-layout

7. Kibana
•Download and unzip to a web server
•Default log stash dashboard is built in
•Default configuration is good
•Edits can be made to app/dashboards/logstash.js
•URL is http://localhost:8090/kibana/index.html#/dashboard/file/logstash.json
•Or replace default.json with logstash.json

8. Proof Of Concept
•Download logs files using SCP Tool
–SCP program used is pscp, supports non-interactive mode.
–SCP should be performed once manually through command line to
enable adding the host to trusted list.
•Convert log file messages into JSON messages
•Place it under log stash monitored directories
when ELK stack is up and running.
•Perform queries on Kibana
•Analyze spikes visually and identify log noises.

9. Demo Logstash
•Run test.SCPTool
•Start elastic search
•Start log stash agent
•Run test.LogstashFileCopier
•Start Kibana server

10. Demo – Kibana
•Run demo.po.lucene.POJsonBuilder
•Build a new dashboard using the data.

11. Q&A
•References
–http://www.elasticsearch.org/overview/elasticsearch
–http://www.elasticsearch.org/overview/logstash
–http://www.elasticsearch.org/overview/kibana
–http://www.elasticsearch.org/guide/en/kibana/current/using-kibana-for-the-
first-time.html
–https://github.com/logstash/log4j-jsonevent-layout