A tale of my adventures to process logs in a production environment. Soon I will link the demo video (in spanish)

1. Using Logstash, ElasticSearch and Kibana
Alejandro E Brito Monedero
@ae_bm
2013 / 05 / 23

2. Business as usual

4. So many hosts to check
Is there life out there?
http://upload.wikimedia.org/wikipedia/commons/a/aa/ESO-The_Milky_Way_above_La_Silla-phot-27-04-hires.jpg

5. Time to play whack a log
http://i102.photobucket.com/albums/m109/niceperson907/121331d1253497450-animated-gif-thre.gif

6. http://brotality.com/wp-content/uploads/2012/12/madness.jpg

7. I need a new toy
http://www.youtube.com/watch?v=8L6Dpq5kY_A
Video time

8. Logstash ✔ collects logs
✔ parses logs
✔ stores logs
✔ indexes logs
✔ searches logs
✔ and fixes timestamps
You only need
● JVM
● logstash.jar

9. $ log_producer | grep ... | sed … | awk … | tee output
| sort | uniq -c | sort -n
Log source Logstash
Logstash
(optional)
ElasticSearch
DB
statsd
Pipes

10. Inputs Filters Outputs
File
Redis
Syslog
Lumberjack
Rabbitmq
SQS
…
Alter
Date
Grok
Multiline
Grep
...
AMQP
Cloudwatch
Elasticsearch
Mongodb
Redis
File
...
Plugins
Not here yet? JRuby to the rescue

11. ElasticSearch
Distributed RESTful search server
● Near real-time search
● RESTful API
● Easy to scale horizontally
● HA
● Full text search
● YAML config file / JSON format!!
● Document oriented JSON
Getting started: Logstash JAR includes it / download and set cluster.name
This is where it will be worth to spend some time tuning

12. Kibana
✔ Nice UI
✔ Better than the old frontend logstash
included
✔ Ruby / framework Sinatra
Web frontend to search / graph and more

13. Original plan
Apache
(ligthweight shipper)
Tomcat
(ligthweight shipper)
broker logstash
ElasticSearch
Kibana

14. After a few workarounds
Apache
(logstash shipper)
Tomcat
(logstash shipper)
Logstash ElasticSearch
Kibana
SSH tunnels

15. Example config 1/3
Logstash-httpd.conf
input {
file {
type => "httpd"
path => ["/var/log/httpd/*-logstash.log"]
exclude => ["*.gz"]
start_position => "beginning"
format => "json_event"
}
}
output {
tcp {
host => "0.0.0.0"
mode => "server"
port => 1666
}
}
Logstash-server.conf
input {
tcp {
type => "httpd"
format => "json_event"
host => "127.0.0.1"
mode => "client"
port => "1666"
}
tcp {
type => "app"
format => "json_event"
host => "127.0.0.1"
mode => "client"
port => "2666"
}
}
output {
elasticsearch {
cluster => "logstash"
}
}

16. Example config 2/3
Logstash-tomcat.conf
filter {
# Tomcat
# Remove blank lines
grep {
type => "tomcat"
match => [ "@message", "(.+)" ]
drop => true
add_tag => [ "no_blank_lines" ]
}
# make the multilines be treated like a single line
multiline {
type => "tomcat"
pattern => "^dddd"
negate => true
what => "previous"
}

17. Example config 3/3
Logstash-tomcat.conf
# mark the exceptions (multiline)
grep {
type => "tomcat"
tags => [ "multiline" ]
match => [ "@message", ".+Exception: .+" ]
drop => false
add_tag => [ "java_exception" ]
}
# get the log level, operation id, module and timestamp as separated fields
grok {
type => "tomcat"
pattern => "%{TIMESTAMP_ISO8601:timestamp} [%{OPERATION_ID:operation_id}]..."
add_tag => [ "groked" ]
}
# fix the timestamp
date {
type => "tomcat"
match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSSZZ" ]
add_tag => [ "timestamp_fix" ]
}
}

18. I need a new toy
Demo

19. Some remarks
● Don't forget about security
● The applications should be flexible enough for allowing to
publish their logs using brokers or other methods beyond
files and syslog
● Logging in JSON format is a nice to have
● Share the log visualization
● Use the brokers Luke
● If you develop internalize this
http://www.masterzen.fr/2013/01/13/the-10-
commandments-of-logging/

20. Extras
● http://logstash.net/
● http://www.logstashbook.com/code/ only $10.09
● https://github.com/logstash/logstash/blob/v1.1.12/patterns/grok-patterns
● http://grokdebug.herokuapp.com/
● http://www.infoq.com/articles/review-the-logstash-book (better diagrams)
●
http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/
● http://kibana.org/
●
https://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/queryparsersyntax.html
● http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/
●
http://blog.lusis.org/blog/2012/01/31/load-balancing-logstash-with-amqp/

21. Do you want to join the <some fancy words here> team?
I am not hiring, but I can tell you about some places where
it is better to stay away
Have a nice day
All the images, videos and stuff are property of their respective owners, look at the cat
and don't sue me
http://stuffpoint.com/cats/image/41633/cute-cat-picture/