Dan Ivovich walks through getting started with Logstash

1. Logstash!
Get to know your logs
Dan Ivovich
BMore on Rails
4/9/13

2. Dan Ivovich
SmartLogic Solutions
http://smartlogicsolutions.com
Twitter - @danivovich

3. What is the goal?
● Collect, Parse, and Store your log events
● Make log events searchable
● Analyze log events

4. Why bother?
● Got logs?
○ syslog
○ nginx access log
○ application logs
○ database logs
Are they all formatted the
same?

6. 3 Parts
● Inputs
● Filters
● Outputs

7. Inputs
● Files
● TCP/UDP
● Redis
● AMQP
● rsyslog
● xmpp
http://logstash.net/docs/1.1.9/ - Full list

8. Filters
● grep
● mutate
● anonymize
● date
● grok
http://logstash.net/docs/1.1.9/ - Full list

9. Outputs
● Files
● TCP/UDP
● Redis
● AMQP
● elasticsearch
http://logstash.net/docs/1.1.9/ - Full list

10. Getting Started
input { stdin { type => "stdin-type"} }
output { stdout { debug => true debug_format =>
"json"} }
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-simple.conf
Type something!

11. See our message!

12. Parse something!
input { stdin { type => "stdin-type"} }
filter { grok { type => "stdin-type" pattern =>
"Hello %{DATA:message}!" } }
output { stdout { debug => true debug_format =>
"json"} }
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-simple.conf
Say Hello!

13. See our message in a field!

14. Life is better with search
input { stdin { type => "stdin-type" } }
output {
stdout { debug => true debug_format => "json" }
elasticsearch { embedded => true }
}
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-search.conf
cURL for it!

15. Search for the data

16. Well that isn't pretty
Enter Kibana

17. Kibana is a friendly
interface for your logs

18. Kibana Connects to Elasticsearch
● Logstash parses and structures data into
Elasticsearch
● Kibana makes that data available
● Apache Lucene Query Syntax (from elasticsearch)
● Field statistics
● Range searches
How do we put it together?

19. It Was Simple to Start
input { stdin { type => "stdin-type" } }
output {
stdout { debug => true debug_format => "json" }
elasticsearch { embedded => true }
}
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-search.conf
But Let's Get Real

20. On a server with logs

21. Logstash/Elasticsearch

22. Demo

23. Thoughts....
● Easy to try out, but for anything real, you'll want a much
more complicated configuration
● The variety of inputs is great
● Easy to build up a nice stack of filters

24. More Thoughts....
● Slow to boot monolithic jar file can be frustrating
○ Flatjar?
● Hard to track down why logs aren't flowing
● Elasticsearch node discovery can be difficult
○ If your cluster doesn't have a node added to it when
your client starts, your client isn't connected

25. More Information
● logstash.net
● grokdebug.herokuapp.com
● www.elasticsearch.org

26. Questions?
http://smartlogicsolutions.com
http://twitter.com/smartlogic
http://github.com/smartlogic
http://fb.me/smartlogic