1. Logstash + Elasticsearch + Kibana
Centralized Log server
(as Splunk replacement)
Marko Ojleski
DevOps Engineer

2. $plunk

3. Business as usual, untill…

4. #Outage @03:00AM

5. Check logs….?!?
10 network devices
40 servers
100 logs

6. Massive RAGE

7. tail
cat
grep
sed
awk
sort
uniq

8. and looots of |

9. tail -10000 access_log | awk '{print $1}' | sort | uniq -c | sort -n

10. it’s just too much

11. 1. collect data
2. parse/filter
3. send data
Logstash
written in JRuby
Author: Jordan Sissel

12. input
parse/filter
output

13. 1. collect data
30+ inputs

14. 1. collect data
file
syslog
tcp
udp
zmq
redis
log4j
Logstash input

15. Log shippers
Logstash
Beaver (Python)
Lumberjack (Go)
Woodchuck (Ruby)
Nxlog (C)

16. Sample conf
input {
tcp {
type => “server1"
host => "192.168.1.1"
port => "5555"
}

17. 2. parse/filter
40+ filters

18. 2. parse/filter
grok
csv
grep
geoip
json
mutate
Logstash
filters
xml
key/value

19. Grok filter
REGEX pattern collection

20. Grok filter

21. Grok filter
(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[09]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[09]{1,2}))(?![0-9])

22. Grok filter
(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[09]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[09]{1,2}))(?![0-9])
IP

24. `$=`;$_=%!;($_)=/(.)/;$==++$|;($.,$/,$,,$,$",$;,$^,$#,$~,$*,$:,@%)=(
$!=~/(.)(.).(.)(.)(.)(.)..(.)(.)(.)..(.)......(.)/,$"),$=++;$.++;$.++;
$_++;$_++;($_,$,$,)=($~.$"."$;$/$%[$?]$_$$,$:$%[$?]",$"&$~,$#,);$,++
;$,++;$^|=$";`$_$$,$/$:$;$~$*$%[$?]$.$~$*${#}$%[$?]$;$$"$^$~$*.>&$=`

25. `$=`;$_=%!;($_)=/(.)/;$==++$|;($.,$/,$,,$,$",$;,$^,$#,$~,$*,$:,@%)=(
$!=~/(.)(.).(.)(.)(.)(.)..(.)(.)(.)..(.)......(.)/,$"),$=++;$.++;$.++;
$_++;$_++;($_,$,$,)=($~.$"."$;$/$%[$?]$_$$,$:$%[$?]",$"&$~,$#,);$,++
;$,++;$^|=$";`$_$$,$/$:$;$~$*$%[$?]$.$~$*${#}$%[$?]$;$$"$^$~$*.>&$=`
Just another Perl hacker.

26. Grok filter
120+ regex patterns
USERNAME
IP
HOSTNAME
SYSLOGTIMESTAMP
LOGLEVEL
etc…

27. Grok filter
2.10.146.54 - 2013-12-01T13:37:57Z - some really boring message

28. Grok filter
2.10.146.54 - 2013-12-01T13:37:57Z - some really boring message
%{IP:client} - %{TIMESTAMP_ISO8601:time} - %{GREEDYDATA:message}

29. Grok filter
client => 2.10.146.54
time => 2013-12-01T13:37:57Z
message = > some really boring message

30. Grok filter
input {
tcp {
type => “server1"
host => "192.168.1.1"
port => "5555"
}
filter {
if [type] == “server1" {
grok {
match => { "message" => "%{IP:client} - %{TIMESTAMP_ISO8601:time} - %{GREEDYDATA:message} "}
}
}

31. 3. send data
50+ outputs

32. 3. send data
Logstash
output
statsd
stdout
tcp
elastic
redis
mongo
zmq

33. 1. RESTful api
2. JSON-oriented
3. Horizontal scale
4. HA
5. Full Text search
6. Based on Lucene
Elasticsearch
Distributed RESTful
search server

34. Logstash => elasticsearch
input {
tcp {
type => “server1"
host => "192.168.1.1"
port => "5555"
}
filter {
if [type] == “server1" {
grok {
match => { "message" => "%{IP:client} - %{TIMESTAMP_ISO8601:time} - %{GREEDYDATA:message} "}
}
}
output {
elasticsearch {}
}

35. 1. Clean and simple UI
2. Fully customizable
3. Bootstrap based
4. Old version running on Ruby
5. Milestone 3 fully rewritten in
HTML/Angular.js
Kibana
Awesome Elasticsearch
Web Frontend to
search/graph

36. Real Life Scenarios

37. Scenario 1
L2 switch
Cisco ASA
L3 switch
UDP
UDP
Elasticsearch
Syslog broker
(lightweight shipper)
UDP
Logstash
(main log server)
Kibana

38. Scenario 2
Apache
(lightweight shipper)
IIS
TCP
TCP
(lightweight shipper)
Jboss
(lightweight shipper)
Elasticsearch
Logstash
(main log server)
TCP
Kibana