Nowadays cloud enviroments are primary platform for applications. We no longer have multipurpose machines, rather multiple smaller virtual servers with dedicated roles. Therefore there is a need to have one place where we can manage applications and system logs. I wish to share my experience gained while building centralized log managment system using Nxlog, Logstash and Kibana. With that tools we are building cost effective and scalable log managment platform. Dariusz Eliasz - Works in Allegro Group as a Solution Architect and is responsible for organizing cooperation with infrastructure teams, also leads some of the infrastructure projects. Earlier as an Expert System Administratorhe was related with building and maintaining the infrastructure shared services (i.e. image hosting platform) within Allegro Group.

1. Centralized log management based on
Logstash and Kibana - case study
Dariusz Eliasz
20.05.2014 Atmosphere Conference

2. • What’s the problem ?
• Solutons
• Transport format
• Architecture
– Sender
– Log router
– Log collector
– Full text search engine
– GUI
• Use case
Agenda

3. What’s the problem ??

6. Solutons ??

11. Transport format - syslog
• RFC3164 (BSD syslog )
• limited size - 1kB
• format of a syslog message:
– PRIORITY (calculated from severity and facility)
– HEADER (tmestamp + hostname or IP)
– MSG (tag + content)
<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8

12. Transport format - json
• JavaScript Object Notaton
• lightweight text-data interchange format
• language independent
• self-describing

13. Transport format - json
{
"LogType": "access_log",
"Vhost": "atmosphere-conference.com",
"HtpsOn": "false",
"Xrealip": "1.2.3.5",
"Clientp": "91.17.13.28",
"UserAgent": "Mozilla/4.0 (compatble; MSIE 6.0; Windows NT 5.1)",
}

14. Architecture

15. Sender

16. Sender
• nxlog (htp://nxlog-ce.sourceforge.net/)
• multple input types:
– tcp socket
– udp socket
– fle input
– unx socket
• multple parser types:
– bsd syslog
– json

17. Sender
Good practce:
• make as much as possible processing on sender site, eg: apache access logs in
json format
• automate confguraton management

18. Log router

19. Log router
nxlog
nxlog logstash
logstash
redis
redissyslog-ng
syslog-ng redis

20. Log router
### JSON PARSER
parser p_json {
json-parser (prefx("_json."));
};
### FILTERS
flter f_someflter {
("${_json.SomeJsonField}” == ”abc.com”)
};
### INPUTS & OUTPUTS
### LOG PATHS
log {
source(s_network_json);
destnaton(d_udp_logstash);
destnaton(d_tcp_hadoop);
#fags(fow-control); # disabled to separate destnatons
};

21. Log router
Good practce:
• good separaton of destnatons
• calculate enough redis size – it’s yor bufer
• batch events writes to redis

22. Log collector

23. Log collector
• Logstash htp://logstash.net/
• collectng, parsing and storing logs tool
• plugins:
Inputs
• fle
• gelf
• tcp
• log4j
• redis
• varnishlog
Codecs
• json
• line
• msgpack
• netlow
• multline
Filters
• grok
• alter
• cidr
• geoip
• grep
• mutate
Outputs
• elastcsearch
• graphite
• jira
• tcp
• zeromq
• zabbix

24. Log collector
Good practce:
• keep up2date version of java & logstash
• use batch & multthread read from redis
• read logs 
• bulk writes to elastcsearch

25. Full text search engine
• Elastcsearch htp://www.elastcsearch.org/
• distributed, real-tme search and analytcs engine
• store documents as a JSON
• high availability
• schema free
• index mult-tenancy
• on top of Lucene

26. Full text search engine
• every index is replicated
• every index sharded
• index parttoning – tme based
• data retenton – tme based

27. Full text search engine
Good practce:
• half memory for ES (<30GB), half for system cache
• bootstrap.mlockall: true
• gateway.recover_afer_nodes
• indices.felddata.cache.size
• authorizaton via proxy
• curator
• Marvel plugin

28. GUI
• Kibana 3 htp://www.elastcsearch.org/overview/kibana/
• search, graph & analyze logs
• JavaScript based (AngularJS)
• only simple htp server needed

30. Q&A

31. Sources
Images:
htp://www.datalife7.com/2014/01/einsteins-secret-to-problem-solving-and.html
htp://www.formengifs.com/victorinox-swiss-army-swiss-champ-multtool-knife/
htp://www.slashgear.com/google-data-center-hd-photos-hit-where-the-internet-lives-gallery-17252451/