1. Goal of Centralized log collection
• Collect, parse and store log events
• Make log events searchable
• Analyze log events

2. Log collection strategy
Indexer
Transporter/Broker
Agent
Machine Instance
UI
Storage and
Search engine
Storage
Log collector and log shipper

3. Agent
• Agent works as log collector and log shipper
• It tails the log on regular interval from log files
• And ships the log to transporter or broker.

4. Transporter/broker
• Event shipping rate at agent vary with time.
• Transportor or broker makes sure that log shipping rate
to indexer is same all the time.

5. Indexer
• Indexer fetches logs from transporter/broker.
• Indexer indexes the log events.
• Indexer calls the API of “Storage and search engine” to
write the data on storage.

6. Storage and search engine
• Stores the indexed data at specific location in a well
defined format
• It provides API for storage, search and analysis of log
events.

7. Tools for Log collection
• Splunk
• Logstash
• Graylog2
• Fluentd

8. Splunk Architecture
Splunk Search
heads
Splunk Indexer
Splunk UI
Forwarder
Server
Forwarder with load
balancer
Splunk Search
heads
Splunk Indexer
Storage
http://docs.splunk.com/Documentation/Splun
k/latest/Deploy/Distributedoverview

9. Logstash Architecture
Logstash Indexer
(10.10.10.215)
Redis server
(10.10.10.215)
Logstash
(10.10.10.73)
Kibana
Elastic search
(10.10.10.215)
Storage
Log collector and log shipper
Logstash
(10.10.10.77)
http://logstash.net/docs/1.2.1/tutorials/gettin
g-started-centralized

10. Log format
• "[DEBUG] [2013-10-13 22:56:20,191] [http-38219-4]
[annotation.DefaultAnnotationHandlerMapping] -
[Mapping [/useractivity/addUserActivity] to handler
'com.firstrain.rest.controller.UserActivityController@44
8d5a91']”
Loglevel Date format Java Thread Java Class
msg

11. Logstash overview
• Input ( 37) – Collects logs from logsource
• Filter ( 39) – Applies regex to fragment the logs
• Output (51) – Writes parsed logs to destination

12. Logstash shipper configuration
input {
file {
path => " "/frlogdir/fruseractivity/cpflogs/fruseractivity.log“
type => "log4j"
tags => “ua"
codec => multiline {
pattern => "^["
negate => true
what => "previous"
multiline_tag => "exception"
}
}
}
filter {
if [type] == "log4j" {
grok {
patterns_dir => "/opt/logstash/patterns"
match => ["message", "[%{FRWORDS:loglevel}] [%{FRTIMESTAMP:frtimestamp}] [%{FRWORDS:javathread}]
[%{FRWORDS:javaclass}] - %{GREEDYDATA:msg}"]
}
date {
match => ["frtimestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
}
}
output {
## stdout { codec => rubydebug }
redis { host => "10.10.10.215" data_type => "list" key => "logstash" }
}

13. Logstash Indexer configuration
input {
redis {
host => "127.0.0.1"
# these settings should match the output of the agent
data_type => "list"
key => "logstash"
# We use the 'json' codec here because we expect to read
# json events from redis.
codec => json
}
}
output {
stdout { debug => true debug_format => "json"}
elasticsearch {
host => "127.0.0.1"
}
}

14. Example of a log event
Log event ::
{
"@timestamp" => "2013-10-14T05:56:20.191Z",
"message" => "[DEBUG] [2013-10-13 22:56:20,191] [http-38219-4]
[annotation.DefaultAnnotationHandlerMapping] - [Mapping [/useractivity/addUserActivity] to
handler 'com.firstrain.rest.controller.UserActivityController@448d5a91']",
"@version" => "1",
"type" => "log4j",
"tags" => [
[0] “ua"
],
"host" => "pfrontend2-rl.ca.firstrain.net",
"path" => "/frlogdir/fruseractivity/cpflogs/fruseractivity.log",
"loglevel" => "DEBUG",
"frtimestamp" => "2013-10-13 22:56:20,191",
"javathread" => "http-38219-4",
"javaclass" => "annotation.DefaultAnnotationHandlerMapping",
"msg" => "[Mapping [/useractivity/addUserActivity] to handler
'com.firstrain.rest.controller.UserActivityController@448d5a91']"
}

15. App’s Tags
• Useractivity (ua)
• Usercollobaration (uc)
• Chartservice (cs)
• Dataprovider (dp)
• Restservice (rs)
• Webapp/portal (wa)
• Solr (solr)
• Searchemailcontentgenerator (secg)
• Searchemailcontentgeneratornode2 (secgn2)
• Searchemaildispatcher (sed)
• Searchemailfilter (sef)

16. Component’s Tags
• Hpthepoint (hpthepoint)
• Hp (hp)
• Pfizer (pfizer)
• Sfdc (sfdc) . . .
• Combined perf logs of components (combinedperf)
• CpfLogs (cpf)
• Corecpf (corecpf)
• ISPN (ispn)
• Matcher (matcher)
• Access log of components (components)

17. Types
• Tomcat logs (log4j)
• Garbage collector (gc)
• Performance logs (perf)
• Mysql (mysql)
• Tomcat and Http access log (accesslog)

18. Dashboard

19. Features
• Centralized log management console (GUI)
• Log search and analysis system
• Histogram, Graphs and Charts
• Log shipping for windows and Linux machine
• Near Real time log processing
• Offline log processing
• Dashboard view
• Data retention and storage policy
• Addons and modules
• Alerting system

20. Splunk vs Logstash
• Paid vs free
• Installation, Configuration and setup
• Regex based field extraction during search from UI
• Regex based field extraction during log shipping
• Correlation graph of multiple graphs
• Zoom in/ Zoom out
• Alerts
• Easy url sharing
• Modules and add-ons support
• Disk space usage

21. Demo
• Find the ip when login is failed

22. Demo
• Find the activity logs for ip where login is failed

23. References
• http://docs.splunk.com/Documentation/Splunk
• http://logstash.net/docs/1.2.1/
• http://semicomplete.com/presentations/logstash-scale11x/#/
1

24. Questions & Answers