Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
Logstash
1. Goal of Centralized log collection
• Collect, parse and store log events
• Make log events searchable
• Analyze log events
2. Log collection strategy
Indexer
Transporter/Broker
Agent
Machine Instance
UI
Storage and
Search engine
Storage
Log collector and log shipper
3. Agent
• Agent works as log collector and log shipper
• It tails the log on regular interval from log files
• And ships the log to transporter or broker.
4. Transporter/broker
• Event shipping rate at agent vary with time.
• Transportor or broker makes sure that log shipping rate
to indexer is same all the time.
5. Indexer
• Indexer fetches logs from transporter/broker.
• Indexer indexes the log events.
• Indexer calls the API of “Storage and search engine” to
write the data on storage.
6. Storage and search engine
• Stores the indexed data at specific location in a well
defined format
• It provides API for storage, search and analysis of log
events.
10. Log format
• "[DEBUG] [2013-10-13 22:56:20,191] [http-38219-4]
[annotation.DefaultAnnotationHandlerMapping] -
[Mapping [/useractivity/addUserActivity] to handler
'com.firstrain.rest.controller.UserActivityController@44
8d5a91']”
Loglevel Date format Java Thread Java Class
msg
11. Logstash overview
• Input ( 37) – Collects logs from logsource
• Filter ( 39) – Applies regex to fragment the logs
• Output (51) – Writes parsed logs to destination
13. Logstash Indexer configuration
input {
redis {
host => "127.0.0.1"
# these settings should match the output of the agent
data_type => "list"
key => "logstash"
# We use the 'json' codec here because we expect to read
# json events from redis.
codec => json
}
}
output {
stdout { debug => true debug_format => "json"}
elasticsearch {
host => "127.0.0.1"
}
}
19. Features
• Centralized log management console (GUI)
• Log search and analysis system
• Histogram, Graphs and Charts
• Log shipping for windows and Linux machine
• Near Real time log processing
• Offline log processing
• Dashboard view
• Data retention and storage policy
• Addons and modules
• Alerting system
20. Splunk vs Logstash
• Paid vs free
• Installation, Configuration and setup
• Regex based field extraction during search from UI
• Regex based field extraction during log shipping
• Correlation graph of multiple graphs
• Zoom in/ Zoom out
• Alerts
• Easy url sharing
• Modules and add-ons support
• Disk space usage