This document compares message and metric management solutions like Fluentd and Logstash. It discusses how these solutions can collect, store, visualize, and alert on log and metric data from heterogeneous environments. While commercial solutions like Splunk are very expensive, open source solutions like Fluentd, Logstash, Elasticsearch, and Kibana provide similar functionality through various "bricks" or components at no cost. The document analyzes key differences between Fluentd and Logstash, such as their configuration, buffering approaches, high availability features, and plugin ecosystems.
11. Importance of Logging Systems
WHAT Actually We NEED?
Collect
Messages
Metrics
●Store
●Visualize
●Alert
12. Importance of Logging Systems
Heterogeneous Environment
Write our own script for each type of log (?)
Not in an enterprise environment with lots of
devices and services!
●Technical Fragility and dependency to
Individuals
●Strong Dependency to knowledge about
underlying process
13.
14.
15. Commercial Solutions
Splunk
(500M/Day is Free, then: 5,000,000 $)
●Nagios
Everything is restricted to Nagios Concept
No separation between metrics and messages
No stylish diagrams (in free solution)
Problems in cloud infrastructure
No realtime monitoring
No manipulating messages
(1,995 $ for commercial solution)
●Online Services
16. Good logging system Specifications
●Have a common interface
●Decouple data sources from data outputs
Prevent mentioned dependencies
No effect of adding new data source/output
Reliability
Persistent Buffering
●Extensibility
High Availability
Load Balancing
Robustness
24. Overview
Fluentd:
Written in Cruby
Used in Google Cloud Platform and Kubernetes
Maintained by Tresure Data
●Logstash:
Written in Jruby
Used in ELK Stack
Maintained by Elastic Co.
➢Both use their own RubyGems Repo
➢Out of the box nature, less dependencies
25. Configuration
Fluentd:
Each Input is tagged
Logs are routed by tags
Logstash:
All inputs are Gathered and Scattered
Conditional Outputs, No tags
28. Full Buffer or Output Exception occurrence
Fluentd:
Exception: streaming
Block input plugin: batch
Drop oldest chunk: monitoring
LogStash:
Retry
Discard
Dead Letter Queuing