4. Fluentd :: format of logs
time
Application tag
2012-02-04 01:33:51
myapp.buylog {
Fluentd
“user”: ”me”,
“path”: “/buyItem”,
“price”: 150,
“referer”: “/landing”
Storage } record
8. Typical architecture before Fluentd
App server App server App server
Application Application Application
File File File ... File File File ... File File File ...
File
High latency
must wait for a day
Log server Hard to analyze
complex text parsers
9. Architecture after Fluentd
App server App server App server
Application Application Application
Fluentd Fluentd Fluentd
Realtime!
Fluentd Fluentd
14. Scribe’s Pros & Cons
• Pros.
> Fast (C++)
• Cons.
> VERY hard to install
> Deals with unstructured logs
you must parse logs before analyzing them
> Hard to extend
you must re-compile C++ programs
> No longer maintained?
15. Fluentd vs Scribe
• Easy to install
> “gem install fluentd”
> stable RPM and DEB packages
http://packages.treasure-data.com/
• Easy to write plugins
> you can use Ruby
• Easy to distribute plugins
> “gem search -rd fluent-plugin”
17. Flume’s Pros & Cons
• Pros.
> Central master server manages all nodes
• Cons.
> Difficult to understand
logical topologies, phisical servers and a configuration of
the logical/phisical mapping
> Dificult to configure
replicated master servers, log servers and agents
> Big footprint
50,000 lines of Java codes
18. Fluentd vs Flume
• Easy to understand
> “syslogd that understands JSON”
• Easy to setup
> “sudo fluentd --setup && fluentd”
• Very small footprint
> small engine (3,000 lines) + plugins
• Easy to configure
19. Fluentd vs Scribe/Flume
Fluentd Scribe Flume
Installation gem/rpm/deb make rpm/deb
3000 lines of 8000 lines of 50,000 lines of
Footprint Ruby C++ Java
Plugin Ruby N/A Java
Plugin distribution RubyGems.org N/A N/A
Master Server No No Yes
License Apache License Apache License Apache License
20. Fluentd
• Documents
> http://fluentd.org
• Source code
> http://github.com/fluent
> 14 committers across
many organizations
• Mailing list
> Google groups
21. • Sadayuki Furuhashi
> twitter: @frsyuki
• Treasure Data, Inc.
> Software Engineer; founder
• Author of MessagePack
• Author of Fluentd