Fluentd presentation at Cloudera World Tokyo

1. Masahiro Nakagawa
Nov 6, 2014
Cloudera World Tokyo
Fluentd
Unified Logging Layer

2. Who are you?
> Masahiro Nakagawa
> github/twitter: @repeatedly
> Treasure Data, Inc.
> Senior Software Engineer
> Fluentd / td-agent developer
> I love OSS :)
> D language - Phobos committer
> Fluentd - Main maintainer
> MessagePack / RPC- D and Python (only RPC)
> The organizer of Presto Source Code Reading
> etc…

3. Structured logging
!
Reliable forwarding
!
Pluggable architecture
http://fluentd.org/

4. Data Processing
Data source
Collect Store Process Visualize
Reporting
Monitoring

5. Related Products
Collect Store Process
Visualize
Cloudera
Horton Works
Treasure Data
Excel
Tableau
R
easier & shorter time
???

6. What’s Fluentd?
• Open sourced log collector written in Ruby
• Customization is essential
small core + many plugins
6
It’s like syslogd, but
uses JSON for log messages

7. Divide & Conquer & Retry
7
error retry
retry
error retry retry

8. Core Plugins
!
• Divide & Conquer
• Buffring & Retrying
• Error handling
• Message routing
• Parallelism
!
!
• read / receive data
• write / send data
8

9. Pluggable Architecture
9
Pluggable Pluggable
Buffer Output
Input
> Forward
> HTTP
> File tail
> dstat
> ...
> Forward
> File
> MongoDB
> ...
> File
> Memory
Engine
Output
> rewrite
> ...

10. Before
10

11. After
11

12. M x N → M + N
12
Alerting
Nagios
Analysis
MongoDB
MySQL
Hadoop
Archiving
Amazon S3
Access logs
Apache
App logs
Frontend
Backend
System logs
syslogd
Databases
buffer / buffer / routing

13. Ecosystem

14. Treasure Agent
• Treasure Data distribution of Fluentd
• including Ruby, core libraries and
third party plugins
• Current stable is 2.1.1
• rpm / deb / dmg
• 2.1.2 will be released soon with fluentd-ui
14

15. fluentd-forwarder
• Forwarding agent written in Go
• mainly for Windows support
• immature than Fluentd
• Bundle TCP input/output and TD output
• No plugin mechanizm
• Similar product: ik, fluent-agent-hydra, etc
15

16. fluentd-ui
• Manage Fluentd instance using Web GUI
16

17. Use-case

18. Users
• Companies
• Web service, Ad Tech, Game, Media,
EC, IoT, PoS, etc..
• Products
• Kubernetes, Cloudn, Harvester,
Bloomberg Clustered Private Cloud, etc..
18

19. Simple forwarding
19
HDFS
HBase
Mongo
S3

20. Separate log stream
20

21. CEP for stream processing
21

22. Roadmap
• v0.10 (current)
• v0.12 (Nov, 2014)
• Filter / Label / at-least-once
• v0.14 (TBD)
• New Plugin APIs, ServerEngine
• v1 (TBD. In the first half of 2015)
22

23. Cloud service for the entire data pipeline
Check: treasuredata.com