2. Who are you?
> Masahiro Nakagawa
> github/twitter: @repeatedly
> Treasure Data, Inc.
> Senior Software Engineer
> Fluentd / td-agent developer
> Living at OSS :)
> D language - Phobos committer
> Fluentd - Main maintainer
> MessagePack / RPC - D and Python (only RPC)
> The organizer of several meetups (Presto, DTM, etc…)
> etc…
4. What’s Fluentd?
> Data collector for unified logging layer
> Streaming data transfer based on JSON
> Written in Ruby
> Gem based various plugins
> http://www.fluentd.org/plugins
> Working in production
> http://www.fluentd.org/testimonials
15. > default second unit
> from data source
Event structure(log message)
✓ Time
> for message routing
> where is from?
✓ Tag
> JSON format
> MessagePack
internally
> schema-free
✓ Record
17. Configuration and operation
> No central / master node
> include helps configuration sharing
> Operation depends on your environment
> Use your deamon management
> Use Chef in Treasure Data
> Apache like syntax and Ruby DSL
18. # receive events via HTTP
<source>
type http
port 8888
</source>
!
# read logs from a file
<source>
type tail
path /var/log/httpd.log
format apache
tag apache.access
</source>
!
# save access logs to MongoDB
<match apache.access>
type mongo
database apache
collection log
</match>
# save alerts to a file
<match alert.**>
type file
path /var/log/fluent/alerts
</match>
!
# forward other logs to servers
<match **>
type forward
<server>
host 192.168.0.11
weight 20
</server>
<server>
host 192.168.0.12
weight 60
</server>
</match>
!
include http://example.com/conf
33. v0.10 (old stable)
> Mainly for log forwarding
> with good performance
> working in production
> almost users use td-agent
> Various plugins
> http://www.fluentd.org/plugins
35. > Apply filtering routine to event stream
> No more tag tricks!
Filter
<match access.**>
type record_reformer
tag reformed.${tag}
</match>
!
<match reformed.**>
type growthforecast
</match>
<filter access.**>
type record_transformer
…
</filter>
v0.10: v0.12:
<match access.**>
type growthforecast
</match>
36. > Internal event routing
> Redirect events to another group
> much easier to group and share plugins
Label
<source>
type forward
</source>
!
<match app1.**>
type record_reformer
</match>
!
…
<source>
type forward
@label @APP1
</source>
<label @APP1>
<match access.**>
type s3
</match>
</label>
v0.10: v0.12:
37. Error stream with Label
> Can handle an error at each record level
> It is still prototype
ERROR!
{"event":1, ...}
{"event":2, ...}
{"event":3, ...}
chunk1
{"event":4, ...}
{"event":5, ...}
{"event":6, ...}
chunk2
…
Input
OK
ERROR!
OK
OK
OK
Output
<label @ERROR>
<match **>
type file
...
</match>
</label>
Error stream
Built-in @ERROR is used
when error occurred in “emit”
38. v0.14 (next stable)
> New plugin APIs
> Actor
> New base classes (#309)
> ServerEngine based core engine
> Robust supervisor
> Sub-second time support (#461)
> Zero downtime restart
39. Actor
> Easy to write popular routines
> Hide implementation details
class TimerWatcher <
Coolio::TimerWatcher
...
end
!
def start
@loop = Coolio::Loop.new
@timer = ...
@loop.attach(@timer)
@thread = ...
end
def configure(conf)
actor.every(@interval) {
router.emit(...)
}
end
!
def start
actor.start
end
v10: v0.14:
40. > Socket manager shared resources with
workers
40
Supervisor
TCP
1. Listen to TCP socket
Zero downtime restart
42. 42
Worker
Supervisor
Worker
TCP
TCP
1. Listen to TCP socket
2. Pass its socket to worker
3. Do same action
at worker restarting
with keeping TCP socket
heartbeat
Zero downtime restart
> Socket manager shared resources with
workers
TODO: How to implement on JRuby?
43. v1 (future stable)
> Fix new features / APIs
> Plugin APIs
> Default configurations
> Clear versioning and stability
> No breaking API compatibility!
> Breaking compatibility by Fluentd v2 ?
45. Other TODO
> Windows support
> Need feedback!
> https://github.com/fluent/fluentd/tree/windows
> Also check: http://qiita.com/okahashi117
> JRuby support
> msgpack / cool.io now work on JRuby
> https://github.com/fluent/fluentd/issues/317
47. Treasure Agent (td-agent)
> Treasure Data distribution of Fluentd
> Treasure Agent 2 is current stable
> Update core components
> We recommend to use v2, not v1
> Next version, 2.2.0, uses fluentd v0.12
> In this week or next week
48. fluentd-forwarder
> Forwarding agent written in Go
> Focusing log forwarding to Fluentd
> Work on Windows
> Bundle TCP input/output and TD output
> No flexible plugin mechanizm
> We have a plan to add some input/output
> Similar product
> fluent-agent-lite, fluent-agent-hydra, ik
50. Embulk
> Bulk Loader version of Fluentd
> Pluggable architecture
> JRuby, JVM languages
> High performance parallel processing
> Share your script as a plugin
> https://github.com/embulk
http://www.slideshare.net/frsyuki/embuk-making-data-integration-works-relaxed
