Fluentd: Data streams in Ruby world #rdrc2014

Fluentd:
Data streams in Ruby world
@tagomoris
RedDotRubyConf 2014
Day1, 26 June 2014
14年6月26日木曜日

TAGOMORI Satoshi a.k.a. @tagomoris

Fluentd
Fluentd is an open source data collector
to simplify log management.
Fluentd is designed to process high-volume data streams reliably. Use cases
include real-time search and monitoring, Big Data analytics, reliable archiving and
more.
http://www.ﬂuentd.org/

Before Fluentd:
Access logs Metrics
Archives
apache
nginx
graphs
Amazon S3
Filesystem
tail -f
scp
python
Error handling? Buffering?

Before Fluentd:
Access logs Metrics
Analytics
Archives
apache
nginx
graphs
Hadoop
MySQL
MongoDB
Redshift
Amazon S3
Filesystem
tail -f
scp
python
ruby
ruby
cmd
Routing? API Keys?

Before Fluentd:
Access logs
App logs
Metrics
Analytics
Archives
apache
nginx
frontend
backend
graphs
Hadoop
MySQL
MongoDB
Redshift
Amazon S3
Filesystem
tail -f
scp
python
ruby
ruby
cmd
ﬁle ruby
logger
Routing? API Keys? Formats?

Before Fluentd:
Access logs
App logs
System logs
Metrics
Analytics
Archives
apache
nginx
frontend
backend
syslogd
snmp data
graphs
Hadoop
MySQL
MongoDB
Redshift
Amazon S3
Filesystem
tail -f
scp
python
ruby
ruby
cmd
ﬁle ruby
logger

Before Fluentd: CHAOS
Access logs
App logs
System logs
Various logs
Metrics
Analytics
Archives
apache
nginx
frontend
backend
syslogd
snmp data
graphs
Hadoop
MySQL
MongoDB
Redshift
Amazon S3
Filesystem
tail -f
scp
python
ruby
ruby
cmd
ﬁle ruby
logger
ﬁle
logger
ruby
cmd
ruby

After Fluentd: Controllable
Access logs
App logs
System logs
Various logs
Metrics
Analytics
Archives
apache
nginx
frontend
backend
syslogd
snmp data
graphs
Hadoop
MySQL
MongoDB
Redshift
Amazon S3
Filesystem

Access logs
App logs
System logs
Various logs
Metrics
Analytics
Archives
apache
nginx
frontend
backend
syslogd
snmp data
graphs
Hadoop
MySQL
MongoDB
Redshift
Amazon S3
Filesystem
Fluentd does:
Format, Buﬀer, Retry, Route
After Fluentd: Controllable

Fluentd
Open source data collector
Written in Ruby, runs on CRuby on UNIX-like OS
With error handling and routing in core
Plugin systems
Input, Output and Buffer (w/ many built-in plugins)
Distributed on rubygems.org
Fluentd and its plugins: gem install fluentd
rpm/deb are also available (td-agent)

Why Fluentd?

Why Fluentd?
Fluentd’s logo is very cute!

He is also very cute...

Why Fluentd?
Simple data structure
tag, time and record(hash)
Apache-like conﬁguration syntax
Simple / powerful routing
Many public plugins
Just few steps for custom plugins
Scalability

Fluentd Event
app.device.ios
2014-06-24 16:28:50
{
“username”: “tagomoris”,
“fullname”: “TAGOMORI Satoshi”,
“age”: 34,
“device”: “iPhone 5”,
...
}
Event

Fluentd Event
app.device.ios
1403512916 (2014-06-23 16:41:56 +0800)
{
“age”: 34,
...
}
tag
time
record

Fluentd Event
app.device.ios
1403512916 (2014-06-23 16:41:56 +0800)
{
“age”: 34,
...
}
tag for routing
record
structured data
time by unix time

# read from a file and parse
<source>
type tail
path /var/log/httpd.log
format apache2
tag web.access
</source>
# logs from client libraries
<source>
type forward
port 24224
</source>
# store logs to MongoDB and S3
<match app.**>
type copy
<store>
type mongo
host mongo.example.com
capped
capped_size 200m
</store>
<store>
type s3
path archive/
</store>
</match>
Fluentd Configuration

<source>
type tail
path /var/log/httpd.log
format apache2
tag web.access
</source>
<source>
type forward
port 24224
</source>
<match app.**>
type copy
<store>
type mongo
host mongo.example.com
capped
capped_size 200m
</store>
<store>
type s3
path archive/
</store>
</match>
Fluentd Configuration
for input for output

source {
type ”tail”
path “/var/log/httpd.log”
format “apache2”
tag ”web.access”
}
source {
type ”forward”
port 24224
}
match(“app.**”) {
type ”copy”
store {
type ”mongo”
host “mongo.example.com”
capped
capped_size “200m”
}
store {
type ”s3”
path “archive/”
}
}
Fluentd Configuration DSL

Tag based routing
input
input
output
output
input
output
output
core
tag
time
record
web.log
sys.*
app.**
**

Tag based routing
input
input
output
output
input
output
output
core
tag
time
record
web.log
sys.*
app.**
**
converted.web.log

300+ Public Plugins
access, add, aes-forward, airbrake-python, amazon_sns, amplifier-filter, amqp, amqp2, andon,
anomalydetect, anonymizer, arango, arduino, axlsx, backlog, bigquery, boundio, buffer-
lightening, buffered-filter, buffered-hipchat, buffered-stdout, bufferize, calc, cassandra,
cassandra-cql, cloudstack, cloudwatch, cloudwatch_ya, combiner, conditional_filter, config-
expander, config_pit, config_reloader, convert-value-to-sha, copy_ex, couch, couch-sharded,
couchbase, dashing, data-rejecter, datacalculator, datacounter, dbi, dd, debug, delay-
inspector, delayed, derive, df, droonga, dstat, dummydata-producer, dynamodb, ec2-metadata,
elapsed-time, elasticsearch, elasticsearch-cluster, elasticsearch-ruby, elb-log, embedded-
elasticsearch, eval-filter, event-tail, extract_query_params, file-alternative, file-sprintf,
filter, filter_keys, flatten, flatten-hash, flowcounter, flowcounter-simple, flume,
fnordmetric, forest, fork, format, forward-aws, ftp, gamobile, ganglia, gc, geoip, glusterfs,
graphite, grassland, gree_community, grep, grepcounter, groonga, groupcounter, growl,
growthforecast, gstore, hash-forward, hato, hbase, hekk_redshift, heroku-postgres, heroku-
syslog, hipchat, histogram, hoop, hostname, hrforecast, http-enhanced, http-ex, http-list,
http-status, https-json, idobata, ikachan, imagefile, imkayac, in-udp-event, incremental,
influxdb, influxdb_metrics, inline-classifier, irc, jabber, json-api, json-nest2flat,
jsonbucket, jstat, jubatus, jvmwatcher, kafka, kanicounter, keep-forward, kestrel, kibana-
server, kinesis-alt, latency, leftronic, librato-metrics, loggly, lossycount, mackerel, mail,
map, measure_time, mecab, metricsense, mixi_community, mixpanel, mobile-carrier, mongo,
mongo-typed, mongokpi, mqtt, msgpack-rpc, mssql, multiprocess, munin, mysql, mysql-binlog,
mysql-bulk, mysql-load, mysql-prepared-statement, mysql-query, mysql-replicator,
mysqlslowquery, mysqlslowquerylog, nats, network-probe, nginx-status, nicorepo, norikra,
notifier, numeric-counter, numeric-monitor, onlineuser, openldap-monitor, opentsdb, order,
out-http, out-http-buffered, out-solr, parser, pgdist, pghstore, pgjson, ping-message,
postgres, qqwry, rambler, rawexec, rds-log, rds-slowlog, reassemble, record
http://www.ﬂuentd.org/plugins

Fluentd patterns

1.
read logs from file
and write these on storages
file in_tail
read, parse
out_file
format, write
file

1.
file
read, parse insert
MongoDBout_mongo
https://github.com/fluent/fluent-plugin-mongo
in_tail

1.
ﬁle
read, parse
out_mysql
insert
MySQL
https://github.com/tagomoris/ﬂuent-plugin-mysql
in_tail

1.
ﬁle
read, parse
out_elasticsearch
send
Elasticsearch
https://github.com/uken/ﬂuent-plugin-elasticsearch
in_tail

1.
file
read, parse
out_webhdfs
format, write
Hadoop HDFS
https://github.com/fluent/fluent-plugin-webhdfs
in_tail

1.
file
read, parse
out_s3
format, write
Amazon S3
https://github.com/fluent/fluent-plugin-s3
in_tail

1.
ﬁle
read, parse
out_redshift
insert
Amazon
Redshift
https://github.com/hapyrus/ﬂuent-plugin-redshift
in_tail

1.
ﬁle
read, parse
out_bigquery
insert
Google
BigQuery
https://github.com/tagomoris/ﬂuent-plugin-bigquery
in_tail

2.
receive and forward data
from/to other node
forward
forward
forward
input
events
input
events
output
events
ﬂuent-logger-ruby
ﬂuent-logger-java
...
send events over TCP

2.
from/to other node
forward
forward
forward
load balance, active-standby forward
forward
forward

datacenter
2’.
from/to other node, over internet & SSL
secure-forward
secure-forward
datacenter
secure-forward
send events over SSL
with authentication
https://github.com/tagomoris/ﬂuent-plugin-secure-forward

3.
connect with other middleware
in_syslog
syslog
Flume
Scribe
Kafka
in_flume
in_scribe
in_kafka
out_flume
in_scribe
in_kafka
Flume
Scribe
Kafka
https://github.com/fluent/fluent-plugin-flume
https://github.com/fluent/fluent-plugin-scribe
https://github.com/htgc/fluent-plugin-kafka/

4.
copy events
forward copy
forward
webhdfs Hadoop HDFS

5.
count events by string values
forward any outputs
count records
by regexp patterns
events
{
“pattern1_count”: 60,
“pattern1_rate” : 1.0,
...
}
datacounter
https://github.com/tagomoris/ﬂuent-plugin-datacounter

5.
count events by numeric values
forward numeric-counter any outputs
count records
by numerical range
https://github.com/tagomoris/ﬂuent-plugin-numeric-counter
events
{
...
}

5.
aggregate numeric values
forward numeric-monitor any outputs
calculate real-time metrics
of numeric values
{
“max”: 128, “min”: 16,
“avg”: 64.0, “sum”: 1024,
“num”: 20,
“percentile_50”: 48,
“percentile_90”: 112,
...
}
https://github.com/tagomoris/ﬂuent-plugin-numeric-monitor
events

6.
various inputs: Linux performance (dstat)
in_dstatdstat
collect server performance data
https://github.com/shun0102/ﬂuent-plugin-dstat

6.
various inputs: SQL execution
in_sql
input from SELECT
RDBMS
https://github.com/ﬂuent/ﬂuent-plugin-sql

6.
various inputs: external command
in_execany commands
input from STDOUT of any commands

7.
various outpus: notiﬁcation on IRC
out_ikachan
notice on IRC channel
IRC
https://github.com/tagomoris/ﬂuent-plugin-ikachan

7.
various outpus: notiﬁcation on IRC
out_ikachan
notice on IRC channel
IRC
https://github.com/tagomoris/ﬂuent-plugin-ikachan
14:56 ikachan: HTTP status_4xx crit [2014-06-23 14:56:29 +0900] serviceX: 100.00 (threshold
75.0) http://graph.tool.local/view_graph/accesslog/httpstatus/serviceX_4xx_percentage
14:57 kazeburo: ↑ 40x 100%...

7.
various outpus: notiﬁcation on HipChat
out_hipchat
notice on HipChat
HipChat
https://github.com/hotchpotch/ﬂuent-plugin-hipchat

7.
various outpus: graph tools
out_growthforecast
POST data into graph tools
GrowthForecast
or
Focuslight
https://github.com/tagomoris/ﬂuent-plugin-growthforecast

7.
various outpus
out_growthforecast
POST data into graph tools
GrowthForecast
or
Focuslight
https://github.com/tagomoris/ﬂuent-plugin-growthforecast

7.
various outpus: external command
out_exec any commands
output into STDIN of any commands

8. ﬁlters:
stream processing: external command
any inputs any outputs
format & write
into STDIN
exec_ﬁlter
any commands
read & parse
from STDOUT
read from STDIN
do WHATEVER you want
write into STDOUT
ex: tail -f | grep ... | sed ... | cat
events

8. ﬁlters:
stream processing w/ external server RPC
any inputs any outputs
send
out_norikra
fetch
stream processing w/ SQL
in_norikra
http://norikra.github.io/
SELECT stage, score, COUNT(*) AS c
FROM results.win:time_batch(1 min)
WHERE stage > 1 AND user.valid
GROUP BY stage, score
events

... And,
Fluentd does
error handling and retries
for all of these plugins!

Before Fluentd: CHAOS
Access logs
App logs
System logs
Various logs
Metrics
Analytics
Archives
apache
nginx
frontend
backend
syslogd
snmp data
graphs
Hadoop
MySQL
MongoDB
Redshift
Amazon S3
Filesystem
tail -f
scp
python
ruby
ruby
cmd
ﬁle ruby
logger
ﬁle
logger
ruby
cmd
ruby

Fluentd: Now and then

Fluentd versions
Latest: v0.10.50
released on Jun 17, 2014
v0.10.x: Stable versions
many minor feature updates, bug ﬁxes
new features for v1

Fluentd v1
Planned as the first major release
someday in 2014 (?)
100% Compatible with v0.10.x
New (and additional) features on v1.x loadmap
https://github.com/fluent/fluentd/issues/251
new configuration syntax, plugin backends
daemon process management
multi core CPU supports

Fluentd on JRuby
Under development!
trying to ﬁx Cool.io to support JRuby

Fluentd on Windows
Under development!
“windows” branch on github ﬂuent/ﬂuend

Use case in LINE

Analytics data ﬂow overview
servers Fluentd
Cluster
archive
visualization
notiﬁcations
Hadoop
Fluentd
Norikra
application
metrics

servers Fluentd
Cluster
archive
visualization
notiﬁcations
Hadoop
Fluentd
Norikra
application
metrics
delivery/stream-map
aggregate/stream-reduce

archive
visualization
notiﬁcations
Hadoop
Norikra
application
metrics
ﬂuent-agent-lite
non-parsed raw logs
non-parsed
access logs
deliver: receive/archive/load-balance
worker:
parse/store/forward
watcher: monitor/notify
cep:
general-purpose
stream processing

Fluentd cluster statistics
Fluentd nodes
access/application logs from 600+ nodes
receiver: 5 server (60 process)
parser/converter: 10 server (90 process)
stream processing: 3 server

Fluentd cluster statistics
Daily:
5.5+ Billion events, 1.5TB+ data
Peak time:
150,000+ events /sec, 300+ Mbps

Fluentd is the best partner
for stream-processing newbies
and rubyists!
Check out sites and code!
http://fluentd.org/
https://github.com/fluent/fluentd

Fault-tolerance?
Node level fault-tolerance
File buffer: processing data can be serialized on disk
Cluster level fault-tolerance
Copy + Forward(load balance, active-standby)
Event level assurance: ACK?
NO (for performance reason)

Performance?
NOT SO BAD:
real throughput depends on plugin/conﬁguration
simple event transferring: 10-20k events/sec

vs Scribe? vs Flume?

vs Storm?

Eco-system? Clones?
ik
ﬂuent-agent-lite
ﬂuenpy

Fluentd: Data streams in Ruby world #rdrc2014

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (8)

Similaire à Fluentd: Data streams in Ruby world #rdrc2014

Similaire à Fluentd: Data streams in Ruby world #rdrc2014 (20)

Plus de SATOSHI TAGOMORI

Plus de SATOSHI TAGOMORI (20)

Dernier

Dernier (20)

Fluentd: Data streams in Ruby world #rdrc2014