Talk by Sada Furuhashi, Treasure Data. To hear about future conferences go to http://dataengconf.com

1. COLLECTING AND MOVING DATA AT SCALE
Sada Furuhashi
Chief Architect
Invented Fluentd, Messagepack

2. BACKGROUND

3. HIGH LEVEL ANALYTICS ARCHITECTURE
Collect Store Process Visualize

4. THE CHALLENGE
Collect Store Process Visualize
How do we shorten
the collection process?
Easier & Shorter Time Excel
Tableau

5. THE PROBLEM

6. TYPICAL ARCHITECTURE BEFORE FLUENTD
Log Server
Application
App Server
File FileFile
High latency
Must wait for a day
Hard to analyze
Complex text parsers
Application
App Server
File FileFile
Application
App Server
File FileFile

7. THE FALSE SOLUTION

8. MULTIPLY CONNECTIONS / COMBINATION EXPLOSION
LOG
File
script to
parse data
cron job for
loading
filtering
script
syslog
script
Tweet-
fetching
script
aggregation
script
aggregation
script
script to
parse data
rsync
server

9. THE SOLUTION

10. CENTRALIZED CONNECTIONS
LOG
FILE

11. FLUENTD INTERNAL ARCHITECTURE

12. INTERNAL ARCHITECTURE (SIMPLIFIED)
Plugin
Input Filter Buffer Output
Plugin Plugin Plugin
2012-02-04 01:33:51
myapp.buylog{
“user”:”me”,
“path”: “/buyItem”,
“price”: 150,
“referer”: “/landing”
}
Time
Tag
Record

13. ARCHITECTURE: INPUT PLUGINS
HTTP+JSON (in_http)
File tail (in_tail)
Syslog (in_syslog)
…
Receive logs
Or pull logs from data sources
In non-blocking manner
Plugin
Input

14. Filter
ARCHITECTURE: FILTER PLUGINS
Transform logs
Filter out unnecessary logs
Enrich logs
Plugin
Encrypt personal data
Convert IP to countries
Parse User-Agent
…

15. Buffer
ARCHITECTURE: BUFFER PLUGINS
Plugin
Improve performance
Provide reliability
Provide thread-safety
Memory (buf_memory)
File (buf_file)

16. ARCHITECTURE: OUTPUT PLUGINS
Output
Write or send event logs
Plugin
File (out_file)
Amazon S3 (out_s3)
MongoDB (out_mongo)
…

17. Buffer
ARCHITECTURE: BUFFER PLUGINS
Chunk
Plugin
Improve performance
Provide reliability
Provide thread-safety
Input
Output
Chunk
Chunk

18. Retry
Error
Retry
Batch
Stream Error
Retry
Retry
DIVIDE & CONQUER & RETRY

19. EXAMPLE USE CASES

20. STREAMING FROM APACHE TO MONGODB PT I
in_tail
/var/log/access.log
/var/log/fluentd/buffer
but_file

21. ERROR HANDLING
in_tail
/var/log/access.log
/var/log/fluentd/buffer
but_file
Buffering for any outputs
Retrying automatically
With exponential wait
and persistence on a disk

22. TAILING FILE INPUT
Supported formats:
Read a log file
Custom regexp
Custom parser in Ruby
• apache
• apache_error
• apache2
• nginx
• json
• csv
• tsv
• ltsv
• syslog
• multiline
• none
pos fileaccess.log

23. OUT TO MULTIPLE LOCATIONS
Routing based on tags
Copy to multiple storages
buffer
access.log
in_tail

24. H.A. CONFIGURATION (HIGH AVAILABILITY)
Retry automatically
Exponential retry wait
Persistent on a disk
buffer
Automatic fail-over
Load balancing
access.log
in_tail

25. FOR HADOOP USERS
Retry automatically
Exponential retry wait
Persistent on a disk
access.log
buffer
Custom text
formatter
Slice files based on time
2016-01-01/01/access.log.gz
2016-01-01/02/access.log.gz
2016-01-01/03/access.log.gz
…
in_tail

26. HADOOP INTEGRATION INTO S3
Retry automatically
Exponential retry wait
Persistent on a disk
buffer
Slice files based on time
in_tail
2016-01-01/01/access.log.gz
2016-01-01/02/access.log.gz
2016-01-01/03/access.log.gz
…
access.log

27. 3RD PARTY INPUT PLUGINS
dstat
df AMQL
munin
jvmwatcher
SQL

28. 3RD PARTY OUTPUT PLUGINS
AMQL
Graphite

29. REAL WORLD USE CASES

30. HIGH-VOLUME FORWARDING
T R E A S U R E
D A T A
-At-most-once / At-least-once
-HA (failover)
-Load-balancing

31. NEAR REALTIME AND BATCH COMBO
Hot data
All data

32. EXAMPLE CONFIGURATION FOR REAL TIME
BATCH COMBO

33. CEP FOR STREAM PROCESSING
Nora is a SQL based CEP engine: http://norikra.github.io/

34. CONTAINER LOGGING
T R E A S U R E
D A T A

35. FLUENTD IN PRODUCTION

36. MICROSOFT
Operations Management Suite uses Fluentd: "The core of the agent uses an existing
open source data aggregator called Fluentd. Fluentd has hundreds of existing
plugins, which will make it really easy for you to add new data sources."
Syslog
Linux Computer
Operating System
Apache
MySQL
Containers
omsconfig (DSC)
PS DSC
Providers
OMI Server
(CIM Server)
omsagent
Firewall/proxy
OMSService
Upload Data
(HTTPS)
Pull
configuration
(HTTPS)

37. ATLASSIAN
"At Atlassian, we've been impressed by Fluentd and have chosen to use it in
Atlassian Cloud's logging and analytics pipeline."
Kinesis
Elasticsearch
cluster
Ingestion
service

38. AMAZON WEB SERVICES
The architecture of Fluentd (Sponsored by Treasure Data) is very similar to Apache
Flume or Facebook’s Scribe. Fluentd is easier to install and maintain and has better
documentation and support than Flume and Scribe.
Types of DataStoreCollect
Transactional
• Database reads & write (OLTP)
• Cache
Search
• Logs
• Streams
File
• Log files (/val/log)
• Log collectors & frameworks
Stream
• Log records
• Sensors & IoT data
Web Apps
IoTApplicationsLogging
Mobile Apps
Database
Search
File Storage
Stream Storage

39. THANK YOU!