Presentation of the project to the Treasure Data team

1. Eduardo Silva
eduardo@treasuredata.com
@edsiper
Fluent Bit
September 08, 2015
Treasure Hunter School

2. About Me
Eduardo Silva
●
Github & Twitter @edsiper
●
Personal Blog http://edsiper.linuxchile.cl
Treasure Data
●
Open Source Engineer
●
Fluentd / Fluent Bit http://github.com/fluent
Projects
●
Monkey HTTP Server http://monkey-project.com
●
Duda I/O http://duda.io

3. Embedded World

4. Embedded World
It's not new...
●
Embedded systems have been around for
decades: automotive, medical devices, cameras, etc.
●
Hardware resources are very limited.
●
Embedded is emerging within a new concept: IoT

5. Internet of Things

6. Internet of Things
Facts
●
IoT will grow to many billions of devices over the
next decade.
●
Now it's about device to device connectivity.
●
Different frameworks and protocols are emerging.
●
It needs Logging.

7. Internet of Things
Alliances
Vendors formed alliances to join forces and develop
generic software layers for their products:

8. Internet of Things
Solutions provided
Alliance Framework
→
→

9. IoT and Big Data
Analytics
IoT requires a generic solution to collect events and
data from different sources for further analysis.
Data can come from a specific framework, radio device,
sensor and others. How do we collect and unify data
properly ?

10. Internet of Things
Market needs
The IoT market is growing and the device to device
connectivity is a must, for hence a strong and fair
software platform is required for a reliable system.

11. IoT Software Layers
Interoperability
Hardware vendors are aware that interoperability with
third party products is a success key. A generic
software layer across products is the answer.

12. In a galaxy
not so far away...

13. @fluentbit

14. Fluent Bit is an open source
data collector
It let's you collect data from IoT/Embedded
devices and transport It to third party
services.

15. Fluent Bit
Targets
●
Services
●
Sensors / Signals / Radios
●
Operating System information
●
Automotive / Telematics

16. Fluent Bit
Requirements
IoT and Embedded environment requires special
handling, specifically on performance and resources:
●
Lightweight
●
Written in C Language
●
Customizable, pluggable architecture
●
Full integration with Fluentd

17. Fluent Bit
I/O

18. Fluent Bit
Integration with Fluentd

19. Fluent Bit
Direct Output

20. Fluent Bit
and Fluentd
Fluentd Fluent Bit
Scope Servers Embedded & IoT devices
Language C & Ruby C
Performance High Performance High Performance
Dependencies Built as a Ruby Gem, it
requires a certain number of
gems.
Zero dependencies, unless some
special plugin requires them.
Plugins More than 300 plugins
available
Less than 10 plugins available

21. Architecture

22. Architecture
Global Overview

23. Architecture
Core Engine
●
Everything related to initialization.
●
Interface plugins.
●
Abstract network operations.
●
Make the magic happens.

24. Architecture
Plugins
●
Abstraction level between the Engine
and data handlers.
●
Provide registration mechanism.
●
Expose callbacks used in the Engine
event loop.

25. Architecture
Input Plugins
●
Collect data.
●
Behave as a network service, built-in
metric or generator.
●
It runs at intervals of time (triggered
by the Engine) or upon file descriptor
events.

26. Architecture
Input Plugins (as of 0.2 dev)
xbee get messages from XBee radio devices.
cpu metric plugin, gather and calculate CPU
usage.
mem metric plugin, gather system memory usage.
kmsg get messages from the Linux Kernel Log.
serial get messages from the Serial input interface.
stdin get messages from the standard input
interface.
mqtt MQTT TCP server, get publish messages.

27. Architecture
Output Plugins
●
Take buffered data and enqueue it
for delivery.
●
Handle plugins
●
Provides the main event loop.
●

28. Architecture
Output Plugins
fluentd Forward messages to Fluentd over TCP.
td Upload data to Treasure Data service.
stdout Print messages to the standard output.

29. Architecture
Stack Providers
Fluent Bit have three dependencies
which are distributed in the source
code and are linked statically.
These components are helpers for
the Engine and Plugins Manager.

30. Architecture
MessagePack
MessagePack is an efficient binary
serialization format. This component
is heavily used to pack JSON into a
binary version.

31. Architecture
mbedTLS
mbedTLS provides a Transport
Layer Security (TLS) for network
operations, for short, It provides TLS
and SSL encryption for network
operations that require secure
channels.

32. Architecture
Monkey
Monkey is a web server that provides
an embeddable HTTP stack and some
extra routines to handle configuration
files, memory handling, event loops,
string manipulation within others.

33. Architecture
Library Mode

34. Architecture
Library Mode

35. Architecture
Configuration Schema

36. Fluent Bit
Development
Contributors Takeshi Hasegawa, Masaya Yamamoto, Naoya Kaneko within others

37. Fluent Bit
Documentation

38. Fluent Bit
Thank you!
●
http://fluentbit.io
●
http://github.com/fluent/fluentd
●
http://twitter.com/fluentbit