SlideShare une entreprise Scribd logo
1  sur  20
Télécharger pour lire hors ligne
Toru Takahashi
June 17, 2015
IoTLT vol.4
IoT Data Connector - Fluent Bit
#iotlt
About Me
> Toru Takahashi
> twitter: @nora96o
> github: toru-takahashi
> Treasure Data, Inc.
> Technical Support Engineer
> I love working in Maid café :)
> Most of staying in 女中酒場幻橙館
2
Today’s Talk focus … on IoT
mbed Device Server
mbed
OS
Wi-Fi
mbed
OS
Wi-Fi
mbed
OS
Wi-Fi
mbed Device Server
mbed
OS
Wi-Fi
mbed
OS
Wi-Fi
mbed
OS
Wi-Fi
mbed Device Server
mbed
OS
Wi-Fi
mbed
OS
Wi-Fi
mbed
OS
Wi-Fi
> Application Layer
> Logging Layer
3
Logging Layer on Web
Before
5
Structured logging
Reliable forwarding
Pluggable architecture
http://fluentd.org/
After
7
What’s Fluentd?
> Data collector for unified logging layer
> Streaming data transfer based on JSON
> Written in Ruby
> Gem based various plugins
> http://www.fluentd.org/plugins
> Working in production
> http://www.fluentd.org/testimonials
8
Logging Layer on IoT
Current
?
10
IoT Data Connector Fluent Bit
Future
12
Fluent Bit
Open Source data collection tool for Embedded Linux
> Sensors
> Services
> Signals / Radios
> Operating System Info
> Automotive / Telematics
Committer: Eduardo Silva (@edsiper)
• https://github.com/fluent/fluent-bit
• http://fluentbit.io (official web site)
13
Features
> Collection & Distribution
> Support Custom Input / Output
> Configurable (through file system files)
> Built-in system metrics
> C API for Developers (WIP)
> Integration with third party services
> Open Source / Apache License v2.0
Fluent Bit
14
Examples
CPU Usage
$ ./fluent-bit -i cpu -o stdout
[2015/06/14 12:13:35] [ info] Flush buf 120 bytes
[0] {"time"=>1434284015, "cpu"=>12.000000}
[1] {"time"=>1434284016, "cpu"=>100.000000}
[2] {"time"=>1434284017, "cpu"=>100.000000}
[3] {"time"=>1434284018, "cpu"=>100.000000}
[4] {"time"=>1434284019, "cpu"=>82.000000}
[4] {"time"=>1434284024, "cpu"=>0.000000}
Built-in Metrics:
16
Kernel Log Message
$ ./fluent-bit -i kmsg -o stdout
[2015/06/14 12:19:00] [ info] starting engine
[0] {"time"=>1434283478, "priority"=>6, "sequence"=>0, "sec"=>0, "usec"=>0,
"msg"=>"Initializing cgroup subsys cpuset"}
[1] {"time"=>1434283478, "priority"=>6, "sequence"=>1, "sec"=>0, "usec"=>0,
"msg"=>"Initializing cgroup subsys cpu"}
[2] {"time"=>1434283478, "priority"=>6, "sequence"=>2, "sec"=>0, "usec"=>0,
"msg"=>"Initializing cgroup subsys cpuacct"}
[3] {"time"=>1434283478, "priority"=>5, "sequence"=>3, "sec"=>0, "usec"=>0,
"msg"=>"Linux version 3.14.35-28.38.amzn1.x86_64 (mockbuild@gobi-
build-64012) (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP
Wed Mar 11 22:50:37 UTC 2015"}
Built-in Metrics:
17
Config File
[TD]
# API
# ===
# The TreasureData API key. To obtain this please log into your
API SOME_API_KEY
# Database
# ========
# Specify the name of your database, it must exists.
Database db_example
# Table
# =====
# Specify the database table name where the records will be stored
Table table_example
Output to TreasureData:
18
Roadmap
• Library mode
• Support a stock of sensors (inputs)
• HTTP input
• Release first stable version
• Documentation
19
IoT❤
https://github.com/fluent/fluent-bit

Contenu connexe

Tendances

Unifying Events and Logs into the Cloud
Unifying Events and Logs into the CloudUnifying Events and Logs into the Cloud
Unifying Events and Logs into the CloudEduardo Silva Pereira
 
FluentD for end to end monitoring
FluentD for end to end monitoringFluentD for end to end monitoring
FluentD for end to end monitoringPhil Wilkins
 
Tracing 2000+ polyglot microservices at Uber with Jaeger and OpenTracing
Tracing 2000+ polyglot microservices at Uber with Jaeger and OpenTracingTracing 2000+ polyglot microservices at Uber with Jaeger and OpenTracing
Tracing 2000+ polyglot microservices at Uber with Jaeger and OpenTracingYuri Shkuro
 
Fluentd Project Intro at Kubecon 2019 EU
Fluentd Project Intro at Kubecon 2019 EUFluentd Project Intro at Kubecon 2019 EU
Fluentd Project Intro at Kubecon 2019 EUN Masahiro
 
Fluentd and Distributed Logging at Kubecon
Fluentd and Distributed Logging at KubeconFluentd and Distributed Logging at Kubecon
Fluentd and Distributed Logging at KubeconN Masahiro
 
Collaborating through data sxsw
Collaborating through data sxswCollaborating through data sxsw
Collaborating through data sxswJeff Branson
 
How to Shrink from 5 Tiers to 2 in a Multitier Microservices Architecture
 How to Shrink from 5 Tiers to 2 in a Multitier Microservices Architecture How to Shrink from 5 Tiers to 2 in a Multitier Microservices Architecture
How to Shrink from 5 Tiers to 2 in a Multitier Microservices Architecturevsoshnikov
 
Fluentd: Unified Logging Layer at CWT2014
Fluentd: Unified Logging Layer at CWT2014Fluentd: Unified Logging Layer at CWT2014
Fluentd: Unified Logging Layer at CWT2014N Masahiro
 
Master-Master Replication and Scaling of an Application Between Each of the I...
Master-Master Replication and Scaling of an Application Between Each of the I...Master-Master Replication and Scaling of an Application Between Each of the I...
Master-Master Replication and Scaling of an Application Between Each of the I...vsoshnikov
 
InfluxData Architecture for IoT | Noah Crowley | InfluxData
InfluxData Architecture for IoT | Noah Crowley | InfluxDataInfluxData Architecture for IoT | Noah Crowley | InfluxData
InfluxData Architecture for IoT | Noah Crowley | InfluxDataInfluxData
 
Introduction to InfluxDB and TICK Stack
Introduction to InfluxDB and TICK StackIntroduction to InfluxDB and TICK Stack
Introduction to InfluxDB and TICK StackAhmed AbouZaid
 
Timur Shemsedinov "Эволюция архитектуры ИС"
Timur Shemsedinov "Эволюция архитектуры ИС"Timur Shemsedinov "Эволюция архитектуры ИС"
Timur Shemsedinov "Эволюция архитектуры ИС"OdessaJS Conf
 
Nick Raienko ''Service-oriented GraphQL''
Nick Raienko ''Service-oriented GraphQL''Nick Raienko ''Service-oriented GraphQL''
Nick Raienko ''Service-oriented GraphQL''OdessaJS Conf
 
Build your own discovery index of scholary e-resources
Build your own discovery index of scholary e-resourcesBuild your own discovery index of scholary e-resources
Build your own discovery index of scholary e-resourcesMartin Czygan
 
InfluxDB Client Libraries and Applications | Miroslav Malecha | Bonitoo
InfluxDB Client Libraries and Applications | Miroslav Malecha | BonitooInfluxDB Client Libraries and Applications | Miroslav Malecha | Bonitoo
InfluxDB Client Libraries and Applications | Miroslav Malecha | BonitooInfluxData
 
Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays N...
Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays N...Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays N...
Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays N...InfluxData
 
MQTT on Raspberry Pi with node.js
MQTT on Raspberry Pi with node.jsMQTT on Raspberry Pi with node.js
MQTT on Raspberry Pi with node.jsPaul Tanner
 
Introduction to InfluxDB, an Open Source Distributed Time Series Database by ...
Introduction to InfluxDB, an Open Source Distributed Time Series Database by ...Introduction to InfluxDB, an Open Source Distributed Time Series Database by ...
Introduction to InfluxDB, an Open Source Distributed Time Series Database by ...Hakka Labs
 

Tendances (20)

Unifying Events and Logs into the Cloud
Unifying Events and Logs into the CloudUnifying Events and Logs into the Cloud
Unifying Events and Logs into the Cloud
 
FluentD for end to end monitoring
FluentD for end to end monitoringFluentD for end to end monitoring
FluentD for end to end monitoring
 
Tracing 2000+ polyglot microservices at Uber with Jaeger and OpenTracing
Tracing 2000+ polyglot microservices at Uber with Jaeger and OpenTracingTracing 2000+ polyglot microservices at Uber with Jaeger and OpenTracing
Tracing 2000+ polyglot microservices at Uber with Jaeger and OpenTracing
 
Fluentd Project Intro at Kubecon 2019 EU
Fluentd Project Intro at Kubecon 2019 EUFluentd Project Intro at Kubecon 2019 EU
Fluentd Project Intro at Kubecon 2019 EU
 
Fluentd and Distributed Logging at Kubecon
Fluentd and Distributed Logging at KubeconFluentd and Distributed Logging at Kubecon
Fluentd and Distributed Logging at Kubecon
 
Collaborating through data sxsw
Collaborating through data sxswCollaborating through data sxsw
Collaborating through data sxsw
 
Fluentd 101
Fluentd 101Fluentd 101
Fluentd 101
 
How to Shrink from 5 Tiers to 2 in a Multitier Microservices Architecture
 How to Shrink from 5 Tiers to 2 in a Multitier Microservices Architecture How to Shrink from 5 Tiers to 2 in a Multitier Microservices Architecture
How to Shrink from 5 Tiers to 2 in a Multitier Microservices Architecture
 
Fluentd: Unified Logging Layer at CWT2014
Fluentd: Unified Logging Layer at CWT2014Fluentd: Unified Logging Layer at CWT2014
Fluentd: Unified Logging Layer at CWT2014
 
Master-Master Replication and Scaling of an Application Between Each of the I...
Master-Master Replication and Scaling of an Application Between Each of the I...Master-Master Replication and Scaling of an Application Between Each of the I...
Master-Master Replication and Scaling of an Application Between Each of the I...
 
InfluxData Architecture for IoT | Noah Crowley | InfluxData
InfluxData Architecture for IoT | Noah Crowley | InfluxDataInfluxData Architecture for IoT | Noah Crowley | InfluxData
InfluxData Architecture for IoT | Noah Crowley | InfluxData
 
Introduction to InfluxDB and TICK Stack
Introduction to InfluxDB and TICK StackIntroduction to InfluxDB and TICK Stack
Introduction to InfluxDB and TICK Stack
 
Timur Shemsedinov "Эволюция архитектуры ИС"
Timur Shemsedinov "Эволюция архитектуры ИС"Timur Shemsedinov "Эволюция архитектуры ИС"
Timur Shemsedinov "Эволюция архитектуры ИС"
 
Nick Raienko ''Service-oriented GraphQL''
Nick Raienko ''Service-oriented GraphQL''Nick Raienko ''Service-oriented GraphQL''
Nick Raienko ''Service-oriented GraphQL''
 
Build your own discovery index of scholary e-resources
Build your own discovery index of scholary e-resourcesBuild your own discovery index of scholary e-resources
Build your own discovery index of scholary e-resources
 
InfluxDB Client Libraries and Applications | Miroslav Malecha | Bonitoo
InfluxDB Client Libraries and Applications | Miroslav Malecha | BonitooInfluxDB Client Libraries and Applications | Miroslav Malecha | Bonitoo
InfluxDB Client Libraries and Applications | Miroslav Malecha | Bonitoo
 
Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays N...
Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays N...Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays N...
Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays N...
 
MQTT on Raspberry Pi with node.js
MQTT on Raspberry Pi with node.jsMQTT on Raspberry Pi with node.js
MQTT on Raspberry Pi with node.js
 
Introduction to InfluxDB, an Open Source Distributed Time Series Database by ...
Introduction to InfluxDB, an Open Source Distributed Time Series Database by ...Introduction to InfluxDB, an Open Source Distributed Time Series Database by ...
Introduction to InfluxDB, an Open Source Distributed Time Series Database by ...
 
Airbox with Gemtek Lora
Airbox with Gemtek LoraAirbox with Gemtek Lora
Airbox with Gemtek Lora
 

En vedette

2015 6-14-raspberry pi×milkcocoa 玄関の明かりを監視!
2015 6-14-raspberry pi×milkcocoa 玄関の明かりを監視!2015 6-14-raspberry pi×milkcocoa 玄関の明かりを監視!
2015 6-14-raspberry pi×milkcocoa 玄関の明かりを監視!Syuhei Hiya
 
代沢アンペアメーター Iot縛りの勉強会!vol4
代沢アンペアメーター Iot縛りの勉強会!vol4代沢アンペアメーター Iot縛りの勉強会!vol4
代沢アンペアメーター Iot縛りの勉強会!vol4Daisuke Hirata
 
Edisonで温度,湿度,気圧を測ってみる
Edisonで温度,湿度,気圧を測ってみるEdisonで温度,湿度,気圧を測ってみる
Edisonで温度,湿度,気圧を測ってみるhrdakinori
 
みんなでつくるスマートハウス
みんなでつくるスマートハウスみんなでつくるスマートハウス
みんなでつくるスマートハウスDaisuke Kimura
 
クラウドファンディングを活用したIo t事例集
クラウドファンディングを活用したIo t事例集クラウドファンディングを活用したIo t事例集
クラウドファンディングを活用したIo t事例集Ouma オーマ
 
Io t縛りの勉強会 #4
Io t縛りの勉強会 #4Io t縛りの勉強会 #4
Io t縛りの勉強会 #4Daichi Morifuji
 
Wireless coffee
Wireless coffeeWireless coffee
Wireless coffeeKenji Ohno
 
2015 06-16 iot縛りの勉強会資料用素材
2015 06-16 iot縛りの勉強会資料用素材2015 06-16 iot縛りの勉強会資料用素材
2015 06-16 iot縛りの勉強会資料用素材Masanori Takano
 
はなまるが欲しいから Milkcocoaでどうにかしてみた!
はなまるが欲しいから Milkcocoaでどうにかしてみた!はなまるが欲しいから Milkcocoaでどうにかしてみた!
はなまるが欲しいから Milkcocoaでどうにかしてみた!angie angie
 
みんなでおばけになる #mlkcca
みんなでおばけになる #mlkccaみんなでおばけになる #mlkcca
みんなでおばけになる #mlkccaHikari Fukasawa
 
とんちきでもmilkcocoaでなんとかできた件について
とんちきでもmilkcocoaでなんとかできた件についてとんちきでもmilkcocoaでなんとかできた件について
とんちきでもmilkcocoaでなんとかできた件についてyaksk
 
ビッグデータ分析基盤を支えるOSSたち
ビッグデータ分析基盤を支えるOSSたちビッグデータ分析基盤を支えるOSSたち
ビッグデータ分析基盤を支えるOSSたちToru Takahashi
 
AngularJS×Milkcocoaでちょっとくらい社会貢献してみたい
AngularJS×Milkcocoaでちょっとくらい社会貢献してみたいAngularJS×Milkcocoaでちょっとくらい社会貢献してみたい
AngularJS×Milkcocoaでちょっとくらい社会貢献してみたいShinnosuke Morimoto
 
MilkcocoaとArduino うまい!やすい!はやい!Lチカからの脱却劇
MilkcocoaとArduino うまい!やすい!はやい!Lチカからの脱却劇MilkcocoaとArduino うまい!やすい!はやい!Lチカからの脱却劇
MilkcocoaとArduino うまい!やすい!はやい!Lチカからの脱却劇Masayuki Kurashita
 
ハードとMilkcocoaをもっと近づける
ハードとMilkcocoaをもっと近づけるハードとMilkcocoaをもっと近づける
ハードとMilkcocoaをもっと近づけるwamisnet
 
IoT超初心者がMilkcocoaからラズベリーパイに遠隔Lチカしてみた
IoT超初心者がMilkcocoaからラズベリーパイに遠隔LチカしてみたIoT超初心者がMilkcocoaからラズベリーパイに遠隔Lチカしてみた
IoT超初心者がMilkcocoaからラズベリーパイに遠隔LチカしてみたKayoko Hashi
 
Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~
Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~
Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~Sotaro Kimura
 
はじめてのMilkcocoa 〜2時間目〜
はじめてのMilkcocoa 〜2時間目〜はじめてのMilkcocoa 〜2時間目〜
はじめてのMilkcocoa 〜2時間目〜Kiyoe Furuichi
 

En vedette (20)

2015 6-14-raspberry pi×milkcocoa 玄関の明かりを監視!
2015 6-14-raspberry pi×milkcocoa 玄関の明かりを監視!2015 6-14-raspberry pi×milkcocoa 玄関の明かりを監視!
2015 6-14-raspberry pi×milkcocoa 玄関の明かりを監視!
 
IotLt vol4_r2
IotLt vol4_r2IotLt vol4_r2
IotLt vol4_r2
 
代沢アンペアメーター Iot縛りの勉強会!vol4
代沢アンペアメーター Iot縛りの勉強会!vol4代沢アンペアメーター Iot縛りの勉強会!vol4
代沢アンペアメーター Iot縛りの勉強会!vol4
 
Edisonで温度,湿度,気圧を測ってみる
Edisonで温度,湿度,気圧を測ってみるEdisonで温度,湿度,気圧を測ってみる
Edisonで温度,湿度,気圧を測ってみる
 
みんなでつくるスマートハウス
みんなでつくるスマートハウスみんなでつくるスマートハウス
みんなでつくるスマートハウス
 
クラウドファンディングを活用したIo t事例集
クラウドファンディングを活用したIo t事例集クラウドファンディングを活用したIo t事例集
クラウドファンディングを活用したIo t事例集
 
Io t縛りの勉強会 #4
Io t縛りの勉強会 #4Io t縛りの勉強会 #4
Io t縛りの勉強会 #4
 
Wireless coffee
Wireless coffeeWireless coffee
Wireless coffee
 
2015 06-16 iot縛りの勉強会資料用素材
2015 06-16 iot縛りの勉強会資料用素材2015 06-16 iot縛りの勉強会資料用素材
2015 06-16 iot縛りの勉強会資料用素材
 
はなまるが欲しいから Milkcocoaでどうにかしてみた!
はなまるが欲しいから Milkcocoaでどうにかしてみた!はなまるが欲しいから Milkcocoaでどうにかしてみた!
はなまるが欲しいから Milkcocoaでどうにかしてみた!
 
みんなでおばけになる #mlkcca
みんなでおばけになる #mlkccaみんなでおばけになる #mlkcca
みんなでおばけになる #mlkcca
 
とんちきでもmilkcocoaでなんとかできた件について
とんちきでもmilkcocoaでなんとかできた件についてとんちきでもmilkcocoaでなんとかできた件について
とんちきでもmilkcocoaでなんとかできた件について
 
ビッグデータ分析基盤を支えるOSSたち
ビッグデータ分析基盤を支えるOSSたちビッグデータ分析基盤を支えるOSSたち
ビッグデータ分析基盤を支えるOSSたち
 
AngularJS×Milkcocoaでちょっとくらい社会貢献してみたい
AngularJS×Milkcocoaでちょっとくらい社会貢献してみたいAngularJS×Milkcocoaでちょっとくらい社会貢献してみたい
AngularJS×Milkcocoaでちょっとくらい社会貢献してみたい
 
MilkcocoaとArduino うまい!やすい!はやい!Lチカからの脱却劇
MilkcocoaとArduino うまい!やすい!はやい!Lチカからの脱却劇MilkcocoaとArduino うまい!やすい!はやい!Lチカからの脱却劇
MilkcocoaとArduino うまい!やすい!はやい!Lチカからの脱却劇
 
ハードとMilkcocoaをもっと近づける
ハードとMilkcocoaをもっと近づけるハードとMilkcocoaをもっと近づける
ハードとMilkcocoaをもっと近づける
 
はじめてのMilkcocoa
はじめてのMilkcocoaはじめてのMilkcocoa
はじめてのMilkcocoa
 
IoT超初心者がMilkcocoaからラズベリーパイに遠隔Lチカしてみた
IoT超初心者がMilkcocoaからラズベリーパイに遠隔LチカしてみたIoT超初心者がMilkcocoaからラズベリーパイに遠隔Lチカしてみた
IoT超初心者がMilkcocoaからラズベリーパイに遠隔Lチカしてみた
 
Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~
Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~
Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~
 
はじめてのMilkcocoa 〜2時間目〜
はじめてのMilkcocoa 〜2時間目〜はじめてのMilkcocoa 〜2時間目〜
はじめてのMilkcocoa 〜2時間目〜
 

Similaire à IoT Data Connector Fluent Bit

Fluentd - RubyKansai 65
Fluentd - RubyKansai 65Fluentd - RubyKansai 65
Fluentd - RubyKansai 65N Masahiro
 
Debugging linux issues with eBPF
Debugging linux issues with eBPFDebugging linux issues with eBPF
Debugging linux issues with eBPFIvan Babrou
 
Reverse engineering Swisscom's Centro Grande Modem
Reverse engineering Swisscom's Centro Grande ModemReverse engineering Swisscom's Centro Grande Modem
Reverse engineering Swisscom's Centro Grande ModemCyber Security Alliance
 
Fluentd and Embulk Game Server 4
Fluentd and Embulk Game Server 4Fluentd and Embulk Game Server 4
Fluentd and Embulk Game Server 4N Masahiro
 
SAP (in)security: Scrubbing SAP clean with SOAP
SAP (in)security: Scrubbing SAP clean with SOAPSAP (in)security: Scrubbing SAP clean with SOAP
SAP (in)security: Scrubbing SAP clean with SOAPChris John Riley
 
Cloud Storage Introduction ( CEPH )
Cloud Storage Introduction ( CEPH )  Cloud Storage Introduction ( CEPH )
Cloud Storage Introduction ( CEPH ) Alex Lau
 
Athens IoT meetup #7 - Create the Internet of your Things - Laurent Ellerbach...
Athens IoT meetup #7 - Create the Internet of your Things - Laurent Ellerbach...Athens IoT meetup #7 - Create the Internet of your Things - Laurent Ellerbach...
Athens IoT meetup #7 - Create the Internet of your Things - Laurent Ellerbach...Athens IoT Meetup
 
Fluentd - road to v1 -
Fluentd - road to v1 -Fluentd - road to v1 -
Fluentd - road to v1 -N Masahiro
 
Microservices Technology Stack
Microservices Technology StackMicroservices Technology Stack
Microservices Technology StackEberhard Wolff
 
Time Series Database and Tick Stack
Time Series Database and Tick StackTime Series Database and Tick Stack
Time Series Database and Tick StackGianluca Arbezzano
 
SecZone 2011: Scrubbing SAP clean with SOAP
SecZone 2011: Scrubbing SAP clean with SOAPSecZone 2011: Scrubbing SAP clean with SOAP
SecZone 2011: Scrubbing SAP clean with SOAPChris John Riley
 
Tick Stack - Listen your infrastructure and please sleep
Tick Stack - Listen your infrastructure and please sleepTick Stack - Listen your infrastructure and please sleep
Tick Stack - Listen your infrastructure and please sleepGianluca Arbezzano
 
Treasure Data and AWS - Developers.io 2015
Treasure Data and AWS - Developers.io 2015Treasure Data and AWS - Developers.io 2015
Treasure Data and AWS - Developers.io 2015N Masahiro
 
Docker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic StackDocker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic StackJakub Hajek
 
Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek PROIDEA
 
IBM Notes Traveler Administration and Log Troubleshooting tips - Part 2
IBM Notes Traveler Administration and Log Troubleshooting tips - Part 2IBM Notes Traveler Administration and Log Troubleshooting tips - Part 2
IBM Notes Traveler Administration and Log Troubleshooting tips - Part 2jayeshpar2006
 
What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...Sveta Smirnova
 
Triangle OpenStack meetup 09 2013
Triangle OpenStack meetup 09 2013Triangle OpenStack meetup 09 2013
Triangle OpenStack meetup 09 2013Dan Radez
 
OpenStack API's and WSGI
OpenStack API's and WSGIOpenStack API's and WSGI
OpenStack API's and WSGIMike Pittaro
 

Similaire à IoT Data Connector Fluent Bit (20)

Fluentd - RubyKansai 65
Fluentd - RubyKansai 65Fluentd - RubyKansai 65
Fluentd - RubyKansai 65
 
Debugging linux issues with eBPF
Debugging linux issues with eBPFDebugging linux issues with eBPF
Debugging linux issues with eBPF
 
Reverse engineering Swisscom's Centro Grande Modem
Reverse engineering Swisscom's Centro Grande ModemReverse engineering Swisscom's Centro Grande Modem
Reverse engineering Swisscom's Centro Grande Modem
 
Fluentd and Embulk Game Server 4
Fluentd and Embulk Game Server 4Fluentd and Embulk Game Server 4
Fluentd and Embulk Game Server 4
 
SAP (in)security: Scrubbing SAP clean with SOAP
SAP (in)security: Scrubbing SAP clean with SOAPSAP (in)security: Scrubbing SAP clean with SOAP
SAP (in)security: Scrubbing SAP clean with SOAP
 
Pycon - Python for ethical hackers
Pycon - Python for ethical hackers Pycon - Python for ethical hackers
Pycon - Python for ethical hackers
 
Cloud Storage Introduction ( CEPH )
Cloud Storage Introduction ( CEPH )  Cloud Storage Introduction ( CEPH )
Cloud Storage Introduction ( CEPH )
 
Athens IoT meetup #7 - Create the Internet of your Things - Laurent Ellerbach...
Athens IoT meetup #7 - Create the Internet of your Things - Laurent Ellerbach...Athens IoT meetup #7 - Create the Internet of your Things - Laurent Ellerbach...
Athens IoT meetup #7 - Create the Internet of your Things - Laurent Ellerbach...
 
Fluentd - road to v1 -
Fluentd - road to v1 -Fluentd - road to v1 -
Fluentd - road to v1 -
 
Microservices Technology Stack
Microservices Technology StackMicroservices Technology Stack
Microservices Technology Stack
 
Time Series Database and Tick Stack
Time Series Database and Tick StackTime Series Database and Tick Stack
Time Series Database and Tick Stack
 
SecZone 2011: Scrubbing SAP clean with SOAP
SecZone 2011: Scrubbing SAP clean with SOAPSecZone 2011: Scrubbing SAP clean with SOAP
SecZone 2011: Scrubbing SAP clean with SOAP
 
Tick Stack - Listen your infrastructure and please sleep
Tick Stack - Listen your infrastructure and please sleepTick Stack - Listen your infrastructure and please sleep
Tick Stack - Listen your infrastructure and please sleep
 
Treasure Data and AWS - Developers.io 2015
Treasure Data and AWS - Developers.io 2015Treasure Data and AWS - Developers.io 2015
Treasure Data and AWS - Developers.io 2015
 
Docker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic StackDocker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic Stack
 
Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek
 
IBM Notes Traveler Administration and Log Troubleshooting tips - Part 2
IBM Notes Traveler Administration and Log Troubleshooting tips - Part 2IBM Notes Traveler Administration and Log Troubleshooting tips - Part 2
IBM Notes Traveler Administration and Log Troubleshooting tips - Part 2
 
What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...
 
Triangle OpenStack meetup 09 2013
Triangle OpenStack meetup 09 2013Triangle OpenStack meetup 09 2013
Triangle OpenStack meetup 09 2013
 
OpenStack API's and WSGI
OpenStack API's and WSGIOpenStack API's and WSGI
OpenStack API's and WSGI
 

Plus de Toru Takahashi

Learn from Case Study; How do people run query on Trino? / Trino japan virtua...
Learn from Case Study; How do people run query on Trino? / Trino japan virtua...Learn from Case Study; How do people run query on Trino? / Trino japan virtua...
Learn from Case Study; How do people run query on Trino? / Trino japan virtua...Toru Takahashi
 
高専後10年の学び - 高専カンファレンス In 仙台
高専後10年の学び - 高専カンファレンス In 仙台高専後10年の学び - 高専カンファレンス In 仙台
高専後10年の学び - 高専カンファレンス In 仙台Toru Takahashi
 
TokyoGirls.rb meetup vol.1 SponsorLT
TokyoGirls.rb meetup vol.1 SponsorLTTokyoGirls.rb meetup vol.1 SponsorLT
TokyoGirls.rb meetup vol.1 SponsorLTToru Takahashi
 
Zendesk Sunshine - Zenlab vol.6
Zendesk Sunshine - Zenlab vol.6Zendesk Sunshine - Zenlab vol.6
Zendesk Sunshine - Zenlab vol.6Toru Takahashi
 
History of TreasureData Support
History of TreasureData SupportHistory of TreasureData Support
History of TreasureData SupportToru Takahashi
 
Dairy of Support Engineering Manager
Dairy of Support Engineering ManagerDairy of Support Engineering Manager
Dairy of Support Engineering ManagerToru Takahashi
 
ZendeskのTriggerを有効活用するためにデータを一元化している話
ZendeskのTriggerを有効活用するためにデータを一元化している話ZendeskのTriggerを有効活用するためにデータを一元化している話
ZendeskのTriggerを有効活用するためにデータを一元化している話Toru Takahashi
 
EmbulkとDigdagとデータ分析基盤と
EmbulkとDigdagとデータ分析基盤とEmbulkとDigdagとデータ分析基盤と
EmbulkとDigdagとデータ分析基盤とToru Takahashi
 
(Draft) lambda architecture by using TreasureData
(Draft) lambda architecture by using TreasureData(Draft) lambda architecture by using TreasureData
(Draft) lambda architecture by using TreasureDataToru Takahashi
 

Plus de Toru Takahashi (10)

Learn from Case Study; How do people run query on Trino? / Trino japan virtua...
Learn from Case Study; How do people run query on Trino? / Trino japan virtua...Learn from Case Study; How do people run query on Trino? / Trino japan virtua...
Learn from Case Study; How do people run query on Trino? / Trino japan virtua...
 
高専後10年の学び - 高専カンファレンス In 仙台
高専後10年の学び - 高専カンファレンス In 仙台高専後10年の学び - 高専カンファレンス In 仙台
高専後10年の学び - 高専カンファレンス In 仙台
 
TokyoGirls.rb meetup vol.1 SponsorLT
TokyoGirls.rb meetup vol.1 SponsorLTTokyoGirls.rb meetup vol.1 SponsorLT
TokyoGirls.rb meetup vol.1 SponsorLT
 
Zendesk Sunshine - Zenlab vol.6
Zendesk Sunshine - Zenlab vol.6Zendesk Sunshine - Zenlab vol.6
Zendesk Sunshine - Zenlab vol.6
 
History of TreasureData Support
History of TreasureData SupportHistory of TreasureData Support
History of TreasureData Support
 
Zenlab - API Night
Zenlab - API NightZenlab - API Night
Zenlab - API Night
 
Dairy of Support Engineering Manager
Dairy of Support Engineering ManagerDairy of Support Engineering Manager
Dairy of Support Engineering Manager
 
ZendeskのTriggerを有効活用するためにデータを一元化している話
ZendeskのTriggerを有効活用するためにデータを一元化している話ZendeskのTriggerを有効活用するためにデータを一元化している話
ZendeskのTriggerを有効活用するためにデータを一元化している話
 
EmbulkとDigdagとデータ分析基盤と
EmbulkとDigdagとデータ分析基盤とEmbulkとDigdagとデータ分析基盤と
EmbulkとDigdagとデータ分析基盤と
 
(Draft) lambda architecture by using TreasureData
(Draft) lambda architecture by using TreasureData(Draft) lambda architecture by using TreasureData
(Draft) lambda architecture by using TreasureData
 

Dernier

OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 

Dernier (20)

OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 

IoT Data Connector Fluent Bit

  • 1. Toru Takahashi June 17, 2015 IoTLT vol.4 IoT Data Connector - Fluent Bit #iotlt
  • 2. About Me > Toru Takahashi > twitter: @nora96o > github: toru-takahashi > Treasure Data, Inc. > Technical Support Engineer > I love working in Maid café :) > Most of staying in 女中酒場幻橙館 2
  • 3. Today’s Talk focus … on IoT mbed Device Server mbed OS Wi-Fi mbed OS Wi-Fi mbed OS Wi-Fi mbed Device Server mbed OS Wi-Fi mbed OS Wi-Fi mbed OS Wi-Fi mbed Device Server mbed OS Wi-Fi mbed OS Wi-Fi mbed OS Wi-Fi > Application Layer > Logging Layer 3
  • 6. Structured logging Reliable forwarding Pluggable architecture http://fluentd.org/
  • 8. What’s Fluentd? > Data collector for unified logging layer > Streaming data transfer based on JSON > Written in Ruby > Gem based various plugins > http://www.fluentd.org/plugins > Working in production > http://www.fluentd.org/testimonials 8
  • 13. Fluent Bit Open Source data collection tool for Embedded Linux > Sensors > Services > Signals / Radios > Operating System Info > Automotive / Telematics Committer: Eduardo Silva (@edsiper) • https://github.com/fluent/fluent-bit • http://fluentbit.io (official web site) 13
  • 14. Features > Collection & Distribution > Support Custom Input / Output > Configurable (through file system files) > Built-in system metrics > C API for Developers (WIP) > Integration with third party services > Open Source / Apache License v2.0 Fluent Bit 14
  • 16. CPU Usage $ ./fluent-bit -i cpu -o stdout [2015/06/14 12:13:35] [ info] Flush buf 120 bytes [0] {"time"=>1434284015, "cpu"=>12.000000} [1] {"time"=>1434284016, "cpu"=>100.000000} [2] {"time"=>1434284017, "cpu"=>100.000000} [3] {"time"=>1434284018, "cpu"=>100.000000} [4] {"time"=>1434284019, "cpu"=>82.000000} [4] {"time"=>1434284024, "cpu"=>0.000000} Built-in Metrics: 16
  • 17. Kernel Log Message $ ./fluent-bit -i kmsg -o stdout [2015/06/14 12:19:00] [ info] starting engine [0] {"time"=>1434283478, "priority"=>6, "sequence"=>0, "sec"=>0, "usec"=>0, "msg"=>"Initializing cgroup subsys cpuset"} [1] {"time"=>1434283478, "priority"=>6, "sequence"=>1, "sec"=>0, "usec"=>0, "msg"=>"Initializing cgroup subsys cpu"} [2] {"time"=>1434283478, "priority"=>6, "sequence"=>2, "sec"=>0, "usec"=>0, "msg"=>"Initializing cgroup subsys cpuacct"} [3] {"time"=>1434283478, "priority"=>5, "sequence"=>3, "sec"=>0, "usec"=>0, "msg"=>"Linux version 3.14.35-28.38.amzn1.x86_64 (mockbuild@gobi- build-64012) (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP Wed Mar 11 22:50:37 UTC 2015"} Built-in Metrics: 17
  • 18. Config File [TD] # API # === # The TreasureData API key. To obtain this please log into your API SOME_API_KEY # Database # ======== # Specify the name of your database, it must exists. Database db_example # Table # ===== # Specify the database table name where the records will be stored Table table_example Output to TreasureData: 18
  • 19. Roadmap • Library mode • Support a stock of sensors (inputs) • HTTP input • Release first stable version • Documentation 19