This is a short training for introduce Pi to use fluentd to collect data and use Google Cloud Logging and BigQuery as backend and then use Apps Script and Google Sheet as presentation layer.

1. Linker Networks Corp.
http://www.linkernetworks.com
Use Raspberry Pi + Fluentd + GCP
Cloud Logging, BigQuery
as IoT Data Collect & Analysis
Simon Su @ LinkerNetworks

2. var simon = {/** I am at GCPUG.TW **/};
simon.GDE = 'https://developers.google.com/experts/people/peihsin-su';
simon.nodejs = ‘http://opennodes.arecord.us';
simon.googleshare = 'http://gappsnews.blogspot.tw'
simon.nodejsblog = ‘http://nodejs-in-example.blogspot.tw';
simon.blog = ‘http://peihsinsu.blogspot.com';
simon.slideshare = ‘http://slideshare.net/peihsinsu/';
simon.email = ‘simonsu.mail@gmail.com’;
simon.say(‘Let’s go Cloud!');

3. Welcome to join us!

4. What we use...
● Docker
● Fluentd
● Google Cloud Logging
● Google BigQuery
● SpreadSheet

5. Docker
● Build, Ship and Run any
App, Anywhere
● DevOps spirit
● Fast to scale

6. Fluentd
● Easy and fast
● Message delivery for at
most once, at least
once, exactly once

7. BigQuery
● TB Level data analysis
● Fast mining tool
● SQL like interface
● Multi-dataset join support
● Cheap & Pay by Use
Google BigQuery - Big data with SQL like query feature, but fast...

8. Cloud Logging

9. Architecture
● BigQuery: Data Warehouse and
support query and analysis
● Cloud Logging: Log gateway
and easy to transfer to one or
multi destination with guarantee
● Fluentd: Device layer gateway
and support multi source for
user to extend http(s) input
tcp/udp input
forward / secure forward
other inputs
● Customize schema
● Co-relation
● Higher quota and
performance
● Web console
● Export available for BQ,
GCS, Pub/Sub
● Configure input source

10. Prepare Log Gateway using
Cloud Logging

11. Still no GCP project?? (https://cloud.google.com/free-trial/)

12. curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh
sudo bash install-logging-agent.sh
service google-fluentd start
Install Cloud Log Agent From: https://cloud.google.com/logging/docs/agent/installation

13. Configure
# /etc/google-fluentd/google-fluentd.conf
<source>
type http
port 9880
</source>
<source>
@type forward
port 24224
</source>
<filter *.**>
@type stdout
</filter>
# Do not collect fluentd's own logs to avoid infinite loops.
<match fluent.**>
type null
</match>
# Configure all sources to output to Google Cloud Logging
<match *.**>
type google_cloud
# Set the chunk limit conservatively to avoid exceeding the limit
# of 10MB per write request.
buffer_chunk_limit 2M
flush_interval 5s
# Never wait longer than 5 minutes between retries.
max_retry_wait 300
# Disable the limit on the number of retries (retry forever).
disable_retry_limit
# Use multiple threads for processing.
num_threads 8
</match>
Set config to open fluentd forward
and also screen output for debug..

14. Test run & watch log...
# service google-fluentd start
# tail -f /var/log/google-fluentd/google-fluentd.log
2016-09-02 09:46:44 +0000 hddusage.simon-project.a_54ERXdr4inKCGiAAAA:
{"catg":"hddusage","ctime":null,"size":null,"used":null,"avail":null,"use":null,"linker_devid":"simon-project.a_54ERXdr4inKCGiAAAA"}
2016-09-02 09:46:44 +0000 hddusage.simon-project.a_54ERXdr4inKCGiAAAA:
{"catg":"hddusage","ctime":null,"size":null,"used":null,"avail":null,"use":null,"linker_devid":"simon-project.a_54ERXdr4inKCGiAAAA"}
2016-09-02 09:46:44 +0000 hddusage.simon-project.a_54ERXdr4inKCGiAAAA:
{"catg":"hddusage","ctime":null,"size":null,"used":null,"avail":null,"use":null,"linker_devid":"simon-project.a_54ERXdr4inKCGiAAAA"}
….(skip)

15. Prepare a Docker enabled Pi

16. Install Docker into Fluentd
http://blog.hypriot.com/downloads/

17. Build the Fluentd Docker for Pi

18. Getting start with fluentd… (https://peihsinsu.gitbooks.io/fluentd-in-action/content/)

19. Build your Raspberry Pi Fluentd - Dockerfile
# Dockerfile
FROM hypriot/rpi-alpine-scratch
MAINTAINER linkeriot
RUN adduser -D -g '' -u 1000 docker
ENV FLUENTD_VERSION 0.14
RUN apk --update add
build-base
ca-certificates
ruby-dev
&&
rm -rf /var/cache/apk/* &&
echo 'gem: --no-document' >> /etc/gemrc &&
gem install fluentd -v $FLUENTD_VERSION &&
apk del build-base
WORKDIR /fluentd
RUN mkdir log etc plugins config.d
COPY fluent.conf /fluentd/etc/
ONBUILD COPY fluent.conf /fluentd/etc/
ONBUILD COPY plugins/* /fluentd/plugins/
ENV FLUENTD_OPT=""
ENV FLUENTD_CONF="fluent.conf"
EXPOSE 24224
EXPOSE 8888
VOLUME ["/fluentd/log"]
CMD fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT

20. Build your Raspberry Pi Fluentd - fluentd.conf
# fluentd.conf
@include /fluentd/config.d/*.conf
<label @mainstream>
<filter **>
@type record_transformer
<record>
linker_devid "#{ENV['TAG']}"
</record>
</filter>
<match **>
@type forward
send_timeout 60s
recover_wait 10s
heartbeat_interval 1s
phi_threshold 16
hard_timeout 60s
<server>
name loggateway
host "#{ENV['FDN']}"
port "#{ENV['FPORT']}"
weight 60
</server>
<secondary>
@type file
path /var/log/fluent/forward-failed
</secondary>
</match>
</label>

21. Build and publish
● docker build -t your-image-name .
● docker push your-image-name

22. Run the Docker...

23. Run from your images
docker run -d -p 24224:24224 --restart=always
-e FDN=[remote-server-ip]
-e FPORT=[remote-server-port]
-e TAG=[your-tag]
-v /var/log:/fluentd/log
-v /data/cfg:/fluentd/config.d
yout-image-name

24. Run using linker build images
docker run -d -p 24224:24224 --restart=always
-e FDN=[remote-server-ip]
-e FPORT=[remote-server-port]
-e TAG=[your-tag]
-v /var/log:/fluentd/log
-v /data/cfg:/fluentd/config.d
linkeriot/iot-log-agent

25. Set log forward to BigQuery

26. Check log insert status...

27. Enable log forward to BigQuery...

28. BigQuery from SpreadSheet

29. Try your query….

30. Create Apps Script from Google SpreadSheet

31. Using my sample...
Get source from gist: http://goo.gl/J90QD6

32. Test run the script

33. Setup the schedule...

34. Create the chart...

35. The End
Q&A