This is a demo that use apps script to demo the lambda dashboard. The apps script publish a endpoint and client using fluentd to post data to apps script and also bigquery. Then, you can see the realtime and batch query in the same view.

1. Lambda Architecture using
Google Cloud plus Apps
collect your log at a large scale scope
Simon Su
Hadoop Conf 2015

2. https://www.facebook.com/groups/GCPUG.TW/
https://plus.google.com/u/0/communities/116100913832589966421
[您知道”GCPUG”要怎麼唸嗎？為什麼會有一隻狗在 Logo裡面呢？]
Google Cloud Platform User Group的縮寫是GCPUG
GCPUG直接唸成G.C.P.U.G？當然可以！
但它也可以分開來，唸成 G.C. PUG喔～
Pug，指的是巴哥犬，所以 GCPUG的Logo中間才會有一隻可愛的巴哥犬喲。
下次聽到別人說G.C. PUG 的時候，您就可以大聲 說：「我也是G.C. PUG社團成員！」

3. var simon = {/** I am at GCPUG.TW **/};
simon.aboutme = 'http://about.me/peihsinsu';
simon.nodejs = ‘http://opennodes.arecord.us';
simon.googleshare = 'http://gappsnews.blogspot.tw'
simon.nodejsblog = ‘http://nodejs-in-example.blogspot.tw';
simon.blog = ‘http://peihsinsu.blogspot.com';
simon.slideshare = ‘http://slideshare.net/peihsinsu/';
simon.email = ‘simonsu.mail@gmail.com’;
simon.say(‘Good luck to everybody!');
哇細項～

4. 今天講的不是這個ㄛ….

5. Lambda Architecture是一個大資料分析的理想模式～
即時 完整

6. ● 存得起來的,叫做Storage (儲存)
● 看得到的,叫做Data (資料)
● 看得懂的,叫做Information (資訊)
● 用得出來的,才能夠叫做Intelligent (智慧)
您需要大資料？！
和沛科技 CEO & Founder - 翟本喬

8. Pain point?
● 沒有資料＠＠
→ 蒐集資料的工程太複雜
● 真的要存...
→ 資料太大、太多
● 存好了，查太難＠＠...
→ 分析工具學習門檻高
● 查出來，結果不是要的 ...
→ 較難回饋查詢
● 終於是我要的...
→ 不過，過了市場需求時間

9. 告訴你，在Cloud上可以輕鬆收集分析大資料...

10. 大資料蒐集的資料傳輸

11. <source>
type forward
port 24224
tag mytag
</source>
<match mytag.**>
type forward
<server>
host 192.168.0.1
port 24224
</server>
</match>
<filter mytag.access>
type record_transformer
<record>
host_param "#{Socket.gethostname}"
</record>
</filter>
Input -> filter 1 -> ... -> filter N -> Output

12. 大資料蒐集的分析工具
select top(title), count(*)
from publicdata:samples.wikipedia
Scanning 1 TB in 1 sec
takes 5,000 disks

13. select
repository.language as lang,
count(*) as total
from publicdata:samples.github_nested
where
repository.language is not null
group by lang
order by total desc limit 100
BigQuery所支援友善的查詢方式

14. BigQuery - User Defined Function (UDF)

15. 大資料的呈現工具

16. 免錢最好～

17. 大資料的分析架構 - Lambda Architecture

21. Cloud
Logging
fluentd / client
(out:cloud logging)
streaming insert
(hot data flow)
bq load (cold data flow)
batch
query
Syslog Apache Log SNMP HTTP(s)...
http / json
post

22. 準備HTTP Data Endpoint
gcloud compute --project "m-plaza" instances create "instance-1"
--zone "asia-east1-b"
--machine-type "n1-standard-1"
--network "default"
--metadata "startup-script-url=https://dl.google.com/cloudagents/install-logging-agent.sh"
--image "https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/backports-
debian-7-wheezy-v20150915"
--boot-disk-size "10"
--boot-disk-type "pd-standard"
--boot-disk-device-name "instance-1"

23. <match **>
type copy
<store>
type https_json
use_https true
buffer_path /tmp/buffer
buffer_chunk_limit 256m
buffer_queue_limit 128
flush_interval 3s
endpoint https://script.google.com/macros/s/AK...Vyl/exec
</store>
<store>
type google_cloud
buffer_chunk_limit 512K
flush_interval 5s
max_retry_wait 300
disable_retry_limit
</store>
</match>
準備你的Fluentd Config
<source>
type http
port 9880
</source>
啟動http listener
啟動http json輸出、
cloud loggin輸出

24. 準備一個Report Server
相關設定可以參考： https://github.com/GoogleCloudPlatform/lambda-dashboard

25. while [ true ]
do
curl -X POST
-d "json={"metric":"cpu","usage":`w | grep load | awk -F',' '{print $3}' | awk
'{print $3}'`}"
http://localhost:9880/test5.tag.cpu
sleep 3;
done
測試

26. 查看我們的Dashboard

27. 有一天... 如果資料再長大....

28. Google還有哪些雲端服務可以幫上忙？
Cloud Monitor Cloud LoggingCloud Storage Cloud DataflowCloud Pub/Sub

29. Cloud
Logging
fluentd / client
(out:cloud
logging)
fluentd / client
(out:secure
forward)
direct import
batch
import
solution 1: for lightware data solution 2: for big data batch query solution 3: for realtime dashboard
Query
Engine
- Unknow format log
data
- Archive purpose
google-fluentd
gateway
(in:secure forward)
Custom Metrics
Plugin
- Well known format
log data
- Detail report use
- Frequency query
Google Cloud Platform
On-premise
Other
Implemented
Client
- Customized program
(mobile, web, IoT...)

30. Custom Metrics
https://github.com/gcpug-tw/gcp-demo/blob/master/demo-cloudmonitor/stock-monitor.md

31. <match **>
type exec
command /opt/node/bin/node /path/to/your/google-metrics-loader.js
format json
time_format %Y-%m-%d %H:%M:%S
flush_interval 5s
buffer_path /tmp/
</match>
Sample config
metric.writeTimeSeries(
project_id,
metricid,
value,
{"custom.cloudmonitoring.googleapis.com/cpu" : cpuUsageValue},
null,
null,
function(err, req, doc){
if(err) console.log('ERROR:', err);
console.log(doc);
});

32. 考慮自動擴展

33. Query Engine

34. Q&A

35. MiTAC - Cloud Team 徵才
FrontEnd engineer
● Skills: HTML (must), Node.js (must) Javascript (must)
Python (better)
● Experience: 3 years related experience
Backend engineer
● Skills: Shell script (must), Node.js or Python, RESTful
API build up experience, Google Cloud Platform user.
● Experience: 1-3 years related experience
工商服務時間
請聯絡：cklouie@mitac.com.tw

36. Qnap - Cloud Solution Team 徵才
FrontEnd engineer
● Skills: HTML, Javascript, Node.js or Python, Love to
Learn!
● Experience: 2 years related experience or more
Backend engineer
● Skills: Shell script (must), Node.js or Python, RESTful
API build up experience, Storage or Network knowledge.
Love to Learn!
● Experience: 1-3 years related experience or more
Billing engineer
● Skills: Billing system related knowledge, Java or Python,
● Experience: 2-5 years related experience or more
工商服務時間
請聯絡：peihsinsu@qnap.com

37. GCPUG.TW Call for Speaker
● 凡對Google Cloud上的應用有使用心得想要分享、想要吐槽、想要
尋求大家幫忙解決的，歡迎來 GCPUG.TW給個Session，跟大家分享
一下在使用Google Cloud Platform的點點滴滴。
● 現場報名，直接送您 T-shirt!
工商服務時間
請聯絡：simonsu.mail@gmail.com