Building Big Data Infrastructure with Cloud Service KCA version

1. Building Big Data
Infrastructure with Cloud
Service
KKBOX/MOPCON
ericpi

2. About ericpi (畢瑄易)
• KKBOX
– RD Center Senior Engineer
– COO Manager / KH Side Leader
• Mobile/Open/Platform CONference
– MOPCON 2012/2013/2014/2015 Organizer
• Kaohsiung Software Developer Group
(KSDG)
– Organizer

3. 關於 Big Data, 可能很多人聽過 3Vs

4. 關於 Big Data, 可能很多人聽過 4Vs

5. IBM 說 4Vs 是 -
http://www.ibmbigdatahub.com/infographic/four-vs-big-data

6. 也有人說 4Vs 應該是
• volume
• variety
• velocity
• variability

7. 一個 Big Data, 各自 表述 ?

8. 還好有個組織很負責任跳出來…
NIST: http://bigdatawg.nist.gov/V1_output_docs.php

9. NIST WG 定義 Big Data
• Big Data refers to the inability of tradit
ional data architectures to efficiently
handle the new datasets.
• characteristics (4Vs)
– volume
– variety
– velocity
– variability

10. NIST WG 還強調了…
• Big Data consists of extensive datase
ts -
primarily in the characteristics of volu
me, variety, velocity, and/or variability
-
that require a scalable architecture for
efficient
storage, manipulation, and analysis.

11. NIST WG 還強調了…
• The Big Data paradigm consists of th
e distribution of data systems across
horizontally
coupled, independent resources to ac
hieve the scalability needed for the eff
icient
processing of extensive datasets.

12. NIST並未著墨 多大 & 什麼硬體

13. efficient & scalability 倒是常出現

14. 本日熱門2 - Cloud Computing
• NIST 定義 Essential Characteristics
– Broad network access
– Resource pooling
– Rapid elasticity
– Measured service

15. 本日熱門2 - Cloud Computing
• NIST 定義 Service Models
– SaaS (Software as a Service)
– PaaS (Platform as a Service)
– IaaS (Infrastructure as a Service)

16. 本日熱門2 - Cloud Computing
• NIST 定義 Service Models
– SaaS (Software as a Service)
– PaaS (Platform as a Service)
– IaaS (Infrastructure as a Service)

17. Big Data 跟 Cloud Computing
• 從資料的角度
– 資料存儲量
– 跨機房存取
– 異地同步

18. Big Data 跟 Cloud Computing
• 從運算規模
及基礎架構
– 硬體投資
– 超高運算量
– 分散運算

19. 所有創新產業的基礎工業

20. 所有創新產業的基礎工業
軟體

21. 2011/08 WSJ:
Why Software Is Eating The World
http://www.wsj.com/articles/SB10001424053111903480904576512250915629460

22. Big Data / Cloud Computing 在
NIST 文件中 99% 在定義軟體架構

23. 未來的珍希資源 - 軟體工程師

24. 未來的珍希資源 - 資料科學家

25. 未來最珍貴的資源都在 - 人才

26. 架構實踐 - 資料來源多元性
• 找到正確資料源
– Website / Mobile
Service
– Open Data
– Open API
• 資訊收集工具
– Centralized Log
Management
– Spider / Crawler
– 3rd Application

27. 架構實踐 - 資料來源多元性
• 對應雲端服務
– IaaS: EC2
– PaaS
• Elastic Beanstalk
• Lambda
• 資訊收集工具
– Centralized Log
Management
– Spider / Crawler
– 3rd Application

28. 架構實踐 - 資料型別多樣性
• RDBMS
• NoSQL

29. 架構實踐 - 資料型別多樣性
• RDBMS
• NoSQL
• 異質資料存儲
• NoSQL 資料塑模

30. 架構實踐 - 資料型別多樣性
• RDBMS
• NoSQL
• 雲端方案
– RDS
– SimpleDB, S3,
DynamoDB
– @EC2: Cassandra,
Hbase.. blah

31. 架構實踐 - 資料處理的流量
• 巨量資料處理的顯學: MapReduce
http://www.pinaldave.com/bimg/mapreduce.jpg

32. 架構實踐 - 資料處理的流量
• cloud service 的 reduce 方案
– Elastic MapReduce

33. 架構實踐 - 資料角色
• 不只異質, 而且可以異位

34. 參考架構探討 – NIST Ref. Arch
http://bigdatawg.nist.gov/V1_output_docs.php

35. 參考架構探討 – AWS Large Scale
Processing and Huge Data sets
http://aws.amazon.com/architecture/

36. THANKS
官方網站
http://mopcon.org/
粉絲群
http://fb.me/mopcon