Enterprise SW로 진화허는 Hadoop - 빅데이터와 Hadoop 현황 및 전망

1. 2014년제3회iTalks 세미나Oct. 1, 2014

2. Contents

3. 빅데이터의짧은역사
출처:HCL Technologies, A History of Big Data, 2013.10.12

4. 빅데이터의짧은역사BigData용어의원조
–JohnR.Mashey(ChiefScientistatSGI),April1998
출처: John R. Mashey, Big Data and the Next Wave of InfraStress,1998

5. 빅데이터의짧은역사BigData용어의도화선
–McKinseyGlobalInstitute,“BigData: Thenextfrontierforinnovation, competition,andproductivity”보고서(2011)
•빅데이타속에서누가먼저가치를추출해내느냐에따라기업의성패가나뉠것이라고언급
•빅데이타가새로운유형의기업자산으로자리잡을것이라고예측
출처: McKinsey Global Institute, “Big Data: The next frontier for innovation, competition, and productivity”, 2011

6. 빅데이터의짧은역사빅데이터의정의(3V)
출처: Industry Insiders Report: Big Data in manufacturing -Part 1

7. 빅데이터의정의(4V)
빅데이터의짧은역사
출처: Gigaom,The Hadoop ecosystem: the (welcome) elephant in the room, 2013.5.5

8. 빅데이터의짧은역사빅데이터의정의(5V)
출처: Data Technocrats, What is big data, 2013.12.27

9. 하둡의더짧은역사HadoopTimeline
–빅데이터의사실상(defacto)의기술표준으로인식,but…
출처: J. Yates Monteith, John D. McGregor, John E. Ingram, Hadoop and its evolving ecosystem, 2013.9.13, 참고수정프로젝트로승격출시설립로전환회명참석
더그커팅합류최대하둡클러스터라주장출시

10. Hadoop생태계
–Distibutions
–3rdPartyMgmt.SW
–OpreationalDatabases
–SQLonHadoop
–Frameworks/Languages
–AnalyticApplications/Platforms
–HadoopasaService(Apps/Alaytics)
–HadoopasaServce(Infra.)
–HadoopRepackaged
–CompetitivePlatforms
–HDFSAlternatives
하둡의더짧은역사
출처: Gigaom,The Hadoop ecosystem: the (welcome) elephant in the room, 2013.5.5

11. 하둡의더짧은역사Hadoop지배자는?(1)
출처: J. Yates Monteith, John D. McGregor, John E. Ingram, Hadoop and its evolving ecosystem, 2013.9.13
: pure-play vendor

12. 하둡의더짧은역사Hadoop지배자는?(2)
출처: Forrester Research, The Forrester Wave™: Big Data Hadoop Solutions, Q1 2014

13. 하둡2.0YARN(YetAnotherResourceNegotiator)
출처: Hortonworks, Hadoop Summit 2014, 2014.6

14. Hadoop1vsHadoop2(1)
하둡2.0
Limited up to 4,000 nodes per cluster
O(# of tasks in a cluster)
JobTracker bottleneck - resource management, job scheduling and monitoring
Only has one namespace for managing HDFS
Map and Reduce slots are static
Only job to run is MapReduce
Potentially up to 10,000 nodes per cluster
O(cluster size)
Supports multiple namespacefor managing HDFS
Efficient cluster utilization (YARN)
MRv1 backward and forward compatible
Any apps can integrate with Hadoop
Beyond Java

15. 하둡2.0Hadoop1vsHadoop2(2)
HADOOP 1.0
HDFS
(redundant, reliable storage)
MapReduce
(cluster resource management
& data processing)
HDFS2
(redundant, highly-available & reliable storage)
YARN
(cluster resource management)
MapReduce
(data processing)
Others
HADOOP 2.0
Single Use System
Batch Apps
Multi Purpose Platform
Batch, Interactive, Online, Streaming, …
출처: Hortonworks, Hadoop Summit 2014, 2014.6

16. 하둡2.0Hadoop1vsHadoop2(3)
출처: Apache Software Foundation, 2013. 10.

17. 하둡2.0YARN:theDataOperatingSystem
출처: Hortonworks, Apache Hadoop YARN-Present and Future, 2014.6

18. 빅데이터, 하둡의현주소및전망국내BigData관심도
출처: Google Trends. 2014.6

19. 빅데이터, 하둡의현주소및전망빅데이터의자화상에대해표현하자면…
–“Bigdataisliketeenagesex:everyonetalksaboutit,nobodyreallyknowshowtodoit,everyonethinkseveryoneelseisdoingit,soeveryoneclaimstheyaredoingit.”(작자미상)
–“빅데이터의현실은아직실체없는경험담과성공사례수집,목적없는기계학습스터디등나침반없는망망대해에있는상황”(최대우교수)
–Conference,세미나,Meetup만하다가끝나겠네(김동한)

20. 잘못된사례-해외(구글독감예측)
빅데이터, 하둡의현주소및전망
출처: * Kate Crawford , Untangling algorithmic illusions from reality in big data, 2013.3측면의데이터이슈측면의데이터이슈

21. 빅데이터, 하둡의현주소및전망잘못된사례-국내(6.4지방선거)
–11개관심선거구,6개적중
–Data의한계,SocialData만이빅데이터?
출처: 와이즈넛, 한국형빅데이터선거분석사이트'초이스화면, 2014.6

22. 빅데이터, 하둡의현주소및전망좋은사례-해외(브라질월드컵)
–SAP,‘매치인사이트’,선수움직임/유형빅데이터로실시간분석
출처: http://blogs.wsj.com/cio/2014/07/10/germanys-12th-man-at-the-world-cup-big-data/

23. 빅데이터, 하둡의현주소및전망좋은사례-국내(서율시심야버스)
–기존노선도:교통카드데이터의기/종착지분석이활용
–심야버스노선:이동통신사의통화량분석(약30억건)

24. 빅데이터, 하둡의현주소및전망빅데이터의전망
출처: KISTI, 빅데이터산업의현황과전망, 2013.4

25. Hadoop도입
빅데이터, 하둡의현주소및전망
출처: Hortonworks, Hadoop Summit 2014, 2014.6

26. 빅데이터, 하둡의현주소및전망Hadoop이기존DW대체?
출처: Wikibon, Q2 2014 Big Data Survey, 2014

27. 빅데이터, 하둡의현주소및전망Hadoop시장전망(1)
출처: Wikibon, Hadoop-NoSQL Software And Services Market Forecast 2012-2017, 2013.9

28. Hadoop시장전망(2)
출처: ExperfyInsights, Hadoop Market Size, Adoption and Growth Through 2020, 2014.6.22

29. 빅데이터, 하둡의현주소및전망하둡의전망:10ReasonstoAdoptHadoop
1.Hadoopisrelativelyinexpensive
2.Hadoophasanactiveopensourcecommunity
3.Hadoopisbeingwidelyadoptedineveryindustry
4.Hadoopcaneasilyscaleoutasyourdatagrows
5.TraditionaltoolsareintegratingwithHadoop
6.Hadoopcanstoredatainanyformat
7.Hadoopisdesignedtoruncomplexanalytics
8.Hadoopcanprocessafulldataset
9.HardwareisbeingoptimizedforHadoop
10.Hadoopcanincreasinglyhandleflexibleworkloads
출처: By Dirk deRoos, Hadoop For Dummies, 2014.4

30. 마무리오늘iTalks을마치며…(1)
출처: 전용준, 산업별Big Data 사례, 이슈및발전방향, 2014.8

31. 마무리오늘iTalks을마치며…(2)
출처: 전용준, 산업별Big Data 사례, 이슈및발전방향, 2014.8

32. iTalks to be continued…