11/12/15に行われたEMR勉強会で使ったスライドです #emrstudy_jp

1. EMR

2. • (@sasata299)
• Hadoop
•
•
• Rails, Hadoop, NoSQL

5. [PR] NoSQL

7. 1. Hadoop
2. EMR
3. EMR
4.

8. 1. Hadoop

10. 2009/9
•
• MySQL
• GROUP BY …
• 7000
• Hadoop

12. 2009/10
• EC2 Hadoop
• Cloudera CDH1
• Ruby Hadoop Streaming
• 7000 →30
• Hadoop

13. Hadoop++
←Hadoop
↓MySQL

14. 2. EMR

15. 2010/7
• Hadoop
• Hadoop
• SocketTimeoutException …
• CDH2
• EMR

17. EMR vs CDH2
AMI
(Amazon Machine
Image)
UP
EMR
CDH2

18. EMR vs CDH2
AMI
(Amazon Machine
Image)
UP
EMR
CDH2

21. 2010/8
• EMR
•
• Hadoop
•

24. 3. EMR

25. DB
• xx UU
• UU
•
• , etc...

26. • MySQL MySQL
• MySQL EMR
- UU
-
-

27. EMR
•
- ○○ xx
• Ruby
•
•

28. 4.

29. •
•
- 1
• 5 …

30. [13930, 29011, 39291, ...] # 50000
1000
{
‘139’ => [13930, 13989, 13991, ...], # 50
‘290’ => [29011, 29098, 29076, ...], # 50
‘392’ => [39291, 39244, 39251, ...], # 50
...
}

31. • …
• mapper → reducer → finalize
• script-runner.jar
•

32. •
• IF
•
• EMR

33. • EMR
•
•
• Hadoop Streaming
• :-)

38. @sasata299