4/2, 4/3に #urapad で使った発表資料

1. Hadoop
in

2. • id:sasata299 ( )
• Hadoop
•
• http://blog.livedoor.jp/sasata299/

4. Hadoop

5. Hadoop MySQL

6. …
Hadoop
MySQL

7. :-)

8. 1. Hadoop
2.
3.
4.
5.

9. Hadoop

10. 915
30 3 1

11. ( )

12. ( )

13. ‣
‣ GROUP BY (
( Д`)
‣ 7000 (
)

14. !!

15. Hadoop

16. ‣ Google MapReduce
‣
‣

17. mapper reducer
( ) ( )

19. ‣ Hadoop Streaming
‣ Ruby
‣ Cloudera CDH1 (0.18.3)
‣ EC2 Hadoop (
10 50 )
‣ Hadoop S3

20. S3 Native FileSystem (s3n://)
‣
‣ 5GB
S3 Block FileSystem (s3://)
‣
‣ HDFS
‣

22. ( )
mapper ( )

23. mapper, reducer

24. master
‣ -file master slave
hadoop jar xxx.jar
-mapper hoge.rb -reducer fuga.rb
-file hoge.rb -file fuga.rb
-file outdata
‣ mapper, reducer File.open
File.open(‘outdata’) {|f| ...}

25. S3
‣ hadoop cat
※
‣ -cacheFile S3
slave (
File.open) ※
hadoop jar xxx.jar
-mapper hoge.rb -reducer fuga.rb
-file hoge.rb -file fuga.rb
-cacheFile s3n://path/to/outdata#othername
mapper reducer

27. 7000 ( )→

28. 7000 ( )→
30

29. Hadoop++

31. Hadoop …
mapper, reducer …
Hadoop
…

32. 1.
2. Hadoop
3. Hadoop
4. Hadoop
5. Hadoop (
)
6. (Excel )
7. Hadoop

33. Hadoop
!!

34. • MySQL
Hadoop
• Hadoop EC2/S3
• Hadoop Streaming Hadoop