Contenu connexe
Plus de Tatsuya Sasaki (6)
Hadoopをemr経由で利用する方法
- 2. • (@sasata299)
• NoSQL
•
• http://blog.livedoor.jp/sasata299/
- 5. •
• EC2 Hadoop & S3
• Cloudera (CDH1)
•
• Hadoop Streaming (Ruby )
•
- 6. •
• ( )
•
• master ssh
• Hadoop (HADOOP-6254)
• S3 cpu
• S3 → …
- 16. EMR CDH2
AMI
(Amazon Machine
UP Image)
EMR
CDH2
- 17. EMR CDH2
AMI
(Amazon Machine
UP Image)
EMR
CDH2
- 22. EMR
BootStrap Action
Step (Hadoop Job)
Job Flow ( )
- 23. EMR
BootStrap Action
Step (Hadoop Job)
Job Flow ( )
- 24. EMR
BootStrap Action
Step (Hadoop Job)
Job Flow ( )
- 25. EMR
BootStrap Action
Step (Hadoop Job)
Job Flow ( )
- 26. ( )
elastic-mapreduce
--create #
--num-instances 10 # master:1 , slave:9
--bootstrap-action s3n://xxx/hoge.sh #
--alive #
- 27. ( )
elastic-mapreduce
--create #
--num-instances 10 # master:1 , slave:9
--bootstrap-action s3n://xxx/hoge.sh #
--alive #
Created job flow j-8IXS98OW1WEE
ID
- 28. ( )
elastic-mapreduce
--stream # Hadoop streaming
--input, --output, --mapper, --reducer #
--cache s3n://xxx/fuga.rb #
--jobconf xxx=yyy #
--jobflow j-xxxxx # ID
- 29. ( )
elastic-mapreduce
--stream # Hadoop streaming
--input, --output, --mapper, --reducer #
--cache s3n://xxx/fuga.rb #
--jobconf xxx=yyy #
--jobflow j-xxxxx # ID
- 30. •
•
•
• --alive
• AMI
• Cloudera AMI
• BootStrap Action
- 31. •
• mapred.child.java.opts
• Java
• Streaming
•
•
• ElasticMapReduce-master 5100
- 32. • EMR
Hadoop
• EMR
•
• --alive