Contenu connexe Similaire à Hadoop Conference Japan 2011 Fallに行ってきました Similaire à Hadoop Conference Japan 2011 Fallに行ってきました (20) Hadoop Conference Japan 2011 Fallに行ってきました2. Hadoop Conference Japan 2011 fall
• 2011/9/26( )
http://hadoop-conference-japan-2011-fall.eventbrite.com/
• Hadoop
1000
•
Cloudera HortonWorks MapR
•
NTT mixi Yahoo
2
9. Hadoop
• Google
GFS(ACM ’03) “The Google File System” → HDFS
MapReduce(OSDI ’04) “Simplified Data Processing on Large Clusters” → MapReduce
BigTable(OSDI ’06) “A Distributed Storage System for Structured Data” → HBase
• Yahoo! Inc. Cloudera Doug Cutting Google
2004 Java OSS
”Hadoop” Doug
9
10. Hadoop
•
index
index etc...
※Google
※ Doug Lucene Nutch OSS
• Hadoop
• Hadoop
10
11. Hadoop
• HBase
BigTable
• ZooKeeper
Chubby
• Hive
HDFS SQL
• Flume
• and more....
11
15. Hadoop 0.23
• 2011 Q4 beta
Integration Test Pig
• Hadoop
NameNode SPoF (NameNode
• MapReduce v2 : MapReduce
MPI(C Giraph Hama Graph Spark etc
•
MapReduce shuffle 30% / 10,000 4000 / etc
15
18. H
• Cloudera
http://www.cloudera.com/
OSS (CDH Sqoop Hue
• HortonWorks
http://www.hortonworks.com/
Yahoo! Inc. Hadoop
• MapR
http://www.mapr.com/
Hadoop
•
Hadoop
18
19. H Cloudera
• CDH
Linux
• OSS
Sqoop SQL HDFS Hue Web Console etc..
• SCM Express
Web Hadoop
50 free
• Cloudera Enterprise
Cloudera Management Suite
•
19
20. H HortonWorks
• Yahoo! Inc. Hadoop 22
Hadoop ※ (
500,000
• 2011/7
• Yahoo! Inc.
Yahoo! Mail
42,000 Hadoop
• Horton the Elephant
http://en.wikipedia.org/wiki/Horton_the_Elephant
20
21. H MapR
• Hadoop
EMC OEM Greenplum
http://www.greenplum.com/
•
HDFS C++ HDFS
random read/write
NIC bonding
snapshot etc..
NFS Hadoop
R
• OSS Hadoop
21
22. • Hadoop
LSP etc..
• Asakusa Framework OSS
https://github.com/asakusafw
DSL
22
23. Yahoo! Japan
•
Hadoop
Geohash Hadoop Iterable
Twitter
※ MPI ...
23
25. • Hadoop
BtoC
Hadoop 0.23 MapReduce v2 Hadoop
• Hadoop Hadoop
Map/Reduce + Graph
BI
25