SlideShare a Scribd company logo
1 of 36
3 Hadoop


Hadoop MR + Cassandra


              2010/06/28
             @ryu_kobayashi
• Hadoop MapReduce
• Cassandra
• MapReduce + Cassandra
• Cassandra       MapReduce

•
•
•
     •                       ...

•                              ...

•   Twitter @ryu_kobayashi
Hadoop MapReduce



         Google MapReduce
Cassandra

Facebook       KVS NoSQL       DB
               0.6.2       0.6.3
Cassandra

Facebook       KVS NoSQL       DB
               0.6.2       0.6.3
Cassandra

Facebook         KVS NoSQL       DB
                 0.6.2       0.6.3




  •   Facebook
Cassandra

Facebook         KVS NoSQL       DB
                 0.6.2       0.6.3




  •   Facebook
  •   Twitter
Cassandra

Facebook         KVS NoSQL       DB
                 0.6.2       0.6.3




  •   Facebook
  •   Twitter
  •
Cassandra

Facebook         KVS NoSQL       DB
                 0.6.2       0.6.3




  •   Facebook
  •   Twitter
  •
  •
Cassandra

Facebook         KVS NoSQL       DB
                 0.6.2       0.6.3




  •   Facebook
  •   Twitter
  •
  •

        @shot6
Hadoop MR + Cassandra
Hadoop MR + Cassandra
Hadoop MR + Cassandra
Hadoop MR + Cassandra


Cassandra Contributer
Hadoop MR + Cassandra
Hadoop MR + Cassandra
Hadoop MR + Cassandra
Hadoop MR + Cassandra




   Hadoop    Cassandra   H
H
Hadoop MR + Cassandra
Hadoop MR + Cassandra
•   MR HDFS
Hadoop MR + Cassandra
•   MR HDFS
•    HDFS       30 40
Hadoop MR + Cassandra
•        MR HDFS
•         HDFS                30 40


•   Cassandra 0.6   Hadoop
                             (API     )
Hadoop MR + Cassandra
•        MR HDFS
•         HDFS                  30 40


•   Cassandra 0.6      Hadoop
                                (API    )
•        KVS                    HDFS
               Hbase
Hadoop MR + Cassandra
•        MR HDFS
•         HDFS                        30 40


•   Cassandra 0.6      Hadoop
                                     (API     )
•        KVS                         HDFS
               Hbase
•
               Cassandra        InputFormat
                RDBS

       AWS EMR(Elastic MapReduce + S3
Hadoop MR + Cassandra
Hadoop MR + Cassandra
Hadoop MR + Cassandra
Hadoop MR + Cassandra
Cassandra               HDFS   +TaskTracker
            Cassandra
Hadoop MR + Cassandra
Cassandra               HDFS   +TaskTracker
            Cassandra

                  Contribute
Cassandra                     MapReduce

•   org.apache.cassandra.hadoop

•                               storage-conf.xml
    # Thrift          storage-conf.xml
•              Jar Cassadra Thrift
               classpath

•   OutputFormat                     HDFS
               (0.7                      )
•   1000                                     Map


    # FAQ Issue        http://goo.gl/5Q3Z
Cassandra            MapReduce




            github


    http://bit.ly/9nwgSu
Cassandra-GUI
http://code.google.com/p/cassandra-gui/
Hadoopソースコードリーディング第3回 Hadopo MR + Cassandra

More Related Content

What's hot

Acunu overview
Acunu overviewAcunu overview
Acunu overview
Acunu
 
Cassandra Hadoop Best Practices by Jeremy Hanna
Cassandra Hadoop Best Practices by Jeremy HannaCassandra Hadoop Best Practices by Jeremy Hanna
Cassandra Hadoop Best Practices by Jeremy Hanna
Modern Data Stack France
 

What's hot (20)

Cloud Optimized Big Data
Cloud Optimized Big DataCloud Optimized Big Data
Cloud Optimized Big Data
 
Messaging architecture @FB (Fifth Elephant Conference)
Messaging architecture @FB (Fifth Elephant Conference)Messaging architecture @FB (Fifth Elephant Conference)
Messaging architecture @FB (Fifth Elephant Conference)
 
Hadoop cassandra training
Hadoop cassandra trainingHadoop cassandra training
Hadoop cassandra training
 
Big Data A La Carte Menu
Big Data A La Carte MenuBig Data A La Carte Menu
Big Data A La Carte Menu
 
Concepts on Hadoop
Concepts on HadoopConcepts on Hadoop
Concepts on Hadoop
 
Hadoop and MapReduce
Hadoop and MapReduceHadoop and MapReduce
Hadoop and MapReduce
 
Nextag talk
Nextag talkNextag talk
Nextag talk
 
Welcome to the Jungle: Distributed Systems for Large Data Sets - StampedeCon ...
Welcome to the Jungle: Distributed Systems for Large Data Sets - StampedeCon ...Welcome to the Jungle: Distributed Systems for Large Data Sets - StampedeCon ...
Welcome to the Jungle: Distributed Systems for Large Data Sets - StampedeCon ...
 
Introduction to Big Data & Hadoop Architecture - Module 1
Introduction to Big Data & Hadoop Architecture - Module 1Introduction to Big Data & Hadoop Architecture - Module 1
Introduction to Big Data & Hadoop Architecture - Module 1
 
Hadoop and Distributed Computing
Hadoop and Distributed ComputingHadoop and Distributed Computing
Hadoop and Distributed Computing
 
Hadoop An Introduction
Hadoop An IntroductionHadoop An Introduction
Hadoop An Introduction
 
Facebook - Jonthan Gray - Hadoop World 2010
Facebook - Jonthan Gray - Hadoop World 2010Facebook - Jonthan Gray - Hadoop World 2010
Facebook - Jonthan Gray - Hadoop World 2010
 
Apache Hadoop at 10
Apache Hadoop at 10Apache Hadoop at 10
Apache Hadoop at 10
 
Intro To Hadoop
Intro To HadoopIntro To Hadoop
Intro To Hadoop
 
RSpec for AWSフルマネージドサービス
RSpec for AWSフルマネージドサービスRSpec for AWSフルマネージドサービス
RSpec for AWSフルマネージドサービス
 
Hadoop
HadoopHadoop
Hadoop
 
Cloudera Hadoop Distribution
Cloudera Hadoop DistributionCloudera Hadoop Distribution
Cloudera Hadoop Distribution
 
Acunu overview
Acunu overviewAcunu overview
Acunu overview
 
Cassandra Hadoop Best Practices by Jeremy Hanna
Cassandra Hadoop Best Practices by Jeremy HannaCassandra Hadoop Best Practices by Jeremy Hanna
Cassandra Hadoop Best Practices by Jeremy Hanna
 
140614 bigdatacamp-la-keynote-jon hsieh
140614 bigdatacamp-la-keynote-jon hsieh140614 bigdatacamp-la-keynote-jon hsieh
140614 bigdatacamp-la-keynote-jon hsieh
 

Viewers also liked

Office live meeting presentation
Office live meeting presentationOffice live meeting presentation
Office live meeting presentation
Texasgirl10
 

Viewers also liked (6)

Portfolio Melt Advertising S.r.l.
Portfolio Melt Advertising S.r.l.Portfolio Melt Advertising S.r.l.
Portfolio Melt Advertising S.r.l.
 
Office live meeting presentation
Office live meeting presentationOffice live meeting presentation
Office live meeting presentation
 
Mobile Learning Using The iPod Touch - A Panel Discussion at Spring CUE 11
Mobile Learning Using The iPod Touch - A Panel Discussion at Spring CUE 11Mobile Learning Using The iPod Touch - A Panel Discussion at Spring CUE 11
Mobile Learning Using The iPod Touch - A Panel Discussion at Spring CUE 11
 
Presentazione Melt Advertising s.r.l.
Presentazione Melt Advertising s.r.l.Presentazione Melt Advertising s.r.l.
Presentazione Melt Advertising s.r.l.
 
Case history Cross-Media Mktg per PMI
Case history Cross-Media Mktg per PMICase history Cross-Media Mktg per PMI
Case history Cross-Media Mktg per PMI
 
Social Media Marketing
Social Media MarketingSocial Media Marketing
Social Media Marketing
 

Similar to Hadoopソースコードリーディング第3回 Hadopo MR + Cassandra

Developers summit cassandraで見るNoSQL
Developers summit cassandraで見るNoSQLDevelopers summit cassandraで見るNoSQL
Developers summit cassandraで見るNoSQL
Ryu Kobayashi
 
Hadoop+Cassandra_Integration
Hadoop+Cassandra_IntegrationHadoop+Cassandra_Integration
Hadoop+Cassandra_Integration
Joyabrata Das
 
Hadoop Conference Japan 2011 Fallに行ってきました
Hadoop Conference Japan 2011 Fallに行ってきましたHadoop Conference Japan 2011 Fallに行ってきました
Hadoop Conference Japan 2011 Fallに行ってきました
moai kids
 
Nosql taxonomy with new nugget
Nosql taxonomy with new nuggetNosql taxonomy with new nugget
Nosql taxonomy with new nugget
Matt Ingenthron
 
Nosql taxonomy with new nugget
Nosql taxonomy with new nuggetNosql taxonomy with new nugget
Nosql taxonomy with new nugget
Matt Ingenthron
 
COOKPADでのHadoop利用
COOKPADでのHadoop利用COOKPADでのHadoop利用
COOKPADでのHadoop利用
Tatsuya Sasaki
 
Microsoft's Big Play for Big Data- Visual Studio Live! NY 2012
Microsoft's Big Play for Big Data- Visual Studio Live! NY 2012Microsoft's Big Play for Big Data- Visual Studio Live! NY 2012
Microsoft's Big Play for Big Data- Visual Studio Live! NY 2012
Andrew Brust
 
Qubole Overview at the Fifth Elephant Conference
Qubole Overview at the Fifth Elephant ConferenceQubole Overview at the Fifth Elephant Conference
Qubole Overview at the Fifth Elephant Conference
Joydeep Sen Sarma
 
800万人の"食べたい"をHadoopで分散処理
800万人の"食べたい"をHadoopで分散処理800万人の"食べたい"をHadoopで分散処理
800万人の"食べたい"をHadoopで分散処理
Tatsuya Sasaki
 

Similar to Hadoopソースコードリーディング第3回 Hadopo MR + Cassandra (20)

Developers summit cassandraで見るNoSQL
Developers summit cassandraで見るNoSQLDevelopers summit cassandraで見るNoSQL
Developers summit cassandraで見るNoSQL
 
Hadoop+Cassandra_Integration
Hadoop+Cassandra_IntegrationHadoop+Cassandra_Integration
Hadoop+Cassandra_Integration
 
Hadoop Conference Japan 2011 Fallに行ってきました
Hadoop Conference Japan 2011 Fallに行ってきましたHadoop Conference Japan 2011 Fallに行ってきました
Hadoop Conference Japan 2011 Fallに行ってきました
 
Partners in Crime: Cassandra Analytics and ETL with Hadoop
Partners in Crime: Cassandra Analytics and ETL with HadoopPartners in Crime: Cassandra Analytics and ETL with Hadoop
Partners in Crime: Cassandra Analytics and ETL with Hadoop
 
Cassandra+Hadoop
Cassandra+HadoopCassandra+Hadoop
Cassandra+Hadoop
 
Hadoop online trainings
Hadoop online trainingsHadoop online trainings
Hadoop online trainings
 
Intro to cassandra + hadoop
Intro to cassandra + hadoopIntro to cassandra + hadoop
Intro to cassandra + hadoop
 
マーケティングのためのHadoop利用
マーケティングのためのHadoop利用マーケティングのためのHadoop利用
マーケティングのためのHadoop利用
 
Nosql taxonomy with new nugget
Nosql taxonomy with new nuggetNosql taxonomy with new nugget
Nosql taxonomy with new nugget
 
Nosql taxonomy with new nugget
Nosql taxonomy with new nuggetNosql taxonomy with new nugget
Nosql taxonomy with new nugget
 
Stratio big data spain
Stratio   big data spainStratio   big data spain
Stratio big data spain
 
COOKPADでのHadoop利用
COOKPADでのHadoop利用COOKPADでのHadoop利用
COOKPADでのHadoop利用
 
Microsoft's Big Play for Big Data
Microsoft's Big Play for Big DataMicrosoft's Big Play for Big Data
Microsoft's Big Play for Big Data
 
Microsoft's Big Play for Big Data- Visual Studio Live! NY 2012
Microsoft's Big Play for Big Data- Visual Studio Live! NY 2012Microsoft's Big Play for Big Data- Visual Studio Live! NY 2012
Microsoft's Big Play for Big Data- Visual Studio Live! NY 2012
 
Qubole Overview at the Fifth Elephant Conference
Qubole Overview at the Fifth Elephant ConferenceQubole Overview at the Fifth Elephant Conference
Qubole Overview at the Fifth Elephant Conference
 
800万人の"食べたい"をHadoopで分散処理
800万人の"食べたい"をHadoopで分散処理800万人の"食べたい"をHadoopで分散処理
800万人の"食べたい"をHadoopで分散処理
 
Real World NoSQL (by Chris Yuen)
Real World NoSQL (by Chris Yuen)Real World NoSQL (by Chris Yuen)
Real World NoSQL (by Chris Yuen)
 
Deep dive into event store using Apache Cassandra
Deep dive into event store using Apache CassandraDeep dive into event store using Apache Cassandra
Deep dive into event store using Apache Cassandra
 
Cassandra Distributions and Variants
Cassandra Distributions and VariantsCassandra Distributions and Variants
Cassandra Distributions and Variants
 
The Hadoop Path by Subash DSouza of Archangel Technology Consultants, LLC.
The Hadoop Path by Subash DSouza of Archangel Technology Consultants, LLC.The Hadoop Path by Subash DSouza of Archangel Technology Consultants, LLC.
The Hadoop Path by Subash DSouza of Archangel Technology Consultants, LLC.
 

More from Ryu Kobayashi (6)

PLAZMA TD Tech Talk 2018 at Shibuya: Hive2 as a new td hadoop core engine
PLAZMA TD Tech Talk 2018 at Shibuya: Hive2 as a new td hadoop core enginePLAZMA TD Tech Talk 2018 at Shibuya: Hive2 as a new td hadoop core engine
PLAZMA TD Tech Talk 2018 at Shibuya: Hive2 as a new td hadoop core engine
 
Treasure Data on The YARN - Hadoop Conference Japan 2014
Treasure Data on The YARN - Hadoop Conference Japan 2014Treasure Data on The YARN - Hadoop Conference Japan 2014
Treasure Data on The YARN - Hadoop Conference Japan 2014
 
Huahin Framework for Hadoop, Hadoop Conference Japan 2013 Winter
Huahin Framework for Hadoop, Hadoop Conference Japan 2013 WinterHuahin Framework for Hadoop, Hadoop Conference Japan 2013 Winter
Huahin Framework for Hadoop, Hadoop Conference Japan 2013 Winter
 
Hadoop Conference Japan 2011 Fall
Hadoop Conference Japan 2011 FallHadoop Conference Japan 2011 Fall
Hadoop Conference Japan 2011 Fall
 
AWSを使ったトラッキングログ収集
AWSを使ったトラッキングログ収集AWSを使ったトラッキングログ収集
AWSを使ったトラッキングログ収集
 
Hadoopソースコードリーディング MapReduce障害時のフロー
Hadoopソースコードリーディング MapReduce障害時のフローHadoopソースコードリーディング MapReduce障害時のフロー
Hadoopソースコードリーディング MapReduce障害時のフロー
 

Recently uploaded

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Victor Rentea
 

Recently uploaded (20)

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptx
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 

Hadoopソースコードリーディング第3回 Hadopo MR + Cassandra

Editor's Notes