Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Map Reduce v2 and YARN - CHUG - 20120604

4 722 vues

Publié le

View the accompanying video on vimeo: https://vimeo.com/43474797

Publié dans : Technologie
  • http://dbmanagement.info/Tutorials/MapReduce.htm
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Map Reduce v2 and YARN - CHUG - 20120604

  1. 1. An Introduction to MapReduce 2 and YARN Tom White, Cloudera @tom_e_white June 4, 2012 Chicago HUGTuesday, June 5, 2012
  2. 2. Road TripTuesday, June 5, 2012
  3. 3. About me • Apache Hadoop Committer, PMC Member, Apache Member • Engineer at Cloudera working on core Hadoop • Founder of Apache Whirr • Author of “Hadoop: The Definitive Guide” • http://hadoopbook.comTuesday, June 5, 2012
  4. 4. First, whatʼs MapReduce 1?Tuesday, June 5, 2012
  5. 5. Tuesday, June 5, 2012
  6. 6. Whatʼs wrong with MR1?Tuesday, June 5, 2012
  7. 7. Motivation 1 • Scaling >4000 nodes • Fewer, larger clustersTuesday, June 5, 2012
  8. 8. Motivation 2 • HA of Job Tracker • Large, complex stateTuesday, June 5, 2012
  9. 9. Motivation 3 • Poor resource utilization • Slots in MR1 are for either map or reduceTuesday, June 5, 2012
  10. 10. Yet Another Resource NegotiatorTuesday, June 5, 2012
  11. 11. Tuesday, June 5, 2012
  12. 12. Tuesday, June 5, 2012
  13. 13. Node Manager is a generalized Task Tracker • Task Tracker • fixed number of map or reduce slots • Node Manager • containers with variable resource limitsTuesday, June 5, 2012
  14. 14. Tuesday, June 5, 2012
  15. 15. Tuesday, June 5, 2012
  16. 16. MR is user space YARN is kernelTuesday, June 5, 2012
  17. 17. Bonus Apps • Distributed shell • MPI (MAPREDUCE-2911) • Master-worker (MAPREDUCE-3315) • Apache Giraph, HamaTuesday, June 5, 2012
  18. 18. Tuesday, June 5, 2012
  19. 19. Tuesday, June 5, 2012
  20. 20. Old API ≠ MR1 New API ≠ MR2Tuesday, June 5, 2012
  21. 21. Old API New API o.a.h.mapred o.a.h.mapreduce MR1 ✓ ✓ MR2 ✓ ✓Tuesday, June 5, 2012
  22. 22. Tuesday, June 5, 2012
  23. 23. Try out MR2 • Apache Hadoop 2.0.0-alpha • hadoop.apache.org • CDH4 and Cloudera Manager • cloudera.com • Cloud - Apache WhirrTuesday, June 5, 2012
  24. 24. MR1 <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>1.0.3</version> </dependency> MR2 <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.0.0-alpha</version> </dependency>Tuesday, June 5, 2012
  25. 25. TODO • Still alpha status • Performance tuning • Usability bug fixes • RM recovery • Security in MR2 not completeTuesday, June 5, 2012
  26. 26. Questions?Tuesday, June 5, 2012

×