This document summarizes Apache Hadoop release 0.23, which is scheduled to be the first stable release since 0.20 in 2009. Key highlights include improvements to HDFS federation, MapReduce, and high availability. The release aims to support large clusters of thousands of machines with high concurrency. Extensive testing is being done to validate performance gains from changes like MapReduce shuffle reimplementation and optimizations for small jobs. The 0.23 branch is expected in August 2011 with an alpha release in October and production release in late Q1 2012.

1. Apache Hadoop 0.23 Arun C. Murthy Hortonworks Founder and Architect @acmurthy (@hortonworks) © Hortonworks Inc. 2011 August 25, 2011

2. Hello! I’m Arun… Architect & Lead, Apache Hadoop MapReduce Development Team at Hortonworks (formerly at Yahoo!) Apache Hadoop Committer and Member of PMC Full-time contributor to Apache Hadoop since early 2006 Apache HadoopRelease Manager for hadoop-0.23

3. hadoop-0.23 On track to be first stable, and widely deployed, release since hadoop-0.20 in 2009 All stable releases of Hadoop today are based on hadoop-0.20 Multiple folks and entities collaborating: Hortonworks, Yahoo, Cloudera, EBay etc. hadoop-0.23 branch in Apache hours away! © Hortonworks Inc. 2011 4

4. Highlights HDFS Federation http://www.hortonworks.com/an-introduction-to-hdfs-federation/ NextGenerationHadoopMapReduce http://www.slideshare.net/hortonworks/nextgen-apache-hadoop-mapreduce Coming soon – HDFS High Availability https://issues.apache.org/jira/browse/HDFS-1623 WIP: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-1623/ © Hortonworks Inc. 2011 5

5. More… Build - Full Mavenization EditLogs re-write https://issues.apache.org/jira/browse/HDFS-1073 HDFS Write pipeline improvements for Hbase Append/flush etc. Re-implementation of MapReduce Shuffle 30% performance gain Stability using netty rather than jetty Small jobs optimizations … © Hortonworks Inc. 2011 6

6. Deployment goals Clusters of 6,000machines Each machine with 16 cores, 48G/96G RAM, 24TB/36TB disks 100,000+ concurrent tasks 10,000 concurrent jobs © Hortonworks Inc. 2011 7

7. Testing Currently tested at reasonable scale - ~500 nodes incl. GridMixv3 Continue to improve on performance benchmarks GridMixv3 Sort Shuffle HDFS Scan HDFS throughput … © Hortonworks Inc. 2011 8

8. Timelines branch-0.23 – August 2011 Alpha (hadoop-0.23.0) - ~October 2011 Production – late Q1 2012 YMMV!  © Hortonworks Inc. 2011 9

9. Thank You.@acmurthy © Hortonworks Inc. 2011