Apache Hadoop Presentation by Steve Watt at Data Day Austin 2011

1. Introduction to Apache Hadoop Steve Watt - IBM Big Data Lead @wattsteve #datadayaustin http://stevewatt.blogspot.com

2. The Origins of Hadoop

5. So what exactly is Apache Hadoop ? It is a cluster technology with a single master and multiple slaves, designed for commodity hardware It consists of two runtimes, the Hadoop distributed file system ( HDFS ) and Map/Reduce As data is copied onto the HDFS, it ensures the data is blocked and replicated to other machines (node) to provide redundancy Self contained jobs are written in Map/Reduce and submitted to the cluster. The jobs run in parallel on each of the machines in the cluster, processing the data on the local machine ( data locality ). Hadoop may execute or re-execute a job on any node in the cluster. Node failures are automatically handled by the framework.

6. Hadoop – The Hadoop Cluster - Distributed File System - Map/Reduce

9. Hadoop - Map/Reduce on the Cluster

10. Hadoop - Map/Reduce Logical Flow

11. Hadoop – Map/Reduce – JobTracker Details

12. Hadoop – Map/Reduce – Job Details

14. The Hadoop Ecosystem ClusterChef / Apache Whirr Hadoop Pig / WuKong Cassandra / HBase Offline Systems (Analytics) Online Systems (OLTP @ Scale) BigSheets / DataMeer Hive Provisioning Nutch / SQOOP / Flume Scripting DBA Non-Programmer Load Tooling https://github.com/tomwhite/hadoop-ecosystem/raw/master/hadoop-ecosystem.dot.png