This document provides an overview of Hadoop and MapReduce. It discusses the origins and evolution of Hadoop from Google's MapReduce paper in 2004 to connectors from IBM, Microsoft and SAP in later years. It also presents a sample use case of using Hadoop for logfile analytics processing 80 TB of data per month. The key components of Hadoop discussed include MapReduce, HDFS, APIs and cluster sizing.

1. © inovex Academy
Hadoop &
map-reduce
1

2. © inovex Academy
Speakers
1
Dr. Kathrin Spreyer
Big Data Engineer
Patrick Thoma
Head of Solution Development

3. © inovex Academy
Inevitable hadoop
2004: Google MapReduce paper
2006: Hadoop team around Doug Cutting at Yahoo!
2010/11: IBM’s Watson
2011/12: Hadoop connectors for Oracle products
Oct 2012: Microsoft (connectors f. Azure, HDInsights)
Oct 2012: SAP (cooperation w/ support companies)
3

4. © inovex Academy
Motivation
1. sample use case: logfile analytics @ 1&1
2. 80 TB/month to be processed
3. too slow on existing hardware
4. further scaling not possible -- or extremely expensive
4

5. © inovex Academy
Amazing performance improvement
4

6. © inovex Academy
Overview
1. Map-Reduce
2. HDFS
3. APIs
4. Cluster sizing
6

7. © inovex Academy
What?
1. framework for distributed data
processing
2. highly scalable: TBs and PBs
3. originated at Google
4. open-source implementation:
Apache Hadoop
7

8. © inovex Academy
The big picture
8
input

9. © inovex Academy
The big picture
8

12. © inovex Academy
Why?
1. too much data for one machine
2. processing speed
3. scaling out vs. scaling up
9
Photo by Flo P.

21. © inovex Academy 14
HDFS
(hadoop distributed file system)
1. Map-Reduce
2. HDFS
3. APIs
4. Cluster sizing

32. © inovex Academy
Apis
20
1. Map-Reduce
2. HDFS
3. APIs
4. Cluster sizing

33. © inovex Academy
Basic map-reduce Apis
1. Java
2. C++ (Pipes)
3. Python (Dumbo)
4. streaming (any language)
21

34. © inovex Academy
Higher-level Apis
1. Apache Pig (data flow language)
2. Apache Hive (SQL dialect)
22
alternative:
graphical ETL tools,
e.g., Pentaho Data Integration

35. © inovex Academy
Cluster sizing
23
1. Map-Reduce
2. HDFS
3. APIs
4. Cluster sizing

37. © inovex Academy
Network topology
1. single data center
2. rack topology
3. bandwidth
25

38. © inovex Academy
Questions?
26

39. © inovex Academy
Contact:
bigdata@inovex.de
27