An introduction to Hadoop, present at Vermont Code Camp 2011.

1. TACKLING BIG DATA WITH
HADOOP
David Howell
Sunday, September 11, 11

2. WHAT IS BIG DATA?
Sunday, September 11, 11

3. WHAT IS BIG DATA?
Google web crawl
Sunday, September 11, 11

4. WHAT IS BIG DATA?
stream of Twitter messages
Sunday, September 11, 11

5. WHAT IS BIG DATA?
Annoying Farmville requests on Facebook
Sunday, September 11, 11

6. WHAT IS BIG DATA?
terabyte-scale data sets
awkward to work with using traditional tools
Sunday, September 11, 11

7. WHAT IS BIG DATA?
requires distributed computing
Sunday, September 11, 11

8. MEDIUM DATA
dozens to hundreds of gigabytes
still awkward to work with using traditional tools
Sunday, September 11, 11

9. MAP-REDUCE
http://labs.google.com/papers/mapreduce.html
Sunday, September 11, 11

10. Sunday, September 11, 11

11. Sunday, September 11, 11

12. COUNTING AT SCALE
Sunday, September 11, 11

13. function map_1(t, search_phrase)
emit(search_phrase, 1)
sort and shuffle
function reduce_1(search_phrase, counts)
total = 0
for count in counts
total += count
emit(search_phrase, total)
function map_2(search_phrase, total)
emit(total, search_phrase)
sort and shuffle
function reduce_2(total, search_phrases)
for search_phrase in search_phrases
emit(search_phrase, total)
Sunday, September 11, 11

14. map shuffle reduce
cat IN | sort | uniq -c > OUT
map shuffle reduce
awk ‘{print $2,$1}’ OUT | sort > FINAL
Sunday, September 11, 11

15. WHY BOTHER?
Sunday, September 11, 11

16. HADOOP
Sunday, September 11, 11

17. DISTRIBUTED COMPUTING
PLATFORM
Sunday, September 11, 11

18. TOOLS IN THE PLATFORM
Map-Reduce APIs Higher Level APIs
•Java •Hive
•C++ •Cascading
•UNIX pipes •Pig
Sunday, September 11, 11

19. THE ORIGIN STORY
Sunday, September 11, 11

20. WHO’S USING IT?
Sunday, September 11, 11

21. HADOOP
How does it work?
Sunday, September 11, 11

22. Sunday, September 11, 11

23. Sunday, September 11, 11

24. Sunday, September 11, 11

25. Sunday, September 11, 11

26. DEMO!
Sunday, September 11, 11

27. YOUR DATA PLATFORM
ad hoc
unstructured
prototyping
experiment
data-driven
curiosity
play
Sunday, September 11, 11

28. LEARN MORE
http://hadoop.apache.org/
http://www.cloudera.com/
Hadoop: The Definitive Guide
@dehowell
dave@poorlytrainedape.com
http://github.com/dehowell/hadoop-crypto-demo
Sunday, September 11, 11