Given at IoT Asia 2014

1. Hadoop 101
A really quick overview of the concepts…

2. A few Terabytes of Data...

5. Text processing--a few hours?

6. But what if you have more data?

7. Network Storage--Petabytes!

8. Network Storage--Petabytes!

9. What if you need compute power for complex algorithms?

10. 8 core? 16 Cores? 64 cores? 512 GB
RAM?

11. A network of commodity computers

12. Run jobs on PART of the data on each computer then
AGGRETAGE the intermediary results from each computer.

13. Let’s add a computer to manage the process of
job delegation, merging the results...
and keeping track of the results...

14. We also need something to keep track of what files are
where, so we know where the data is that needs to be
computed...

15. When you have a lot of computers, and even more hard
drives,
one thing I can guarantee...

16. Computers will eventually fail.

17. Computers will eventually fail.

18. Hard drives will eventually fail.

19. Hard drives will eventually fail.

20. Hard drives will eventually fail.

21. Hard drives will eventually fail.

22. Even whole racks will fail.

23. If a computer fails and you only have one copy of your
data...

24. You will be very, very unhappy.

25. So lets store multiple copies of the data. Hard drives are
CHEAP!

26. So lets store multiple copies of the data. Hard drives are
CHEAP!

27. So lets store multiple copies of the data. Hard drives are
CHEAP!

28. So lets store multiple copies of the data. Hard drives are
CHEAP!

29. If one hard drive fails... we are still OK

30. If one computer fails... we are still OK

31. Even if a whole rack fails... we are still OK

32. Once we find a failure let’s have the system recopy the
copies.

33. Send the compute job to all nodes.

34. And let it run on it’s part of the data….

35. And let it run on it’s part of the data….

36. And let it run on it’s part of the data….

37. And let it run on it’s part of the data….

38. One is stuck….

39. We have three copies—we can redistribute the compute

40. And take the one that finishes fastest

41. Merge sorted sets based on some key…
A-E F-J K-O P-T U-Z

42. …and write partial results
PART-01 PART-02 PART-03 PART-04 PART-05

43. Guess, what? We’ve just invented Hadoop!
PART-03
PART-01
PART-02
A-E F-J

44. So let’s talk about the pieces of Hadoop.

45. Data nodes store and manage the data on a single “slave”
computer
Data Node

46. Task trackers manage the compute
Data Node
Task Tracker

47. Job tracker manages task trackers, ships code to compute
nodes
Data Node
Task Tracker
Job Tracker

48. Name node manages distribution and replication on the
data nodes
Data Node
Task Tracker
Job Tracker
Name Node

49. Map Reduce
Task Tracker
Job Tracker

50. HDFS (Hadoop Distributed File System)
Data Node
Name Node

51. HDFS

52. Visual Example

53. Map

54. Shuffle

55. Reduce

56. Putting It All Together