Introduction to HDInsight service (aka Hadoop on Azure)

1. BigData

2. The problem is simple
• While the storage capacities of hard drives
have increased massively over the
years, access speeds—the rate at which data
can be read from drives have not kept up.
• One typical drive from 1990 could store 1,370
MB of data and had a transfer speed of 4.4
MB/s

3. • so you could read all the data from a full drive
in around five minutes.
• Over 20 years later, one terabyte drives are
the norm, but the transfer speed is around
100 MB/s, so it takes more than two and a half
hours to read all the data off the disk.

4. GoParallel

5. Cloud computing changes
the way applications grow
http://journals.worldnomads.com/davidsgibson/photo/22804/664941/USA/Elephant-shaped-cloud!

6. BIG-TIME:
Introducing Hadoop
on Azure
Yaniv Rodenski
Senior Consultant, Sela Group
http://blogs.microsoft.co.il/blogs/roadan
Twitter: @YRodenski
yanivr@sela.co.il
David Ginzburg
Big Data infrastructure consultant
Twitter: @David_Ginzburg
davidginzburg@gmail.com

7. AGENDA

8. Apache™ Hadoop™

9. Apache™ Hadoop™

10. Hadoop Distributed File System (HDFS)
HDFS
Client

11. Hadoop Distributed File System (HDFS)
HDFS
Client

12. Hadoop Distributed File System (HDFS)
HDFS
Client

13. MapReduce via WordCount
1
Hello
World 1
1 1 2
1 1 2
1
Hello
Azure 1 1
1
1 1
1 1
Goodbye 1
Cruel
World 1
1

14. DEMO
A new way to MapReduce

15. Hadoop MapReduce Processing
Input
Split
Input
Merge
Split
Input
Split

16. Hadoop MapReduce Processing
Job
Client

17. MapReduce TMI
Partition,
Sort, and
spill to
disk
Fetch
Input
Buffer
Split

18. MapReduce TMI
Sort
Map
Output
Merge
result
Map
Output
Output
Map
Output
Merge
result
Map
Output

19. Partitioners

20. Combiners

21. The TeraSort Use case

22. The TeraSort Use case

23. Beginners Pitfalls

24. Beginners Pitfalls

25. Distinct Values Problem Statement
http://highlyscalable.wordpress.com/2012/02/01/mapreduce-patterns/

26. Distinct Values Problem Statement
http://highlyscalable.wordpress.com/2012/02/01/mapreduce-patterns/

27. Distinct Values Problem Statement
http://highlyscalable.wordpress.com/2012/02/01/mapreduce-patterns/

28. Distinct Values Problem Statement
http://highlyscalable.wordpress.com/2012/02/01/mapreduce-patterns/

29. DEMO
Administrating Hadoop in the real world

30. Why did Microsoft choose Hadoop?

31. Hadoop on Azure

32. DEMO
Using hadooponazure.com

33. Windows Azure Compute
Supporting service
Application
Configuration

34. Hadoop on Azure Roles
Monitoring service (RdAdmin)
Hadoop services
Configuration

35. Hadoop MapReduce Processing
Fabric
Controller

36. Hadoop MapReduce Processing
Fabric
Controller

37. Hadoop MapReduce Processing
Fabric
Controller

38. The Head Node Template

39. The Worker Node Template

40. Node VM Templates

41. Cloud Storage

42. High Availability on Azure
Azure
Storage
Fabric
Controller

43. Elastic MapReduce

44. Elastic MapReduce
Storage
Client
Azure Amazon
Storage S3

45. Elastic MapReduce
Storage
Client
Azure Amazon
Storage S3

46. Elastic MapReduce
Storage
Client
Azure Amazon
Storage S3
$ $ $ $
$ $ $ $

47. DEMO
Using Elastic MapReduce

48. Azure Blob Considerations

49. Storage Size Limitations

50. IsotopeJS

51. DEMO
Using the JavaScript interactive console

52. DEMO
Using Hive

53. Summary

54. Q&A

55. Resources
My Blog Windows Azure Developer center
http://bit.ly/roadan http://www.windowsazure.com/en-us/develop/overview
Apache™ Hadoop™
http://hadoop.apache.org
Hadoop on Azure
http://www.hadooponazure.com
Hadoop: The Definitive Guide
Tom White
http://shop.oreilly.com/product/9780596521981.do
Thanks!
Yaniv Rodenski
Twitter: @YRodenski