A comprehensive overview on the entire Hadoop operations and tools: cluster management, coordination, injection, streaming, formats, storage, resources, processing, workflow, analysis, search and visualization

1. Hadoop Ecosystem
Lior Sidi
Sep 2016

2. Hello!
I am Lior Sidi

3. Big data V’s
Volume
Velocity
Variety

4. What is Hadoop?
• Hadoop – Open source implementation of MapReduce (MR)
• Perform MR Jobs fast and efficient
Goal
generating Value from large datasets
That cannot be analyzed
using traditional technologies

5. Hadoop Concepts
Requirements
• Linear horizontal scalability
• Jobs run in isolation
• Simple programming model
Challenges and solution
• Ch1: Data access bottleneck
• Sol: Store and process data on same node
• Ch1: Distributed Programming is Difficult
• Sol: Use high level languages API

6. Hadoop Timeline
2003 Oct
Google File System
paper released
2004 Dec
MapReduce: Simplified Data
Processing on Large Clusters
2006 Oct
Hadoop 1.0 released
2007 Oct
Yahoo Labs creates Pig
2008 Oct
Cloudera, Hadoop
distributor is founded
2010 Sep
Hive and Pig Graduates
2011 Jan
Zookeeper Graduates
2013 Mar
Yarn deployed in Yahoo
2014 Feb
Apache Spark top
Level Apache Project

7. Data
Streaming
Analysis
Data Injection
Resource
Management
coordination
Processing
workflow
Visualization
Cluster
management
Storage
Search
Data
Formats
Hadoop Ecosystem

8. Storage
Hadoop Ecosystem

9. Storage / HDFS
• “Hadoop Distributed File System”
• Design:
• Write once – read many times pattern
• Cheap hardware
• Low latency data access
• Concepts:
• Block – File is split to Size 128 MB blocks, redundancy - 3
• NameNode (Master) – per cluster - file system namespace for blocks (single point of
failure)
• DataNode (Worker) – per Node - store and retrieve blocks
• Functions:
• High availability – run a second NameNode
• Block caching – block cached in only one DataNode
• Locality - Rack sensitive, network topology
• File permissions – like POSIX – r w x – owner/group/mode file/directory
• Interfaces – HTTP (proxy/direct), Java API
• Cluster balance – evenly spread the block on the cluster

10. 2Rack
1Rack
Data
Block 1
Block 2
Block 3
DataNodeDataNodeDataNodeDataNode
Block 1
Block 1
Block 2
Block 2
Block 3
Block 3
Block 1
DataNode
Block 2
Block 3
NameNode
HDFS proxy Client
file is distribution and
accessed on Hadoop HDFS

11. Resource
Management
Storage
Hadoop Ecosystem

12. Resource Management / YARN
• “Yet Another Resource Negotiator”
• Manage and schedule the cluster resource
• Daemons:
• Resource Manager – Per Cluster – manage resource across the cluster
• Node Manager – Per Node – launch and monitor a Container
• Container – execute an app process
• Resource requests for containers:
• Amount of computers (CPU & Memory)
• Locality (node/rack)
• Lifespan: application per user job or long-running apps shared by users
• Scheduling:
• Allocate resource by policy (FIFO, capacity (ordanisation), Fair

13. Hadoop Cluster
Nodemanager
node
NodeManager
Container
Master
Client node
application
Resource manager node
ResourceManager
Client
Nodemanager
node
NodeManager
Container
Worker
Nodemanager
node
NodeManager
Container
Worker
launch
launch
launch
launch
Launch
YARN app
heartbeat Job scheduling on top
Hadoop Cluster

14. Resource
Management
Processing
Storage
Hadoop Ecosystem

15. Processing / MapReduce
• Simplify, large scale, automatic, Fault tolerant development data
processing
• origin - Google paper 2004
• Batch processing
• Hadoop MR:
• JobTracker – 1per cluster - master process, schedule tasks on workers,
monitor progress
• taskTracker – 1 per worker - execute map/reduce tasks locally and
report progress

16. Processing / MapReduce
LiorRonLior
RonRonAndrey
LiorAndreyLior
CountName
1Lior
1Ron
1Lior
CountName
1Lior
1Andrey
1Lior
CountName
1Andrey
1Ron
1Ron
CountName
4Lior
CountName
3Ron
CountName
2Andrey
Data
Map
ReduceShuffle
& Sort

17. Hadoop Cluster
Nodemanager
node
NodeManager
Container
JobTracker
Client node
MR program
Resource manager node
ResourceManager
Client
Nodemanager
node
NodeManager
Container
TaskTracker
Nodemanager
node
NodeManager
Container
TaskTracker
launch
launch
launch
launch
Launch
YARN app
heartbeat
MR Job scheduling on top
Hadoop Cluster

18. Resource
Management
Processing
Storage
Hadoop Ecosystem

19. Storage / HBase
• Distributes Column Base database on top HDFS
• Real time read/write random access for large data-sets
• Region – tables splitting by row
• Pheonex - SQL on HBase
RowKey Column Family 1 Column Family 2
Col 1.1
Version Data
Col 1.2 Col 1.3
Version Data
Version Data
Hbase Data Model

20. Resource
Management
coordination
Processing
Storage
Hadoop Ecosystem

21. Coordination / ZooKeeper
• Hadoop’s distributed coordination service
• Coordinate read/write action on data
• high availability filesystem
• Implementation:
• Data model:
• Tree build from Znodes (1MB data)
• Znode – data changes, ACL (access control list )
• Leader - perform write and broadcast an update
• Follower – pass atomic request to leader
• Lock service
• User groups
• Replicate mode

22. Coordination / ZooKeeper
Hadoop Cluster
ZooKeaper Service
Leader
HDFSHBase
DataNodeDataNodeDataNode
HMaster Other
Client
RegionRegionRegion
NameNode
/
/HBase HDFS/
Follower
/
/HBase HDFS/
Follower
/
/HBase HDFS/
LOCK LOCK
ZooKeeper
Coordination
example

23. Resource
Management
coordination
Processing
Storage
Data
Formats
Hadoop Ecosystem

24. Row Based Avro
• Language natural data serialization system
• Share many data formats with many code language
• Split able and sortable - Allow easy map reduce
• Rich schema resolution – flexible scheme
• Other Row Based formats
• sequenceFile - Logfile format
• MapFile - Sorted sequenceFile

25. Row Based Avro
Header Block 1 Block 2 Block N
Count objs Serialized objs SyncMarker
identifier Metadata: Schema & codec SyncMarker
Size objs
{
"Type":"record"
"Name":"Person"
"Fields":
[{
"name":"firstName",
"type":"string"
"order":"descending"
},{
"name":"age",
"type":"int"
},{...
]
}
Schema
File Structure

26. File Structure
Parquet
• Columnar storage format
• Skip unneeded columns
• Fast queries & small size
• Efficient nested data store Header Block 1 Block 2 Block N
Column chunk Column chunk Column chunk
Page Page Page Page
Magic Number File Metadata
Footer
Message Person {
Required binary name (UTF8);
Required int32 age (UTF8);
Required group hobbies (LIST) {
Required binary array (UTF8);
}
}
Schema

27. Data Injection
Resource
Management
coordination
Processing
Storage
Data
Formats
Hadoop Ecosystem

28. Data Integration / Sqoop
• Import/export structural data
• Sqoop connector:
• import/export from a database
• Sqoop1- command line
• Sqoop2 – service
• Connectors – connect RDBs
Hadoop Cluster
Export MapReduce Job
Database
Table
Sqoop client
Import MapReduce Job
Hdfs Hdfs
Map Map
Hdfs Hdfs
Map Map
metadata
launch launch
ExportImport

29. Data Integration / Flume
• Event base data injection into Hadoop
• Flume agent components:
• Sources – spoolingDir (create events), Avro(RPC), Http (requests)
• Channel
• Sink – Avro, HDFS, HBase, Solr(=near real time)
• Reliability - Use separate transaction
• Fan out – one source many sinks
• Scale - agent tiers for aggregation multiple sources
• Sink grouping- avoid failure and load balancing

30. Fan Out
Data Integration / Flume
Hadoop Data
File
system
Flume Agent
Source Channel Sink
Tier 1
Flume Agent
Tier 1
Flume Agent
Tier 1
Flume Agent
Tier 2
Flume Agent
Tier 2
Flume Agent
Tier 3
Flume Agent
Tier 3
Flume Agent
File
system Sink
GroupingScale
HDFS
HBase
Data

31. Data Integration / Kafka
• distributed publish-subscribe messaging system
• Fast, scalable, durable
• Components:
• Topics – categories of feeds messages
• Procedures – process that publish messages to topic
• Message consumer – processes that subscribe for topic
• Broker – kafka servers on cluster
• Distribution
• Leader – allow read/write
• Follower – replicate

32. Data
Streaming
Data Injection
Resource
Management
coordination
Processing
Storage
Data
Formats
Hadoop Ecosystem

33. Data Integration / Streaming
• Stream processing
• Kafka Stream - Process and analyze data in Kafka
• Storm – real-time computation
• Spark streaming – process live data and can apply Spark MLib and
graphX
Flume Agent 1
Data
Kafka
Spark Streaming
Flume Agent 2 Storm
Topic
A
Topic
B
HDFS
1
1
1
2
2

34. Data
Streaming
Analysis
Data Injection
Resource
Management
coordination
Processing
Storage
Data
Formats
Hadoop Ecosystem

35. • Cluster Computing Framework
• In Memory processing
• Language: Scala, Java and Python
• RDD – resilience Distributed dataset
• Read only collection spread in the cluster
• Computation of transformation happened when Action
• DAG engine – schedule many transformations to one optimal Job
• Spark context
• parallel jobs
• Caching
• Broadcast variables (Data/Functions)
• Cluster Manager of executors:
• Local, Standalone, Mesos , Yarn
Computation / Spark

36. Computation / Spark
Hadoop
Driver
SparkContext
Spark Program
DAG Scheduler
Task Scheduler
Scheduler backend
Executer Executer Executer
Job
Job
Stages
Tasks
Task Task Task

37. Scripting / Pig
• Data flow programming language - Map reduce abstraction
• support: User defined functions (UDF), Streaming, nested data
• Don’t support: random read/write
• Pig Latin - Scripting language
• Load, store, filtering, Group, Join, Sort, Union and Split, UDF, Co-group
• Modes
• Local – small datasets
• MR mode – run on cluster
• Execution - script, grunt (shell), embedded (java)
• Parameter substitution – run script with different parameters
• Similar
• Crunch – MR pipeline with Java (no UDF)

38. Query / Hive
• Components
• MetaStore – tables description
• HiveQL – SQL dialect (SQL: 2003)
• tables Management
• warehouse directory
• external tables
• functionality
• Bucketing and Partitions by column
• Support UDF and UDAF (aggregate)
• Insert Update Delete:
• Saved in delta files
• Background MR Jobs
• (Available Transaction context)
• Lock table (avoid drop)

39. Query / Comparison
SparkSql (shark)ImpalaHive
Procedural
development
BI & SQL analyticsBatchUsage
OKBestbadSpeed
MemoryDedicated Deamons on
DataNode
MapReduceimplementation
Persto ,
Drill (SQL: 2011)
Hive On sparkSimilar tools

40. Data
Streaming
Analysis
Data Injection
Resource
Management
coordination
Processing
workflow
Storage
Data
Formats
Hadoop Ecosystem

41. Workflow / Oozie
• Schedule Hadoop jobs
• Job types:
• Workflows – sequence of jobs via Directed Graphs (DAGs)
• Coordinator - trigger jobs by time or availability
start Sqoop Fork
Pig
PigMR
Sub
workflow
FS
(HDFS)
Join End
Control flow
Action
Email

42. Data
Streaming
Analysis
Data Injection
Resource
Management
coordination
Processing
workflow
Storage
Search
Data
Formats
Hadoop Ecosystem

43. Search / Solr
• Full- text search over Hadoop
• Near real time indexing
• REST API
• Based on Apache Lucene java search library

44. Data
Streaming
Analysis
Data Injection
Resource
Management
coordination
Processing
workflow
Visualization
Storage
Search
Data
Formats
Hadoop Ecosystem

45. Visualization / Hue
• Open source Web interface for analyzing data with any Hadoop.
• Application:
• File Browser: HDFS, Hbase
• Scheduling of jobs and workflows : Oozie
• Job Browser: YARN
• SQL : Hive, Impala
• Data analysis: Pig, UDF
• Dynamic Search: Solr
• Notebooks: Spark
• Data Transfer: Sqoop 2

46. Data
Streaming
Analysis
Data Injection
Resource
Management
coordination
Processing
workflow
Visualization
Cluster
management
Storage
Search
Data
Formats
Hadoop Ecosystem

47. Cluster Management / Cloudera
• 100% open source
• The most complete and tested distribution of Hadoop
• Integrate all Hadoop project
• Express – free, end to end administration
• Enterprise – Extra features and support

48. Cluster Management / Comparison
https://talendexpert.com/cloudera-vs-honworks-vs-mapr

49. MasterMasterMaster
Other Servers
Worker
Basic Cluster configuration
Resource manager
Standby
Resource Manager
NodeManager
DataNode
Cloudera Manager
Hive GW
ZooKeeper
Impala Daemon
Impala State
Sqoop GW
Spark GW
NameNode
Master
ZooKeeper
Secondary
NameNode
Worker
NodeManager
DataNode
Impala Daemon
Worker
NodeManager
DataNode
Impala Daemon
Worker
NodeManager
DataNode
Impala Daemon

50. Data
Streaming
Analysis
Data Injection
Resource
Management
coordination
Processing
workflow
Visualization
Cluster
management
Storage
Search
Data
Formats
Hadoop Ecosystem

51. Thanks!
Any questions?