A Container-based Sizing Framework for Apache Hadoop/Spark Clusters

1. A Container-based Sizing Framework
for Apache Hadoop/Spark Clusters
October 27, 2016
Hokkaido University
Akiyoshi SUGIKI, Phyo Thandar Thant

2. Agenda
Hokkaido University Academic Cloud
A Docker-based Sizing Framework for Hadoop
Multi-objective Optimization of Hadoop
1

3. Information Initiative Center, Hokkaido
University
Founded in 1962 as a national supercomputing center
A member of
– HPCI (High Performance Computing Infrastructure) - 12 institutes
– JHPCN (Joint Usage/Research Center for Interdisciplinary Large-
scale Information Infrastructure) - 8 institutes
University R&D center for supercomputing, cloud computing,
networking, and cyber security
Operating HPC twins
– Supercomputer (172 TFLOPS) and Academic Cloud System (43 TFLOPS)
2

4. Hokkaido University Academic Cloud (2011-)
Japan’s largest academic cloud system
– > 43 TFLOPS (> 114 nodes)
– ~2,000 VMs
3
Supercomputer Cloud System
Data-science
Cloud System
(Added, 2013-)
SR16000 M1
172 TF/176 nodes
22 TB (128 GB/node)
AMS2500 (File System)
600 TB (SAS, RAID5)
300 TB (SATA, RAID6)
BS2000
44 TF/114 nodes
14 TB (128 GB/node)
Cloud Storage
1.96 PB
AMS2300 (Boot File System)
260 TB (SAS, RAID6)
VFP500N+AMS2500 (NAS)
500 TB (near-line NAS/RAID6)
HA8000/RS210HM
80 GB x 25 nodes
32 GB x 2 nodes
CloudStack 3.x CloudStack 4.x
Hadoop Package for “Bigdata”
(Hadoop, Hive, Mahout, and R)

5. Supporting “Big Data”
“Big Data” cluster package
• Hadoop, Hive, Mahout, and R
• MPI, OpenMP, and Torque
– Automatic deployment of VM-based clusters
– Custom scheduling policy
• Spread I/O on multiple disks
4
VM
VM
VM
VM
#1
#2
#3
#4
Storage #1
Storage #2
Storage #3
Storage #4
Hadoop Cluster
Virtual Disks
Hadoop
Hive
Mahout
R
Big Data Package

6. Lessons Learned (So Far)
No single Hadoop (a little like silos)
– Hadoop instance for each group of users
Version problem
– Upgrades and expansion of Hadoop ecosystem
Strong demand of middle person
– Gives advice with deep understanding of research domains,
statistical analysis, and Hadoop-based systems
5
VM VM VM
Hadoop #1
VM VM VM
Hadoop #2
VM VM VM
Hadoop #3
Research Group #1 Research Group #2 Research Group #3
Research
Data

7. Going Next
A new system will be installed in April, 2018
– x2 CPU cores, x5 storage space
– Bare-metal, accelerating performance at every layer
– Supports both interclouds and hybrid clouds
Still supports Hadoop as well as Spark
– Cluster templates
– Build user community
6
Supercomputer
System Hokkaido U.
Regions
(Tokyo,
Osaka,
Okinawa)
Cloud
Systems
(In other universities
and public clouds)
Cluster Templates (Hadoop, Spark, …)

8. Requirements
Run Hadoop on multiple Clouds
– Academic Clouds (Community Clouds)
• Hokkaido University Academic Cloud, ...
– Public Clouds
• Amazon AWS, Microsoft Azure, Google Cloud, …
Offer best choice for researches (our users)
– Under multiple criteria
• Cost
• Performance (time constraints)
• Energy
…
7

9. Our Solution
A Container-based Sizing Framework for Hadoop Clusters
– Docker-based
• Light-weight, easily migrate to other clouds
– Emulation (rather than simulation)
• Close to actual execution times on multiple clouds
– Output:
• Instance type
• Number of instances
• Hadoop configuration (*-site.xml files)
8

10. Architecture
9
Emulation
Engine
Docker Runtime
Application (HPC, Big Data)
Application (HPC, Big Data)
Docker
Application (HPC, Big Data,…)
CPU
Memory
DiskI/O
NetworkI/O
Interpose
Collect Metrics
Run Profiles
Instance
Profiles
t2 m4 r3c4
Public Clouds
Cost
Estimator

11. Why Docker?
10
Virtual Machines OS Containers
Size Large Small
Machine Emulation Complete Partial (Share OS kernel)
Launch time Large Small
Migration Sometime requires image
conversion
Easy
Software Xen, KVM, VMware Dockers, rkt, …
App
Lib
App
Lib
OS
Container Container
App
Lib
App
Lib
OS
VM VM
OS
Hypervisor

12. Container Execution
Cluster Management
– Docker Swarm
– Multi-host (VXLAN-based) networking mode
Container
– Resources
• CPUs, memory, disk, and network I/O
– Regulation
• Docker run options, cgroups, and tc
– Monitoring
• Docker remote API and cgroups
11

13. Docker Image
“Hadoop all in the box”
– Hadoop
– Spark
– HiBench
The same image for master/slaves
Exports
– (Environment variables)
– File mounts
• *-site.xml files
– (Data volumes)
12
Hadoop
Spark
HiBench
Hadoop
Spark
HiBench
Volume mounts
Hadoop all in the box
core-site.xml
hdfs-site.xml
yarn-site.xml
mapred-site.xml

14. Resources
Resources How Command
CPU cores Change CPU set Docker run/cgroups
clock rate Change quota & period Docker run/cgroups
Memory size Set memory limit Docker run/cgroups
Out-of-
memory
(OOM)
Change out-of-memory
handling
Docker run/cgroups
Disk IOPS Throttle read/write IOPS Docker run/cgroups
bandwidth Throttle read/write bytes/sec Docker run/cgroups
Network IOPS Throttle TX/RX IOPS Docker run/cgroups
bandwidth Throttle TX/RX bytes/sec Docker run/cgroups
latency Insert latency (> 1 ms) tc
Freezer freeze Suspend/resume cgroups
13

15. Hadoop Configuration
Must be adjusted according to
– Instance type (CPU, memory, disk, and network)
– Number of instances
Targeting all parameters in *-site.xml
Dependent parameters
– (Instance type)
– YARN container size
– JVM heap size
– Map task size
– Reduce task size
14
Machine Instance Size
YARN Container Size
JVM Heap Size
Map/Reduce
Task Size

16. Optimization
Multi-objective GAs
– Trading cost and performance (time constraints)
– Other factors: energy, …
– Future: multi-objective to many-objective (> 3)
Generate “Pareto-optimal Front”
Technique: non-dominated sorting
15
Objective 1
Objective 2
X
X X X
X
X
X
X
X
X
X
X
XX
X

17. (Short) Summary
A Sizing Framework for Hadoop/Spark Clusters
– OS container-based approach
– Combined with Genetic Algorithms
• Multi-objective optimization (cost & perf.)
Future Work
– Docker Container Executor (DCE)
• DCE runs YARN containers into Docker ones
• Designed to provide custom environment for each app.
• We believe DCE can also be utilized for slow-down and speeding-
up of Hadoop tasks
16

18. Slow Down - Torturing Hadoop
Make strugglers
No intervention is required
17
Map 1 Map 2 Map 3 Map 4 Map 5
Master
Red 1 Red 2 Red 3 Red 4
Map Tasks
Reduce Tasks
Struggler
Struggler

19. Speeding up - Accelerating Hadoop
Balance resource usage of tasks on the same node
18
Map 1 Map 2 Map 3 Map 4 Map 5
Master
Red 1 Red 2 Red 3 Red 4
Map Tasks
Reduce Tasks
Struggler
Struggler

20. MHCO: Multi-Objective Hadoop
Configuration Optimization Using
Steady-State NSGA-II

21. 20
Introduction
BIG
DATA
◦ Increasing use of connected devices at the hands of the Internet of
Things and data growth from scientific researches will lead to an
exponential increase in the data
◦ Portion of these data is underutilized or underexploited
◦ Hadoop MapReduce is very popular programming model for large
scale data analytics

22. Problem Definition I
21
◦ Objective 1  Parameter Tuning for Minimizing Execution Time
mapred-site
.xml
core-site
.xml
hdfs-site
.xml
yarn-site
.xml
Configuration settings
for HDFS core such as
I/O settings
Configuration settings
for HDFS daemons
Configuration settings
for MapReduce
daemons
Configuration settings
for YARN daemons
◦Hadoop provides tunable options have significant effect on
application performance
◦Practitioners and administers lack the expertise to tune
◦Appropriate parameter configuration is the key factor in Hadoop

23. Problem Definition II
22
◦ Appropriate machine instance selection for Hadoop cluster
◦ Objective 2  Instance Type Selection for Minimizing Hadoop
Cluster Deployment Cost
request
Service
provider
Applic
ation
result
Machine instance type
- small
- medium
- large
- x-large Pay Per
Use

24. Proposed Search based Approach
23
ssNSGA-II
Performance Optimization
Hadoop Parameter
Tuning
1
Deployment Cost
Optimization
Cluster Instance Type
Selection
2
◦ Chromosome encoding can solve dynamic nature of Hadoop on
version changes
◦ Use Steady State approach for computation overhead reduction
in generic GA approach
◦ Bi-objective optimization (execution time, cluster deployment
cost)

25. Objective Function
min t(p) , min c(p)
where,
p = [p1,p2,…,pm] ,
configuration parameter
list and instance type
t(p) = execution time of MR job
c(p)= machine instance usage
cost
24
t(p) = twc
c(p) = (SP*NS)*t(p)
where,
twc = workload execution time
SP= instance price
NS=no of machine instances
Assumption
- two objective functions are black-box functions
- no of instances in the cluster is static
Instance
type
Mem(GB)
/ cpu cores
Price per
second (Yen)
X-large 128/40 0.0160
Large 30/10 0.0039
Medium 12/4 0.0016
Small 3/1 0.0004

26. Parameter Grouping
I. HDFS and MAPREDUCE PARAMETERS
II. YARN PARAMETERS
III.YARN related MAPREDUCE PARAMETERS
25
17
6
7
30
machine instance
type specification
(cpu, mem)
reference from
previous researches

27. Group I Parameter Values
26
Parameter Name Value Range
dfs.namenode.handler.count 10, 20
dfs.datanode.handler.count 10, 20
dfs.blocksize 134217728,
268435456
mapreduce.map.output.compress True, False
mapreduce.job.jvm.numtasks 1: limited,
-1: unlimited
mapreduce.map.sort.spill.percent 0.8, 0.9
mapreduce.reduce.shuffle.input.buffer.p
ercent
0.7, 0.8
mapreduce.reduce.shuffle.memory.limit.
percent
0.25, 0.5
mapreduce.reduce.shuffle.merge.percent 0.66, 0.9
mapreduce.reduce.input.buffer.percent 0.0, 0.5
Parameter Name Value Range
dfs.datanode.max.transfer.threads 4096, 5120,
6144, 7168
dfs.datanode.balance.bandwidthPer
Sec
1048576,
2097152,
194304, 8388608
mapreduce.task.io.sort.factor 10, 20, 30, 40
mapreduce.task.io.sort.mb 100, 200, 300,
400
mapreduce.tasktracker.http.threads 40, 45, 50, 60
mapreduce.reduce.shuffle.parallelco
pies
5, 10, 15, 20
mapreduce.reduce.merge.inmem.thr
eshold
1000, 1500,
2000, 2500

28. Group II and III Parameter Values
YARN Parameters x-large large medium small
yarn.nodemanager.resource.memory.mb 102400 26624 10240 3072
yarn.nodemanager.resource.cpu-vcores 39 9 3 1
yarn.scheduler.maximum.allocation-mb 102400 26624 10240 3072
yarn.scheduler.minimum.allocation-mb 5120 2048 2048 1024
yarn.scheduler.maximum.allocation-vcores 39 9 3 1
yarn.scheduler.minimum.allocation-vcores 10 3 1 1
mapreduce.map.memory.mb 5120 2048 2048 1024
mapreduce.reduce.memory.mb 10240 4096 2048 1024
mapreduce.map.cpu.vcores 10 3 1 1
mapreduce.reduce.cpu.vcores 10 3 1 1
mapreduce.child.java.opts 8192 3277 1638 819
yarn.app.mapreduce.am.resource-mb 10240 4096 2048 1024
yarn.app.mapreduce.am.command-opts 8192 3277 1638 819
27

29. Chromosome Encoding
28
HDFS and MAPREDUCE Parameters
Binary Chromosome
Machine Instance Type
Single bit or two consecutive bits
represents parameter values,
instance type
Dependent Parameters
YARN Parameters
small
YARN related MapReduce Parameters
Chromosome Length = 26 bits

30. System Architecture
29
ssNSGA-II
optimization
workload
Resourc
e
Manager
Node
Manager
Node
Manager
Node
Manager
List of optimal
setting
Time Cost
…Cluster
deployment
cost

31. ssNSGA-II Based Hadoop Configuration Optimization
30
Generate n Sample Configuration Chromosomes
C1,C2,…,Cn
Select 2 Random Parents P1,P2
Perform 2 Point Crossover on P1, P2 (probability Pc =1)
Generate Offspring Coffspring
Perform Mutation on Coffspring (probability Pm= 0.1)
Coffspring Fitness Calculation
Update Population P
Perform Non-dominated Sorting
Update Population P
Output Pareto Solutions List, Copt
REPEAT
CONDITION = YES

32. Experiment Benchmark
31
Type Workload Input Size Benchmark
MicroBenchmark - Sort
- TeraSort
- Wordcount
2.98023GB - measure cluster
performance
(intrinsic behavior
of the cluster)
Web Search - Pagerank 5000 pages with
3 Iterations
- measure the
execution
performance for
real world big data
applications
Benchmark used : Hibench Benchmark suite version 4.0,
https://github.com/intel-hadoop/HiBench/releases

33. Experiment Environment
32
Setup Information Specification
CPU Intel ® Xeon R
E7-8870(40
cores)
Memory 128 GB RAM
Storage 400 TB
Hadoop version 2.7.1
JDK 1.8.0
NameNode
DataNode1 DataNode2 DataNode3
DataNode4 DataNode 5
User
Public
network
6-node cluster
1 NameNode
5 DataNodes
ssNSGA-II
optimization

34. Experimental Results
33
0
1
2
3
4
5
6
7
8
0 50 100 150 200
cost(¥)
execution time (sec)
sort workload result
small medium large x-large
0
1
2
3
4
5
6
30 40 50 60 70 80cost(¥)
execution time (sec)
terasort workload result
small medium large x-large
Population Size =30 Number of Evaluations=180
Number of Objectives = 2 Mutation Probability = 0.1
Crossover Probability = 1.0
* significant effects on HDFS and MapReduce Parameters

35. Experimental Results Cont’d
34
0
2
4
6
8
10
12
14
16
18
50 100 150 200 250 300cost(¥)
execution time (sec)
pagerank workload result
medium large x-largesmall
0
1
2
3
4
5
6
7
8
0 100 200 300 400 500 600
cost(¥)
execution time (sec)
wordcount workload result
small medium large x-large
* depend on YARN / related Parameters compared to HDFS and MapReduce Parameters
Population Size =30 Number of Evaluations=180
Number of Objectives = 2 Mutation Probability = 0.1
Crossover Probability = 1.0

36. Conclusion & Continuing Work
35
◦ Offline Hadoop configuration optimization using the ssNSGA-II
based search strategy
◦ x-large instance type cluster is not a suitable option for the
current workloads and input data size
◦ Large or medium instance type cluster show the balance for our
objective functions
◦Continuing process - dynamic cluster resizing through containers
and online configuration optimization of M/R workloads for
scientific workflow applications for effective Big Data Processing