1. Profiling the network performance of data transfers in Hadoop jobs
Team : Pramod Biligiri & Sayed Asad Ali
Abstract
We have attempted to reproduce existing research which shows that the Shuffle phase of
Hadoop is network intensive and can constitute a bottleneck for many Hadoop jobs. We ran the
Terasort and Ranked Inverted Index jobs on an Amazon Elastic MapReduce cluster. Our
experiments show that the Shuffle phase can form a significant fraction (upto nearly 30%) of the
time consumed in these jobs.
We do not have decisive results showing that the network is saturated during this phase. This is
due to a) lack of precise documentation on the network capacity of EMR, and b) inconsistent
results between our network benchmark tests and the results from the Hadoop jobs. See
Section 6 for a detailed discussion of both these factors.
1. Introduction
Data intensive computing on large scale, commodity clusters is becoming commonplace.
Hadoop[9] is a popular framework used in such computing environments. While performance
analysis is traditionally focused on the algorithm, the processing unit, memory and disk, the rise
of cluster computing adds the communication patterns of the algorithm and the underlying
network capacity as factors to consider while evaluating performance.
In this project we profile the data transfers that happen between the different stages of a Hadoop
job, with an aim to understand the utilization of network resources during the process. We hope
to reproduce some well known results which show that network utilization is a bottleneck in
MapReduce. We intend to focus on the shuffle phase of the MapReduce pipeline, and the
many­to­many pattern of data movement therein.
1.1. Hadoop
Hadoop is a framework for distributed processing of large data sets across clusters of
computers using simple programming models based on Google’s MapReduce [7]. Hadoop is
open source and implemented in Java.
Hadoop can be characterized by the following distinct features:
● Designed for commodity hardware
● Fault tolerant
● Horizontally scalable
● Push computation to data
1.2. MapReduce
MapReduce is a programming model for processing large data sets with a parallel, distributed
algorithm on a cluster. In this model, a program consists of two phases: Map and Reduce. In the
Map phase, each input record is processed to generate a (key, value) pair. In the Reduce phase,

2. values associated with the same key are grouped together and an operation is applied on them
to obtain the final results.
The following figure illustrates the flow of a MapReduce job:
1.3. Shuffle
The Shuffle is a phase where each reducer fetches its part of the sorted map outputs from all
the mapper nodes. This phase results in a n­to­n communication among a set of n nodes.

3. 2. Related Work:
We have studied a few papers which cite that the shuffle phase is an expensive operation. As
stated by the Orchestra paper[1]:
“On average, the shuffle phase accounts for 33% of the running time in these jobs. In addition,
in 26% of the jobs with reduce tasks, shuffles account for more than 50% of the running time,
and in 16% of jobs, they account for more than 70% of the running time. This confirms widely
reported results that the network is a bottleneck in MapReduce”
More information from the Hedera paper[2] corrobrates this:
“A data shuffle is an expensive but necessary operation for many MapReduce/Hadoop
operations in which every host transfers a large amount of data to every other host participating
in the shuffle. In this experiment, each host sequentially transfers 500MB to every other host
using TCP (a 120GB shuffle).”
Furthermore, the VL2 paper[3] also establishes this observation:
“we consider an all­to­all data shuffle stress test: all servers simultaneously initiate TCP
transfers to all other servers. This data shuffle pattern arises in large scale sorts, merges and
join operations in the data center. We chose this test because, in our interactions with
application developers, we learned that many use such operations with caution, because the
operations are highly expensive in today’s data center network. However, data shuffles are
required, and, if data shuffles can be efficiently supported, it could have large impact on the
overall algorithmic and data storage strategy.”

4. 3. Choice of Benchmarks:
3.1. Terasort
3.1.1. Why Terasort?
Terasort [4] is a popular benchmark for Hadoop and is also shipped with most Hadoop
distributions. This benchmark program sorts 1 terabyte of data. Each data item is 100 bytes in
size. The first 10 bytes of a data item constitute its sort key.
Each key is represented as:
<key 10 bytes><rowid 10 bytes><filler 78 bytes>rn
key : random characters from ASCII 32­126
rowid : an integer
filler : random characters from the set A­Z
The Terasort workload utilizes all aspects of the cluster ­ cpu, network, disk and memory ­ and
also has a large amount of data to shuffle (240 GB). Moreover, this is representative of real world
workloads, as mentioned in the VL2 paper[3]:
“we consider an all­to­all data shuffle stress test: all servers simultaneously initiate TCP
transfers to all other servers. This data shuffle pattern arises in large scale sorts, merges and
join operations in the data center. We chose this test because, in our interactions with application
developers, we learned that many use such operations with caution, because the operations are
highly expensive in today’s data center network. However, data shuffles are required, and, if data
shuffles can be efficiently supported, it could have large impact on the overall algorithmic and
data storage strategy.”
3.1.2. How it works?
The Map phase of Terasort partitions input keys into different buckets and then leverages
Hadoop’s default sorting of Map output. Finally, the reducer only collects outputs from different
maps and does not perform a computation­intensive task. Due to its simple application logic and
usage of Hadoop’s default sorting mechanism, Terasort is considered a good benchmarking
application.
3.2. Ranked Inverted Index
3.2.1. Why Ranked Inverted Index?
This benchmark was chosen as it is mentioned in the Tarazu[4] paper as a Shuffle heavy
workload. Also, a ranked inverted index is used often in text processing and information retrieval
tasks and is therefore a commonly executed job. For a given text corpus, for each word it
generates a list of documents containing the word in decreasing order of frequency
word ­> (count1 | file1), (count2 | file2), ...
count1 > count2 > …

5. 4. Experimental Setup:
4.1. Configuration
We utilised three configurations as a testbed for our experiments. Two of these were configured
on Amazon’s Elastic MapReduce (EMR) clusters and we used a cluster at SDSC as a learning
testbed. However, Config 1 on EMR is the one we chose for a majority of our tests and our
results are based on that.
Both the EMR configurations have 1 NameNode, 10 DataNode/Tasktrackers.
Instance
type
Memory CPU ECU Disk Network
performance
Config 1 m1.large 7.5 GB 64­bit 4 2 x 420 GB Moderate
Config 2 m1.xlarge 15 GB 64­bit 8 4 x 420 GB High
SDSC custom 8 GB 64­bit/ Intel Xeon
CPU 5140 @2.33
GHz, 4 cores
2 x 1.5 TB 1 Gb/s
4.2. Network Test
Source 1 : with AppNeta pathtest[8]
average : 753 Mb/s
Source 2 : “The available bandwidth is still 1 Gb/s, confirming anecdotal evidence that EC2 has
full bisection bandwidth."[5]
Source 3 : “The median TCP/UDP throughput of medium instances are both close to 760 Mb/s."
[6]
5. Results:
5.1. Terasort
5.1.1. Comparison of running Terasort on different Configurations
Total job
Time (min)
Map Time
(min)
Reduce
Time (min)
Shuffle Average
Time
Shuffle Time %
Config 1 205 84 205 60 29.3
SDSC 166 60 90 36 21.7
Config 2 86 40 75 22 25.5

6. 5.1.2. CDF of Transferred Data
The CDF shows that network traffic happens in two distinct phases. First is the Map phase
during which there is steady traffic, although not at high rates. Approximately half the amount of
the total volume of 240GB is transferred during this time.
Following the Map, the traffic reduces as the map outputs are sorted locally. Then the job enters
the Shuffle phase, where the data is transferred to all the reducers. During this phase, the
network traffic saturates the links, as we show in subsequent sections. This phase transfers the
remaining half of the 240GB of data.

7. 5.1.3. Network activity
The following figure shows the network transfer rates over the lifetime of the job. It shows that
during the Shuffle phase the network traffic reaches 700 Mbps, which was the peak transfer rate
as measured by one of our tests.
5.1.4. Disk activity
The figure shows that the map phase has a good mix of Read and Write. However, once the
map is done which is around the 5100sec mark, a marked reduction in the data read is observed
and this phase shows a increase in writing activity of the cluster. This pattern continues till the
end of the shuffle phase around 6900s, where shuffle starts and the pattern shifts to read/write
but a marked reduction in read activity.
The observed peak value is around 60 to 80 MB/s which is well below the threshold value of 100
MB/s according to the dd performance metric which was executed on the Amazon EC2
machines.

8. 5.1.5. Memory Activity
The captured logs indicate that throughout the timeline of the job, the memory shows a
somewhat consistent utilisation of nearly 4.5 GB on all boxes and never overshoots this mark.
As nearly 7.5 GB of memory is available on each box, this proves that the memory was never
stressed to capacity, therefore hinting that the bottleneck lies elsewhere.

9. 5.1.6. CPU Utilisation
The graphs below show the CPU utilisation on the EC2 boxes over the lifetime of the project. As
indicated by the figure, the CPU may have been stressed to capacity with 100% utilisation but
the pattern seen in the initial portion of the graph is very erratic and shows a high oscillation
between full and partial CPU utilisation. However, during the shuffle phase, it is quite evident that
the CPU utilisation drops below the 50% threshold, thus the CPU utilisation cannot be the limiting
factor in the case of Shuffle phase of a job.

10. 5.2. Ranked Inverted Index (RII)
5.1.1. Data for a RII run on Config 1
Total job
Time (min)
Map Time
(min)
Reduce
Time (min)
Shuffle Average
Time
Shuffle Time %
Config 1 12 5.5 11.5 3.5 27.14
5.1.2. CDF of data
The CDF shows that network traffic happens in three distinct phases. First is the Map phase
during which there is steady traffic, although not at high rates. Once, shuffle is activated then the
network traffic picks up, this is where 13 GB of data gets transferred across the network in a
short duration and we see the saturation point of the network. The burst of traffic which happens
after this is the replication of the results to 3 nodes.
5.1.3. Network Activity
The following figure shows the network transfer rates over the lifetime of the job. It shows that
during the Shuffle phase the network traffic reaches 1.5 Gbps, which is something we have not
been able to explain as the maximum expected rate should have been in the range of 700 ­ 800
Mbps.

11. 5.1.4. Disk Activity
The figure shows that the map phase has a good mix of Read and Write. However, once the
map is done which is around the 350sec mark, a marked reduction in the data read is observed
and this phase shows an increase in writing activity of the cluster. This pattern continues till the
end of the shuffle phase around 550s, where shuffle starts and the pattern shifts to read/write but
a marked reduction in read activity.
The sporadic maximum read value is around 140 to 150 MB/s which does not stress the cluster
as the consistent read rate is well below that.

12. 5.1.5. Memory Activity
The captured logs indicate that throughout the timeline of the job, the memory shows a
somewhat consistent utilisation of nearly 4.5 GB on all boxes and never overshoots this mark.
As nearly 7.5 GB of memory is available on each box, this proves that the memory was never
stressed to capacity, therefore hinting that the bottleneck lies elsewhere.

13. 5.1.6. CPU Utilisation
he graphs below show the CPU utilisation on the EC2 boxes over the lifetime of the project. As
indicated by the figure, the CPU may have been stressed to capacity with 100% utilisation during
the map phase but the pattern seen in the initial portion of the graph is very erratic and shows a
high oscillation between full and partial CPU utilisation. However, during the shuffle phase, it is
quite evident that the CPU utilisation drops below the 50% threshold, thus the CPU utilisation
cannot be the limiting factor in the case of shuffle phase of a job.

14. 6. Unresolved Issues
6.1 Maximum network bandwidth of EMR
Amazon does not provide maximum network bandwidth rates for EMR. The network
performance is only described in qualitative terms (Low, Moderate and High). We measured the
network bandwidth between 2 nodes using the pathtest application from AppNeta [8]. We found a
peak transfer rate of 753Mb/s.
We surveyed existing literature for the same, and found two conflicting versions:
1. “The available bandwidth is still 1 Gb/s, confirming anecdotal evidence that EC2 has full
bisection bandwidth" ­ Opening Up Black Box Networks with CloudTalk, by Costin Raiciu et al
2. “The median TCP/UDP throughput of medium instances are both close to 760 Mb/s" ­ The
Impact of Virtualization on Network Performance of Amazon EC2 Data Center, by Guohui Wang
et al
Further, the Terasort job maxed out at 800 Mb/s during our runs, whereas the Ranked Inverted
Index crossed 1Gb/s. We are not able to reconcile these results, and believe it needs further
investigation.
6.2 Sorting phase on Ranked Inverted Index
The sorting phase of the Ranked Inverted Index lasts for a very short duration. In fact it is not
noticeable on the network CDF graph. But from the Disk activity graph it can be seen that there
is write heavy activity between the 300­400 seconds mark, which correspond to a period of low
network activity. It should be investigated why the network transfer does not flatline during this
period for this job, whereas it does in the case of Terasort.
7. Summary
We have shown that both for the Terasort and Ranked Inverted Index jobs, the shuffle phase of
MapReduce can constitute a large fraction of the overall job runtime (nearly 30%). We infer that
for this phase, the network is potentially a bottleneck, as there is low activity on CPU, disk and
memory. A better understanding of EMR’s network performance can lead to more a conclusive
result in this regard.
8. Future Work
Apart from the issues raised in Section 6, we see other avenues of investigation for this project.
The experiments should be run on different kinds of hardware, and different values for certain
Hadoop parameters. The important Hadoop parameters to consider are: io.sort.mb,
io.sort.factor, and fs.inmemory.size.mb.
Jobs should be investigated with and without the presence of Combiners, which help to reduce
the amount of data shuffled. The number of map and reduce tasks can be varied to see if that
has an impact on the results.
Also, the locality of the tasks is an important factor to be considered while evaluating Hadoop

15. jobs eg a rack­local or machine local setup may perform better.
There is scope for extensive work to determine the topology and bandwidth expectations of
Amazon’s EMR clusters.

16. 9. References
1. Mosharaf Chowdhury, Matei Zaharia, Justin Ma, Michael I. Jordan, Ion Stoica. Managing Data
Transfers in Computer Clusters with Orchestra ­ in SIGCOMM ’11
2. M. Al­Fares, S. Radhakrishnan, B. Raghavan, N. Huang, and A. Vahdat. Hedera: Dynamic flow
scheduling for data center networks. In NSDI, 2010
3. A. Greenberg, J. R. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri,
D. A. Maltz, P. Patel, and S. Sengupta. VL2: A scalable and flexible data center network. In
SIGCOMM, 2009
4. Tarazu: Optimizing MapReduce on Heterogeneous Clusters
Faraz Ahmad, Srimat T. Chakradhar, Anand Raghunathan, T.N. Vijaykumar
5. Opening Up Black Box Networks with CloudTalk, by Costin Raiciu et al
6. The Impact of Virtualization on Network Performance of Amazon EC2 Data Center, by Guohui
Wang et al
7. MapReduce: Simplified Data Processing on Large Clusters, by Jeffrey Dean and Sanjay
Ghemawat
8. AppNeta pathtest ­ http://www.appneta.com/resources/pathtest­download.html
9. Apache Hadoop ­ http://hadoop.apache.org/