1. CASE STUDY
Enhancing, Monitoring and
Managing a Hadoop Based
Analytics Solution
In this engagement, Imaginea team contributed over 14 patches to the Hadoop community and
all of them were verified and accepted.
COPYRIGHT © 2012, IMAGINEA TECHNOLOGIES, INC.
COPYRIGHT © 2012, IMAGINEA TECHNOLOGIES, INC. THIS DOCUMENT IS CONFIDENTIAL AND NOT FOR DISTRIBUTION WITHOUT
WRITTEN PERMISSION FROM IMAGINEA TECHNOLOGIES, INC.

2. CASE STUDY
1. Executive Summary
One of Imaginea’s clients is a video marketing company that deals with branding,
real time media buying, ad serving, targeting, optimization and brand
measurement.
Imaginea enhanced and managed a platform for video playtime statistical
analysis for our client. The solution used Hadoop (Cloudera distribution) and
Hive. The cluster was 500 nodes with 300 TB of existing data and over 200 GBs
data being streamed in and processed every day.
2. Hadoop Migration and New Features
We helped in migration of the entire platform from 0.19 to 0.20.2, porting all the
MR jobs. Migration also included back-porting some feature from .21 to 20.
Features that were back-ported included:
 Map-Side join
 CompositeInputFormat
3. Cluster Monitoring, Management & Resolution
We helped in monitoring and managing the cluster during IST Business hours.
We were able to uncover workflow instability issues and lack of resume feature
during these phases, which we resolved later.
The solution had used a custom workflow manager; it had stability issues
especially as the load increased by orders of magnitude.
Zookeeper was introduced as the central workflow status manager and changes
were made to the workflow manager to use the same. This helped the system
stability improve by about 90%.
We also discovered problems in publishing configuration and code changes to all
the nodes in the cluster during this phase. We used Ganglia and Nagios for
COPYRIGHT © 2012, IMAGINEA TECHNOLOGIES, INC. THIS DOCUMENT IS CONFIDENTIAL AND NOT FOR DISTRIBUTION WITHOUT
WRITTEN PERMISSION FROM IMAGINEA TECHNOLOGIES, INC.

3. CASE STUDY
monitoring. We also solved some of the memory overflow issues in the Hadoop
nodes.
4. Configuration Management using Puppet
As part of the engagement, Imaginea worked to introduce Puppet in to the system
removing a custom configuration management tool. We had developed some
recipes and were able to solve many issues that were raised with replication of
configuration changes and deployment of new codebase
5. Performance Improvements
Imaginea contributed to enhance performance in a variety of ways. Below are a
couple of highlight scenarios
Job Starvation
Problem: Many cases of data overflow at a collector level
The solution had business analytics Hive queries, which used to starve the
normal MR jobs. Imaginea helped in development of a fair scheduling algorithm
which balances the production tasks and hive query jobs. Before this solution
there were many cases of data overflow at the collector level.
Job Optimization
Problem: Job to identify if the user was unique took over 8 hours
Imaginea helped in optimizing the job from 8-10 hours to 4 hours using better
distribution of keys and better hashing algorithm.
COPYRIGHT © 2012, IMAGINEA TECHNOLOGIES, INC. THIS DOCUMENT IS CONFIDENTIAL AND NOT FOR DISTRIBUTION WITHOUT
WRITTEN PERMISSION FROM IMAGINEA TECHNOLOGIES, INC.

4. CASE STUDY
6. Apache Hadoop involvement and other
contributions
We have worked on Apache Hadoop and other components. Following are the list
of patches that were contributed to the community by Imaginea.
Jira Id Severity/ Priority Component Brief Description
MAPREDUCE-3360 Critical-Improvement mrv2 Provide information about lost nodes in the UI
MAPREDUCE-3686 Critical-Bug mrv2 history server web ui - job counter values for map/reduce not
shown properly
MAPREDUCE-3532 Critical-Bug mrv2,nodema When 0 is provided as port number in
nager yarn.nodemanager.webapp.address, NMs webserver
component picks up random port, NM keeps on Reporting 0
port to RM
MAPREDUCE-3952 Major-Bug mrv2 In MR2, when Total input paths to process == 1,
CombinefileInputFormat.getSplits() returns 0 split.
MAPREDUCE-3686 Critical-Bug mrv2 history server web ui - job counter values for map/reduce not
shown properly
MAPREDUCE-3532 Critical-Bug mrv2,nodema When 0(zero) is provided as port number in
nager yarn.nodemanager.webapp.address, NMs webserver
component picks up random port, NM keeps on Reporting
0(zero) port to RM
MAPREDUCE-3316 Major-Bug Resource Rebooted Link not working
Manager
MAPREDUCE-3708 Major-Bug mrv2 Metrics: Incorrect Apps submitted count
MAPREDUCE-3723 Major-Bug mrv2, test, TestAMWebServicesJobs & TestHSWebServicesJobs
webapp incorrectly asserting tests
MAPREDUCE-4050 Major-Bug mrv2 Invalid Node link
MAPREDUCE-3870 Major – Bug mrv2 Invalid App Metrics
MAPREDUCE-4102 Major – Bug Webapps Job counter not available in Job History Web UI for killed jobs
MAPREDUCE-4002 Major – Bug Examples MultiFileWordCount job fails if the input path is not from default
file system
MAPREDUCE-4040 Minor-Bug mrv2, History links should use hostname rather than IP address.
jobhistoryserv
er
MAPREDUCE-3212 Minor-Bug mrv2 Message displays while executing yarn command should be
proper
COPYRIGHT © 2012, IMAGINEA TECHNOLOGIES, INC. THIS DOCUMENT IS CONFIDENTIAL AND NOT FOR DISTRIBUTION WITHOUT
WRITTEN PERMISSION FROM IMAGINEA TECHNOLOGIES, INC.