2. What is YARN?
• YARN stands for Yet Another Resource Negotiator. It’s a framework introduced in 2010 by a group ay Yahoo!. This is
considered as next generation of MapReduce. YARN is not specific for MapReduce. It can be used for running any
application.
Why YARN?
• In a Hadoop cluster which has more than 4000 nodes, Classic MapReduce hit scalability bottlenecks. This is because
JobTracker does too many things like job scheduling, monitoring task progress by keeping track of tasks, restarting failed or
slow tasks and doing task bookkeeping (like managing counter totals).
How YARN solves the problem?
• The problem is solved by splitting the responsibility of JobTracker (in Classic MapReduce) to different components. Because
of which, there are more entities involved in YARN (compared to Classic MR). The entities in YARN are as follows;
• Client: which submits the MapReduce job
• Resource Manager: which manages the use of resources across the cluster. It creates new containers for Map and
Reduce processes.
• Node Manager: In every new container created by Resource Manager, a Node Manager process will be run which
oversees the containers running on the cluster nodes. It doesn’t matter if the container is created for Map or Reduce
or any other process. Node Manager ensures that the application does not use more resources than what it is
allocated with.
• Application Master: which negotiates with the Resource Manager for resources and runs the application-specific
process (Map or Reduce tasks) in those clusters. The Application Master & the MapReduce tasks run in containers
that are scheduled by the resource manager and managed by the node manager.
• HDFS
How to activate YARN?
• By setting the property ‘mapreduce.framework.name’ to ‘yarn’, the YARN framework will be activated. From then on when a
Job is submitted, YARN framework will be used to execute the Job.

3. FIRST A JOB HAS TO BE SUBMITTED TO
HADOOP CLUSTER. LET’S SEE HOW JOB
SUBMISSION HAPPENS IN CASE OF YARN.

4. MR Program
Job
Client JVM
ResourceManager
HDFS
Folder in name of Application ID
getNewApplicationId()
submit() submitApplication()
Job Jar
Configuration Files
Computed Input
Splits
Resource Manager node
• Job submission in YARN is very similar to Classic MapReduce. In YARN its not called as Job
instead its called as Application.
• Client calls the submit() (or waitForCompletion()) method on Job.
• The Job.submit() does the following;
• A new application ID is retrieved from Resource Manager.
• Checks the input and output specification.
• Computes input slits.
• Creates a directory in the name of Application ID in HDFS.
• Copies Job jar, configuration files and computed input splits to this directory.
• Informs the Resource manager by executing the submitApplication() on the Resource
manager.

5. NEXT THE SUBMITTED JOB WILL BE INITIALIZED.
NOW LET’S SEE HOW JOB INITIALIZATION
HAPPENS IN YARN.

6. • ResourceManager.submitApplication() method hands over the job to scheduler.
• Scheduler creates a new Container.
• All containers in YARN will have an instance of NodeManager running in it, which manages
the actual process which is scheduled to run in the container.
• The actual process in this case is an application master. For a MR Job, the main class for an
application master is ‘MRAppMaster’.
• MRAppMaster initilizes the job by creating number of bookkeeping objects to track the job’s
progress as it receives progress & completion reports from the tasks.
• Next the MRAppMaster retrieves input splits from HDFS.
ResourceManag
er
submitApplication(
)
Scheduler
HDFS
Map Tasks
Reduce Tasks
Other Tasks
Bookkeeping Info
Input Splits stored in Job ID
directory in HDFS.
T1 T2 T3
T1
J
S
J
CS
1
S
2
S
3
• Now application master, creates one map task per split and it checks the
‘mapreduce.job.reduces’ property & creates those many number of reducers.
Container
NodeManager
Application
Master
MRAppMaster

7. • At this point, application master knows how big the job is and it decides if it can execute the
job in the same JVM as itself or should it run each of these tasks parallel in different
containers. Such small jobs are said to be uberized or run as uber task.
• This decision is made by the application master based on the following property
configurations;
• mapreduce.job.ubertask.maxmaps
• mapreduce.job.ubertask.maxreduces
• mapreduce.job.ubertask.maxbytes
• mapreduce.job.ubertask.enable
• Before any task can run, application master executes the job setup methods to create job’s
output directory.

8. IF THE JOB IS NOT RUN AS UBER TASK, THEN
APPLICATION MASTER REQUESTS CONTAINERS
FOR ALL MAP AND REDUCE TASK. THIS
PROCESS IS CALLED TASK ASSIGNMENT. NOW
LET’S SEE HOW TASK ASSIGNMENT HAPPENS
IN YARN.

9. • Application master sends heartbeat signal to Resource Manager every few seconds.
Application Master uses this signal to request containers for Map and Reduce tasks.
Resource
Manager
Application Master sends a heartbeat
signal with request for map and reduce
tasks
• The container request includes information about map task’s data locality i.e., host and rack
in which the split resides.
• Unlike MR 1, where there are fixed number of slots and fixed amount of resources allocated,
YARN is pretty flexible in resource allocation. The request (which is sent along with the
heartbeat signal) for container can include a request for amount of memory needed for the
task. The default for map and reduce task is 1024 MB.
• Scheduler uses this information to make scheduling decisions. Scheduler tries to do a local
placement. If not possible, it tried for rack-local placement. Else non-local placement. Refer :
Replica placement slide.
Container
NodeManager
MRAppMaster
• Once the Resource Manager gets this request, it creates a new container and starts Node
Manager instance in it to manage the Map or Reduce task for which the container was
created for. It also ensures that the requested amount of resources are allocated to the

10. NOW TASKS ARE ASSIGNED TO CONTAINER
WHICH FOLLOWS A SERIES OF STEPS TO
EXECUTE A TASK. LET’S SEE HOW TASKS ARE
EXECUTED IN A YARN CONTAINER.

11. Distributed
Cache
HDFS
Folder created in
container’s local.
• Application Master starts a container through the Node Manager running in the other
container.`• Node Manager (in the other container) spawns a new JVM process and launches a new Java
application called ‘YarnChild’. The reason for a new JVM process is same as MR 1. And
YARN doesn’t support JVM reuse.
• The work of YarnChild is to execute the actual process (Map or Reduce).
• First YarnChild tries to localize the resources like Job jar, configuration files and supporting
files from Distribute Cache.
• Once the resources are localized, YarnChild begins executing the Map or Reduce task.
Container
NodeManager
Container
NodeManager
MRAppMaster
JVM Process
YarnChild
Map/Reduce
Task
Un-jar the job jar
contents

12. SINCE TASKS ARE EXECUTED IN A DISTRIBUTED
ENVIRONMENT, TRACKING THE PROGRESS AND
STATUS OF JOB IS TRICKY. LET’S SEE HOW
PROGRESS AND STATUS UPDATES ARE TAKEN
CARE IN YARN.

13. • Clients poll the Application master every second to receive the progress updates. This can be
configured using the property mapreduce.client.progressmonitor.pollinterval.
• The Application Master then aggregates this to build the overall job progress.
• The task sends its progress and counters are sent to Application Master once in three
seconds.
Container
NodeManager
Container
NodeManager
MRAppMaster
JVM Process
YarnChild
Map/Reduce
Task
Client Node
Job
getStatus()
MapReduce
Program
Job: SFO Crime
Job Status: Running
Task & task status
• The Application Master Web UI displays all the running applications with links to the web Uis
of respective application masters, each of which displays further details on the MR job,
including its progress.

14. THIS EXECUTION PROCESS CONTINUES TILL ALL
THE TASKS ARE COMPLETED. ONCE THE LAST
TASK IS COMPLETED, MR FRAMEWORK ENTERS
THE LAST PHASE CALLED JOB COMPLETION.

15. • When the job is completed, the application master and task containers clean up their working
state, and the OutputCommiter’s job cleanup method is executed.
• If the property ‘job.end.notification.url’ is set, the Job Tracker will send a HTTP job notification
to the client.
• Job information is archived by the job history server to enable later interrogation by users if
desired.

16. THE END
PLEASE SEND YOUR VALUABLE FEEDBACK TO
RAJESH_1290K@YAHOO.COM