1. Apache Hadoop YARN
Page 1

2. A Cursory Look At The Architecture
Node
Node
Manager
Manager
Container App Mstr
App Mstr
Client
Resource Node
Node
Resource
Manager
Manager Manager
Manager
Client
Client
App Mstr Container
Container
MapReduce Status Node
Node
MapReduce Status
Manager
Manager
Job Submission
Job Submission
Node Status
Node Status
Resource Request
Resource Request Container Container
© Hortonworks Inc. 2012. Confidential and Proprietary. Page 2

3. Global Scheduler (ResourceManager)
• Pure resource arbitration
• Multiple resource dimensions
–<priority, data-locality, memory, cpu, …>
• In-built support for data-locality
–Node, Rack etc.
– Unique to YARN
© Hortonworks Inc. 2012. Confidential and Proprietary. Page 3

4. Scheduler Concepts
• Input from AM(s) is a dynamic list of ResourceRequests
– <resource-name, resource-capability>
– Resource name: (hostname / rackname / any)
– Resource capability: (memory, cpu, …)
– Essentially an inverted <name, capability> request map from AM to RM
– No notion of tasks!
• Output - Container
–Resource(s) grant on a specific machine
–Verifiable grant
© Hortonworks Inc. 2012. Confidential and Proprietary. Page 4

5. Scheduling Walkthrough
MapReduce job with 2 maps and 1 reduce
© Hortonworks Inc. 2012. Confidential and Proprietary. Page 5

6. Scheduling Walkthrough
Container allocation on r22/h2121:
© Hortonworks Inc. 2012. Confidential and Proprietary. Page 6

7. Scheduling Walkthrough
Container allocation on r11/h1010:
© Hortonworks Inc. 2012. Confidential and Proprietary. Page 7

8. Writing Custom Applications
• Grand total of 3 protocols
–ClientRMProtocol
– Application launching program
– submitApplication
–AMRMProtocol
– Protocol between AM & RM for resource allocation
– registerApplication / allocate / finishApplication
–ContainerManagerProtocol
– Protocol between AM & NM for container start/stop
– startContainer / stopContainer
© Hortonworks Inc. 2012. Confidential and Proprietary. Page 8

9. API improvements
• Overload of the ‘*’ entry.
• Release / reject containers
• Ask for specific nodes/racks (only)
• Don’t give me containers on this racks/nodes
• Single client thread allowed to request containers
• Overloaded allocate call
Page 9
© Hortonworks Inc. 2012

10. Recent advancements
• Tools for debugging AMs
–Unmanaged AM
• Generic AM – Utility libraries for writing
–YARN-103, YARN-29
• YARN project split and how multiple versions of
MapReduce can coexist.
Page 10
© Hortonworks Inc. 2012

11. Roadmap
• MapReduce container reuse
• RM restart capability
• Multi-resource scheduling
• Generic application history server
Page 11
© Hortonworks Inc. 2012

12. Questions?
Thank You!
© Hortonworks Inc. 2012. Confidential and Proprietary. Page 12