Apache Hadoop YARN is the modern distributed operating system for big data applications. It morphed the Hadoop compute layer to be a common resource management platform that can host a wide variety of applications. Many organizations leverage YARN in building their applications on top of Hadoop without themselves repeatedly worrying about resource management, isolation, multi-tenancy issues, etc. In this talk, we’ll start with the current status of Apache Hadoop YARN—how it is used today in deployments large and small. We'll then move on to the exciting present and future of YARN—features that are further strengthening YARN as the first class resource management platform for data centers running enterprise Hadoop. We’ll discuss the current status as well as the future promise of features and initiatives like: powerful container placement, global scheduling, support for machine learning and deep learning workloads through GPU and FPGA support, extreme scale with YARN federation, containerized apps on YARN, support for long-running services (alongside applications) natively without any changes, seamless application upgrades, powerful scheduling features like application priorities, intra-queue preemption across applications, and operational enhancements including insights through Timeline Service V2, a new web UI, and better queue management. Speaker: Sanjay Radia, Chief Architect, Founder, Hortonworks

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved
Apache Hadoop YARN:
State of the Union
Sanjay Radia
Founder, Chief Architect Hortonworks
Apache Hadoop PMC

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved
About the Speaker
• Sanjay Radia
• Chief Architect, Founder, Hortonworks
• Apache Hadoop PMC and Committer
• Part of the original Hadoop team at Yahoo! since 2007
• Chief Architect of Hadoop Core at Yahoo!
• Prior
• Data center automation, virtualization, Java, HA, OSs, File Systems
• Startup, Sun Microsystems, INRIA…
• Ph.D., University of Waterloo
Page 2Architecting the Future of Big Data

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved
• Introduction
• Past
• State of Union for YARN
Agenda

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved
A Brief Timeline from Past Year: GA Releases
2.8.0 2.9.0 3.0.0 3.1.0
• GPU/FPGA
• YARN Native
Service
• Placement
Constraints
• YARN Federation
• Opportunistic
Container
(Backported from 3.0)
• New YARN UI
• Timeline V2
• Global Scheduling
• Multiple Resource types
• New YARN UI
• Timeline service V2
Ever involving requirements (computation intensive, larger, services)
• Application Priority
• Reservations
• Node labels
improvements
22 March ’17 17 Nov ’17 13 Dec ‘17 02 Aug ‘182.8.4 2.9.1 3.0.3 3.1.1

5. 8 © Hortonworks Inc. 2011–2018. All rights reserved
Apache Hadoop 3.0/3.1

6. 9 © Hortonworks Inc. 2011–2018. All rights reserved
Key Themes
Scale
Platform Themes Workload Themes
Scheduling Usability Containers Resources Services

7. 10 © Hortonworks Inc. 2011–2018. All rights reserved
Key Themes
Scale
Platform Themes Workload Themes
Scheduling Usability Containers Resources Services

8. 11 © Hortonworks Inc. 2011–2018. All rights reserved
• Many customers with clusters made up of large number of nodes
• Oath (Yahoo!), Twitter, LinkedIn, Microsoft, Alibaba etc.
• Previous largest clusters: 6K-8K
• Now: 50K nodes under YARN in Microsoft [1]
• Roadmap: To 100K and beyond
[1] https://azure.microsoft.com/en-us/blog/how-microsoft-drives-exabyte-analytics-on-the-world-s-largest-yarn-cluster/
Looking at the Scale!

9. 12 © Hortonworks Inc. 2011–2018. All rights reserved
• Enables applications to scale to 100k of thousands of nodes
• Federation divides a large (10-100k nodes) cluster into smaller units called sub-clusters
• Federation negotiates with sub-clusters RM’s and provide resources to the application
• Applications can schedule tasks on any node
YARN Federation

10. 13 © Hortonworks Inc. 2011–2018. All rights reserved
Key Themes
Scale
Platform Themes Workload Themes
Scheduling Usability Containers Resources Services

11. 14 © Hortonworks Inc. 2011–2018. All rights reserved
• YARN-5139
• Problems
• Current design of one-node-at-a-time allocation cycle can lead to suboptimal decisions.
• Performance and also hard to do global placement policies
• Several coarse grained locks
• With this, we improved to
• Look at several nodes at a time
• Fine grained locks
• Multiple allocator threads
• YARN scheduler can allocate 3k+ containers per second ≈ 10 mil allocations / hour!
• 10X throughput gains
• Much better placement decisions
Moving Towards Global & Fast Scheduling

12. 16 © Hortonworks Inc. 2011–2018. All rights reserved
Better Placement Strategies (YARN-6592)
• Past
• Supported constraints in form of Node Locality
• Now YARN can support a lot more use cases
• Co-locate the allocations of a job on the same rack (affinity)
• Spread allocations across machines (anti-affinity) to minimize resource interference
• Allow up to a specific number of allocations in a node group (cardinality)

13. 17 © Hortonworks Inc. 2011–2018. All rights reserved
Better Placement Strategies (YARN-6592)
Affinity Anti-affinity
Region
Server
Region
Server

14. 18 © Hortonworks Inc. 2011–2018. All rights reserved
Absolute Resources Configuration in CS – YARN-5881
• The cloud model! “Give me X resources, not X%”
• Gives ability to configure Queue resources as below
<memory=24GB, vcores=20, yarn.io/gpu=2>
• Enables admins to assign different quotas of different resource-types
• No more “Single percentage value limitation for all resource-types”
root
(memory=60G, vcores=40,
gpu=10)
sales
(memory=40G, vcores=20,
gpu=6)
marketing
(memory=10G, vcores=10,
gpu=4)
engineering
(memory=10G, vcores=10,
gpu=0)

15. 20 © Hortonworks Inc. 2011–2018. All rights reserved
Useful Scheduling Features from Recent 2.X Releases
• Application Priorities
• Queue Priorities
• Previously : Resources to least satisfied queue
• Queues can have priorities e.g. (production over dev)

16. 21 © Hortonworks Inc. 2011–2018. All rights reserved
Key Themes
Scale
Platform Themes Workload Themes
Scheduling Usability Containers Resources Services

17. 22 © Hortonworks Inc. 2011–2018. All rights reserved
Usability: UI (1)

18. 23 © Hortonworks Inc. 2011–2018. All rights reserved
Usability: UI (2)

19. 24 © Hortonworks Inc. 2011–2018. All rights reserved
Timeline Service 2.0
• Application History
• Where did my container run
• Why is my app slow
• Why is failing
• What happened with my app
• Cluster History
• Which user had most utilization
• Largest App
• What happened in my cluster
• …
• Scalable & Robust Service
• Using HBase as backend for better scale
• More robust storage fault tolerance
• Migration and compatibility with v.1.5

20. 25 © Hortonworks Inc. 2011–2018. All rights reserved
Usability: Queue & Logs
API based queue management
Decentralized
(YARN-5734)
Improved logs
management
(YARN-4904)
Live application logs

21. 26 © Hortonworks Inc. 2011–2018. All rights reserved
Key Themes
Scale
Platform Themes Workload Themes
Scheduling Usability Containers Resources Services

22. 27 © Hortonworks Inc. 2011–2018. All rights reserved
• Better Packaging model
• Lightweight mechanism for packaging and resource isolation
• Popularized and made accessible by Docker
• Native integration ++ in YARN
• Support for “Container Runtimes” in LCE: YARN-3611
• Native process runtime
• Docker runtime
• Many security/usability improves added to 3.0.x / 3.1.x
Containers

23. 28 © Hortonworks Inc. 2011–2018. All rights reserved
Containers
• Run both with and without Docker on the same cluster
• Choose at run-time!

24. 29 © Hortonworks Inc. 2011–2018. All rights reserved
• Apache Spark applications have a complex set of required software dependencies
• Docker on YARN helps to solve Spark package isolation issues with
• PySpark - Python versions, packages
• R - packages
Spark on Docker in YARN

25. 31 © Hortonworks Inc. 2011–2018. All rights reserved
Hadoop Apps
YARN on YARN… Towards a Private Cloud …YCloud
YARN
MR Tez Spark
Tensor
Flow
YARN
MR Tez Spark
Spark

26. 32 © Hortonworks Inc. 2011–2018. All rights reserved
YCloud: YARN Based Container Cloud We Use for Testing
ü Testing Hadoop on Hadoop!

27. 33 © Hortonworks Inc. 2011–2018. All rights reserved
Key Themes
Scale
Platform Themes Workload Themes
Scheduling Usability Containers Resources Services

28. 34 © Hortonworks Inc. 2011–2018. All rights reserved
• YARN supported only Memory
and CPU
• Now
• A generalized vector for all resources
• Admin could add arbitrary resource types!
Resource Profiles and Custom Resource Types
• Ease of resource requesting model
using profiles for apps
Profile Memory CPU GPU
Small 2 GB 4 Cores 0 Cores
Medium 4 GB 8 Cores 0 Cores
Large 16 GB 16 Cores 4 CoresMemory
CPU
GPU
FPGA
Node Manager

29. 35 © Hortonworks Inc. 2011–2018. All rights reserved
• Why?
• No need to setup separate clusters
• Leverage shared compute!
• Why need isolation?
• Multiple processes use the single GPU will be:
• Serialized.
• Cause OOM easily.
• GPU isolation on YARN:
• Granularity is for per-GPU device.
• Use Cgroups / Docker to enforce isolation.
GPU Support on YARN
Tensorflow 1.2
Nginx AppUbuntu 14:04
Nginx AppHost OS
GPU Base Lib v1
Volume Mount
CUDA Library 5.0

30. 36 © Hortonworks Inc. 2011–2018. All rights reserved
• FPGA isolation on YARN: .
• Granularity is for per-FPGA device.
• Use Cgroups to enforce the isolation.
• Currently, only Intel OpenCL SDK for FPGA is supported.
• But implementation is extensible to other FPGA SDK.
FPGA on YARN

31. 37 © Hortonworks Inc. 2011–2018. All rights reserved
Key Themes
Scale
Platform Themes Workload Themes
Scheduling Usability Containers Resources Services

32. 38 © Hortonworks Inc. 2011–2018. All rights reserved
Services Support in YARN
• In the past we have focused on Jobs and have an advanced fine grained scheduler
• A native YARN services framework (YARN-4692)
• Manage services and their lifecycle
• Scale up and down, restart etc
• Support for Upgrades (YARN-4726
• “Do an upgrade of my HBase app with minimal impact to end-users
• Apache Slider retired from Incubator – lessons and key code carried over to YARN
• Simplified discovery of services via DNS mechanisms: YARN-4757
• regionserver-0.hbase-app-3.hadoop.yarn.site

33. 39 © Hortonworks Inc. 2011–2018. All rights reserved
• Applications need simple APIs
• Need to be deployable “easily”
• Simple REST API layer fronting YARN
• YARN-4793 Simplified API layer
• Spawn services & Manage them
Simplified APIs for Service Definitions

34. 41 © Hortonworks Inc. 2011–2018. All rights reserved
LLAP on YARN
• Apache Hive LLAP is a key long running application
• Used for query processing
• Designed to run on a shared multi-tenant YARN cluster

35. 42 © Hortonworks Inc. 2011–2018. All rights reserved
Application Timeout – YARN-3813
• Controlling running time of workloads in YARN
• Define lifetime for an application anytime for YARN to manage.
• “Give me resources for this app/service but kill it after 15 days”
Reservation-based Scheduling: If You’re Late Don’t Blame Us! - Carlo, et al. 2015

36. 43 © Hortonworks Inc. 2011–2018. All rights reserved
Apache Hadoop 3.2 and Beyond

37. 44 © Hortonworks Inc. 2011–2018. All rights reserved
• Every user says “Give me 16GB for my task”, even though it’s only needed at peak
• Each node has some allocated but unutilized capacity. Use such capacity to run opportunistic tasks
• Preempt such tasks when needed
Container Overcommit (YARN-1011)
Un-allocated
Allocated
Utilized
Allocated-
but-unutilized
Node1
Guaranteed
Containers
Node1
Opportunistic
Containers
Preemption
When necessary

38. 45 © Hortonworks Inc. 2011–2018. All rights reserved
• “Take me to a node with JDK 10”
• Node Partition vs. Node Attribute
• Partition:
• One partition for one node
• ACL
• Shares between queues
• Preemption enforced.
• Attribute:
• For container placement
• No ACL/Shares on attributes
• First-come-first-serve
Node Attributes (YARN-3409) Partition - 1 (15 nodes)
Queue-A (40%)
Queue-B (60%)
Partition - 2 (15 nodes)
A (25%)
B (25%)
C (25%)
D (25%)
Node 1
os.type=ubuntu
os.version=14.10
glibc.version=2.20
JDK.version=8u20
Node 2
os.type=RHEL
os.version=5.1
GPU.type=x86_64
JDK.version=7u20
Node 16
os.type=windows
os.version=7
JDK.version=8u20
Node 17
os.type=SUSE
os.version=12
GPU.type=i686
JDK.version=7u20

39. 48 © Hortonworks Inc. 2011–2018. All rights reserved
Questions?