Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
© Cloudera, Inc. All rights reserved.
NEXT GENERATION SCHEDULING for YARN & K8s:
For Hybrid Cloud/On-prem Environment to r...
© Cloudera, Inc. All rights reserved. 2
SPEAKER
SUNIL GOVINDAN
Engineering Manager @Cloudera
@sunilgovind
Apache Hadoop PM...
© Cloudera, Inc. All rights reserved. 3
AGENDA
Journey through Big Data Experience
Where are we today on scheduling experi...
© Cloudera, Inc. All rights reserved. 4
BATCH
WORKLOADS
DEEP LEARNING APPS
CUSTOMER JOURNEY - BIG DATA ECOSYSTEM
BIG DATA ...
© Cloudera, Inc. All rights reserved. 5
BIG DATA ECOSYSTEM
NEW Trends
Big Data &
Containerization
• Portability for apps
•...
© Cloudera, Inc. All rights reserved. 6
AGENDA
Journey through Big Data Experience
Where are we today on scheduling experi...
© Cloudera, Inc. All rights reserved. 7
WHERE ARE WE TODAY ?
RESOURCE ORCHESTRATOR PERSPECTIVE - STRENGTHS
APACHE YARN
Big...
© Cloudera, Inc. All rights reserved. 8
WHERE ARE WE TODAY ?
QUICK RECAP TO THE NEW TRENDS
Moving to Cloud
Big Data &
Cont...
© Cloudera, Inc. All rights reserved. 9
WHERE ARE WE TODAY ?
CHALLENGES - APACHE YARN
APACHE YARN WORLD
• Challenges in ma...
© Cloudera, Inc. All rights reserved. 10
WHERE ARE WE TODAY ?
CHALLENGES - KUBERNETES
KUBERNETES WORLD
• Challenges in run...
© Cloudera, Inc. All rights reserved. 11
“WHERE IS MY SILVER BULLET
TO SOLVE ALL THESE
CHALLENGES? ”
© Cloudera, Inc. All rights reserved. 12
“SORRY, THERE ISN’T ANY”
© Cloudera, Inc. All rights reserved. 13
SO HOW TO SOLVE THIS ?
Assessment
Native Big Data Apps — Moving
batch workloads (...
© Cloudera, Inc. All rights reserved. 14
AGENDA
Journey through Big Data Experience
Where are we today on scheduling exper...
© Cloudera, Inc. All rights reserved. 15
“WE NEED TO IMPROVE ON
WHAT THEY ARE NOT GOOD AT
TODAY ”
© Cloudera, Inc. All rights reserved. 16© Cloudera, Inc. All rights reserved.
APACHE YARN WORLD
Strengthening Dynamic
Envi...
© Cloudera, Inc. All rights reserved. 17© Cloudera, Inc. All rights reserved.
KUBERNETES WORLD
Demand for Better support o...
© Cloudera, Inc. All rights reserved. 18
MOTIVATION & EFFORTS TO IMPROVE
Not Optimized — to balance use-
cases like batch ...
© Cloudera, Inc. All rights reserved. 19
AGENDA
Journey through Big Data Experience
Where are we today on scheduling exper...
© Cloudera, Inc. All rights reserved. 20
PROPOSAL
YuniKorn (/ˈyo͞ onəˌkôrn/, Y for YARN, K for K8s, uni- for
Unified)
• A ...
© Cloudera, Inc. All rights reserved. 21
WHAT YuniKorn IS (IS NOT) ?
YuniKorn is
• A better scheduler for the K8s world, f...
© Cloudera, Inc. All rights reserved. 22© Cloudera, Inc. All rights reserved.
YuniKorn - A UNIFIED RESOURCE SCHEDULER
Capa...
© Cloudera, Inc. All rights reserved. 23© Cloudera, Inc. All rights reserved.
CAPACITY PLANNING
Hierarchy of queues
Queues...
© Cloudera, Inc. All rights reserved. 24© Cloudera, Inc. All rights reserved.
RESOURCE SCHEDULING
Resource Fairness
Queue/...
© Cloudera, Inc. All rights reserved. 25© Cloudera, Inc. All rights reserved.
RESOURCE MONITORING
Common dashboard to moni...
© Cloudera, Inc. All rights reserved. 26© Cloudera, Inc. All rights reserved.
APPLICATION MANAGEMENT
Track applications in...
© Cloudera, Inc. All rights reserved. 27© Cloudera, Inc. All rights reserved.
ARCHITECTURE
YuniKorn Core
Scheduler
Shim
Ma...
© Cloudera, Inc. All rights reserved. 28
YuniKorn SCHEDULER vs. OTHERS
Disclosure: this table is summarized based on speak...
© Cloudera, Inc. All rights reserved. 29
AGENDA
Journey through Big Data Experience
Where are we today on scheduling exper...
© Cloudera, Inc. All rights reserved. 30© Cloudera, Inc. All rights reserved.
DEMO
K8shim API Server
YuniKorn Core
etcd
yu...
© Cloudera, Inc. All rights reserved. 31
AGENDA
Journey through Big Data Experience
Where are we today on scheduling exper...
© Cloudera, Inc. All rights reserved. 32© Cloudera, Inc. All rights reserved.
YARN cluster
TAKE AWAY
BEFORE AFTER
Scenario...
© Cloudera, Inc. All rights reserved. 33© Cloudera, Inc. All rights reserved.
OPEN SOURCE
Yes, We are going to Open Source...
© Cloudera, Inc. All rights reserved. 34© Cloudera, Inc. All rights reserved.
ACKNOWLEDGMENTS
A big shout out to the folks...
© Cloudera, Inc. All rights reserved.
THANK YOU
Prochain SlideShare
Chargement dans…5
×

Next Generation Scheduling for YARN and K8s: For Hybrid Cloud/On-prem Environment to run Mixed Workloads

266 vues

Publié le

Scheduler of a container orchestration system, such as YARN and K8s, is a critical component that users rely on to plan resources and manage applications.

And if we assess where we are today, in YARN effectively it had two power schedulers (Fair and Capacity scheduler) and both serve many strong use cases in big data ecosystem. It can scale up to 50k nodes per cluster, and schedule 20k containers per second, and extremely efficient to manage batch workloads.

K8s default scheduler is an industry-proven solution to efficiently manage long-running services. As more big data apps are moving to K8s and cloud world, but many features like hierarchical queues to support multi-tenancy better, fairness resource sharing, and preemption, etc. are either missing or not mature enough at this point of time to support big data apps running on K8s.

At this point, there is no solution that exists to address the needs of having a unified resource scheduling experiences across platforms. That makes it extremely difficult to manage workloads running on different environments, from on-premise to cloud.

Hence evolving a common scheduler powered from YARN and K8s’s legacy capabilities and improving towards cloud use cases will focus more on use cases like:

Better bin-packing scheduling (and gang scheduling)

Autoscale up and shrink policy management

Effectively run batch workloads and services with clear SLA’s

In summary, we are improving core scheduling capabilities to manage both K8s and YARN cluster which is cloud aware as a separate initiative and above-mentioned cases will be the core focus of this initiative. More details of our works will be presented in this talk.

Publié dans : Technologie
  • Soyez le premier à commenter

Next Generation Scheduling for YARN and K8s: For Hybrid Cloud/On-prem Environment to run Mixed Workloads

  1. 1. © Cloudera, Inc. All rights reserved. NEXT GENERATION SCHEDULING for YARN & K8s: For Hybrid Cloud/On-prem Environment to run Mixed Workloads Sunil Govindan & Weiwei Yang
  2. 2. © Cloudera, Inc. All rights reserved. 2 SPEAKER SUNIL GOVINDAN Engineering Manager @Cloudera @sunilgovind Apache Hadoop PMC & Committer WEIWEI YANG Staff Software Engineer @Cloudera @abvclouds Apache Hadoop PMC & Committer
  3. 3. © Cloudera, Inc. All rights reserved. 3 AGENDA Journey through Big Data Experience Where are we today on scheduling experience ? Motivation to improve scheduling Introduction to YuniKorn Demo Future & Open Source Story
  4. 4. © Cloudera, Inc. All rights reserved. 4 BATCH WORKLOADS DEEP LEARNING APPS CUSTOMER JOURNEY - BIG DATA ECOSYSTEM BIG DATA ECOSYSTEM - TODAY PUBLIC CLOUD STORAGE 100100100 101001000 010010101 STORAGECOMPUTE (on-prem/on-cloud) HIVE on LLAP SERVICES
  5. 5. © Cloudera, Inc. All rights reserved. 5 BIG DATA ECOSYSTEM NEW Trends Big Data & Containerization • Portability for apps • Resource Isolation • Dependency Management Moving to Cloud for better elasticity • Budget Control • Infinite Resources • Cloud Burst Decoupled Compute & Storage • Low Cost • Hybrid in nature • Agility • Mixed Workloads
  6. 6. © Cloudera, Inc. All rights reserved. 6 AGENDA Journey through Big Data Experience Where are we today on scheduling experience ? Motivation to improve scheduling Introduction to YuniKorn Demo Future & Open Source Story
  7. 7. © Cloudera, Inc. All rights reserved. 7 WHERE ARE WE TODAY ? RESOURCE ORCHESTRATOR PERSPECTIVE - STRENGTHS APACHE YARN Big Data Ecosystem KUBERNETES Cloud Ecosystem CLOUD NATIVE Public Cloud Big Data—Optimized to run Big Data workloads. Batch Workloads—High Throughput scheduling for batch workloads. SLA—Better SLA for Big Data workloads. Multi tenant—Quota Management. Services—Optimized for containerized microservices. Networking—Strong Network management support. Cloud Aware—Better tuned for cloud use cases. Storage—Persistent volumes Cost—Budget Centric ∞—Infinite resource for infinite $$.
  8. 8. © Cloudera, Inc. All rights reserved. 8 WHERE ARE WE TODAY ? QUICK RECAP TO THE NEW TRENDS Moving to Cloud Big Data & Containerization Mixed Workloads
  9. 9. © Cloudera, Inc. All rights reserved. 9 WHERE ARE WE TODAY ? CHALLENGES - APACHE YARN APACHE YARN WORLD • Challenges in managing THREE YARN schedulers for different use cases • Should Deploy & Manage containerized microservices in the same Hadoop Cluster • Need strong networking and persistent volume support • Need more powerful Auto scaling and budget control
  10. 10. © Cloudera, Inc. All rights reserved. 10 WHERE ARE WE TODAY ? CHALLENGES - KUBERNETES KUBERNETES WORLD • Challenges in running Big Data workloads along with microservices together • Need much better quota management and better SLA • No first class Application management concept for Big Data workloads
  11. 11. © Cloudera, Inc. All rights reserved. 11 “WHERE IS MY SILVER BULLET TO SOLVE ALL THESE CHALLENGES? ”
  12. 12. © Cloudera, Inc. All rights reserved. 12 “SORRY, THERE ISN’T ANY”
  13. 13. © Cloudera, Inc. All rights reserved. 13 SO HOW TO SOLVE THIS ? Assessment Native Big Data Apps — Moving batch workloads (MR, TEZ .. ) from YARN to Kubernetes looking into High throughput, Low Latency and With notion of job etc is costly. Adaptability — Optimized for services and running batch workloads exposes hard-to-bridge gaps such as run few workloads with or without docker Services on YARN — Services and web farms can run on YARN however it is not as feature-rich as Kubernetes. Multiple Schedulers (YARN) — Different schedulers are focussed on specific use cases and not very easy to drive continuous feature enhancements . One cannot replace another — Neither Kubernetes can replace YARN or vice versa in the near future considering some of the fundamental architecture differences. EXPENSIVE Higher cost to achieve the goal
  14. 14. © Cloudera, Inc. All rights reserved. 14 AGENDA Journey through Big Data Experience Where are we today on scheduling experience ? Motivation to improve scheduling Introduction to YuniKorn Demo Future & Open Source Story
  15. 15. © Cloudera, Inc. All rights reserved. 15 “WE NEED TO IMPROVE ON WHAT THEY ARE NOT GOOD AT TODAY ”
  16. 16. © Cloudera, Inc. All rights reserved. 16© Cloudera, Inc. All rights reserved. APACHE YARN WORLD Strengthening Dynamic Environments • Improve YARN to work well for cloud (public & private) • Focus on Autoscaling, Smarter Scheduling etc. Refer: YARN-9548 Improving capabilities for persistent volumes • Added CSI (Container Storage Interface) support • Enhancing CSI implementation to expand and support storages such S3, Ozone etc as mounted volume to YARN containers Native Service enhancements • Improving native services support in YARN • Micro Service upgrades What’s happening now in YARN today ?
  17. 17. © Cloudera, Inc. All rights reserved. 17© Cloudera, Inc. All rights reserved. KUBERNETES WORLD Demand for Better support of batch workloads • Early efforts to support batch scheduling is in progress by K8s community. Efforts on running Spark on K8s • Spark and Kubernetes community is working towards Spark on K8s deployments. • Gaps in running Spark such dynamic resource allocation, security etc is still open. CDP and Kubernetes Cloudera Few CDP microservices will be running on Kubernetes What’s happening now in KUBERNETES today ?
  18. 18. © Cloudera, Inc. All rights reserved. 18 MOTIVATION & EFFORTS TO IMPROVE Not Optimized — to balance use- cases like batch workloads to the needs like running web farms or services with respect to scheduling challenges. Poor Resource Utilization — Not able to effectively utilize complete resources in cluster for services and workloads. FRAGMENTED Multiple YARN & Kubernetes schedulers are UNIFIED RESOURCE SCHEDULER AND APPLICATION MANAGEMENT What we need is an effort to improve both YARN and Kubernetes scheduling worlds. Multiple schedulers power YARN & Kubernetes for different use cases
  19. 19. © Cloudera, Inc. All rights reserved. 19 AGENDA Journey through Big Data Experience Where are we today on scheduling experience ? Motivation to improve scheduling Introduction to YuniKorn Demo Future & Open Source Story
  20. 20. © Cloudera, Inc. All rights reserved. 20 PROPOSAL YuniKorn (/ˈyo͞ onəˌkôrn/, Y for YARN, K for K8s, uni- for Unified) • A common resource scheduler • Platform independent • Enhanced scheduling capabilities
  21. 21. © Cloudera, Inc. All rights reserved. 21 WHAT YuniKorn IS (IS NOT) ? YuniKorn is • A better scheduler for the K8s world, for services and batch workloads • A unified scheduler for the YARN world (FiFo, Fair and Capacity Scheduler) • Providing unified resource scheduling experience across the YARN and K8s (and beyond) • Suitable for both finite resource (datacenter) and infinite resources/dollars (cloud) worlds Is NOT • A system to port YARN applications to run on K8s w/o modification, or vice versa
  22. 22. © Cloudera, Inc. All rights reserved. 22© Cloudera, Inc. All rights reserved. YuniKorn - A UNIFIED RESOURCE SCHEDULER Capacity Planning Capacity Planning Divide cluster resources into resource pools (queues), define capacity range based on needs. Enforce resource quotas and limits. Resource scheduling Resource fairness, preemption, high-throughput, multi-tenant, placement, etc. Application Management A central place to monitor application states Resource Monitoring A unified view of cluster resources, a dashboard to easy track resource usage by queue, user or organization. Resource Scheduling Application Management Resource Monitoring ... Explore the feature set
  23. 23. © Cloudera, Inc. All rights reserved. 23© Cloudera, Inc. All rights reserved. CAPACITY PLANNING Hierarchy of queues Queues can be organized in honor of user groups or organizations, with multiple levels. Elastic Capacity Each queue has its min-max capacity, usage is elastic within this range for multiple users. Resource Quotas Resource cap for queues or users. Limited amount of resources, number of applications etc. Partition A set of instances (nodes) that are physically isolated
  24. 24. © Cloudera, Inc. All rights reserved. 24© Cloudera, Inc. All rights reserved. RESOURCE SCHEDULING Resource Fairness Queue/User/App level fairness ensures each entity gets its own fair share of resources. Priorities Queue priority + App priority Preemption Queue demands for more resources have the chance to preempt resources from other queues for high priority apps. Placement Constraints Affinity/anti-affinity, node constraints etc Services Low Latency Long Running Batch High-throughput Short-lived YuniKorn
  25. 25. © Cloudera, Inc. All rights reserved. 25© Cloudera, Inc. All rights reserved. RESOURCE MONITORING Common dashboard to monitor resources Hierarchy of queues Cluster resources are divided into hierarchy of queues, all queue state is visible Common View A common view of resources, cross platform. Resource Centric Focus on resources, total/available/used, and all Resource Dashboar d
  26. 26. © Cloudera, Inc. All rights reserved. 26© Cloudera, Inc. All rights reserved. APPLICATION MANAGEMENT Track applications in a consistent fashion Application originated GUI to manage workloads (instead of individual pods in K8s). Entire application lifecycle is visible and trackable.
  27. 27. © Cloudera, Inc. All rights reserved. 27© Cloudera, Inc. All rights reserved. ARCHITECTURE YuniKorn Core Scheduler Shim Master Scheduler Interface Api-server etcd kubelets Resource Manager Node Managers GPRC/API Resource requests, new application, node updates GPRC/API Container allocation, preemption Master Node Slave nodes MR Spark Flink Tez MySQL Spark Web Server Kafka Client API Allocate, release container
  28. 28. © Cloudera, Inc. All rights reserved. 28 YuniKorn SCHEDULER vs. OTHERS Disclosure: this table is summarized based on speakers’ analysis Scheduler Capabilities Resource Sharing Resource Fairness Preemption Throughput Gang Scheduling Hierarchy queues Queue prioritoy Queue elastic capacity Cross queue fairness User level fairness App level fairness Basic preemption With fairness With priority Kube-default x x x x x x v x v 100+ allocs/s x Kube-batch x x x x x v v x v ? v YARN CS/FS v v v v v v v v v 4k+ allocs/s x YuniKorn v x (wip) v v v v v v x (wip) ? x (wip) Key capabilities of a resource scheduler from our perspective
  29. 29. © Cloudera, Inc. All rights reserved. 29 AGENDA Journey through Big Data Experience Where are we today on scheduling experience ? Motivation to improve scheduling Introduction to YuniKorn Demo Future & Open Source Story
  30. 30. © Cloudera, Inc. All rights reserved. 30© Cloudera, Inc. All rights reserved. DEMO K8shim API Server YuniKorn Core etcd yunikorn-pod Master Kubelet ● YuniKorn is deployed in ONE pod ● YuniKorn core only talks to K8shim via scheduler-interface, K8shim talks to api-server Let’s explore what YuniKorn can do for KUBERNETES !!! HERE WE GO... YuniKorn UI
  31. 31. © Cloudera, Inc. All rights reserved. 31 AGENDA Journey through Big Data Experience Where are we today on scheduling experience ? Motivation to improve scheduling Introduction to YuniKorn Demo Future & Open Source Story
  32. 32. © Cloudera, Inc. All rights reserved. 32© Cloudera, Inc. All rights reserved. YARN cluster TAKE AWAY BEFORE AFTER Scenario 1 Scenario 2 Scenario 3 K8s cluster on cloud/prem Existing K8s schedulers K8s cluster on cloud/prem YuniKorn scheduler K8s cluster Existing K8s schedulers YARN cluster Capacity Scheduler Fair Scheduler K8s cluster YuniKorn Scheduler YARN cluster Capacity Scheduler Fair Scheduler YARN cluster YuniKorn scheduler
  33. 33. © Cloudera, Inc. All rights reserved. 33© Cloudera, Inc. All rights reserved. OPEN SOURCE Yes, We are going to Open Source soon. STAY TUNED...
  34. 34. © Cloudera, Inc. All rights reserved. 34© Cloudera, Inc. All rights reserved. ACKNOWLEDGMENTS A big shout out to the folks who helped to design, develop and make this project possible. ❏ Vinod Kumar Vavilapalli ❏ Wangda Tan ❏ Wilfred Spiegelenburg ❏ Akhil PB ❏ Suma Shivaprasad ❏ and many others...
  35. 35. © Cloudera, Inc. All rights reserved. THANK YOU

×