Big Data Day LA 2015 - Introducing Myriad, a Mesos framework for dynamically scaling Hadoop workloads by Will Ochandarena of MapR
•
3 likes•1,101 views
"Myriad is a Mesos framework for dynamically scaling Hadoop workloads. Myriad, produced by collaboration between eBay, MapR, and Mesosphere, allows YARN and Mesos to work harmoniously. If you are looking to deploy Mesos in your data centre you are likely to find that Hadoop and YARN are already there. Mesos and YARN are both resource managers, each with a unique architecture, protocol and ecosystem around them. Running these two alongside statically partitions the datacenter, something that both managers are designed to avoid, when they are the only resource manager in control. Myriad helps to avoid the need for static partitioning. "
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (15)
More from Data Con LA
More from Data Con LA (20)
Recently uploaded
Recently uploaded (20)
Big Data Day LA 2015 - Introducing Myriad, a Mesos framework for dynamically scaling Hadoop workloads by Will Ochandarena of MapR
- 1. © 2015 MapR Technologies 1© 2014 MapR Technologies
- 2. © 2015 MapR Technologies 2 Agenda • Challenges of the Modern Data Center • Schedulers – Mesos, YARN, and more • Introducing Apache Myriad
- 3. © 2015 MapR Technologies 3 What’s in your data center??? • Your applications – Tier 1 – Tier 2 – CI/Build • Primary/secondary storage • Some databases – relational, NoSQL, and more • Messaging • High-priority analytics • Low-priority best-effort analytics
- 4. © 2015 MapR Technologies 4 The utilization problem… Application Servers Hadoop Servers Utilizatiion long lived excess capacity situations • Wouldn’t it be nice if you could? • “Scale up” Hadoop during long periods of low utilization • “Scale down” Hadoop ahead of anticipated high utilization
- 5. © 2015 MapR Technologies 5 What about multi-tenancy? Tenant #1 Tenant #2 Tenant #3 Cluster #1 Cluster #2 Cluster #3Cluster Tenant #1 Tenant #2 Tenant #3 Pros: ● No data movement ● Selective data sharing ● Single cluster to manage ● Full use of capacity Cons: ● Hadoop-only server infrastructure ● Not all Hadoop components fully multi-tenant ● IT defines Hadoop components to offer Pros: ● Discrete, custom per-tenant Hadoop clusters ● Sharing of infra between Hadoop and other applications Cons: ● Virtualization impacts performance ● Lots of clusters to manage ● No shared data – additional duplication & movement ● Difficult to do short term “borrowing” of capacity
- 6. © 2015 MapR Technologies 6 Schedulers to the rescue! A wish list - • Applications request resources when they need them – Without user intervention • Custom scheduling algorithms – – Some apps want resources ASAP – Others want specific resources and are willing to wait • Multi-tenancy with strong isolation • Efficient use of resources with preemption & oversubscription
- 7. © 2015 MapR Technologies 7 Great. But which one? YARN Approach Multi-level Scheduling Application decides what’s best Single-level Scheduling Scheduler decides what’s best Ideal For Long-lived and short-lived apps Short-lived, task-based jobs Ecosystem
- 8. © 2015 MapR Technologies 8 Mesos Architecture Mesos Master Mesos Master Mesos Master Myriad Framework Marathon Framework Mesos Slave Mesos Slave Myriad Executor Mesos Executor Mesos Executor Docker Executor Task ./ruby XYZ java –jar XYZ.jar ./xyz Tas k Zookeeper Quorum
- 9. © 2015 MapR Technologies 9 YARN Architecture
- 10. © 2015 MapR Technologies 10 Life with Both Data Created Here Data Processed Here
- 11. © 2015 MapR Technologies 11© 2014 MapR Technologies Apache Myriad (Incubating) Enables Mesos & YARN to co-exist on same physical data center infrastructure.
- 12. © 2015 MapR Technologies 12 How it works • Mesos creates virtual clusters • YARN uses Mesos resources • YARN can release resources • Or get more • Myriad manages conversation between RM and Mesos master – between NM and Mesos too Mesos YARN cluster Web Servers YARN cluster
- 13. © 2015 MapR Technologies 13 Without using more than it needs Mesos Master Myriad RM Myriad NM YARN Task Myriad NM YARN Task Myriad NM YARN Task Mesos Slave Mesos SlaveMesos Slave Job
- 14. © 2015 MapR Technologies 14 What about my storage???
- 15. © 2015 MapR Technologies 15 Add some persistent, shared storage • FS and DB as a service – Outside the scheduler • Rapid clusters-on-demand – Hadoop or other • Stop cluster, data persists • Cluster restart doesn’t need to copy data in • Share data across clusters Mesos Shared Data Services (FS, DB, …)
- 16. © 2015 MapR Technologies 16 Putting it all together Physical Machines Distributed Applications Data Center Operating Services Data Services
- 17. © 2015 MapR Technologies 17 The Future • Incubator – Proposal at http://wiki.apache.org/incubator/MyriadProposal – Initial team from Mesosphere, Paypal, MapR • Community building – Diversity is good already – Starting with very lean team
- 18. © 2015 MapR Technologies 18 Myriad Project • Blog “Project Myriad: No Hadoop is an Island” http://bit.ly/myriad- mapr-blog • Proposal to be an incubator project of the Apache Foundation submitted 12 February 2015 http://bit.ly/myriad-asf-proposal • Initial code on github: http://bit.ly/github-myriad • Join us! Twitter for Myriad community @ApacheMyriad [actual logo coming soon]
- 19. © 2015 MapR Technologies 19© 2014 MapR Technologies Thank You
- 20. © 2015 MapR Technologies 20 Myriad Services Architecture Node ManagerResource Manager Executor Mesos Scheduler Mesos Container Container App YARN Scheduler (fairshare) Offers Launch Tasks Launch Tasks Task Status Launch containers via HB Submit Map<Node, Capacity>