An approach to scheduling batch loads across multiple globally distributed datacenters using Mesos. Includes maths ;)

1. Aaron Carey
Production Engineer - ILM London
acarey@ilm.com

2. Federated
Clusters in
Mesos

3. Why?

4. Who wins?

5. Why?
Sites in 3 time zones
Need to share render resources
Went through a project to prepare for cloud burst rendering
Renders mostly come at night (mostly)
What happens when our farm is full?
Can we burst to our other locations?

6. Approaches

7. Huawei Design
Led by the master and gossip protocol
Includes policy model
Master decides if a framework gets an offer
Master is in control
Based on two master plugins, consul deployment, gossip protocol
https://www.youtube.com/watch?v=kqyVQzwwD5E
http://www.slideshare.net/mKrishnaKumar1/federated-mesos-clusters-for-global-
data-center-designs

8. Our hack design
Needs to be simple
Decisions made in the framework
Framework connects to all masters
Masters don’t care about each other
We don’t need a policy engine
Keep code out of the Master

9. Diversion...

10. A note on scheduling...
Historically, schedulers in VFX are tyrannical micro managers
Full knowledge of the whole cluster and all tasks allow better informed decisions
In Mesos you only know what the Master tells you
No knowledge of other frameworks
At the mercy of the Master
Offers only deal in the present
We could hoard all offers we get, but we want to play nice
We don’t know if a better offer is just around the corner

11. Making dynamic scheduling decisions...
Can we intelligently schedule tasks without knowing the whole cluster state?

12. Schedule penalty
Every datacentre has a penalty for scheduling a task
Golf rules
Penalty = Interactivity Penalty + Data Penalty + Utilisation Penalty

13. Interactive Penalty
Framework regularly checks current latency to connected datacentres
Lo = maximum latency for interactive applications (around 35ms)
Lm = latency for datacentre m
I = 0 for non interactive, 1 for interactive

14. Data Penalty
Total Input Data Required - Input Data Already at Location
Bandwidth

15. Utilisation Penalty
Framework checks current utilisation of datacentres
Utarget = target utilisation of datacentre (e.g. 95%)
Um = utilisation of datacentre m

16. Time Penalty
Optional
Penalty decreases based on length of time in the queue

17. Putting it together
Set a cost threshold above which jobs don’t run
Tasks will get dispatched to the datacentre with the lowest cost
Thresholding can ensure jobs wait for optimum resources without consuming all
offers

18. Where were we?

19. Framework

20. System

21. What’s Next?

22. Peer to Peer vs Hierarchical

23. Get involved!
Proposal for federated clusters:
https://docs.google.com/document/d/1U4IY_ObAXUPhtTa-
0Rw_5zQxHDRnJFe5uFNOQ0VUcLg/edit?usp=sharing
Federated Marathon:
https://github.com/schibsted/triathlon
Current Discussion (favouring hierarchical design):
user@mesos.apache.org

24. We’re Hiring
londonrecruitment@ilm.com