Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

mesos-devoxx14

Intro to Mesos

mesos-devoxx14

  1. 1. Kernel for Distributed Applications Sam BESSALAH @samklr ROOM 9 #devoxx #DV14 #mesos #docker @samklr
  2. 2. #Devoxx #DV14 #mesos #docker @samklr
  3. 3. Example of a statically partionned cluster #Devoxx #DV14 #mesos #docker @samklr
  4. 4. Akka/Play Jenkins Spark/Hadoop Storm Redis #Devoxx #DV14 #mesos #docker @samklr
  5. 5. Issues with static partitionning - Increased Complexity : Machines/vm sprawl Software failures gets harder to handle as machines number grow - Huge overheard, on development, - Planning for failure and maintenance gets harder - Not really developer friendly. Long cycles to deployments. #Devoxx #DV14 #mesos #docker @samklr
  6. 6. Resources utilisation under static partitionning Play/Akka Redis Hadoop #Devoxx #DV14 #mesos #docker @samklr
  7. 7. Resources utilisation under static partitionning Play/Akka Redis #Devoxx #DV14 #mesos #docker @samklr Hadoop Put together, we would get
  8. 8. Enters Google stage left ... Google “Borg” .Unofficial name. Moved to a new system called Omega Bundles all the cluster’s machines into a single pooled view of resources. Evolved to Omega. #Devoxx #DV14 #mesos #docker @samklr
  9. 9. #Devoxx #DV14 #mesos #docker @samklr
  10. 10. Mesos : Level of indirection 1. Run multiple types of distributed systems on the same cluster of machines, and dynamically share resources more eficiently. 2. Provide failure detection, task distribution, task tracking, task monitoring, low level resources management, and fine grained resources sharing. And scale to the thousands of nodes. #Devoxx #DV14 #mesos #docker @samklr
  11. 11. #Devoxx #DV14 #mesos #docker @samklr
  12. 12. #Devoxx #DV14 #mesos #docker @samklr Quick View.
  13. 13. Mesos Architecture #Devoxx #DV14 #mesos #docker @samklr
  14. 14. http://mesos.apache.org/documentation/latest/mesos-architecture/ #Devoxx #DV14 #mesos #docker @samklr
  15. 15. #Devoxx #DV14 #mesos #docker @samklr Mesos Framework A framework is an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : gets information from the framework like environmental variables,configuration information and run tasks inside the mesos slaves.
  16. 16. Mesos Framework A framework is an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : run tasks in mesos slaves. Mesos provides an API to build its own frameworks. And there are SDKs in mulitple languages. #Devoxx #DV14 #mesos #docker @samklr
  17. 17. #Devoxx #DV14 #mesos #docker @samklr
  18. 18. Resources allocation process - Mesos allocate resources via “resources offers” - Resources are a snapshot of available resources that a scheduler will use to run tasks on mesos slaves. #Devoxx #DV14 #mesos #docker @samklr
  19. 19. #Devoxx #DV14 #mesos #docker @samklr
  20. 20. #Devoxx #DV14 #mesos #docker @samklr
  21. 21. #Devoxx #DV14 #mesos #docker @samklr
  22. 22. 1- resourceOffers() #Devoxx #DV14 #mesos #docker @samklr
  23. 23. 1- resourceOffers() 2. launchTasks() #Devoxx #DV14 #mesos #docker @samklr
  24. 24. 1- resourceOffers() 2. launchTasks() 3. lauchTask() #Devoxx #DV14 #mesos #docker @samklr
  25. 25. 1- resourceOffers() 2. launchTasks() 3. lauchTask() 4. statusUpdate() #Devoxx #DV14 #mesos #docker @samklr
  26. 26. 1- resourceOffers() 2. launchTasks() 5. statusUpdate() 3. lauchTask() 4. statusUpdate() #Devoxx #DV14 #mesos #docker @samklr
  27. 27. #Devoxx #DV14 #mesos #docker @samklr Mesos slaves
  28. 28. #Devoxx #DV14 #mesos #docker @samklr Mesos slaves Linux Containers
  29. 29. Mesos uses Linux Containers for resource isolation. Mesos containers are built around cgroups , to provide isolation of CPU, memory, disk … and namespaces for network isolation. Started leveraging containerirsation since 2010 #Devoxx #DV14 #mesos #docker @samklr
  30. 30. #Devoxx #DV14 #mesos #docker @samklr
  31. 31. #Devoxx #DV14 #mesos #docker @samklr Linux Containers
  32. 32. Linux Containers #Devoxx #DV14 #mesos #docker @samklr LXC
  33. 33. Linux Containers #Devoxx #DV14 #mesos #docker @samklr LXC Cgroups
  34. 34. #Devoxx #DV14 #mesos #docker @samklr
  35. 35. Mesos provides a native support for Docker #Devoxx #DV14 #mesos #docker @samklr Containers since 0.20
  36. 36. #Devoxx #DV14 #mesos #docker @samklr
  37. 37. Fault recovery & HA - Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus #Devoxx #DV14 #mesos #docker @samklr
  38. 38. Fault recovery & HA - Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus - Mesos has a slave recovery mechanism. So user’s tasks keep running whenever a slave proces goes under. #Devoxx #DV14 #mesos #docker @samklr
  39. 39. Fault recovery & HA - Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus - Mesos has a slave recovery mechanism. So user’s tasks keep running whenever a slave proces goes under. Slaves checkpoint information to their local disks (task info, status updates, etc ), they recover checkpointed data and reconnect with old tasks when they restart. #Devoxx #DV14 #mesos #docker @samklr
  40. 40. #Devoxx #DV14 #mesos #docker @samklr
  41. 41. Mesos Framework A framework is an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : run tasks in mesos slaves. Mesos provides an API to build its own frameworks. And there are SDKs in mulitple languages. #Devoxx #DV14 #mesos #docker @samklr
  42. 42. #Devoxx #DV14 #mesos #docker @samklr Marathon
  43. 43. What is Marathon ? - A generic mesos framework to run long running services (web apps, etc …) - A distributed Init.d for the cluster. - Runs any linux binary without modification (Tomcat, Play, …) - Cluster wide process supervisor -Private PaaS - Service discovery - Provide a self service Rest API for deployment - Authentification & SSL - Placement constrainst (nodes, racks, etc. (Like with Spark) ) - Service discovery and load balancing via HAProxy #Devoxx #DV14 #mesos #docker @samklr
  44. 44. Marathon workflow #Devoxx #DV14 #mesos #docker @samklr
  45. 45. Marathon workflow #Devoxx #DV14 #mesos #docker @samklr
  46. 46. POST /v2/apps { Marathon Rest APi “id” : “IPython Notebook” “container “ : { “image “ : “docker://ipython/notebook” “options” : [ . . . ] } } #Devoxx #DV14 #mesos #docker @samklr
  47. 47. Marathon Rest APi Wants to scale by addind more instances .. PATCH /v2/apps { “instances” : 4 } #Devoxx #DV14 #mesos #docker @samklr
  48. 48. #Devoxx #DV14 #mesos #docker @samklr Conclusion
  49. 49. #Devoxx #DV14 #mesos #docker @samklr Conclusion
  50. 50. #Devoxx #DV14 #mesos #docker @samklr Conclusion Aurora
  51. 51. #Devoxx #DV14 #mesos #docker @samklr Conclusion Kubernetes
  52. 52. #Devoxx #DV14 #mesos #docker @samklr
  53. 53. #Devoxx #DV14 #mesos #docker @samklr
  54. 54. #Devoxx #DV14 #mesos #docker @samklr
  55. 55. speakerdeck.com/samklr #Devoxx #DV14 #mesos #docker @samklr

×