Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Enabling Microservice @ Orbitz - GOTO Chicago 2016

298 vues

Publié le

In this talk we will discuss how we enabled decomposition of one of our 500+ system components into a continuously deployed microservice cluster. Our platform is comprised of Apache Mesos/Marathon, Docker, and a number of local services including Consul for service discovery, Logstash for diskless logging, and a custom metrics forwarder to Graphite. Building on this, we'll detail our CI pipeline using Jenkins workflows to build and publish microservices as Docker images, test and deploy via Marathon/Mesos, and automated change tickets. Finally, we'll discuss lessons learned from building our own enterprise PaaS and scaling it out to a large organization.

Publié dans : Technologie
  • Soyez le premier à commenter

Enabling Microservice @ Orbitz - GOTO Chicago 2016

  1. 1. Enabling Microservices
 @ Orbitz Steve Hoffman Senior Principal Engineer @bacoboy Rick Fast Principal Engineer @tortiepoint
  2. 2. #GOTOChgo 2016
  3. 3. #GOTOChgo 2016 Agenda • Brief Architecture Overview/History • From Monolithic to Services to Microservices with Docker • Automated Pipelines • Questions
  4. 4. #GOTOChgo 2016
  5. 5. #GOTOChgo 2016 2000
  6. 6. #GOTOChgo 2016 2000 Web Layer Business Layer
  7. 7. #GOTOChgo 2016 2000
  8. 8. #GOTOChgo 2016 2000
  9. 9. #GOTOChgo 2016 2000 Business Layer
  10. 10. #GOTOChgo 2016 2000 Business Layer
  11. 11. #GOTOChgo 2016 Business Layer 2000
  12. 12. #GOTOChgo 2016 Business Layer 2003
  13. 13. #GOTOChgo 2016 Business Layer 2003
  14. 14. #GOTOChgo 2016 Business Layer 2003
  15. 15. #GOTOChgo 2016 2003
  16. 16. #GOTOChgo 2016 2004
  17. 17. #GOTOChgo 2016 2004
  18. 18. #GOTOChgo 2016 2004
  19. 19. #GOTOChgo 2016 2012
  20. 20. #GOTOChgo 2016 2012
  21. 21. #GOTOChgo 2016 2012 • Multiple Brands • Websites • Webservices • Multiple Backends • 500+ apps / thousands of instances • Deployments Daily (sometimes more)
  22. 22. #GOTOChgo 2016 2015 • Multiple Brands • Websites • Webservices • Multiple Backends • 500+ apps / thousands of instances • Deployments Daily (sometimes more)
  23. 23. #GOTOChgo 2016 • Multiple Brands • Websites • Webservices • Multiple Backends • 500+ apps / thousands of instances • Deployments Daily (sometimes more) 2015
  24. 24. #GOTOChgo 2016 2016 • Multiple Brands • Websites • Webservices • Multiple Backends • 500+ apps / thousands of instances • Deployments Daily (sometimes more)
  25. 25. #GOTOChgo 2016 Process Overkill
  26. 26. #GOTOChgo 2016 Different Provisioning Tools DEV OPS
  27. 27. #GOTOChgo 2016 Different Provisioning Tools DEV OPS Application Platform!=
  28. 28. #GOTOChgo 2016 A New Experiment • Microservices? • Decompose single “service” into the actual 40+ sub-services • Any change to sub-service was a deployment of many • Could it be a simple Spring Boot App in Docker (12 factor?) • Backward compatible with existing service infrastructure. • Code to Production w/o help from other Humans
  29. 29. #GOTOChgo 2016 The Docker Slave aka “The Rickbot”
  30. 30. #GOTOChgo 2016 The Docker Slave aka “The Rickbot” App App App
  31. 31. #GOTOChgo 2016 App App App
  32. 32. #GOTOChgo 2016 App App App
  33. 33. #GOTOChgo 2016 App App App Register & Lookup
  34. 34. #GOTOChgo 2016 10.10.10.10 10.10.10.10:31002 App App App Register & Lookup
  35. 35. #GOTOChgo 2016 App App App
  36. 36. #GOTOChgo 2016 App App App
  37. 37. #GOTOChgo 2016 10.10.10.10 App App App
  38. 38. #GOTOChgo 2016 10.10.10.10 10.10.10.10:51515 App App App
  39. 39. #GOTOChgo 2016 App App App
  40. 40. #GOTOChgo 2016 App App App
  41. 41. #GOTOChgo 2016 App App App
  42. 42. #GOTOChgo 2016 10.10.10.10 App App App
  43. 43. #GOTOChgo 2016 10.10.10.10 10.10.10.10:1337 App App App
  44. 44. #GOTOChgo 2016 Almost done… App App App
  45. 45. #GOTOChgo 2016 Mesos Scheduler http://mesos.apache.org/
  46. 46. #GOTOChgo 2016 Mesos Scheduler Marathon Scheduler Slave xNMaster x3 ZK x3
  47. 47. #GOTOChgo 2016 App App App
  48. 48. #GOTOChgo 2016 App App App
  49. 49. #GOTOChgo 2016 App App App
  50. 50. #GOTOChgo 2016 App App App
  51. 51. #GOTOChgo 2016 App App App VIP
  52. 52. #GOTOChgo 2016 App App App VIP
  53. 53. #GOTOChgo 2016 App App App VIP
  54. 54. #GOTOChgo 2016 App App App
  55. 55. #GOTOChgo 2016 App App App
  56. 56. #GOTOChgo 2016 App App App
  57. 57. #GOTOChgo 2016 App App App
  58. 58. #GOTOChgo 2016 A Little More Background…
  59. 59. #GOTOChgo 2016
  60. 60. #GOTOChgo 2016 Module Module Module Module
  61. 61. #GOTOChgo 2016 Editorial Module A Continuously Deployed Microservice www.orbitz.com (scroll down)
  62. 62. #GOTOChgo 2016 Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module
  63. 63. #GOTOChgo 2016 Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module Hotel Team Search Team Content Team
  64. 64. #GOTOChgo 2016 Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module
  65. 65. #GOTOChgo 2016 Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module Hotel Team Search Team Content Team
  66. 66. #GOTOChgo 2016 Pre-Continuous Delivery 1.2-BETA-20150401-113002 Test Discard Test Discard 1.2-BETA-20150402-093002 Build Build And so on…
  67. 67. #GOTOChgo 2016 Pre-Continuous Delivery 1.2-BETA-20150401-113002 Test Discard Test Discard 1.2-BETA-20150402-093002 Build Build Test Deploy 1.2 Build
  68. 68. #GOTOChgo 2016 Continuous Delivery with Jenkins, Docker, Ansible and Marathon
  69. 69. #GOTOChgo 2016 Yo rfast-mbp:git rfast$ yo microservice _-----_ | | .--------------------------. |--(o)--| | Welcome to the kickass | `---------´ | Microservice | ( _´U`_ ) | generator! | /___A___ '--------------------------' | ~ | __'.___.'__ ´ ` |° ´ Y ` ? Enter the name of your service. (E.G. "my-service")
  70. 70. #GOTOChgo 2016 Committer Pull Request
  71. 71. #GOTOChgo 2016 Committer Reviewer Pull Request
  72. 72. #GOTOChgo 2016 Committer Reviewer Pull Request
  73. 73. #GOTOChgo 2016 Committer Reviewer Pull Request
  74. 74. #GOTOChgo 2016 Jenkins Pipeline
  75. 75. #GOTOChgo 2016 if(…) { x ++; } Merge
  76. 76. #GOTOChgo 2016 Pipeline (Simplified) build unit test publish deploy dev acceptance tests deploy qa deploy staging open RFC deploy prod close RFC
  77. 77. #GOTOChgo 2016 Build if(…) { x ++; } 1.2 Merge
  78. 78. #GOTOChgo 2016 Build if(…) { x ++; } 1.2 Merge 1.2. editorial-module.jar ./gradlew build {BUILD_NUMBER}
  79. 79. #GOTOChgo 2016 Build 1.2 Merge editorial-module.jar if(…) { x ++; } 171.2.
  80. 80. #GOTOChgo 2016 Build 1.2 Merge editorial-module.jar FROM orbitz/java-8 COPY build/editorial-module.jar /opt/orbitz WORKDIR /opt/orbitz ENTRYPOINT [“java”,”-jar”,”editorial-module.jar”] if(…) { x ++; } 171.2.
  81. 81. #GOTOChgo 2016 Build 1.2 Merge editorial-module.jar orbitz/editorial-module:1.2.17 if(…) { x ++; } 171.2.
  82. 82. #GOTOChgo 2016 Build orbitz/editorial-module:1.2.17
  83. 83. #GOTOChgo 2016 Deploy - hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image docker: {{previous}} git pull playbook.yml (abridged)
  84. 84. #GOTOChgo 2016 - hosts: dev host-001 host-002 host-003 host-004
  85. 85. #GOTOChgo 2016 - hosts: dev host-001 host-002 host-003 host-004
  86. 86. #GOTOChgo 2016 - hosts: dev host-001 host-002 host-003 host-004
  87. 87. #GOTOChgo 2016 Downsides • How to handle failure with YAML files? • What happens when a VM is moved (destroyed/created)? • What if I need to add capacity?
  88. 88. #GOTOChgo 2016 Deploy (Second Attempt) - hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image docker: {{previous}} git pull playbook.yml (abridged)
  89. 89. #GOTOChgo 2016 Deploy (Second Attempt) - hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image git pull playbook.yml (abridged) - hosts: localhost - tasks: - name: marathon deploy marathon: image=orbitz/{{application}}:{{version}} instances=3
  90. 90. #GOTOChgo 2016 Deploy (Second Attempt) git pull playbook.yml (abridged) - hosts: localhost - tasks: - name: marathon deploy marathon: image=orbitz/{{application}}:{{version}} instances=3
  91. 91. #GOTOChgo 2016 Deploy (Second Attempt) New host setup
  92. 92. #GOTOChgo 2016 Deploy (Second Attempt) Mesos Agent New host setup 1.2.16
  93. 93. #GOTOChgo 2016 - tasks: marathon: … playbook.yml 1.2.16 1.2.16 1.2.16
  94. 94. #GOTOChgo 2016 - tasks: marathon: … 1.2.16 1.2.16 1.2.16
  95. 95. #GOTOChgo 2016 - tasks: marathon: … 1.2.16 1.2.16 1.2.16
  96. 96. #GOTOChgo 2016 - tasks: marathon: … 1.2.16 1.2.16 1.2.16 PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … }
  97. 97. #GOTOChgo 2016 PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … } - tasks: marathon: … 1.2.16 1.2.16 1.2.16 app = GET /v2/apps/editorial-module if not app then deploy_id = POST /v2/apps { “image”: “orbitz/editorial-module:1.2.17”, “id”: “editorial-module” } else deploy_id = PUT /v2/apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” } end if while GET /v2/deployments contains deploy_id // still deploying end // deploy complete
  98. 98. #GOTOChgo 2016 - tasks: marathon: … 1.2.16 1.2.16 1.2.16 “env”: { “SOME_DATABASE_URL”: “jdbc://whatever”, “SOME_DATABASE_USER”: “rickfast”, “SOME_OTHER_VALUE”: “whatever” } PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … }
  99. 99. #GOTOChgo 2016 - tasks: marathon: … PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … } 1.2.16 1.2.16 1.2.16
  100. 100. #GOTOChgo 2016 - tasks: marathon: … 1.2.16 1.2.16 1.2.16
  101. 101. #GOTOChgo 2016 - tasks: marathon: … 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17
  102. 102. #GOTOChgo 2016 - tasks: marathon: … /health 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17
  103. 103. #GOTOChgo 2016 - tasks: marathon: … /health 200 OK 200 OK 200 OK 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17
  104. 104. #GOTOChgo 2016 - tasks: marathon: … 1.2.17 1.2.17 1.2.17
  105. 105. #GOTOChgo 2016 And off to the next environment… 1.2.17 1.2.17 1.2.17
  106. 106. #GOTOChgo 2016 1.2.17 1.2.17 1.2.17 What if?
  107. 107. #GOTOChgo 2016 1.2.17 1.2.17
  108. 108. #GOTOChgo 2016 1.2.17 1.2.17
  109. 109. #GOTOChgo 2016 1.2.17 1.2.17 1.2.17
  110. 110. #GOTOChgo 2016 /health 200 OK 1.2.17 1.2.17 1.2.17
  111. 111. #GOTOChgo 2016 1.2.17 1.2.17 1.2.17
  112. 112. #GOTOChgo 2016 Smoke/Acceptance Testing 1.2.17 1.2.17 1.2.17
  113. 113. #GOTOChgo 2016 Paper Trail create tickets commit(s)
  114. 114. #GOTOChgo 2016 Paper Trail fail! close/fail
  115. 115. #GOTOChgo 2016 Paper Trail ok close
  116. 116. #GOTOChgo 2016 Build Unit Test Deploy Dev Deploy Prod Deploy Staging Acceptance Test Code Review & Push Production Pre-Production Open RFC Close RFC
  117. 117. #GOTOChgo 2016 Pipeline as code! node { git ‘https://stash.orbitz.com/editorial-module.git‘ sh ‘./gradlew clean build’ def image = docker.build “editorial-module:${env.BUILD_TAG}" image() }
  118. 118. #GOTOChgo 2016 Pipeline as code! node { git ‘http://stash.orbitz.com/deploy-playbook.git' sh ‘./ansible-playbook … editorial-module:${env.BUILD_TAG} test’ } node { git ‘https://stash.orbitz.com/editorial-module.git‘ sh ‘./gradlew clean build’ def image = docker.build “editorial-module:${env.BUILD_TAG}" image.push() }
  119. 119. #GOTOChgo 2016 Pipeline as code! pipeline.json { “steps”: [ { “type”: “gradle-build” }, { “type”: “marathon-deploy”, “environment”: “test” } ] }
  120. 120. #GOTOChgo 2016 Pipeline as code! def pipeline = new JsonSluper().parse(‘pipeline.json’) pipeline.steps.each { step -> if (step.type == ‘gradle-build’) { node(‘java’) { git ‘https://stash.orbitz.com/${service}.git’ sh ‘./gradlew clean build’ def image = docker.build “${service}:${env.BUILD_TAG}” image.push() } } … }
  121. 121. #GOTOChgo 2016 Jenkins CI/CD - on Mesos? • Existing Solution • Dedicated Jenkins Slaves • Hand created • Snapshotted & Rolled Back to “Clean” state after each Job • Hard to Manage Build Environment for 300+ apps across many OSes, Java versions, Ruby versions, perl versions, python versions, protocol buffer compiler versions, etc…
  122. 122. #GOTOChgo 2016 Master Slave Commit/Push or Pull Request/Merge Before
  123. 123. #GOTOChgo 2016 Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Clone & Build Push Artifacts Before
  124. 124. #GOTOChgo 2016 Master Slave Before
  125. 125. #GOTOChgo 2016 Jenkins mesos-plugin
  126. 126. #GOTOChgo 2016 Master Master Commit/Push or Pull Request/Merge After Slave env:jenkins
  127. 127. #GOTOChgo 2016 Master Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Clone & Build Push Artifacts After Slave env:jenkins
  128. 128. #GOTOChgo 2016 Master Master After Slave env:jenkins
  129. 129. #GOTOChgo 2016 Master Master Slave Slave env:jenkins env:ci
  130. 130. #GOTOChgo 2016 Master Master Slave Slave Deploy App Clone & Run Slave env:jenkins env:ci docker
 pull
  131. 131. #GOTOChgo 2016 Master Master Slave App Slave env:jenkins env:ci
  132. 132. #GOTOChgo 2016 Prod Network Pre-Prod Network env:ci env:qa env:staging env:jenkins Master env:prod
  133. 133. #GOTOChgo 2016 DC: East DC: West env:ci
 dc:west env:qa
 dc:west env:staging
 dc:west env:jenkins
 dc:west Master env:prod
 dc:east env:prod
 dc:west
  134. 134. #GOTOChgo 2016 Learnings • Create a shared platform for docker deployments using shared and app-specific “localhost” helpers — this was ours, yours SHOULD look different — adapt to change, don’t fight it. • Take people out of the release process • Docker & Packer AMIs - repeatable applications/servers • Chef - repeatable infrastructure & configuration — Environment aware • Jenkins - repeatable releases • Delineate configuration concerns: • Known at Compile time — Bake into Docker image • Known at Boot time — Pass from Playbook/Launcher/Jenkins - parameter to Docker run/AMI • Changes Anytime — Externalize (consul K/V, etcd, zookeeper)
  135. 135. #GOTOChgo 2016

×