Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Airbnb’s Great Migration: Building Services at Scale

119 vues

Publié le

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2DESAwC.

Jessica Tai recaps her QCon SF 2018 “Great Migration” presentation then continues the story with a focus on how Airbnb is building, operating, and scaling its expanding network of services. Though their re-architecture to SOA is still ongoing, they are already seeing various benefits including improved performance, developer productivity, build and deploy times, and site reliability. Filmed at qconlondon.com.

Jessica Tai has worked at Airbnb for 4 years, starting as a full-stack product engineer for the guest and host booking flow and is now an infrastructure engineer on the Core Services team. She leads the user data service, which is one of Airbnb’s highest QPS services and integrates with all business verticals. She is a member of Leadership & Development committee for women in tech at Airbnb.

Publié dans : Technologie
  • Soyez le premier à commenter

Airbnb’s Great Migration: Building Services at Scale

  1. 1. Airbnb’sGreatMigration:
 BuildingServicesatScale JESSICA TAI / MARCH 4, 2019 / QCON LONDON
  2. 2. InfoQ.com: News & Community Site Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ airbnb-services-scalability • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week
  3. 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
  4. 4. @jessicamtai 2014
  5. 5. @jessicamtai 2015+
  6. 6. @jessicamtai Hi,I’mJessica.
 Ipairprogram
 withmycorgi.
  7. 7. Why migrate? Service design tenets Incremental
 comparison Best practices Results Decomposition @jessicamtai
  8. 8. Why migrate? Best practices Results Incremental
 comparison Service design tenets Decomposition @jessicamtai
  9. 9. Why migrate? Best practices Results Incremental
 comparison Service design tenets Decomposition @jessicamtai
  10. 10. Why migrate? Best practices Results Incremental
 comparison Service design tenets Decomposition @jessicamtai
  11. 11. Why migrate? Best practices Results Incremental
 comparison Service design tenets Decomposition @jessicamtai
  12. 12. Why migrate? Best practices Results Incremental
 comparison Service design tenets Decomposition @jessicamtai
  13. 13. Monorail,ourRubyonRailsmonolith @jessicamtai
  14. 14. Easystartwithmonoliths EARLY AIRBNB Client traffic Shared database Monorail Data access query Business logic Presentation view @jessicamtai
  15. 15. @jessicamtai
  16. 16. @jessicamtai
  17. 17. WHY DECIDE TO MIGRATE? @jessicamtai
  18. 18. @jessicamtai
  19. 19. Moreincidents Slowerdeploytrains @jessicamtai
  20. 20. Moreincidents Slowerdeploytrains @jessicamtai
  21. 21. Oursolution:Service-orientedarchitecture(SOA) NETWORK OF LOOSELY-COUPLED SERVICES Client API gateway Service1 Service2 Database2Database3 Service3 @jessicamtai
  22. 22. CheckoutpageinSOA Business travel service Cancellation service Home demand service Pricing service Home service Reservation service Messaging service @jessicamtai
  23. 23. SERVICE DESIGN TENETS @jessicamtai
  24. 24. Servicesownreads&writes
 totheirdata @jessicamtai
  25. 25. Servicesaddressaspecificconcern @jessicamtai
  26. 26. Avoidduplicatefunctionality https://www.flickr.com/photos/popilop/331357312 @jessicamtai
  27. 27. Datamutationspropagate
 viastandardevents @jessicamtai
  28. 28. Buildforproduction @jessicamtai
  29. 29. DECOMPOSE BY
 REQUEST LIFE CYCLE @jessicamtai
  30. 30. Requestlifecycle Client traffic Shared database Monorail Data access query Business logic Presentation view V1: MONORAIL @jessicamtai
  31. 31. Requestlifecycle Monorail API traffic Routing & view Business logic, model, data 
 via services Client traffic V2: MONORAIL & SERVICES @jessicamtai
  32. 32. MIDDLE TIER Shared business logic Servicetypes STRICT FLOW OF DEPENDENCIES PRESENTATION Synthesize DERIVED DATA Shared context, multiple sources DATA Entity read and writes @jessicamtai
  33. 33. MIDDLE TIER Shared business logic PRESENTATION Synthesize DERIVED DATA Shared context, multiple sources DATA Entity read and writes Servicetypes STRICT FLOW OF DEPENDENCIES @jessicamtai
  34. 34. MIDDLE TIER Shared business logic PRESENTATION Synthesize DERIVED DATA Shared context, multiple sources DATA Entity read and writes Servicetypes STRICT FLOW OF DEPENDENCIES @jessicamtai
  35. 35. MIDDLE TIER Shared business logic DERIVED DATA Shared context, multiple sources DATA Entity read and writes PRESENTATION Synthesize Servicetypes STRICT FLOW OF DEPENDENCIES @jessicamtai
  36. 36. PRESENTATION Synthesize MIDDLE TIER Shared business logic DERIVED DATA Shared context, multiple sources DATA Entity read and writes Servicetypes STRICT FLOW OF DEPENDENCIES @jessicamtai
  37. 37. Homes H O S T E D B Y K I T T Y   ·   A P T O S , C A L I F O R N I A Mushroom Dome @jessicamtai
  38. 38. Homes data service Shared database Monorail Data access query Business logic 1.Migratingcoredatamodels Presentation view Homes database @jessicamtai
  39. 39. Homes data service Shared database Monorail Data access query Business logic 2.Migratingcorebusinesslogic Presentation view Homes database Pricing derived data service Pricing trends data store @jessicamtai
  40. 40. Homes data service Shared database Monorail Data access query Business logic 3.Migratingcoreproductviews Presentation view Homes database Pricing derived data service Pricing trends data store Checkout presentation service @jessicamtai
  41. 41. Homes data service Shared database Monorail Data access query Business logic 4.Migratingcoreproductwrites Presentation view Homes database Homes validation middle-tier Pricing derived data service Pricing trends data store Checkout presentation service @jessicamtai
  42. 42. Requestlifecycle Monorail API traffic Routing & view Client traffic V2: MONORAIL & SERVICES @jessicamtai Business logic, model, data 
 via services
  43. 43. Requestlifecycle API gateway Middleware Session data service Authentication data service Oauth data service Risk derived 
 data service ... Request
 context Presentation, logic, data V3: SOA & API GATEWAY Routing @jessicamtai
  44. 44. Requestlifecycle API gateway Middleware Session data service Authentication data service Oauth data service Risk derived 
 data service ... Request
 context Web rendering service HTML viewV3: SOA & API GATEWAY Routing @jessicamtai Presentation, logic, data
  45. 45. Monolith world Services world TheFutureTM @jessicamtai
  46. 46. Migration world Monolith world Services world @jessicamtai
  47. 47. COMPARE FOR DIFFERENCES @jessicamtai
  48. 48. Dualreadcomparison WaitRamp&waitCompareGate Admin web UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Read path A Read path BMonorail Service Database @jessicamtai
  49. 49. Dualreadcomparison WaitRamp&waitCompareGate Admin web UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monorail Service Database Response A Response B Consumer + 
 offline comparison framework @jessicamtai
  50. 50. Dualreadcomparison WaitRamp&waitCompareGate Admin web UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monorail Service Database Response A Response B Consumer + 
 offline comparison framework @jessicamtai
  51. 51. Dualreadcomparison WaitRamp&waitCompareGate Admin web UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monorail Service Database Response A Response B Consumer + 
 offline comparison framework @jessicamtai
  52. 52. Dualreadcomparison WaitRamp&waitCompareGate Admin web UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Production
 traffic Monorail Service Database Response A Response B Consumer + 
 offline comparison framework @jessicamtai
  53. 53. Dualreadcomparison WaitRamp&waitCompareGate Admin web UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch 100% Production
 traffic Monorail Service Database Response A Response B Consumer + 
 offline comparison framework @jessicamtai
  54. 54. Dualreadcomparison WaitRamp&waitCompareGate Admin web UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch 100% Monorail Service Database Response A Response B Consumer + 
 offline comparison framework @jessicamtai
  55. 55. Dualreadcomparison WaitRamp&waitCompareGate Admin web UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monorail Service Database Response A Response B Consumer + 
 offline comparison framework @jessicamtai
  56. 56. Dualreadcomparison WaitRamp&waitCompareGate Admin web UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monorail Service Database The only read path @jessicamtai
  57. 57. Writecomparison DUAL WRITE TO SEPARATE DATABASES Presentation service Write validation middle tier service Write path A Write path B Data service (production) Data service (shadow) Consumer + 
 offline comparison framework Monorail Payload A Payload B @jessicamtai
  58. 58. s Writecomparison DUAL WRITE TO SEPARATE DATABASES Presentation service Write validation middle tier service The only write path Data service (shadow) Monorail @jessicamtai
  59. 59. APIgatewaycomparison API gateway Presentation service Product logic Monorail Product logic Original request 
 (no middleware applied) Shadow request 
 with context Add request context @jessicamtai Middleware @jessicamtai Shadow
 request copy
  60. 60. APIgatewaycomparison API gateway Presentation service Product logic Request 
 with context Middleware @jessicamtai
  61. 61. Incrementalmigration @jessicamtai
  62. 62. ! Production traffic with partially complete service ○ e.g. batch API /loadUsers ○ Fetch users only by user id ! Unblock clients Migratebyendpoint @jessicamtai
  63. 63. Migratebyattribute Service Monorail Database Read migrated
 attributes Read not-yet-migrated 
 attributes Database Presentation service Production traffic @jessicamtai
  64. 64. SOA BEST PRACTICES @jessicamtai
  65. 65. Frameworks Auto-generate code Testing&deploying Replay production traffic Observability Standard templates Standardizeservicebuilding SCALE WITH CONSISTENCY @jessicamtai
  66. 66. Service Service&clientsetup Business logic @jessicamtai
  67. 67. Service Service&clientsetup Business logic Endpoint logic Server transport @jessicamtai
  68. 68. Service Service&clientsetup Business logic Endpoint logic Server transport Java client Ruby client Client transport Client transport @jessicamtai
  69. 69. Service Service&clientsetup Business logic Server metrics Server diagnostics Startup / teardown Endpoint logic Metrics Data validation Server transport Server resilience Java client Ruby client Metrics Client transport Data validation Error handling Resilience Metrics Client transport Data validation Error handling Resilience Type checking @jessicamtai
  70. 70. Service Endpoint logic Service&clientsetup Business logic Server metrics Server diagnostics Startup / teardown Dashboard Dashboard Alert Alert Alert Runbook documentation Metrics Data validation Server transport Server resilience Java client Ruby client Metrics Client transport Data validation Error handling Resilience Metrics Client transport Data validation Error handling Resilience Type checking @jessicamtai
  71. 71. Service Endpoint logic Service&clientsetup Business logic Server metrics Server diagnostics Startup / teardown Dashboard Dashboard Alert Alert Alert Runbook documentation Metrics Data validation Server transport Server resilience Java client Ruby client Metrics Client transport Data validation Error handling Resilience Metrics Client transport Data validation Error handling Resilience Type checking @jessicamtai
  72. 72. FrameworksusingThriftIDL @jessicamtai
  73. 73. IDL Service Endpoint logic Business logic Server metrics Server diagnostics Startup / teardown Dashboard Dashboard Alert Alert Alert Runbook documentation Metrics Data validation Server transport Server resilience Java client Ruby client Metrics Client transport Data validation Error handling Resilience Metrics Client transport Data validation Error handling Resilience Type checking @jessicamtai
  74. 74. ThriftIDL API FRAMEWORK /** Batch request for demo data */ struct LoadSomeDataRequest { 1: optional set<i64> ids /** Some extra context baz */ 2: optional bool fooBar (personal) } @jessicamtai
  75. 75. ThriftIDL API FRAMEWORK /** id to data response */ struct LoadSomeDataResponse { 1: optional map<i64, SomeData> data } @jessicamtai /** Batch request for demo data */ struct LoadSomeDataRequest { 1: optional set<i64> ids (personal) /** Some extra context baz */ 2: optional bool fooBar }
  76. 76. /** /loadSomeData batch endpoint */
 LoadSomeDataResponse loadSomeData 
 (1: LoadSomeDataRequest request)
 throws (1: SomeException exception1) 
 (accept_replay = "true", rate_limit = "true") ThriftIDL API FRAMEWORK @jessicamtai
  77. 77. Thrift annotations @jessicamtai
  78. 78. Block comments 
 from .thrift file @jessicamtai
  79. 79. TRY TRY AGAIN SUCCESS FAIL FAST @jessicamtai
  80. 80. Separateasync workerthreadpools Graceful degradation @jessicamtai
  81. 81. Testing&deploying TIMELINE ProductionCanaryDiffyStaging @jessicamtai Localdev
  82. 82. Testing&deploying TIMELINE ProductionCanaryDiffyStagingLocaldev Dev environment supports shared dev services @jessicamtai
  83. 83. Testing&deploying TIMELINE Replayed production traffic with other staging services ProductionCanaryDiffyStagingLocaldev @jessicamtai
  84. 84. Testing&deploying TIMELINE ProductionCanaryDiffyStagingLocaldev Compare responses from staging against production @jessicamtai
  85. 85. RegressionTesting DIFFY Staging (new code) Primary 
 (old code) Secondary (old code) Raw response differences Non-
 deterministic noise Filtered response differences Diffy Replayed traffic github.com/twitter/diffy @jessicamtai
  86. 86. Testing&deploying TIMELINE ProductionCanaryDiffyStagingLocaldev Deploy to single instance of production @jessicamtai
  87. 87. Testing&deploying TIMELINE ProductionCanaryDiffyStagingLocaldev Confidently deploy to prod @jessicamtai
  88. 88. @jessicamtai
  89. 89. @jessicamtai
  90. 90. EMPOWERING THE MIGRATION @jessicamtai
  91. 91. 2016: 
 One small infra team Airbnb’s SOA progress @jessicamtai
  92. 92. Productculture:
 shipthingsquickly @jessicamtai
  93. 93. Orgchallenges:
 servicebuildinginparallel @jessicamtai
  94. 94. Product Frontend
 Monorail Infrastructure Backend Monorail+services Volunteer
 sysops
 on-call @jessicamtai
  95. 95. @jessicamtai Product Frontend
 Monorail Infrastructure Backend Monorail+services Volunteer
 sysops
 on-call
  96. 96. On-callrotationperteam SERVICE OWNERSHIP Checkout service Pricing service User service Product Team Infrastructure Team @jessicamtai
  97. 97. PROGRESS SO FAR? @jessicamtai
  98. 98. 2016: 
 One small infra team 2019: 
 Whole engineering org Airbnb’s SOA progress @jessicamtai
  99. 99. ! Faster build & deploy times ○ Hours (Monorail) to minutes (service) ○ Fewer reverts ! Quicker bug fixes ! Increased developer productivity happiness Promisinginitialresults SUCCESS @jessicamtai
  100. 100. ! Lower latency from parallelization ○ Ruby monorail single-threaded ○ Java services multi-threaded ! Search results page 3x faster ! Homes description page 10x faster! Latencyresults SUCCESS @jessicamtai
  101. 101. Monorailfreeze @jessicamtai
  102. 102. 500+ 67% Deploys in MonorailEngineers 2016 @jessicamtai
  103. 103. 500+ 67% Deploys in MonorailEngineers 1200+ 7% 2016 2019 @jessicamtai
  104. 104. Production traffic via API Gateway 40% @jessicamtai
  105. 105. IDL services in production0 400+ @jessicamtai
  106. 106. CheckoutpagerequiredmessageinSOA Checkout presentation Home data Reservation data Review data Pricing derived data Home demand derived data Cancellation derived data Business travel derived data Messaging data @jessicamtai
  107. 107. SOAhasitschallenges CAUTION @jessicamtai
  108. 108. Distributedservices CAUTION @jessicamtai
  109. 109. Multiple,isolateddatabases CAUTION @jessicamtai
  110. 110. Complexserviceorchestration CAUTION @jessicamtai
  111. 111. ! Prepare for a long commitment ! Decompose incrementally ! Scale with auto-generating frameworks, tools ! Shift development culture SOAmigration TAKEAWAYS @jessicamtai
  112. 112. Lookbothways
 duringyourGreatMigration @jessicamtai
  113. 113. @JESSICAMTAI QCON LONDON 2019
  114. 114. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ airbnb-services-scalability

×