Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Five Rules

127 vues

Publié le

From a talk at DeveloperWeek Seattle: Cloud Edition on May 31, 2019.

Publié dans : Technologie
  • Soyez le premier à commenter

Five Rules

  1. 1. New Rules Expedia Group @sallamar https://www.subbu.org Subbu Allamaraju
  2. 2. Faster, better, and cheaper Safer, reliable
  3. 3. Change Everything else
  4. 4. Source: “Evolve of Die: High-Availability Design Principles Drawn from Googles Network Infrastructure” by Ramesh Govindan et al, from the Proceedings of the 2016 ACM SIGCOMM Conference We find that failures are evenly distributed across different network types and across data, control, and management planes, but that a large number of failures happen when a network management operation is in progress within the network.
  5. 5. ROOT CAUSE On Monday 11 March 2019, Google SREs were alerted to a significant increase in storage resources for metadata used by the internal blob service. On Tuesday 12 March, to reduce resource usage, SREs made a configuration change which had a side effect of overloading a key part of the system for looking up the location of blob data. The increased load eventually lead to a cascading failure. “On Tuesday 12 March 2019, Google's internal blob storage service experienced a service disruption for a duration of 4 hours and 10 minutes.”
  6. 6. © 2019 Expedia Group 9 Invest for speed and safety Rule 1
  7. 7. © 2019 Expedia Group 10Image ID 130985919 © Andrey Kryukov | Dreamstime.com [changes, traffic, dependencies, side effects …] [cloud services and building blocks…]
  8. 8. Releasing changes safely Progressive deployments Separating deploy step from release Feature flags Canary releases Compatible interfaces Additive data model changes Shadow traffic Predictable pipelines Rollback Change freeze
  9. 9. Sharded, replicated, or passive non-fate-sharing architectures
  10. 10. Observability Improve steady-state understanding
  11. 11. © 2019 Expedia Group 14 There is no substitute for architecture and operational hygiene Rule 2
  12. 12. Big App Big Database Big Ops
  13. 13. Retired App Big App Tech Debt Another Big App
  14. 14. © 2019 Expedia Group 18 When change is constant, hybrid is perpetual Rule 3 tech, process, culture
  15. 15. What you built yesterday What you are building now What you want to build tomorrow
  16. 16. Tech debt Re- platform Grow the business Something else ?
  17. 17. © 2019 Expedia Group 21 Stay calm and learn to make value-based decisions Rule 4
  18. 18. © 2019 Expedia Group 22 Embrace the messiness Rule 5
  19. 19. Thank you @sallamar https://www.subbu.org Expedia Group