Zero Downtime Migrations at Scale
•
6 j'aime•2,364 vues
Already have a system that serves user traffic and it has become so popular that it's hitting scaling limitations? It's probably time to upgrade its architecture or move its data to a more scalable database. Learn how to do this upgrade with zero downtime and no user visible effects in my talk!
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Zero Downtime Migrations at Scale
Similaire à Zero Downtime Migrations at Scale (20)
Plus de Aysylu Greenberg
Plus de Aysylu Greenberg (20)
Dernier
Dernier (20)
Zero Downtime Migrations at Scale
- 1. Zero Downtime Migrations at Scale Aysylu Greenberg ∽ April 28, 2018 ∽ Medellín Photo by Iván Erre Jota / CC BY-SA
- 3. Zero Downtime Migrations at Scale
- 4. Zero Downtime Migrations at Scale
- 12. Zero Downtime Migrations at Scale
- 13. Zero Downtime Migrations at Scale We will be performing scheduled site maintenance on Saturday from 3 am to 5 am!
- 14. Zero Downtime Migrations at Scale
- 15. New data storage and architecture Zero-Downtime Migration for Backups
- 16. Zero Downtime Migrations at Scale
- 17. Considerations Before Migration ● Scale: O(100M) of users with 1B+ backups, representing 1T+ objects
- 18. Considerations Before Migration ● Scale: O(100M) of users with 1B+ backups, representing 1T+ objects ● No sharing, no search, no folders
- 19. Considerations Before Migration ● Scale: O(100M) of users with 1B+ backups, representing 1T+ objects ● No sharing, no search, no folders ● Mainly write traffic, read traffic is rare and high priority
- 20. Considerations Before Migration ● Scale: O(100M) of users with 1B+ backups, representing 1T+ objects ● No sharing, no search, no folders ● Mainly write traffic, read traffic is rare and high priority ● Spanner: global strong consistency, SQL
- 21. Considerations Before Migration ● Scale: O(100M) of users with 1B+ backups, representing 1T+ objects ● No sharing, no search, no folders ● Mainly write traffic, read traffic is rare and high priority ● Spanner: global strong consistency, SQL ● Backups can get large
- 22. Considerations Before Migration ● Scale: O(100M) of users with 1B+ backups, representing 1T+ objects ● No sharing, no search, no folders ● Mainly write traffic, read traffic is rare and high priority ● Spanner: global strong consistency, SQL ● Backups can get large
- 23. So how do you migrate all this data? Photo by Richard Evea / CC BY-SA 2.0
- 24. So how do you migrate all this data? copy/paste... Photo by Richard Evea / CC BY-SA 2.0
- 26. Data migration ● Dual writes Architecture migration So how do you migrate all this data?
- 27. Data migration ● Dual writes Architecture migration So how do you migrate all this data? Are we storing data correctly?
- 28. Data migration ● Dual writes Architecture migration ● Dual writes So how do you migrate all this data?
- 29. Data migration ● Dual writes Architecture migration ● Dual writes So how do you migrate all this data? No effect on latency and error rates!
- 30. Data migration ● Dual writes ● Backfill data Architecture migration ● Dual writes So how do you migrate all this data?
- 31. Data migration ● Dual writes ● Backfill data Architecture migration ● Dual writes So how do you migrate all this data? Do we understand all client behavior and adapt the data correctly?
- 32. Data migration ● Dual writes ● Backfill data Architecture migration ● Dual writes ● Prove the stack So how do you migrate all this data?
- 33. Data migration ● Dual writes ● Backfill data Architecture migration ● Dual writes ● Prove the stack So how do you migrate all this data? Is the response from the new system same as from the old?
- 34. Data migration ● Dual writes ● Backfill data ● Learn the new storage Architecture migration ● Dual writes ● Prove the stack So how do you migrate all this data?
- 35. Data migration ● Dual writes ● Backfill data ● Learn the new storage Architecture migration ● Dual writes ● Prove the stack So how do you migrate all this data? New storage mechanism or schema?
- 36. Data migration ● Dual writes ● Backfill data ● Learn the new storage Architecture migration ● Dual writes ● Prove the stack ● Harden the system So how do you migrate all this data?
- 37. Data migration ● Dual writes ● Backfill data ● Learn the new storage Architecture migration ● Dual writes ● Prove the stack ● Harden the system So how do you migrate all this data? How to get it to production readiness to serve full load?
- 38. Data migration ● Dual writes ● Backfill data ● Learn the new storage ● Migrate slowly Architecture migration ● Dual writes ● Prove the stack ● Harden the system So how do you migrate all this data?
- 39. Data migration ● Dual writes ● Backfill data ● Learn the new storage ● Migrate slowly Architecture migration ● Dual writes ● Prove the stack ● Harden the system So how do you migrate all this data? Validate, validate, validate Resource constraints? Scale migration
- 40. Data migration ● Dual writes ● Backfill data ● Learn the new storage ● Migrate slowly Architecture migration ● Dual writes ● Prove the stack ● Harden the system ● Roll out slowly So how do you migrate all this data?
- 41. Data migration ● Dual writes ● Backfill data ● Learn the new storage ● Migrate slowly Architecture migration ● Dual writes ● Prove the stack ● Harden the system ● Roll out slowly So how do you migrate all this data? Scale carefully & proactively
- 42. Zero-Downtime Migrations at Scale FOCUS ON INTERMEDIATE STATE
- 43. Zero-Downtime Migrations at Scale FOCUS ON INTERMEDIATE STATE ● Prepare to write code for intermediate state >>> Quality of code corresponds to the expected lifetime
- 44. Zero-Downtime Migrations at Scale FOCUS ON INTERMEDIATE STATE ● Prepare to write code for intermediate state >>> Quality of code corresponds to the expected lifetime ● Migrate backends first
- 45. Zero-Downtime Migrations at Scale FOCUS ON INTERMEDIATE STATE ● Prepare to write code for intermediate state >>> Quality of code corresponds to the expected lifetime ● Migrate backends first
- 46. Zero-Downtime Migrations at Scale FOCUS ON INTERMEDIATE STATE ● Prepare to write code for intermediate state >>> Quality of code corresponds to the expected lifetime ● Migrate backends first
- 47. Zero-Downtime Migrations at Scale FOCUS ON INTERMEDIATE STATE ● Prepare to write code for intermediate state >>> Quality of code corresponds to the expected lifetime ● Migrate backends first ● Invest into visibility into system & migration state
- 48. Zero-Downtime Migrations at Scale FOCUS ON INTERMEDIATE STATE ROLL OUT INCREMENTALLY
- 49. Zero-Downtime Migrations at Scale FOCUS ON INTERMEDIATE STATE ROLL OUT INCREMENTALLY ● Validate scalability while affecting fewest users
- 50. Zero-Downtime Migrations at Scale FOCUS ON INTERMEDIATE STATE ROLL OUT INCREMENTALLY ● Validate scalability while affecting fewest users ● Decouple launch of services
- 51. Zero-Downtime Migrations at Scale FOCUS ON INTERMEDIATE STATE ROLL OUT INCREMENTALLY ● Validate scalability while affecting fewest users ● Decouple launch of services
- 52. Zero-Downtime Migrations at Scale FOCUS ON INTERMEDIATE STATE ROLL OUT INCREMENTALLY ● Validate scalability while affecting fewest users ● Decouple launch of services
- 53. Zero-Downtime Migrations at Scale FOCUS ON INTERMEDIATE STATE ROLL OUT INCREMENTALLY VALIDATE & PRACTICE ROLLOUT
- 54. Zero-Downtime Migrations at Scale
- 55. Zero-Downtime Migrations at Scale
- 56. Zero-Downtime Migrations at Scale
- 57. Zero-Downtime Migrations at Scale
- 58. Zero-Downtime Migrations at Scale
- 59. Zero-Downtime Migrations at Scale
- 60. Zero-Downtime Migrations at Scale FOCUS ON INTERMEDIATE STATE ROLL OUT INCREMENTALLY VALIDATE & PRACTICE ROLLOUT
- 61. Gratitude Steve Clark Thomas Escobar Matt Welsh Ranjodh Mathial Tatiana Marquez
- 62. Zero Downtime Migrations at Scale Aysylu Greenberg ∽ April 28, 2018 ∽ Medellín Photo by Iván Erre Jota / CC BY-SA