Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Migrating from PostgreSQL to MySQL Without Downtime

10 607 vues

Publié le

How Etsy migrates live data into MySQL shards without downtime.

Publié dans : Technologie
  • ⇒⇒⇒WRITE-MY-PAPER.net ⇐⇐⇐ I love this site. It always finds me the best tutors in accordance with my needs. I have been using it since last year. The prices are not expensive compared to other sites. I am glad I discored this site:)
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD THE BOOK INTO AVAILABLE FORMAT (New Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://redirect.is/fyxsb0u } ......................................................................................................................... Download Full EPUB Ebook here { https://redirect.is/fyxsb0u } ......................................................................................................................... Download Full doc Ebook here { https://redirect.is/fyxsb0u } ......................................................................................................................... Download PDF EBOOK here { https://redirect.is/fyxsb0u } ......................................................................................................................... Download EPUB Ebook here { https://redirect.is/fyxsb0u } ......................................................................................................................... Download doc Ebook here { https://redirect.is/fyxsb0u } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THE can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THE is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBOOK .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookBOOK, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, EBOOK, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THE Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THE the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THE Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • Follow the link, new dating source: ♥♥♥ http://bit.ly/39pMlLF ♥♥♥
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • Dating for everyone is here: ❶❶❶ http://bit.ly/39pMlLF ❶❶❶
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Migrating from PostgreSQL to MySQL Without Downtime

  1. 1. Migrating from PostgreSQL to MySQL ... without downtimeMatthew GrahamPercona Live NYC 2011
  2. 2. Global marketplace for buying and selling handmade goods Total April 2011• 8 million members • $38.7 million goods sold• 800k active shops • 2 million items sold• 1 billion page views / month • 1.8 million new items listed • 390k new members joined
  3. 3. Global marketplace for buying and selling handmade goods Total April 2011• 8 million members • $38.7 million = $900 / min• 800k active shops • 2 million items sold• 1 billion page views / month • 1.8 million new items listed • 390k new members joined
  4. 4. Downtime = -$
  5. 5. $320m$240m$160m$80m $0m 2006 2007 2008 2009 2010 Gross Merchandise Sales
  6. 6. Reasons To Migrate • Horizontal Scaling • Reduce Types of Databases • Licensing Costs • Functional Partitioning • Schema Refactor
  7. 7. Deciding to Switch expected quality - transition ----------------- net quality quality > actual
  8. 8. Deciding to Switch expected quality - transition----------------- net quality > actual quality
  9. 9. Deciding to Switch expected quality - transition----------------- net quality > actual quality
  10. 10. Source Migration Target
  11. 11. Why ?
  12. 12. Foundational Principles • Migrate One Table at a Time • Progressive Ramp Up • Data Duplication During Transition
  13. 13. 5 Steps Per Table 1. Create Target Tables 2. Tee Writes 3. Backfill 4. Read from Target Tables 5. Wrap Up
  14. 14. Create the Target
  15. 15. Merge into the Target
  16. 16. Separate the Target
  17. 17. Source OnlySource Target Application Writes
  18. 18. Teed WritesSource Target Application Writes
  19. 19. Generating IDs Source TargetID Name ID Name1 Adam 1 Adam2 Bill 2 Bill3 Charlie 3 David4 David 4 Charlie Ticket Server: http://bit.ly/dbtickets
  20. 20. Backfill
  21. 21. Application Code Backfillfunction backfill_user(user_id) { old_row = find_old_user(user_id); new_row = create_new_user(); new_row.id = old_row.id; ... store_new_user(new_row);}
  22. 22. ETL BackfillSource Target Load Extract Transform
  23. 23. Don’t Overload • Script bulk inserts 100-2000 row batchesINSERT INTO account (id, firstname, lastname)VALUES(1, ‘Alan’, ‘Alda’),(2, ‘Barry’, ‘Bonds’),(3, ‘Charlie’, ‘Chaplin’);
  24. 24. Backfill SpeedApplication Code ETL• Easier to Write • Faster Run Time
  25. 25. Backfill ExtractApplication Code ETL• Easier to Write • Faster Run Time• Less Likely to Get Out • Needs an Extract Source Of Sync
  26. 26. Backfill RerunsApplication Code ETL• Easier to Write • Faster Run Time• Less Likely to Get Out • Needs an Extract Source Of Sync• Handles Duplicates from • REPLACE and INSERT Multiple Executions ON DUPLICATE KEY UPDATE
  27. 27. Teed Writes then Backfill orBackfill then Teed Writes
  28. 28. Verification by Diff$ diff_user 111222333target user row is missing$ diff_user 123456789- source.user.address_state = ‘CA’+ target.user.address_state = ‘NY’
  29. 29. Verification by DiffCOMPARING 200 ROWS From: 111222197User Id: 111222333target user row is missingUser Id: 111222345- source.user.address_state = ‘CA’+ target.user.address_state = ‘NY’SUMMARY: total rows with errors: 2/200
  30. 30. Read from Target
  31. 31. Progressive Ramp Up 100% 0%Source Target Application Reads
  32. 32. Progressive Ramp Up 99% 1%Source Target Application Reads
  33. 33. Progressive Ramp Up 95% 5%Source Target Application Reads
  34. 34. Progressive Ramp Up 75% 25%Source Target Application Reads
  35. 35. Progressive Ramp Up 0% 100%Source Target Application Reads
  36. 36. Ramp Up Example# global configurationuse_new_tables: employees: true percent: 1# application codeif (enabled(‘use_new_tables’)) { $result = read_from_target();} else { $result = read_from_source();}
  37. 37. Enabled or Disabled?• Check cookies if user already assigned• $enabled = configured threshold > random %• Store $enabled in a cookie for future requests
  38. 38. Continuous Deployment • Ramp Up / Ramp Down • Backfill Fixes • Need Code Running on Prod to Proceed • Makes it Easier
  39. 39. No Foreign Keys
  40. 40. Wrapping Up
  41. 41. Analytical Data
  42. 42. Teed WritesSource Target Application Writes
  43. 43. Target OnlySource Target Application Writes
  44. 44. Things to Remove• Code to use old tables• Code to switch on configuration• Configuration• Drop the old tables... eventually
  45. 45. 5 Steps Per Table 1. Create Target Tables 2. Tee Writes 3. Backfill 4. Read from Target Tables 5. Wrap Up
  46. 46. Foundational Principles • Migrate One Table at a Time • Progressive Ramp Up • Data Duplication During Transition
  47. 47. Questions? • Yes, we’re hiring! Specifically MySQL Ops http://bit.ly/etsywantsawesome • http://codeascraft.etsy.com • http://twitter.com/lapsuMatthew GrahamPercona Live NYC 2011

×