PalominoDB's David Turner and Ben Black cover common operations implemented in production and how you can minimize downtime and customer impact

1. Online Schema Changes
for Maximizing Uptime
PALOMINODB
OPERATIONAL EXCELLENCE
FOR DATABASES
David Turner
Ben Black
www.palominodb.com

2. Palominodb: Who are we and what do we do?
• Proactive database awesomeness so you
can focus on other things
• Systems and Performance Optimization
• Operational Best Practices
• Devops
• Ask us how we like working from anywhere

3. Online Schema Changes
• Best practices
• Online Schema Tools
• PT Online Schema Change
• Innodb Online DDL
• Case Study

4. Best Practices
• Backups
• Benchmark
o Start small
o Step up
o Time
o Performance_schema and Information_schema

5. Repository
• Prevent
o Broken Replication
o Data loss
o Broken applications

6. Creating the repository
• Dump
• Script

7. Syncing the repository
• Migrations
• Schema Changes
• Scheduled

8. Problems
• Schema changes
• Fragmentation

9. You want to do what to that 40G
table?
• Down for Maintenance/blocking DDL
• Disable writes
• Remaster
• Facebook OSC
• Large Hadron Migrator
• Openark-kit
• Pt Online Schema Change
• MySQL Innodb DDL - IOD

10. OSC Tools or how I learned
to love the alter...
• copy original table structure to new table
• alter new table
• create triggers to copy dml from original
table to new table
• copy data in original table to new table in
chunks
• swap table names and drop original table

11. pt-online-schema-change
Percona toolkit is your friend.
Mature product
Pay attention to version you are using!!!
FK issues, log_bin with 2.0
screen - use it along with tee and time

12. It's all ball bearings nowadays
--progress time,10 (default 30 seconds)
--max-lag
--recursion-method (how to find slaves)
show processlist
show hosts
dsn table (great for ignoring some slaves)

13. What could possibly go wrong?
• PK/UK required
• FK names will change on altered table
• FK's reference table to alter
--alter-foreign-keys-method drop_swap
sets foreign_key_checks=0
drops original table (hardlink!!!)
renames new table

14. What could possibly go wrong?
• non xfs table drops (create the hardlink!)
o Can even cause innodb to crash mysqld
• PKs with gaps
• Largest table to alter vs free disk space
• Disk space (2x for RBR)
• Global mutexes (table drops)
• table metadata locks (triggers)

15. NOOOOOO!!
• Running PT-OSC against a slave with RBR
Replication started erroring after pt-osc
why???
• And how about syncing a table using pt-osc
with RBR?

16. Set it and forget it???
• watch
o w
o df -h
o ls -alh ibd files
o mysql -e"show processlist;"|egrep -v "Sleep|repl"
o slave lag
• How is it affecting the application?
• max-lag, max-load,

17. Features we like
• Throttling
• Nibbling
• Reporting
• Replicating
• Reorgs

18. Innodb Online DDL - IOD
• Looking back 5.1 and 5.5
o Fast index creation
• Enhancements
o Online DDL
o Inplace

19. Silver Bullets
http://dev.mysql.com/doc/refman/5.6/en/innodb-
create-index-overview.html#innodb-online-
ddl-summary-grid
• Drop and add Index together
• Temporary table index creation

20. Alter Online lock modes
• Exclusive
• Shared
• Default
• None
* Note

21. Alter Online Algorithms
• Inplace
• Copy

22. Grouping DDL
• Copy
• Sequence
• Lock

23. IOD Behavior
• Metadata
• Data volume
• Type of index
• Foreign keys
• Partitioning
o Add partition, drop partition
o Truncate partition
o Add partition and coalesce partition
• Algorithm
• Autoincrement
• Locking

24. IOD Errors
• Locking
• Timeouts
• Tmpdir
• DML

25. IOD Crash Recovery
• Secondary Indexes
• Clustered Indexes (don't crash!)

26. Of Interest
• Tmpdir
• Dropping Indexes with Foreign keys
• Cascade on
• Inconsistent .frm

27. Monitoring
• Rows Affected
• Performance_schema and
Information_schema
• Progress
• Replication lag

28. IOD Server parms
• innodb_online_alter_log_max_size
• sort_buffer_size

29. Case Study
• Ticketing system

30. Architecture
• 1 master
• 1 active slave
• 1 analytics/indexing slave
• 1 backup/reorg slave

31. Issues
• Growth
• Rebalancing Shards
• Deletions
• Reorgs

32. Timeline
• Difficult failovers
• Ticket Growth
• FB OSC
• PT OSC
• Reorgs

34. Questions
David Turner
dturner@palominodb.com
Ben Black
ben@palominodb.com
www.palominodb.com