2. Aspirin for your MySQL Headaches
Eugene L. Simpkins, II - Nexcess.net
Max Bubenick - Percona

3. Topics Covered
● Case study in replication
● Types of replication
● Their uses
● Advantages of each

4. Topics Covered
● Case study in replication
● Types of replication
● Their uses
● Advantages of each
● Possible pitfalls of asynchronous replication

5. Topics Covered
● Case study in replication
● Types of replication
● Their uses
● Advantages of each
● Possible pitfalls of asynchronous replication
● Percona Toolkit

6. Database Replication
An Overview

7. Two Basic Replication Types
• Asynchronous
– Stripped down race
car
– Super fast
– Not reliable
• Synchronous
– Family minivan
– Not the fastest
– Super reliable

8. ● Difference in performance lies in cooperation
What’s the difference?

9. ● Difference in performance lies in cooperation
● Asynchronous replication
● No coordination between master and slave
● Doesn’t care if slave server is not in sync with master
What’s the difference?

10. ● Difference in performance lies in cooperation
● Asynchronous replication
● No coordination between master and slave
● Doesn’t care if slave server is not in sync with master
● Synchronous replication
● Total coordination between nodes
● Doesn’t allow its nodes to be out of sync
What’s the difference?

11. The Scenario

12. Initial Configuration

13. ● Run reports and process data
● Data processing could impact performance
● Queries can be resource intensive
● Queries will only be read only (SELECT statements)
● Some queries may hold locks “SELECT … FOR UPDATE”
● Customer experience should not degrade as a result of
reports and data processing
● Location of a separate server not important
Requirements

14. Run reports against single database server

15. Run reports against single database server (cont’d)

16. Implement an asynchronous slave server

17. Run reports against asynchronous slave server

18. The Problem
Data Consistency

19. ● Data inconsistency in two manners
1. Reports could possibly be pulling incorrect / outdated
information because of replication lag
2. Inherent to delayed replication - time delay between
COMMIT on master and COMMIT on slave
3. Reports run at a later time may contain up-to-date
information
4. Dependent on rate of COMMITs, rate of reports being run,
replication lag
Asynchronous Replication = No guarantee

20. ● Data inconsistency in two manners (cont’d)
2. Reports could possibly be pulling incorrect / outdated
information because of missing replication transactions
3. Information may never be recovered
4. Reports run at a later time may not use the correct
information
5. THIS IS A PROBLEM!!
Asynchronous Replication = No guarantee

21. ● Operations to database are STATEFUL
● Statements that are run on master server operate on the
state of database caused by previous queries
● This state is replicated to SLAVE servers
● A missing update to the state on SLAVE server may cause
divergence in consistency of data set between master and
slave
Asynchronous Replication = No guarantee

22. What did that mean?!

23. The Solution
Percona Toolkit

24. ● Three very helpful tools
● pt-heartbeat measures replication lag on a database
server.
Percona Toolkit & Replication

25. ● Three very helpful tools
● pt-heartbeat measures replication lag on a database
server.
● pt-table-checksum verifies replication integrity by
leveraging the stateful nature of STATEMENT based
replication
Percona Toolkit & Replication

26. ● Three very helpful tools
● pt-heartbeat measures replication lag on a database
server.
● pt-table-checksum verifies replication integrity by
leveraging the stateful nature of STATEMENT based
replication
● pt-table-sync will synchronise tables across replication
(master to slave)
Percona Toolkit & Replication

27. Early detection is key to prevent
total disaster!
In Summary

28. Monitoring

29. Monitoring
• Why?
– We need to know

30. Monitoring
• Why?
– We need to know
• Trending
– Proactive

31. Monitoring
• Why?
– We need to know
• Trending
– Proactive
• Alerting
– Reactive

32. Monitoring
- What do I need to monitor?
● OS
○ Di
sk
sp
ac
e
○ R
AI
D
he
● MySQ
L
○ Co
nn
ec
tio
ns
○ Th
re
ad

33. Monitoring
• Nagios
– Alerting
– Plugins

34. Monitoring
• Cacti
– Trending Graphs
– Not hard to deploy
– Plugins
– Hard to scale

35. Monitoring
• Zabbix
– Trending
– Alerting
– Dashboards

36. Monitoring
• Percona Cloud Tools (beta)
– Trending
– Easy to install

37. Monitoring
• Percona Cloud Tools (beta)
– Query analytics
– Summary
– Explain

38. Backups

39. Backups
Why we need backups?
- Anything can and will fail
- Hardware fail
- Application bugs
- Operational mistakes
- Attacks
- Stage dev env or new slaves
- Feed ETL processes

40. Backups
We need to be ready to:
• Restore a complete server
• Restore a complete DB
• A complete table
• A few rows

41. Backups
Different Backup types
• Logical
– Plain text files
– Can be Remote
– Slower on large datasets
– Can be restored on different versions
– Sometimes the only option

42. Backups
Different Backup types (cont’d)
• Binary or Raw
– File-system level access
– Local
– Faster on large datasets
– Storage engine dependent
– Can be useless on filesystem corruption

43. Backups
Different Backup types (cont’d)
• Binlog
– Can be done almost real time
– For point in time recovery PITR
– Even for a single schema or table

44. Backups
Different Backup types
• Environment
– my.cnf
– SSL certificates
– Scripts

45. Backups
• Backup Tips

46. Backups
• Backup Tips
– Logical backups for partial restores

47. Backups
• Backup Tips
– Logical backups for partial restores
– Binary for full restores

48. Backups
• Backup Tips
– Logical backups for partial restores
– Binary for full restores
– Binlogs to point in time recovery

49. Backups
• Backup Tips
– Logical backups for partial restores
– Binary for full restores
– Binlogs to point in time recovery
– Off site storage

50. Backups
• Backup Tips
– Logical backups for partial restores
– Binary for full restores
– Binlogs to point in time recovery
– Off site storage
– Encrypt

51. Backups
• Backup Tips
– Logical backups for partial restores
– Binary for full restores
– Binlogs to point in time recovery
– Off site storage
– Encrypt
– Document all possible restores

52. Backups
• Backup Tips
– Logical backups for partial restores
– Binary for full restores
– Binlogs to point in time recovery
– Off site storage
– Encrypt
– Document all possible restores
– Test them!!

53. Backups
• Backup Tips
– Logical backups for partial restores
– Binary for full restores
– Binlogs to point in time recovery
– Off site storage
– Encrypt
– Document all possible restores
– Test them
– Slaves are not Backups!!!

54. Thank you
Eugene L. Simpkins, II - Nexcess.net
Max Bubenick - Percona