An overview of MySQL HA solutions, And a further in depth example with Pacemaker/ Linux-HA

1. MySQL HA with PaceMaker Kris Buytaert

3. „ Infrastructure Architect“

4. I don't remember when I started using MySQL :)

5. Specializing in Automated , Large Scale Deployments , Highly Available infrastructures, since 2008 also known as “the Cloud”

6. Surviving the 10 th floor test

7. DevOp

9. MySQL HA Solutions

10. MySQL Replication

11. Linux HA / Pacemaker

13. Not designed for high-performance

14. Not designed for high troughput (load balancing)

16. You mis out on $$$

17. Your boss complains

18. New users don't return

19. Lies, Damn Lies, and Statistics Counting nines (slide by Alan R)

21. Keep it Simple

22. Prepare for Failure

23. Complexity is the enemy of reliability

24. Test your HA setup

26. Realitime

28. Most of the time

30. Fans

32. Going Out Of Memory

34. If those partitions each try and take control of the cluster, then it's called a split-brain condition

37. Multiple read only nodes

38. Application needed to be modified

40. DRBD

41. MySQL Cluster NDBD

42. Multi Master Replication

43. MySQL Proxy

44. MMM

45. Flipper

48. Proxy

49. Heartbeat / Pacemaker

51. Monitor MySQL node

52. Stonith

53. $$$ 1+1 <> 2

54. Storage = SPOF

55. Split Brain :(

57. In the Linux Kernel (as of very recent)

59. Only 1 MySQL instance Active accessing data

60. Upon Failover MySQL needs to be started on other node

62. Why did the crash happen ?

63. Is my data still correct ?

65. Check your BinLog size

67. Automatic partitioning

68. Synchronous replication

69. Fast automatic fail-over of data nodes

70. In-memory indexes

71. Not suitable for all query patterns (multi-table JOINs, range scans)

76. Does not fit for all apps

79. It catches up on the updates

80. It waits for the master to notify it of new update s

83. The I/O thread copies the statements to the relay log

85. Allows the slave to keep up with the master better

86. In case of master crash the slave is more likely to have all statements

88. RESET SLAVE

89. SHOW SLAVE STATUS

90. CHANGE MASTER TO…

91. LOAD DATA FROM MASTER

93. PURGE MASTER LOGS…

94. Show slave status Slave_IO_State: Waiting for master to send event Master_Host: 172.16.0.1 Master_User: repli Master_Port: 3306 Connect_Retry: 60 Master_Log_File: XMS-1-bin.000014 Read_Master_Log_Pos: 106 Relay_Log_File: XMS-2-relay.000033 Relay_Log_Pos: 251 Relay_Master_Log_File: XMS-1-bin.000014 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: xpol Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 106 Relay_Log_Space: 547 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: 1 row in set (0.00 sec)

96. Smaller log files

97. Auditing of actual SQL statements

99. Similar technology used by other RDBMSes

101. Log file size increases (backup/restore implications)

102. Replicated tables require explicit primary keys

103. Possible different result sets on bulk INSERTs

105. Both nodes are slave

106. Write in 1 get updates on the other M|S M|S

110. No Application change required

111. Sends specific queries to a specific node

113. User

114. Table

115. Availability

123. OpenARK

125. Advanced Routing

126. LVS

127. Or the upcoming slides

130. master-master MySQL Servers

131. Clients machines do not connect &quot;directly&quot; to either node instead,

132. One IP for read,

133. One IP for write.

134. Flipper allows you to move these IP addresses between the nodes in a safe and controlled manner.

135. http://provenscaling.com/software/flipper/

137. Manages more than MySQL

138. ...v3 .. don't even think about the rest anymore

139. http://clusterlabs.org/

141. No finegrained resources

143. Stability issues

144. Forking ?

146. Lrmd : Local Resource Management Daemon. Interacts directly with resource agents (scripts).

147. pengine Policy Engine. Computes the next state of the cluster based on the current state and the configuration.

148. cib Cluster Information Base. Contains definitions of all cluster options, nodes, resources, their relationships to one another and current status. Synchronizes updates to all cluster nodes.

149. crmd Cluster Resource Management Daemon. Largely a message broker for the PEngine and LRM, it also elects a leader to co-ordinate the activities of the cluster.

150. openais messaging and membership layer.

151. heartbeat messaging layer, an alternative to OpenAIS.

152. ccm Short for Consensus Cluster Membership. The Heartbeat membership layer.

154. Only CRM Code taken out of Heartbeat

156. Different Release Cycles as Heartbeat

158. Initially only Heartbeat

159. OpenAIS

160. Heartbeat got unmaintained

161. OpenAIS has heisenbugs :(

162. Heartbeat maintenance taken over by LinBit

163. CRM Detects which layer

164. or OpenAIS Heartbeat Pacemaker Cluster Glue

166. Configuring Heartbeat heartbeat::hacf {&quot;clustername&quot;: hosts => [&quot;host-a&quot;,&quot;host-b&quot;], hb_nic => [&quot;bond0&quot;], hostip1 => [&quot;10.0.128.11&quot;], hostip2 => [&quot;10.0.128.12&quot;], ping => [&quot;10.0.128.4&quot;], } heartbeat::authkeys {&quot;ClusterName&quot;: password => “ClusterName &quot;, } http://github.com/jtimberman/puppet/tree/master/heartbeat/

168. Heartbeat resource (+status)

169. OCF (Open Cluster FrameWork) (+monitor)

170. Clones (don't use in HAv2)

171. Multi State Resources

173. Replication state

174. ....

176. Keeps Nodes in Sync

177. XML Based

178. cibadm

179. Cli manageable

180. Crm configure property $id=&quot;cib-bootstrap-options&quot; stonith-enabled=&quot;FALSE&quot; no-quorum-policy=ignore start-failure-is-fatal=&quot;FALSE&quot; rsc_defaults $id=&quot;rsc_defaults-options&quot; migration-threshold=&quot;1&quot; failure-timeout=&quot;1&quot; primitive d_mysql ocf:local:mysql op monitor interval=&quot;30s&quot; params test_user=&quot;sure&quot; test_passwd=&quot;illtell&quot; test_table=&quot;test.table&quot; primitive ip_db ocf:heartbeat:IPaddr2 params ip=&quot;172.17.4.202&quot; nic=&quot;bond0&quot; op monitor interval=&quot;10s&quot; group svc_db d_mysql ip_db commit

181. Hardware Cluster Stack Resource MySQL Replication Adding MySQL to the stack Node A Node B HeartBeat Pacemaker “ MySQLd” “ MySQLd” Service IP MySQL

184. OpenARK

186. Think about your Data

187. Think about getting Queries to that Data

188. Complexity is the enemy of reliability

189. Keep it Simple

190. Monitor inside the DB

191. ` Kris Buytaert < [email_address] > Further Reading http://www.krisbuytaert.be/blog/ http://www.inuits.be/ http://www.virtualization.com/ http://www.oreillygmt.com/ ? !