Advanced MySQL Replication techniques

1. Advanced MySQL
Replication Techniques
Giuseppe Maxia, QA Director, Continuent, Inc
Facts. And Demos. Possibly fun
Monday, April 11, 2011 1

2. about me - Giuseppe Maxia
• a.k.a. The Data Charmer
• QA Director at Continuent, Inc
• Long time hacking with MySQL features
• Formerly, community manager,db consultant, designer,
coder.
• A passion for QA and open source
• Blogger
• http://datacharmer.blogspot.com
Monday, April 11, 2011 2

3. Agenda
• Replication basics
• Replication blues
• Master switch and failover
• Slave lagging
• Gotchas
• Replication power techniques and features
• row-based, semi-synch, delayed
• Circular, bi-directional
• leveraging replication
• Tools and tips
• Replication outside the box
•
Monday, April 11, 2011
seamless failover, parallel, multi-master
3

4. DA
AG
EN Replication basics
• Principles
• How to set up replication
• Monitoring replication
Monday, April 11, 2011 4

5. DA
AG
EN Principles
Monday, April 11, 2011 5

6. database server
a simple web
application
scheme
r/w requests
web server
clients
6
Monday, April 11, 2011 6

7. database server
scaling web
requests
r/w requests
web servers
load balancer
clients
7
Monday, April 11, 2011 7

8. read/write
master a web
application
read/only
slaves
scheme with
replication
load balancer
R/W
R/O
web servers
load balancer
clients
8
Monday, April 11, 2011 8

9. client master
transaction
binary log
reads
slave
IO thread
relay log replication
concepts
SQL thread reads
9
Monday, April 11, 2011 9

10. DA
AG
EN replication setup
Monday, April 11, 2011 10

11. 1 SHUT DOWN THE DATABASE SERVER
Master
11
Monday, April 11, 2011 11

12. 2 MAKE A BACKUP COPY
Master
12
Monday, April 11, 2011 12

13. 3 ENABLE THE MASTER
Master
Configuration file
[mysqld]
log-bin=mysql-bin
server-id=1
13
Monday, April 11, 2011 13

14. 4 RESTART THE MASTER
Master
14
Monday, April 11, 2011 14

15. 5 CREATE REPLICATION USER
Master
SQL command
GRANT REPLICATION SLAVE ON *.*
to 'slave_user@'10.10.100.%'
IDENTIFIED BY 'slave_pass';
15
Monday, April 11, 2011 15

16. 6 INSTALL MySQL on the slave
Slave 1
Make sure that:
• You're using the same version of MySQL
• You have the same directory structure
• The server is not started yet
16
Monday, April 11, 2011 16

17. 7 COPY THE MASTER DATA to the slave
Slave 1
17
Monday, April 11, 2011 17

18. 8 ENABLE THE SLAVE
Slave 1
Configuration file
[mysqld]
server-id=2
relay-log=mysql-relay
read-only
# optional:
log-bin=mysql-bin 18
Monday, April 11, 2011 18

19. 9 START THE SLAVE SERVER
Slave 1
19
Monday, April 11, 2011 19

20. 10 INITIALIZE THE SLAVE
Slave 1
SQL command
CHANGE MASTER TO
MASTER_HOST=master_IP,
MASTER_PORT=3306,
MASTER_USER=slave_user,
MASTER_PASSWORD='slave_pwd';
20
Monday, April 11, 2011 20

21. 11 START THE SLAVE SERVICE
Slave 1
SQL command
START SLAVE;
21
Monday, April 11, 2011 21

22. 12 CHECK THE SLAVE
Slave 1
SQL command
SHOW SLAVE STATUS G
...
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...
22
Monday, April 11, 2011 22

23. Troubleshooting
• SHOW SLAVE STATUS says
SLAVE_IO_RUNNING=No
• Make sure that the slave host can connect
to the master
• Make sure that master and slave have
different Server-id
• Check the error log of both master and slave
Monday, April 11, 2011 23

24. Testing the slave
• Create a table in the master.
• Make sure that the slave has replicated
the table.
• Insert data in the master
• read that data in the slave
Monday, April 11, 2011 24

25. DA
AG
EN Monitoring
Monday, April 11, 2011 25

26. Sample get slave
status
monitoring
slave
Running? No
Yes
master
Get master
binlog and
position
Same or later
No
Yes
binlog/position?
check table
contents
alert 26
Monday, April 11, 2011 26

27. monitoring replication
master> SHOW MASTER STATUS
slave> SHOW SLAVE STATUS
FULL SCRIPTS:
http://datacharmer.blogspot.com/2011/04/refactored-again-poor-mans-
mysql.html
http://forge.mysql.com/tools/tool.php?id=6
Monday, April 11, 2011 27

28. show master status
File: mysql-bin.000002
Position: 78045744
Binlog_Do_DB:
Binlog_Ignore_DB:
Monday, April 11, 2011 28

29. show slave status
Slave_IO_State: Waiting for master to send event
Master_Host: 127.0.0.1
Master_User: rsandbox
Master_Port: 27371
Connect_Retry: 60
Master_Log_File: mysql-bin.000002
Read_Master_Log_Pos: 78045744
Relay_Log_File: mysql_sandbox27372-relay-bin.000055
Relay_Log_Pos: 78045889
Relay_Master_Log_File: mysql-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Monday, April 11, 2011 29

30. show slave status
...
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
...
Monday, April 11, 2011 30

31. show slave status
...
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 78045744
Relay_Log_Space: 78046100
...
Seconds_Behind_Master: 0
...
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Monday, April 11, 2011 31

32. DA
AG
EN Replication blues
• Master switch and failover
• Slave lagging
• Gotchas
Monday, April 11, 2011 32

33. DA
AG
EN Master switch and failover
Monday, April 11, 2011 33

34. Replacing the Master
master crashe
s
Let all slaves
catch up with
execution STOP
replication in
all slaves
FIND the most make it the
up to date slave master
FIND which run missing
connect all
transactions are transactions
slaves to the
missing from to other
new master
other slaves slaves
Stop
34
Monday, April 11, 2011 34

35. Planned master switch
• Stop accepting writes
• Wait until all slaves have caught up
• Stop replication in all slaves
• Promote a slave to master
• Point all slaves to the new master
Monday, April 11, 2011 35

36. Changing a failed master
• Pre-requisite:
• log_bin and log_slave_updates must be
enabled in all the slaves
• If not, there is more manual labor
Monday, April 11, 2011 36

37. Changing a failed master (1)
• Wait until all slaves have caught up
• Identify the most advanced slave
• Make that slave the new master
• ... so far, so good
Monday, April 11, 2011 37

38. Changing a failed master (2)
• For each remaining slave:
• Find the missing statements
• Find the LAST statement replicated by the slave
• Find the same statement in the new master
binlog (*)
• get the position of the NEXT statement
(*) if log_slave_updates was not enabled, you
need to convert the relay log statements to SQL
and do the next step manually
Monday, April 11, 2011 38

39. Changing a failed master (3)
• For each remaining slave:
• Apply the missing statements
• CHANGE MASTER TO
master_host=”new_master_hostname”,
master_port=new_master_port,
master_log_file=”mysql-bin.xxxxxx”,
master_log_pos=YYYY
Monday, April 11, 2011 39

40. Reasons for complexity
• No global transaction ID
Monday, April 11, 2011 40

41. What is a global transaction ID
• A unique identifier of a transaction
• Unique for the whole cluster, not for each node
• Generated by the ultimate source (the master)
• Does not change when the transaction goes
through an intermediate master
Monday, April 11, 2011 41

42. Why should you care
• Failure recovery
• MySQL DOES NOT have it
Monday, April 11, 2011 42

43. Defining the problem
Slave 3
Master Replication Slave 2
Slave 1
Binary log
Transactions
Relay log Binary log
--------
-------- Transactions Transactions
-------- -------- ========
-------- -------- ========
-------- ========
-------- ========
Monday, April 11, 2011 43

44. The master crashes
Slave 3
Master Replication Slave 2
Slave 1
Binary log
Transactions
Relay log Binary log
--------
-------- Transactions Transactions
-------- -------- ========
-------- -------- ========
-------- ========
-------- ========
Monday, April 11, 2011 44

45. Where do the slaves
stand?
Slave 1 Slave 2 Slave 3
Binary log Binary log Binary log
Transactions Transactions Transactions
======== ======== ========
======== ======== ========
======== ======== ========
======== ======== ========
Monday, April 11, 2011 45

46. Slave 3 becomes
master
New master
Slave 1 Slave 2
Binary log
Transactions
========
Binary log Binary log ========
========
Transactions Transactions ========
======== ========
======== ========
======== ========
======== ========
Monday, April 11, 2011 46

47. Need to synch the
missing transactions
New master
Slave 1 Slave 2
Binary log
Transactions
========
Binary log Binary log ========
========
Transactions Transactions ========
======== ========
======== ========
======== ========
======== ========
Monday, April 11, 2011 47

48. Transaction position in
the new master binlog
New master
Slave 1 Slave 2
Binary log
Transactions
========
Binary log Binary log ========
========
Transactions Transactions ========
======== ========
======== ========
======== ? ======== ?
======== ========
Monday, April 11, 2011 48

49. An existing solution
• Google has made a patch to implement global
transaction IDs
• HOWEVER
• It only works with 5.0
• It doesn't work with row based replication
Monday, April 11, 2011 49

50. Hack
ahead!
Monday, April 11, 2011 50

51. CREATE TABLE Global_Trans_ID (
number INT UNSIGNED AUTO_INCREMENT PRIMARY KEY
) ENGINE = MyISAM;
CREATE TABLE Last_Exec_Trans (
server_id INT UNSIGNED,
trans_id INT UNSIGNED
) ENGINE = InnoDB;
Monday, April 11, 2011 51

52. CREATE PROCEDURE commit_trans ()
BEGIN
DECLARE trans_id, server_id INT UNSIGNED;
SET SQL_LOG_BIN = 0;
INSERT INTO global_trans_id() values ();
SELECT LAST_INSERT_ID() INTO trans_id,
@@server_id INTO server_id;
DELETE FROM global_trans_id where number < trans_Id;
SET SQL_LOG_BIN = 1;
INSERT INTO last_exec_trans(server_id, trans_id)
VALUES (server_id, trans_id);
COMMIT;
END
Monday, April 11, 2011 52

53. Hacking
the hack
Monday, April 11, 2011 53

54. Simplifying the scheme
• No MyISAM table and LAST_INSERT_ID()
• Use UUID_SHORT() instead
• Replace the InnoDB table with a BlackHole one
Monday, April 11, 2011 54

55. Advantages
• 10 times faster
• No additional tables and storage required
Monday, April 11, 2011 55

56. Disadvantages
• The master SERVER ID must be < 256
Monday, April 11, 2011 56

57. Another solution
• We’ll see that in the second part of the
talk
Monday, April 11, 2011 57

58. DA
AG
EN Slave lagging
Monday, April 11, 2011 58

59. Slave lagging
• Common causes:
• Excess of traffic
• Slave restart
• Long DDL operations
Monday, April 11, 2011 59

60. Slave lagging
transaction
transaction
MySQL transaction
transaction
transaction
DBMS transaction
transaction
transaction
transaction
transaction
transaction
transaction
BINARY LOG
ns act act tio n
tra csio s ac ctio
ac ion ion n
an n s an
nr tr tra aniton
tsa a n csio
nts ct n
rn a o
taa scti
r
trnsa
n
n
tio
t
tra
tra
REPLICATION
MySQL replication
is SINGLE THREAD
transaction
transaction
MySQL
DBMS
Monday, April 11, 2011 60

61. Slave lagging remedies
• row-based replication (helps in some
cases)
• Expensive query split
• Double apply (dangerous)
• Slave query prefetch (requires external
app)
• Parallel replication (requires external
app)
Monday, April 11, 2011 61

62. statement-based replication
master> set global binlog_format=‘statement’;
master> insert into test.t1 select count(*) from
information_schema.columns;
master> show global status like 'opened_tables';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Opened_tables | 41 |
+---------------+-------+
slave> show global status like 'opened_tables';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Opened_tables | 41 |
+---------------+-------+
Monday, April 11, 2011 62

63. row-based replication
master> set global binlog_format=‘row’;
master> insert into test.t1 select count(*) from
information_schema.columns;
master> show global status like 'opened_tables';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Opened_tables | 41 |
+---------------+-------+
slave> show global status like 'opened_tables';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Opened_tables | 17 |
+---------------+-------+
Monday, April 11, 2011 63

64. Expensive query split
# instead of this:
insert into t1 select benchmark(10000000, sha('abc'));
Query OK, 1 row affected (4.19 sec)
Records: 1 Duplicates: 0 Warnings: 0
# you may do this
set @result=(select benchmark(10000000, sha('abc')));
Query OK, 0 rows affected (4.19 sec)
insert into t1 values (@result);
Query OK, 1 row affected (0.00 sec)
Monday, April 11, 2011 64

65. Expensive query split
# at 471
#110410 8:21:21 server id 1 end_log_pos 518
User_var
SET @`result`:=0/*!*/;
# at 518
#110410 8:21:21 server id 1 end_log_pos 612 Query
thread_id=3 exec_time=0 error_code=0
SET TIMESTAMP=1302448881/*!*/;
insert into t1 values (@result)
Monday, April 11, 2011 65

66. Slave query prefetch
• What is it?
• a technique that fetches queries from the
relay logs, and runs them as SELECT
statements, to warm up the indexes
• How do you do it?
• mk-slave-prefetch (http://www.maatkit.org)
• Slave readahead http://sourceforge.net/
projects/slavereadahead/
Monday, April 11, 2011 66

67. DA
AG
EN Gotchas
Monday, April 11, 2011 67

68. Gotchas
• When you thought that you had figured
out everything ...
Monday, April 11, 2011 68

69. Gotchas: LOAD DATA
• LOAD DATA INFILE is replicated as a
temporary file
• The slave needs three times the size of
the data:
• the relay log
• the temporary file
• the data itself
Monday, April 11, 2011 69

70. Gotchas: default engine
• If you define storage_engine in the
master, it does not get replicated
Monday, April 11, 2011 70

71. Gotchas: binlog format
• In circular replication, and relay slaves
• Changes to binlog_format are not
propagated
Monday, April 11, 2011 71

72. Gotchas: binlog format
server id: 101 insert into t1
format: statement values (@@server_id) 101
server id: 102
format: row
102
server id: 103 102
format: row
Monday, April 11, 2011 72

73. Gotchas: set global not really global
• set global binlog_format=row
• Works for all new connections after the
change
• It does not work for the event
scheduler!
Monday, April 11, 2011 73

74. Gotchas: triggers and row replication
• In statement based replication, triggers
are fired both on master and slave
• in row-based replication, triggers are
fired on master only, and the resulting
rows are moved to the slave
Monday, April 11, 2011 74

75. DA
AG
EN Replication power techniques
and features
• row-based
• semi-synch
• delayed
• Circular
• bi-directional
• Leveraging replication
Monday, April 11, 2011 75

76. DA
AG
EN row-based replication
Monday, April 11, 2011 76

77. row-based replication
• Available in 5.1 and later
• can be changed using “binlog_format”
• row
• mixed
• statement (default)
Monday, April 11, 2011 77

78. row-based replication
• DO
• when you have expensive queries in the
master
• when you are using non-deterministic
functions
• DON’T
• When you are updating a lot of rows
Monday, April 11, 2011 78

79. DA
AG
EN semi-synch replication
Monday, April 11, 2011 79

80. semi-synchronous
replication
• Available in 5.5 and higher
• Makes sure that at least one slave has
copied the data.
• Increases reliability
Monday, April 11, 2011 80

81. client
master transaction
1 with regular
commit replication
execute 2
slave
returns
4 to client
binary log 3
5 replication
81
Monday, April 11, 2011 81

82. client transaction
master
1 with semi-
commit synchronous
replication
execute 2
slave
7 returns
to client
binary log 3
sends
4 transaction relay log 5
to slave
gets
6 acknowledgement
82
Monday, April 11, 2011 82

83. semi-synchronous
replication in practice
• installation:
• it’s a plugin.
• Actually, two plugins
Monday, April 11, 2011 83

84. semi-synch replication install
# in the master
plugin-load=rpl_semi_sync_master=semisync_master.so
rpl_semi_sync_master_enabled=1
# in each slave
plugin-load=rpl_semi_sync_slave=semisync_slave.so
rpl_semi_sync_slave_enabled=1
# restar all servers
Monday, April 11, 2011 84

85. semi-synch replication check
# in the master
show variables like 'rpl_semi%';
+------------------------------------+-------+
| Variable_name | Value |
+------------------------------------+-------+
| rpl_semi_sync_master_enabled | ON |
| rpl_semi_sync_master_timeout | 10000 |
| rpl_semi_sync_master_trace_level | 32 |
| rpl_semi_sync_master_wait_no_slave | ON |
+------------------------------------+-------+
Monday, April 11, 2011 85

86. semi-synch replication check
show status like "rpl_semi_%tx";
+-----------------------------+-------+
| variable_name | value |
+-----------------------------+-------+
| RPL_SEMI_SYNC_MASTER_NO_TX | 0 |
| RPL_SEMI_SYNC_MASTER_YES_TX | 0 |
+-----------------------------+-------+
Monday, April 11, 2011 86

87. semi-synch replication test
master> create table t1 ( i int);
Query OK, 0 rows affected (0.01 sec)
master> show status like "rpl_semi_%tx";
+-----------------------------+-------+
| Variable_name | Value |
+-----------------------------+-------+
| Rpl_semi_sync_master_no_tx | 0 |
| Rpl_semi_sync_master_yes_tx | 1 |
+-----------------------------+-------+
Monday, April 11, 2011 87

88. disabling semi-synch
# for each slave
set global rpl_semi_sync_slave_enabled=0;
stop slave io_thread;
start slave io_thread;
Monday, April 11, 2011 88

89. disabled semi-synch replication test
master> insert into t1 values (1);
Query OK, 1 row affected (10.00 sec)
master> show status like "rpl_semi_%tx";
+-----------------------------+-------+
| Variable_name | Value |
+-----------------------------+-------+
| Rpl_semi_sync_master_no_tx | 1 |
| Rpl_semi_sync_master_yes_tx | 1 |
+-----------------------------+-------+
2 rows in set (0.00 sec)
Monday, April 11, 2011 89

90. disabled semi-synch replication test
master> insert into t1 values (2);
Query OK, 1 row affected (0.01 sec)
master> show status like "rpl_semi_%tx";
+-----------------------------+-------+
| Variable_name | Value |
+-----------------------------+-------+
| Rpl_semi_sync_master_no_tx | 2 |
| Rpl_semi_sync_master_yes_tx | 1 |
+-----------------------------+-------+
2 rows in set (0.00 sec)
Monday, April 11, 2011 90

91. re-enabling semi-synch
# in one slave
set global rpl_semi_sync_slave_enabled=1;
stop slave io_thread;
start slave io_thread;
Monday, April 11, 2011 91

92. reenabled semi-synch replication test
master> insert into t1 values (3);
Query OK, 1 row affected (0.01 sec)
master> show status like "rpl_semi_%tx";
+-----------------------------+-------+
| Variable_name | Value |
+-----------------------------+-------+
| Rpl_semi_sync_master_no_tx | 2 |
| Rpl_semi_sync_master_yes_tx | 2 |
+-----------------------------+-------+
2 rows in set (0.00 sec)
Monday, April 11, 2011 92

93. DA
AG
EN delayed replication
Monday, April 11, 2011 93

94. delayed replication in practice
STOP SLAVE;
change master to master_delay=60;
START SLAVE;
Monday, April 11, 2011 94

95. delayed replication
• DEMO
Monday, April 11, 2011 95

96. DA
AG
EN Circular replication
Monday, April 11, 2011 96

97. Circular
A
A MySQL B
DBMS MySQL
DBMS B
master
master
MySQL
DD
DBMS MySQL
DBMS
C
C
Monday, April 11, 2011
master 97

98. DA
AG
EN bi-directional replication
Monday, April 11, 2011 98

99. bi-directional replication
MySQL MySQL
DBMS DBMS
Monday, April 11, 2011 99

100. bi-directional replication
MySQL MySQL
DBMS DBMS
Los Angeles New York
MySQL
MySQL MySQL DBMS
DBMS DBMS
MySQL
DBMS
Monday, April 11, 2011 100

101. DA
AG
EN Leveraging replication
Monday, April 11, 2011 101

102. Overview of MySQL partitions
Monday, April 11, 2011 102

103. Partition pruning
1a - unpartitioned table - SINGLE RECORD
I
D select *
N from table_name
A
D where colx =
T
E 120
A
X
Monday, April 11, 2011 103

104. Partition pruning
2a - table partitioned by colx - SINGLE REC
1-99
100-199 select *
from table_name
200-299
where colx =
120
300-399
400-499
500-599
Monday, April 11, 2011 104

105. Partition pruning
1a - unpartitioned table - RANGE
select *
I from
D
N table_name
A where colx
D between 120
T
E and 230
A
X
Monday, April 11, 2011 105

106. Partition pruning
2a - table partitioned by colx - RANGE
1-99
select *
100-199 from
table_name
200-299 where colx
between 120
and 230
300-399
400-499
500-599
Monday, April 11, 2011 106

107. Storage comparison
engine storage
(GB)
innodb (with PK) 330
myisam (with PK) 141
archive 13
innodb partitioned (no PK) 237
myisam partitioned (no PK) 107
archive partitioned 13
Monday, April 11, 2011 107

108. Benchmarking results
engine 6 month range
InnoDB 4 min 30s
MyISAM 25.03s
Archive 22 min 25s
InnoDB partitioned by month 13.19
MyISAM partitioned by year 6.31
MyISAM partitioned by month 4.45
Archive partitioned by year 16.67
Archive partitioned by month 8.97
Monday, April 11, 2011 108

109. Partitions limits
§ Little concurrency
§ Need of using the partitioning column
Monday, April 11, 2011 109

110. The ARCHIVE storage engine
Monday, April 11, 2011 110

111. Size matters
Innodb
Archive
Monday, April 11, 2011 111

112. REPLICATION PARTITIONING
large
data
ARCHIVE
Monday, April 11, 2011 112

113. Replication
=
let the slave do the dirty work
Monday, April 11, 2011 113

114. master
backup
slaves
STOP SLAVE
remove slave
perform
from load
backup
balancer
attach slave
START Let slave
to load
SLAVE catch up
balancer
Monday, April 11, 2011 114

115. master
make
summary
tables
slaves
STOP SLAVE
calculate remove slave
summary from load
tables balancer
attach slave
Let slave
START SLAVE to load
catch up
balancer
Monday, April 11, 2011 115

116. master
Partitions
for heavy innodb
non partitioned
statistics
slave slave
innodb
innodb non partitioned
partitioned by range
slave
MyISAM
partitioned by range
Monday, April 11, 2011 116

117. Simulating master
multiple innodb
non partitioned
dimensions
slave
slave
innodb
non partitioned
ARCHIVE
partitioned by range
(date)
slave
slave
ARCHIVE
partitioned by range
ARCHIVE (location)
partitioned by range
(product)
Monday, April 11, 2011 117

118. DA
AG
EN Tools and tips
• MySQL Sandbox
• Replaying binary logs
• Filtering binary logs
• Baron Schwartz’s Maatkit
• Shlomi Noach’s openark kit
• Facebook’s Online Schema Change
• Measuring replication speed
• Detecting if replication is on
Monday, April 11, 2011 118

119. DA
AG
EN MySQL Sandbox
Monday, April 11, 2011 119

120. MySQL Sandbox
http://mysqlsandbox.net
• Free software (Perl under GPL)
• One (unix) host
• Many database servers
• Single or multiple sandboxes
• Customized scripts to use the servers
• Standard or circular replication
• Installs IN SECONDS
Monday, April 11, 2011 120

121. overview
MySQL MySQL
server server
Data DB1 Data DB1
DB2 DB3
DATA DIRECTORY DB2 DB3
PORT
SOCKET
Monday, April 11, 2011 121

122. overview
MySQL MySQL
server server
Data DB1
SAME Data DB1
DB2 DB3
DATA DB2 DB3
DIRECTORY?
/var/lib/mysql /var/lib/mysql
DATA CORRUPTION
Monday, April 11, 2011 122

123. overview
MySQL MySQL
server server
SAME
PORT or
SOCKET?
3306 3306
/tmp/mysql.sock /tmp/mysql.sock
DOES NOT START
Monday, April 11, 2011 123

124. The hard way (1)
Monday, April 11, 2011 124

125. the hard way (2)
Monday, April 11, 2011 125

126. The easy way
$ make_sandbox
/path/to/mysql-5.1.54_linux.tar.gz
# it should work always
Monday, April 11, 2011 126

127. The easier way
$ make_sandbox 5.1.54
# Needs some preliminary work
Monday, April 11, 2011 127

128. The easiest way
$ sb 5.1.54
# Needs the same preliminary work
Monday, April 11, 2011 128

129. MySQL Sandbox
VERSION
MySQL
server
Data DB1
DB2 DB3
$SANDBOX_HOME/msb_VERSION/data
VERSION
/tmp/mysql_VERSION.sock
Monday, April 11, 2011 129

130. MySQL Sandbox
5.1.54
MySQL
server
Data DB1
DB2 DB3
$SANDBOX_HOME/msb_5_1_54/data
5154
/tmp/mysql_5154.sock
Monday, April 11, 2011 130

131. MySQL Sandbox
5.5.9
MySQL
server
Data DB1
DB2 DB3
$SANDBOX_HOME/msb_5_5_09/data
5509
/tmp/mysql_5509.sock
Monday, April 11, 2011 131

132. Single Sandbox
MySQL customized scripts
server
start
stop
restart
status
clear
send_kill
use
Monday, April 11, 2011 132

133. Multiple Sandbox
MySQL customized scripts
server
start_all
stop_all
restart_all m n1
status_all s1 n2
clear_all s2 n3
send_kill_a
ll
use_all
Monday, April 11, 2011 133

134. Where do you get it
•from CPAN
sudo cpan MySQL::Sandbox
•from launchpad
http://launchpad.net/mysql-sandbox
Monday, April 11, 2011 134

135. The easy replication way
$ make_replication_sandbox
/path/to/mysql-5.1.54_linux.tar.gz
# or, after some preparation
$ make_replication_sandbox 5.1.54
Monday, April 11, 2011 135

136. default architecture
$HOME
/sandboxes opt
expanded
tarballs
$SANDBOX_HOME
mysql
$SANDBOX_BINARY
installed
sandboxes
Monday, April 11, 2011 136

137. default architecture
$HOME
/sandboxes opt
msb_5_0_91
mysql
msb_5_1_48
5.0.91
rsandbox_5_1_48
5.1.45
master
5.1.48
node1
5.5.4
node2
Monday, April 11, 2011 137

138. Monday, April 11, 2011 138

139. creating a single sanbox
make_sandbox
/path/to/mysql-X.X.XX-OS.tar.gz
Monday, April 11, 2011 139

140. using a single sanbox
# after
# make_sandbox
# /path/to/mysql-X.X.XX-OS.tar.gz
$ cd $SANDBOX_HOME/msb_X_X_XX
$ ./use
Monday, April 11, 2011 140

141. creating a single sanbox
with a specific options file
make_sandbox
/path/to/mysql-X.X.XX-OS.tar.gz
--my_file=/path/to/my.cnf
Monday, April 11, 2011 141

142. easily create a sandbox after the first
one
The long way
$ cd $HOME/opt/mysql # $SANDBOX_BINARY
$ gunzip -c
/path/to/mysql-5.1.34-osx10.5-x86.tar.gz
| tar -xf -
$ mv mysql-5.1.34-osx10.5-x86 5.1.34
$ make sandbox 5.1.34
Monday, April 11, 2011 142

143. easily create a sandbox after the first
one
The short way
$ make_sandbox
path/to/mysql-5.1.34-osx10.5-x86.tar.gz
--export_binaries
Monday, April 11, 2011 143

144. starting a single sanbox
$ cd $SANDBOX_HOME/msb_X_X_XX
$ ./start
Monday, April 11, 2011 144

145. starting a single sanbox
with temporary options
$ cd $SANDBOX_HOME/msb_X_X_XX
$ ./start --option=value
$ ./restart --option=value
$ ./start --key-buffer=20000000
Monday, April 11, 2011 145

146. creating a sandbox with custom port
and directory
$ make_sandbox 5.1.34
--sandbox_port=7800
--sandbox_directory=mickeymouse
Monday, April 11, 2011 146

147. creating a sandbox with automatic
port checking
$ make_sandbox 5.1.34 --check_port
# if 5.1.34 is free
# port=5134
# directory=msb_5_1_34
# else
# port=5135 (or the first free)
# directory=msb_5_1_34_a
Monday, April 11, 2011 147

148. create a replication sandbox
$ make_replication_sandbox
path/to/mysql-5.1.34-osx10.5-x86.tar.gz
Monday, April 11, 2011 148

149. create a circular replication sandbox
$ make_replication_sandbox
--circular=4
path/to/mysql-5.1.34-osx10.5-x86.tar.gz
Monday, April 11, 2011 149

150. changing port to an existing sandbox
$ sbtool -o port
-s /path/to/source/sandbox
--new_port=XXXX
Monday, April 11, 2011 150

151. installing the innodb plugin
$ sbtool -o plugin
--plugin=innodb
-s /path/to/source/sandbox
Monday, April 11, 2011 151

152. creating a replication sandbox with
new base port
$ make_replication_sandbox
--replication_directory=newwdir
--check_base_port 5.0.79
# Creates a replication directory under
# $SANDBOX_HOME/newdir
# The previous one is preserved.
# No conflicts happen
Monday, April 11, 2011 152

153. DA
AG
EN Replaying binary logs
Monday, April 11, 2011 153

154. DA
AG
EN Filtering binary logs
Monday, April 11, 2011 154

155. DA
AG
EN Maatkit
Monday, April 11, 2011 155

156. maatkit
• mk-heartbeat Monitor MySQL replication delay.
• mk-purge-logs Purge binary logs on a master based on purge rules.
• mk-slave-delay Make a MySQL slave server lag behind its master.
• mk-slave-find Find and print replication hierarchy tree of MySQL
slaves.
• mk-slave-move Move a MySQL slave around in the replication
hierarchy.
• mk-slave-prefetch Pipeline relay logs on a MySQL slave to pre-warm
caches.
• mk-slave-restart Watch and restart MySQL replication after errors.
• mk-table-checksum Perform an online replication consistency check,
or checksum MySQL tables efficiently on one or many servers.
• mk-table-sync Synchronize MySQL table data efficiently.
• http://www.maatkit.org/
Monday, April 11, 2011 156

157. Facebook Online Schema Change
DA
N
AGE
Monday, April 11, 2011 157

158. Facebook Online Schema Change
• Based on one of Shlomi Noach OpenArk utilities
• Developed in PHP by Facebook
• http://www.facebook.com/note.php?note_id=430801045932
Monday, April 11, 2011 158

159. DA
AG
EN OpenArk
Monday, April 11, 2011 159

160. openark
• oak-get-slave-lag: print slave replication lag and
terminate with respective exit code.
• oak-online-alter-table: perform a non-blocking ALTER
TABLE operation.
• oak-purge-master-logs: purge master logs, depending
on the state of replicating slaves.
• oak-show-replication-status: show how far behind are
replicating slaves on a given master.
• http://code.openark.org/forge/openark-kit
Monday, April 11, 2011 160

161. DA
AG
EN Check replication speed
Monday, April 11, 2011 161

162. check replication speed
• Write a time to the master
• Retrieve it from the slave
• Compare times
• http://forge.mysql.com/tools/tool.php?
id=5
Monday, April 11, 2011 162

163. DA
AG
EN Replication outside the box
• Seamless failover
• Parallel replication
• Multiple source replication
• Multiple master replication
Monday, April 11, 2011 163

164. Advanced MySQL
Replication for the
masses
Monday, April 11, 2011 164

165. Once Upon A Time,
In The Life Of A
Database
Consultant ...
Monday, April 11, 2011 165

166. The story of a steel
foundry
• Used MySQL databases to store
production monitoring data
• Inserted a zillion records per second.
• Slaves often lagged behind
Monday, April 11, 2011 166

167. transaction
transaction
MySQL transaction
transaction
transaction
DBMS transaction
transaction
transaction
transaction
transaction
transaction
transaction
BINARY LOG
ns act act tio n
tra ansio ns sac ntio
ac ion ion n
ac
nr tr tra aniton
tsa a n csio
nts ct n
rn a o
taa scti
r
trnsa
n
n
tio
ct
tra
tra
REPLICATION
transaction
transaction
MySQL
DBMS
Monday, April 11, 2011 167

168. The story of a shoe
maker
• Had a successful business, spread to a
dozen stores.
• Needed to aggregate the data from the
stores in his headquarters database.
Monday, April 11, 2011 168

169. MySQL
MySQL
DBMS MySQL
DBMS
DBMS
store
store store
MySQL
DBMS
headquarters
Monday, April 11, 2011 169

170. The story of a widgets
seller
• Had a successful business, designed for one
server.
• Products were created in several sites.
• Needed to allow insertions from more
than one master at the time
Monday, April 11, 2011 170

171. MySQL MySQL
DBMS DBMS
master master
MySQL MySQL
DBMS DBMS
master master
Monday, April 11, 2011 171

172. All these stories tell us:
Nice dream, but MySQL
can’t do it
Monday, April 11, 2011 172

173. Enter Tungsten Replicator
Monday, April 11, 2011 173

174. Tungsten Replicator 2.0
• What is it?
Monday, April 11, 2011 174

175. http://code.google.com/p/tungsten-replicator
Open Source
100% GPL v2
Monday, April 11, 2011 175

176. What can it do?
• Easy failover
• Multiple masters
• Multiple sources to a single slave
• Parallel replication
• Replicate to Oracle and PostgreSQL database
Monday, April 11, 2011 176

177. MySQL to foreign services
MySQL setup
to run as
MySQL master
Master DB
Data is applied to
PostgreSQL or
Oracle
Bin
Logs
(Binlogs
enabled) Slave DB
© Continuent 2010
Monday, April 11, 2011 177

178. From the beginning ...
Monday, April 11, 2011 178

179. Fail-over (1)
Monday, April 11, 2011 179

180. Fail-over (2)
Monday, April 11, 2011 180

181. Fail-over (3)
Monday, April 11, 2011 181

182. Fail-over (4)
Monday, April 11, 2011 182

183. Fail-over (5)
Monday, April 11, 2011 183

184. Fail-over (6)
Monday, April 11, 2011 184

185. Fail-over (7)
Monday, April 11, 2011 185

186. Fail-over (8)
Monday, April 11, 2011 186

187. Failover
• DEMO
Monday, April 11, 2011 187

188. master/slave with an attitude
Monday, April 11, 2011 188

189. The steel foundry dream
or parallel replication
From here ...
Monday, April 11, 2011 189

190. The steel foundry dream
or parallel replication
To here.
Monday, April 11, 2011 190

191. Parallel replication facts
• Sharded by database
• Good choice for slave lag problems
• Bad choice for single database projects
Monday, April 11, 2011 191

192. Testing parallel
replication
Monday, April 11, 2011 192

193. db0
preparation (1)
db1 sysbench
db2 sysbench
db3 sysbench
db4 sysbench
db5 sysbench
db6 sysbench
db7 sysbench
db8 sysbench
db9 sysbench
sysbench
Monday, April 11, 2011 193

194. preparation (2)
db0
db1
db2
db3
db4
db5
db6
db7
db8
db9
Monday, April 11, 2011 194

195. before the test (1)
db0
db1
db2
db3
db4
db5
db6
db7
db8
db9
Monday, April 11, 2011 195

196. before the test (2)
SQL thread
RELAY logs
IO thread
binary logs
MySQL slave
RELAY logs
Tungsten slave direct:
alpha
(slave)
replicator alpha
Monday, April 11, 2011 196

197. starting the test
SQL thread
RELAY logs
IO thread
binary logs
MySQL slave
RELAY logs
Tungsten slave direct:
alpha
(slave)
replicator alpha
Monday, April 11, 2011 197

198. MySQL native
replication
slave catch up in 00:02:30
Monday, April 11, 2011 198

199. Tungsten parallel
replication
slave catch up in 00:01:20
Monday, April 11, 2011 199

200. The widget seller
dream, or multi masters
• Tungsten Replicator recipe: use more
services
Monday, April 11, 2011 200

201. Bi-directional replication
Monday, April 11, 2011 201

202. Bi-directional replication with slaves
Monday, April 11, 2011 202

203. True multiple master
We’ll see that in a moment.
But first
Monday, April 11, 2011 203

204. The shoe maker dream,
or multiple sources
• Tungsten Replicator recipe is still valid: use
more services
Monday, April 11, 2011 204

205. Multiple source replication
Monday, April 11, 2011 205

206. Multiple masters replication: 3 nodes
Monday, April 11, 2011 206

207. Multiple masters replication: 4 nodes
Monday, April 11, 2011 207

208. Updating 4 masters : 1 flow
Monday, April 11, 2011 208

209. Updating 4 masters : 2 flows
Monday, April 11, 2011 209

210. Updating 4 masters : 3 flows
Monday, April 11, 2011 210

211. Updating 4 masters : 4 flows
Monday, April 11, 2011 211

212. Tungsten in practice
• installation
Monday, April 11, 2011 212

213. Installation:
the long way
• Get the binaries
• Expand the tarball
• Check the requirements in the manual
• Run ./configure
• Run ./configure-service
Monday, April 11, 2011 213

214. Installation:
the quick way
• Get the tarball
• Get the tungsten deployer from Google
Code
• Sit back and enjoy!
Monday, April 11, 2011 214

215. Tools
• replicator
• trepctl
• thl
Monday, April 11, 2011 215

216. replicator
• It’s the service provider
• You launch it once when you start
• You may restart it when you change config
Monday, April 11, 2011 216

217. trepctl
• Tungsten Replicator ConTroLler
• It’s the driving seat for your replication
• You can start, update, and stop services
• You can get specific info
Monday, April 11, 2011 217

218. thl
• Transaction History List
• Gives you access to the Tungsten relay logs
Monday, April 11, 2011 218

219. DA
AG
EN More info on replication
Monday, April 11, 2011 219

220. Books
Monday, April 11, 2011 220

221. High Performance MySQL
Monday, April 11, 2011 221

222. MySQL High Availability
Monday, April 11, 2011 222

223. Web Operations
Monday, April 11, 2011 223

224. Cloud Application Architectures
Monday, April 11, 2011 224

225. Talks
• Monday, 1:30pm “Learn how to cure replication
deprivation with Tungsten”
• Tuesday 10:50am “Replication Update”
• Tuesday 2:00pm “Diagnosing replication failures”
• Tuesday 3:05 “Advanced replication monitoring”
• Thursday 10:50am “Replication for Availability &
Durability with MySQL and Amazon RDS”
• Thursday 11:55am “Build your own PaaS for MySQL
with Tungsten Enterprise”
• Thursday 2:00pm “Error detection and correction with
MySQL Replication”
• Thursday 2:50pm “Performance comparisons and
trade-offs for various MySQL replication schemes”
Monday, April 11, 2011 225

226. ADVERTISING
Monday, April 11, 2011 226

227. WE ARE HIRING!
QA and support
engineers
http://www.continuent.com/about/careers
Monday, April 11, 2011 227

228. http://opendbcamp.org
Open Database Camp
Sardinia
Technology Park
6-7-8 May 2011
Monday, April 11, 2011 228

229. Contact Information
Worldwide
560 S. Winchester Blvd., Suite 500
San Jose, CA 95128
Tel (866) 998-3642
Fax (408) 668-1009
e-mail: sales@continuent.com
Continuent Web Site:
http://www.continuent.com
Tungsten Project
http://code.google.com/p/tungsten-replicator
© Continuent 2011
Monday, April 11, 2011 229