This session was presented in Open source India 2013. The presentation covers MySQL multi-source replication which is released under MySQL Labs.

1. MySQL Labs :
Multi Source Replication
Shivji Kumar Jha,
Software Developer,
MySQL Replication Team
1
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

2. Safe Harbour Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract.
It is not a commitment to deliver any material, code, or functionality, and should not
be relied upon in making purchasing decisions. The development, release, and
timing of any features or functionality described for Oracle’s products remains at the
sole discretion of Oracle.
2
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

3. MySQL Labs features are not fit for production.
They are provided solely for testing purposes, to try the latest bug fixes and
generally to keep up with the development.
Please, do not use these binaries in production.
Instead, install them on a spare server.
If you are looking for production ready binaries, please visit MySQL Downloads.
3
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

4. 
Agenda
Background: Why Use Replication?

Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitoring

4
Reading More about Multi-source Replication
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

5. Background: What is Replication Used For?
 Read scale-out
S
M
S
More
reads?
More
slaves!
M
S
S
write clients
write clients
5
read clients
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
read clients

6. What about Write scale-out?
M?
M
More
writes?
More
Masters?
M?
M?
write clients
Enter MySQL Fabric ...
6
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
write clients

7. Background: What is Replication Used For?
 Redundancy: If master crashes, promote slave to master
B
A
B
Uh Oh!
C
B
Whew!
Crash
A
A
C
C
B is the
new master
7
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

8. Background: What is Replication Used For?
 On-line Backup and Reporting
S
M


write clients

8
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

business intelligent client apps
reporting client apps
big queries client apps

9. Background: What is Replication Used For?
A
C
A
B
B
C
Image from
www.ginkgomaps.com
9
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

10. 
Agenda
Background: Why Use Replication?

Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitoring

10
Reading More about Multi-source Replication
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

11. Multi-Source Replication: Introduction
 A Slave can have more than one master.
M1
M2
M3
M4
11
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
S

12. Multi-Source Replication: Introduction
 A Slave can have more than one master.
db1
12
M1
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

13. Multi-Source Replication: Introduction
 A Slave can have more than one master.
db1
db2
13
M1
M2
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

14. Multi-Source Replication: Introduction
 A Slave can have more than one master.
db1
db2
M2
db3
M3
db4
14
M1
M4
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

15. Multi-Source Replication: Introduction
 A Slave can have more than one master.
db1
M1
db2
M2
S
db3
db4
15
M3
M4
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
db1
db2
db3
db4

16. 
Agenda
Background: Why Use Replication?

Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitoring

16
Reading More about Multi-source Replication
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

17. Multi-Source Replication: Use Cases
 The main use cases of Multi-source replication are related to data aggregation.
17
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

18. Multi-Source Replication: Use Cases
Database 1
M1
Database 2
Database 1, 2, 3
M2
Database 3
M3
18
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
S
Business Intelligence
Data analytics
Backup
etc

19. Multi-Source Replication: Use Cases
Shard 1
M
Shard 2
Full table
M
Shard 3
M
19
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
S

20. 
Agenda
Background: Why Use Replication?

Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitoring

20
Reading More about Multi-source Replication
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

21. Multi-Source Replication: More details..
 Slave can have more than one
master.
M1
– Receive transactions from
several MySQL servers
simultaneously.
M2
S
– Apply transactions from
different masters
simultaneously.
M3
– No conflict detection or
resolution.
M4
21
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

22. Multi-Source Replication: What is a Channel?
 A channel is an an abstraction for a sender-receiver-applier path.
Client
Insert...
Receiver
thread
Sender
thread
Applier
thread
binary log
binary log
Insert...
A
relay log
Insert...
Insert...
Network
Channel
22
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
B

23. Multi-Source Replication: How Many Channels?
 Number of channels on slave = Number of sources.
M1
Network
relay log1
M2
binary log
Network
relay log2
M3
23
binary log
binary log
Network
relay log3
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
S
binary log

24. Multi-Source Replication: Slave Appliers
 Each channel has its own single-threaded slave applier.
M1
Network
relay log1
M2
binary log
Network
relay log2
M3
24
binary log
binary log
Network
relay log3
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
S
binary log

25. Multi-Source Replication: Slave Appliers
 Each channel has its own multi-threaded slave applier.
M1
Network
relay log1
M2
binary log
Network
relay log2
M3
25
binary log
binary log
Network
relay log3
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
S
binary log

26. Adding / Configuring A Channel
 Each channel can be configured individually.
 To add or alter a channel configuration use:
CHANGE MASTER TO master_def …
FOR CHANNEL=”<channel_name>”
26
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

27. Replication Commands on a Channel
 The FOR CHANNEL=”<channel_name>” clause is consistent
across different replication commands.
– start slave [...]
FOR CHANNEL=”<channel_name>"
– stop slave [thread_types] FOR CHANNEL=”<channel_name>”
– reset slave [all]
FOR CHANNEL=”<channel_name>”
– flush relay logs
FOR CHANNEL=”<channel_name>”
– show relay log events
FOR CHANNEL=”<channel_name>”
...
27
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

28. Working With All Channels
 The FOR ALL CHANNELS clause.
– START SLAVE [thread_type] FOR ALL CHANNELS
– STOP SLAVE [thread_type]
28
FOR ALL CHANNELS
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

29. Compatibility With Other Replication Features
Multi-Threaded Slaves
Global Transaction Identifiers
Yes
Semi-Synchronous Replication
Yes
Crash Safe Slaves
Yes
Filters
29
Yes
Yes
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

30. 
Agenda
Background: Why Use Replication?

Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitoring

30
Reading More about Multi-source Replication
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

31. Replication Monitoring: P_S Replication Tables
Slave Status
Connection
Configuration
Connection
Status
Execution
Configuration
Execution
Status
Applier / Coordinator
Status
31
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Workers
Status

32. Replication Monitoring: P_S Replication Tables
 We have six performance schema tables for replication (MySQL-5.7.2):
– replication_connection_configuration
– replication_connection_status
One row per CHANNEL
– replication_execute_configuration
– replication_execute_status
– replication_execute_status_by_coordinator
– replication_execute_status_by_worker
One row per CHANNEL per WORKER
 Consistent semantics across tables. Lets explore one of them.
32
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

33. Replication Monitoring: Connection Status
mysql> select * from performance_schema.replication_connection_statusG
*************************** 1. row ***************************
CHANNEL_NAME
: CHANNEL1
SOURCE_UUID
: 7cff7406-23ca-11e3-ac3e-5c260a83b12b
THREAD_ID
: 13
SERVICE_STATE
: ON
RECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4
LAST_ERROR_NUMBER
:0
LAST_ERROR_MESSAGE
:
LAST_ERROR_TIMESTAMP
: 0000-00-00 00:00:00
*************************** 2. row ***************************
CHANNEL_NAME
: CHANNEL2
...
One row per CHANNEL
33
33
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

34. Replication Monitoring: Connection Status
mysql> select * from performance_schema.replication_connection_statusG
*************************** 1. row ***************************
CHANNEL_NAME
: CHANNEL1
SOURCE_UUID
: 7cff7406-23ca-11e3-ac3e-5c260a83b12b
THREAD_ID
: 13
SERVICE_STATE
: ON
RECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4
LAST_ERROR_NUMBER
:0
LAST_ERROR_MESSAGE
:
Receiver thread &
LAST_ERROR_TIMESTAMP
: 0000-00-00 00:00:00
*************************** 2. row *************************** its service state
CHANNEL_NAME
: CHANNEL2
...
34
34
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

35. Replication Monitoring: Connection Status
mysql> select * from performance_schema.replication_connection_statusG
*************************** 1. row ***************************
CHANNEL_NAME
: CHANNEL1
SOURCE_UUID
: 7cff7406-23ca-11e3-ac3e-5c260a83b12b
THREAD_ID
: 13
SERVICE_STATE
: ON
RECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4
LAST_ERROR_NUMBER
:0
LAST_ERROR_MESSAGE
:
LAST_ERROR_TIMESTAMP
: 0000-00-00 00:00:00
*************************** 2. row ***************************
CHANNEL_NAME
: CHANNEL2
Set of transactions received
...
through this channel
35
35
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

36. Replication Monitoring: Connection Status
mysql> select * from performance_schema.replication_connection_statusG
*************************** 1. row ***************************
CHANNEL_NAME
: CHANNEL1
SOURCE_UUID
: 7cff7406-23ca-11e3-ac3e-5c260a83b12b
THREAD_ID
: 13
SERVICE_STATE
: ON
RECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4
LAST_ERROR_NUMBER
: 1045
LAST_ERROR_MESSAGE
: error connecting to master 'replssl@127.0.0.1:13000' ...
LAST_ERROR_TIMESTAMP
: 2013-11-04 13:37:23
*************************** 2. row ***************************
CHANNEL_NAME
: CHANNEL2
...
Oops! There was an error in the
receiver thread on this channel
36
36
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

37. Replication Monitoring: Connection Status
mysql> select * from performance_schema.replication_connection_statusG
*************************** 1. row ***************************
CHANNEL_NAME
: CHANNEL1
Receiver
SOURCE_UUID
: 7cff7406-23ca-11e3-ac3e-5c260a83b12b thread on this
channel stopped serving...
THREAD_ID
: NULL
SERVICE_STATE
: OFF
RECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4
LAST_ERROR_NUMBER
: 1045
LAST_ERROR_MESSAGE
: error connecting to master 'replssl@127.0.0.1:13000' ...
LAST_ERROR_TIMESTAMP
: 2013-11-04 13:37:23
*************************** 2. row ***************************
CHANNEL_NAME
: CHANNEL2
...
37
37
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

38. 
Agenda
Background: Why Use Replication?

Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitoring

38
Reading More about Multi-source Replication
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

39. Read More About Multi Source Replication



39
Read more about Multi-source replication from the design notes:
http://dev.mysql.com/worklog/task/?id=1697
Read a feature preview of Multi-source replication on Rith's blog:
http://on-mysql-replication.blogspot.in/2013/09/feature-preview-mysqlmulti-source-replication.html
Read more about Multi-source replication on Rith's blog:
http://on-mysql-replication.blogspot.in/2013/09/mysql-labs-multi-sour
ce-replication.html
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

40. Read More About Multi Source Replication


40
Official MySQL documentation for replication P_S tables:
http://dev.mysql.com/doc/refman/5.7/en/performance-schema-replicati
on-tables.html
Read more about replication P_S tables on Shiv's blog:
http://shivjijha.blogspot.com/2013/09/Monitoring-Replication-with-t
he-NEW-performance-schema-tables.html
| Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |