SlideShare une entreprise Scribd logo
1  sur  55
Télécharger pour lire hors ligne
MySQL Parallel Replication:
all the 5.7 and 8.0 details
(LOGICAL_CLOCK)
by Jean-François Gagné
System and MySQL Expert at HubSpot
Presented at Percona Live Austin 2022
jfg.mysql AT gmail DOT com
Twitter: @jfg956
Benchmark Preview
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
2
Replication throughput (TPS) without and with parallel replication
Terminology [1 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
MySQL Terminology Updates in 2020:
https://dev.mysql.com/blog-archive/mysql-terminology-updates/
• master / slave à primary / replica
• the master of a slave à the source of a replica
This presentation uses above terminology, with below additions:
• Multi-Threaded Slave (MTS) à Multi-Threaded Replication (MTR)
• intermediary master à intermediary source or non-primary source
(Exceptions for citing content from before the terminology change)
3
Terminology [2 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
From MySQL 8.0.22 (not in 5.7), new commands were introduced:
• START/STOP SLAVE à START/STOP REPLICA
• And more…
• Some missing: START REPLICA UNTIL SQL_AFTER_MTS_GAPS
From 8.0.26 (not in 5.7 either), new variables were introduced:
• log_slave_updates à log_replica_updates
• slave_parallel_workers à replica_parallel_workers
• And more…
Because this talk applies to 5.7, and not everyone is on 8.0-latest,
this presentation uses old command and variable names
Summary
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
• Replication reminders
• Why parallel replication is important (and complicated)
• How to enable parallel replication and what is LOGICAL_CLOCK
• How parallel replication works on replicas
• How dependency tracking works on primaries
• Benchmarks
• Closing thoughts
5
MySQL Replication
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
• The primary records transactions in a journal (binary logs), and replicas
• downloads the journal and saves them locally in the relay logs (IO thread)
• executes the relay logs on their local database (SQL thread)
• could also write binlogs to be themselves a source (log-slave-updates)
Other Replication Concepts: Durability
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Persistence / Durability (D of ACID):
• Is there data lost after a crash ?
(different types of crash: MySQL process (mysqld) or OS (Linux))
• Parameters controlling durability: sync_binlog and trx_commit
(trx_commit is short for innodb_flush_logs_at_trx_commit)
• High Durability (HD) or no data lost after crashes
(sync_binlog = 1 and trx_commit = 1)
(most expensive / slower because flush to disk after each trx)
• No Durability (ND sometimes called low durability)
(sync_binlog = 0 and trx_commit = 2)
(faster, with data lost on OS crash, but no loss on mysqld crash)
• (Data lost for mysqld crash in case trx_commit = 0 but not covered here) 7
Other Replication Concepts: Group Commit
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Group Commit is an optimization making high durability faster
• Before Group Commit (GC), each trx needed its disk flush
• With GC, many trx can be persisted in the same flush
à better write throughput on primaries with concurency
• (GC was introduced in MariaDB xx and then in MySQL 5.6)
• (Single threaded replication cannot Group Commit)
• (MariaDB 10.0 introduced Replica Group Commit)
8
From Single to Multi-Threaded Replication
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Without MTR, a single CPU is used for applying writes on replicas:
• on the primary, many CPUs are used for writes (one per session)
WO MTR, a single read IO can be “in flight” for writes on replicas:
• this is a bottleneck, especially for high latency disks (cloud)
• on primary, many IOs can be in flights for writes (one per session)
WO MTR, no group commit on replicas:
• on the p., a single flush can persist many trx in binlogs and redo logs
• on replicas, flushing each trx is a bottleneck
Multi-Threaded Replication is a solution to all above problems 9
Challenges of Multi-Threaded Replication
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
But running transactions in parallel on replica is not trivial:
• Updating a row cannot be run in parallel with the insert of this row
• MTR implementation needs to keep data consistency
Enabling MTR is also not straightforward:
• Many parameters and complex tunning
MTR also still has some rough edges:
• In 5.7, START SLAVE UNTIL not supported with MTR (ok from 8.0.0)
• Latest -1 Percona Server 5.7 crashes on STOP SLAVE with MTR
(PS-8030: 5.7.27 to .36 affected and 8.0.17 to .21, Oracle MySQL not affected)
• I have also seen deadlock with PS 5.7.36, and rumor of another crash
(to be investigated, unknown if both in Percona Server and Oracle MySQL)
Solutions for MTR Data Consistency
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
• MySQL 5.6: schema based parallel replication
• MariaDB 10.0: group commit based parallel replication
• MariaDB 10.1: optimistic parallel replication
• Improvement on 10.0, probably the fastest solution, but high CPU cost
• MySQL 5.7: same schema parallel replication (Logical Clock)
• Not as fast as optimistic, but not wasteful in CPU
• MySQL 8.0: Write-Set improvement (still Logical Clock)
• Lot to say, including need for RBR and back-ported in 5.7.22
This talk is about Logical Clock in 5.7 and 8.0 11
Solutions for MTR Data Consistency
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
• MySQL 5.6: schema based parallel replication
• MariaDB 10.0: group commit based parallel replication
• MariaDB 10.1: optimistic parallel replication
• Improvement on 10.0, probably the fastest solution, but high CPU cost
• MySQL 5.7: same schema parallel replication (Logical Clock)
• Not as fast as optimistic, but not wasteful in CPU
• MySQL 8.0: Write-Set improvement (still Logical Clock)
• Lot to say, including need for RBR and back-ported in 5.7.22
This talk is about Logical Clock in 5.7 and 8.0 12
Enabling Parallel Replication
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Set slave_parallel_type to LOGICAL_CLOCK
• This is the new default in 8.0.27
(was DATABASE before (schema-based solution of 5.6), including in 5.7)
(in 8.0.26, this variable is deprecated for dropping DATABASE support)
And set slave_preserve_commit_order to ON
• This is also the new default in 8.0.27(was OFF before, including in 5.7)
(more about this parameter later in the talk)
And set slave_parallel_workers to 2 or more
• New default to 4 in 8.0.27 (was 0 before including in 5.7)
These enable MTR, but are not sufficient for best throughput 13
What is LOGICAL_CLOCK ?
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
From the MySQL Reference Manual for LOGICAL_CLOCK:
• Transactions that are part of the same binary log group commit on a
source are applied in parallel on a replica.
• But this is not 100% exact (Bug#85977)
When using LOGICAL_CLOCK, MTR is scheduling trx on replica
using dependency tracking information from the binary logs:
• We will name these information Intervals
• By default, trx intervals are generated with Commit Timestamps
(which is close, but not strictly the same as group commit)
• Much better intervals can be generated with Write Set
(and Write Set is completely different to group commit) 14
Intervals
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Each trx is tagged with two numbers in the binlogs (starting with 5.7):
• sequence_number: increasing id for each trx (not to confuse with GTID)
• last_committed: id of the latest trx on which this trx depends
The last_committed / sequence_number is the trx interval
Here an example of intervals from a real system:
15
mysqlbinlog $f | awk '$11 ~ "last_committed"{print $1,$2,$11,$12}'
[...]
#170206 20:08:33 last_committed=6201 sequence_number=6203
#170206 20:08:33 last_committed=6203 sequence_number=6204
#170206 20:08:33 last_committed=6203 sequence_number=6205
#170206 20:08:33 last_committed=6203 sequence_number=6206
#170206 20:08:33 last_committed=6205 sequence_number=6207
Parallel Scheduling on Replicas [1 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
16
A transaction can be started when its last_committed and all
previous transactions have been executed
The intervals below mean that:
• When transaction #6203 (and all previous) have been executed,
transactions #6204, #6205 and #6206 can be started
• When transaction #6205 (and all previous) have been executed,
transaction #6207 can be started
#170206 20:08:33 last_committed=6201 sequence_number=6203
#170206 20:08:33 last_committed=6203 sequence_number=6204
#170206 20:08:33 last_committed=6203 sequence_number=6205
#170206 20:08:33 last_committed=6203 sequence_number=6206
#170206 20:08:33 last_committed=6205 sequence_number=6207
Parallel Scheduling on Replicas [2 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
17
Trx #6204 and #6205 (and #6206) can be run in parallel but #6205
could complete before #6204
slave_preserve_commit_order (SPCO) does what you expect:
• If SPCO is OFF, #6205 can commit before #6204
• If SPCO is ON , #6205 waits for #6204 completion before committing
SPCO to OFF generates data states that never existed on the primary
• Use with care (you probably want ON most of the time)
#170206 20:08:33 last_committed=6201 sequence_number=6203
#170206 20:08:33 last_committed=6203 sequence_number=6204
#170206 20:08:33 last_committed=6203 sequence_number=6205
#170206 20:08:33 last_committed=6203 sequence_number=6206
#170206 20:08:33 last_committed=6205 sequence_number=6207
Parallel Scheduling on Replicas [3 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
SPCO to OFF generates data states that never existed on the primary
• Use with care (you probably want ON most of the time)
Also, SPCO to OFF generates temporary gaps in GTIDs
• For filling the gaps, use START SLAVE UNTIL SQL_AFTER_MTS_GAPS
SPCO to ON needed log_slave_updates until 8.0.19 (Bug#75396)
Before 5.7.33 and 8.0.23, SPCO to ON could deadlock (Bug#89247)
• If replication gets stuck, kill -9 mysqld (no data lost unless trx_commit = 0)
• Thanks Percona / Venkatesh for the fixing this, but…
• But I have recently seen deadlocks in PS 5.7.36: not fully fixed 18
Parallel Scheduling on Replicas [4 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
19
When run in parallel, trx could fail to execute (below from the doc):
• If replication fails to execute a trx because of an InnoDB deadlock or
the trx's execution time exceeded innodb_lock_wait_timeout…
Such failed trx will be retried (below more from the doc):
• …it automatically retries slave_transaction_retries times before
stopping with an error.
The column COUNT_TRANSACTIONS_RETRIES from the the P_S
table replication_applier_status indicates how many retries
have happened, and the table repl._app._status_by_worker
contains details about the retries
Reminder: What is LOGICAL_CLOCK ?
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
From the MySQL Reference Manual for LOGICAL_CLOCK:
• Transactions that are part of the same binary log group commit on a
source are applied in parallel on a replica.
• But this is not 100% exact (Bug#85977)
When using LOGICAL_CLOCK, MTR is scheduling trx on replica
using dependency tracking information from the binary logs:
• We will name these information Intervals
• By default, trx intervals are generated with Commit Timestamps
(which is close, but not strictly the same as group commit)
• Much better intervals can be generated with Write Set
(and Write Set is completely different to group commit) 20
Commit Timestamps [1 of 3]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Commit Timestamps (CTS) is the default for 5.7 and 8.0
(binlog_transaction_dependency_tracking = COMMIT_ORDER)
It is a dependency tracking method exploiting parallelism on the
source to generate the intervals:
• While running a DML, the current trx notes the last committed trx
(this is where last_committed from mysqlbinlog is coming from)
(this is close to Group Commit, but not strictly the same)
Without tunning, CTS is not producing efficient intervals
(by default, MTR will not give much better replication speed) 21
Commit Timestamps [2 of 3]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
In below, the update id=3 ran while the update id=1 was not yet committed:
• Insert id=4 and update id=1 can run in parallel on a replica
• (Note that nothing commit at the same time à no group commit)
Session #1: CREATE TABLE t(id INT PRIMARY KEY, v INT);
Session #1: INSERT INTO t VALUES (1,1),(2,2),(3,3);
Session #2: BEGIN; UPDATE t SET v = 11 WHERE id = 1;
Session #3: UPDATE t SET v = 33 WHERE id = 3;
Session #1: INSERT INTO t VALUES (4,4);
Session #2: COMMIT;
#220410 11:21:47 last_committed=0 sequence_number=1 -- CREATE
#220410 11:21:49 last_committed=1 sequence_number=2 –- INSERT id=1,2,3
#220410 11:21:55 last_committed=2 sequence_number=3 -- UPDATE id=3
#220410 11:21:59 last_committed=3 sequence_number=4 –- INSERT id=4
#220410 11:22:03 last_committed=2 sequence_number=5 -- UPDATE id=1 22
Commit Timestamps [3 of 3]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
More binlog: note that the DML order is different from the commit order
mysqlbinlog $f |
awk '$11 ~ "last_committed"{print $1,$2,$11,$12} $10 ~ "_rows|Xid"{print $1,$2,$10}'
#220410 11:21:47 last_committed=0 sequence_number=1 -- CREATE
#220410 11:21:49 last_committed=1 sequence_number=2 -- INSERT id=1,2,3
#220410 11:21:49 Write_rows:
#220410 11:21:49 Xid
#220410 11:21:55 last_committed=2 sequence_number=3 -- UPDATE id=3
#220410 11:21:55 Update_rows:
#220410 11:21:55 Xid
#220410 11:21:59 last_committed=3 sequence_number=4 -- INSERT id=4
#220410 11:21:59 Write_rows:
#220410 11:21:59 Xid
#220410 11:22:03 last_committed=2 sequence_number=5 -- UPDATE id=1
#220410 11:21:52 Update_rows:
#220410 11:22:03 Xid
23
Tunning Commit Timestamp
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
If very few trx run in parallel, or if commit is quick (auto-commit, very
fast disks or low durability), CTS do not identify much parallelism
(low durability: sync_binlog != 1 and trx_commit != 1)
A solution is to delay commit for having more transactions running
in parallel, which leads to better interval generation:
• binlog_group_commit_sync_delay
• binlog_group_commit_sync_no_delay_count
I call this slowing-down the primary to speed-up the replicas
But performing this tunning just looking at replica speed is tedious24
Intervals Quality
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
To ease the tunning, I came-up with a metric for interval quality:
• the Average Modified Interval Length (AMIL)
It is based on the following observation:
• the larger the intervals are, the better MTR throughput will be
All the details about the AMIL in
http://jfg-mysql.blogspot.com/2017/02/metric-for-tuning-parallel-replication-mysql-5-7.html
Computing the AMIL needs parsing the output of mysqlbinlog
for extracting interval information, MySQL should make this easier:
• Bug#85965: Expose, on the master, counters for monitoring // information quality
• Bug#85966: Expose, on slaves, counters for monitoring // information quality 25
Tunning Commit Timestamps with AMIL [1 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Monitoring the AMIL provides an immediate tunning feedback:
• After increasing binlog_group_commit_sync_delay, the AMIL
should increase (if not, increase more for better parallel replication,
or rollback for not penalizing commit latency)
But while doing this, also look at the commit-rate of the primary:
• If the commit-rate noticeably drop, consider reverting the tuning
(it is penalizing the application throughput)
But penalizing throughput might be ok for solving replication lag:
• lowering the primary throughput from 200 to 100 tps might be
acceptable to increase replica throughput from 50 to 150 tps 26
Tunning Commit Timestamps with AMIL [2 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
AMIL without and with tuning on four primaries:
(speed-up the replicas by increasing binlog_group_commit_sync_delay)
27
Other Commit Timestamp Tunning
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
28
To alleviate drop in throughput caused by delay,
increase the number of operations done in parallel in MySQL
Also, more transactions run in parallel by the application leads to
more parallelism identified by Commit Timestamps and better
replication speed on replicas
CTS and Non-Primary Sources
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Without MTR, CTS leads to no parallelism identification on replicas
Similarly, even with MTR and compared to the primary,
less parallelism is identified on replicas by Commit Timestamps:
• MTR loses efficiency when replicating from a non-primary source
Also, after promoting a replica as the new primary, its binlogs have
smaller intervals than the one from the old primary:
• Using these leads to less efficient MTR (eg: after restore backup)
With SBR, no good solutions to that (except Binlog Servers)
https://medium.com/booking-com-infrastructure/better-parallel-replication-for-mysql-14e2d7857813 29
Write Set [1 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
If using RBR, a better dependency tracking method can be used:
• Write Set was developed for Group Replication (in MySQL 5.7)
• It can also be used for dependency tracking (introduced in 8.0)
• It was back-ported in 5.7.22 (thanks Oracle for this back-port)
The Write Set is a data structure used to generate intervals:
• It stores a hash of all primary keys (and unique keys) modified by trx
(tunable size with binlog_transaction_dependency_history_size)
The intervals generated by Write Set on replicas are as good as on
the primary (and better if the primary is using CTS)
(Write Set can be used on a RBR replica of a SBR source) 30
Write Set [2 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Enabling Write Set dependency tracking (on both primary and replicas):
• Set BTDT to WRITESET (or WRITESET_SESSION)
(BTDT for binlog_transaction_dependency_tracking)
• This needs TWSE to XXHASH64 (or MURMUR32)
(TWSE for transaction_write_set_extraction)
• Which also needs binlog_format to ROW
With BTDT to WRITESET, a second trx run by a single session does
not depend on the first:
• With BTDT to WRITESET, SPCO should be enabled on replica
• Or set BTDT to WRITESET_SESSION on the primary
(not a good setting for replicas, we will come back to this) 31
Write Set [3 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
In below, updates id=3 and id=1, and insert id=4 can execute in parallel:
• Without SPCO (and the update), insert id=4 could commit before id=1,2,3
SET GLOBAL binlog_transaction_dependency_tracking = WRITESET;
Session #1: CREATE TABLE t(id INT PRIMARY KEY, v INT);
Session #1: INSERT INTO t VALUES (1,1),(2,2),(3,3);
Session #2: BEGIN; UPDATE t SET v = 11 WHERE id = 1;
Session #3: UPDATE t SET v = 33 WHERE id = 3;
Session #1: INSERT INTO t VALUES (4,4);
Session #2: COMMIT;
#220410 12:01:19 last_committed=0 sequence_number=1 -- CREATE
#220410 12:01:21 last_committed=1 sequence_number=2 -- INSERT id=1,2,3
#220410 12:01:29 last_committed=2 sequence_number=3 -- UPDATE id=3
#220410 12:01:34 last_committed=1 sequence_number=4 -- INSERT id=4
#220410 12:01:38 last_committed=2 sequence_number=5 -- UPDATE id=1 32
Barriers in Parallel Scheduling on Replicas
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
#220410 12:01:19 last_committed=0 sequence_number=1 -- CREATE
#220410 12:01:21 last_committed=1 sequence_number=2 -- INSERT id=1,2,3
#220410 12:01:29 last_committed=2 sequence_number=3 -- UPDATE id=3
#220410 12:01:34 last_committed=1 sequence_number=4 -- INSERT id=4
#220410 12:01:38 last_committed=2 sequence_number=5 -- UPDATE id=1 33
In below, the update id=3 (seq_no 3) depends on the multi-value insert:
• Parallel transaction scheduling blocks until the dependent trx completes
A dependent transaction acts as a barrier in parallel scheduling:
• Even if insert id=4 could be run at the same time as the multi-value insert,
the update id=3 prevents this
This might change in a future MySQL version
(it is also the M in AMIL – Modified)
Write Set [4 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
If running with slave_preserve_commit_order to OFF,
WRITESET_SESSION avoid inconsistencies: insert id=4 runs after id=1,2,3
SET GLOBAL binlog_transaction_dependency_tracking = WRITESET_SESSION;
Session #1: CREATE TABLE t(id INT PRIMARY KEY, v INT);
Session #1: INSERT INTO t VALUES (1,1),(2,2),(3,3);
Session #2: BEGIN; UPDATE t SET v = 11 WHERE id = 1;
Session #3: UPDATE t SET v = 33 WHERE id = 3;
Session #1: INSERT INTO t VALUES (4,4);
Session #2: COMMIT;
#220410 12:12:30 last_committed=0 sequence_number=1 -- CREATE
#220410 12:12:32 last_committed=1 sequence_number=2 -- INSERT id=1,2,3
#220410 12:12:39 last_committed=2 sequence_number=3 -- UPDATE id=3
#220410 12:12:43 last_committed=2 sequence_number=4 -- INSERT id=4
#220410 12:12:47 last_committed=2 sequence_number=5 -- UPDATE id=1 34
Write Set and Non-Primary Sources
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
We can enable Write Set on intermediary source:
• Even on a single-threaded replica
• But we should use BTDT to WRITESET because
WRITESET_SESSION would create new dependencies
(without MTR, all intervals of WRITESET_SESSION are of size one)
• This solves the CTS interval degradation on intermediary source
This was done to get the first (E*) benchmark results (in 2017):
• Primary in 5.7 (pre .22),
intermediary source in 8.0 with WRITESET, replica with MTR
This is useful to “try” MTR before big migration:
• SBR to RBR for Write Set, or 5.6 to 5.7 for MTR 35
AMIL for Write Set [1 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
AMIL from a CTS primary and a Write Set intermediary source:
• The primary does not need RBR, only the intermediary source does
• Write Set AMIL is usually better than untuned and tunned CTS
36
AMIL for Write Set [2 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Sometimes, combining delay with Write Set gives a better AMIL:
• Discovered by accident, tunning opportunity?, more research needed
(Write Set AMIL below 3x better with delay compared to without)
37
More Tunning: Smaller Transaction Sizes
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Because transaction scheduling sometimes/often blocks (barriers
and dependencies), some replica workers might be idle or waiting
on another transaction completion:
• In the worse case, many workers are waiting for a single big trx
These big trx reduce the potential for speedups:
• Avoiding big transactions in the application will make MTR faster
Ironically, when commit was expensive, or to amortize round-trip to
database, doing many things in a single (big) trx was / is the right
thing to do, but this makes MTR run slower 38
Recommended Settings for MTR Readiness
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Without RBR, CTS is the only available dependency tracking
à not worth “preparing” for MTR (implement and tune when needed)
With RBR and MySQL 8.0:
• binlog_transaction_dependency_tracking to WRITESET
• slave_parallel_type to LOGICAL_CLOCK (new default in 8.0.27)
• slave_preserve_commit_order to ON (new default in 8.0.27)
With RBR and MySQL 5.7, all above plus:
• transaction_write_set_extraction to XXHASH64
(this is the default in 8.0)
With these, faster repl. only needs updating slave_parallel_workers
Thoughts on MySQL 8.0.27 new Defaults
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
In MySQL 8.0.27 we have:
• slave_parallel_type to LOGICAL_CLOCK
• slave_preserve_commit_order to ON (was OFF before)
• and slave_parallel_workers to 4 in 8.0.27 (was 0 before)
But intervals are still generated with Commit Timestamps:
• Having 4 workers will probably not improve replication speed much
• But this opens possibility for MTR rough-edges
Enabling parallel replication by default might be a little early
and when these bugs will be fixed, Write Set should also be enabled
40
Benchmark
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Benchmarks done a few years ago and at HubSpot:
• Used 0 to 256 threads (slave_parallel_workers)
• E1 to E8: different environments / workloads (2017-2018)
• F1 to F4: more environments / workloads (from HubSpot)
• HD: High Durability: sync_binlog = 1 and trx_commit = 1
• ND: No Durability: sync_binlog = 0 and trx_commit = 2
• IS: Intermediary Source: with log_slave_updates
• NO: No Order: slave_preserve_commit_order = OFF
• (WO: with order: slave_preserve_commit_order = ON)
• (RB: replica with binary logs: without log_slave_updates)
41
Benchmark Presentation [1 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
42
E5 IS-HD Single-Threaded: 6138 seconds
E5 IS-ND Single-Threaded: 2238 seconds
Going from HD to ND for E5 IS gives a speedup of 2.74
Benchmark Presentation [2 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
All in a single graph:
• HD et ND: previous speedups
• HD vs ND single threaded: the speedup from HD to ND
43
Old Benchmark Setup
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
The test environments involve three nodes:
+---+ +---+ +---+
| A | -------> | B | -------> | C |
+---+ +---+ +---+
• A is a production primary (MySQL 5.6 or 5.7: this was in 2017-2018)
• Some of the primaries are SBR, some are RBR
• B is a MySQL 8.0.3 Intermediary Source in RBR with Write Set
(transaction_write_set_extraction = XXHASH64)
(binlog_transaction_dependency_tracking = WRITESET)
• C is a replica where timing is done 44
Previous Benchmark Results [1 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Previous Benchmark Results [2 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Observations:
• High variation of speedups, very workload dependent
(probably possible to do better if spending time optimizing the application)
• HD gives better speedups than ND (normal because group commit)
• ND gives some interesting speedups, but do not expect 16x faster
not possible to use all CPUs of modern hardware on replica for writes
HubSpot Benchmark Setup
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
The test environments involve two nodes:
+---+ +---+
| A | -------> | B |
+---+ +---+
• A is a production primary (Percona Server 5.7.36 with Write Set)
(transaction_write_set_extraction = XXHASH64)
(binlog_transaction_dependency_tracking = WRITESET)
• B is a replica where timing is done
47
HubSpot Benchmarks
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Observations:
• Higher HD speedups for F{1,2} than E*
• Better HD vs ND single-threaded
for F* than E* (F1 being much better)
Something is different between E and F.
Benchmark Discussion
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Previous was on local SSD, HubSpot is EBS in AWS
• Disk-sync more expensive for HubSpot
• Which explains better HD results
• Also explains better HD vs ND single-threaded speedup
• (Also, make benchmarking hard because AWS variance)
(HubSpot is running production with ND + semi-sync)
Also, HubSpot is running with the previously recommended settings:
• Write Set enabled on primaries and replicas
• slave_parallel_type to LOGICAL_CLOCK
• slave_preserve_commit_order to ON
à Ready for MTR by updating slave_parallel_workers 49
Other Benchmarks Results
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
I also have results for:
• WO, and NO vs WO
• RB, and IS vs RB
Too long to present everything, contact me if interested
TL&DR:
• NO is faster than WO
• RB is faster than IS
• Both highly variable depending on workload and durability
50
Other Parallel Replication Subjects
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
MySQL Parallel Replication Simulator
https://blog.koehntopp.info/2021/11/08/mysql-parallel-replication.html
Replica Group Commit (Slave Group Commit in MariaDB)
https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2
• Replicating trx sequentially, in many threads for Group Committing
• Almost as good as ND single-threaded speed, with benefit of HD
Optimistic Parallel Replication (in MariaDB)
https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-4-more-benchmarks-in-production-49ee255043ab
• Go beyond scheduling barriers on replica, achieving replication prefetching
Facebook partial transaction parallel execution in MySQL 5.6
https://www.percona.com/live/e18/sessions/faster-mysql-replication-using-row-dependencies
• Using Write Set on replicas for fine grain parallel statement excution 51
Links [1 of 3]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Content related to this talk from the speaker:
• A Metric for Tuning Parallel Replication in MySQL 5.7
https://jfg-mysql.blogspot.com/2017/02/metric-for-tuning-parallel-replication-mysql-5-7.html
• Better Parallel Replication for MySQL (Binlog Server)
https://medium.com/booking-com-infrastructure/better-parallel-replication-for-mysql-14e2d7857813
• An update on Write Set (parallel replication) bug fix in MySQL 8.0
https://jfg-mysql.blogspot.com/2018/01/an-update-on-write-set-parallel-replication-bug-fix-in-mysql-8-0.html
Binlog Server content from the speaker:
• MySQL Slave Scaling (and more)
https://medium.com/booking-com-infrastructure/mysql-slave-scaling-and-more-a09d88713a20
• Abstracting Binlog Servers and MySQL Master Promotion without Reconfiguring all Slaves
https://medium.com/booking-com-infrastructure/abstracting-binlog-servers-and-mysql-master-promotion-without-reconfiguring-all-slaves-44be1febc8a0
Other parallel replication content from the speaker:
• Slave Group Commit
https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2
• Optimistic Parallel Replication
https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-4-more-benchmarks-in-production-49ee255043ab 52
Links [2 of 3]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Related bugs (some fixed, some still open) for the full history:
• Bug#75396: Allow slave_preserve_commit_order without log-slave-updates.
• Bug#81840: Automatic Replication Recovery Does Not Handle Lost Events (applies to MTR)
• Bug#85965: Expose, on the master, counters for monitoring // information quality.
• Bug#85966: Expose, on slaves, counters for monitoring // information quality.
• Bug#85977: The doc. for LOGICAL_CLOCK wrongly references Group Commit
• Bug#89247: Deadlock with MTS when slave_preserve_commit_order = ON
• PS-8030: Multithreaded replica crashes [at STOP SLAVE] inside rli::cannot_safely_rollback()
• I think there is still a deadlock bug in 5.7 (at least in Percona Server 5.7.36),
troubleshooting in progress at HubSpot
• Also, there might be other bugs
53
Links [3 of 3]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
And more related content:
• Solving MySQL Replication Lag with LOGICAL_CLOCK and Calibrated Delay
https://orangematter.solarwinds.com/2017/01/13/solving-mysql-replication-lag-with-logical_clock-and-calibrated-delay/
• How to Fix a Lagging MySQL Replication
https://thoughts.t37.net/fixing-a-very-lagging-mysql-replication-db6eb5a6e15d
• Estimating potential for MySQL 5.7 parallel replication
https://www.percona.com/blog/2016/02/10/estimating-potential-for-mysql-5-7-parallel-replication/
• How Binary Logs (and Filesystems) Affect MySQL Performance
https://www.percona.com/blog/2018/05/04/how-binary-logs-and-filesystems-affect-mysql-performance/
• A Dive Into MySQL Multi-Threaded Replication
https://www.percona.com/blog/a-dive-into-mysql-multi-threaded-replication/
• MySQL Parallel Replication Simulator
https://blog.koehntopp.info/2021/11/08/mysql-parallel-replication.html
54
Thanks !
by Jean-François Gagné
System and MySQL Expert at HubSpot
Presented at Percona Live Austin 2022
jfg.mysql AT gmail DOT com
Twitter: @jfg956

Contenu connexe

Tendances

MMUG18 - MySQL Failover and Orchestrator
MMUG18 - MySQL Failover and OrchestratorMMUG18 - MySQL Failover and Orchestrator
MMUG18 - MySQL Failover and OrchestratorSimon J Mudd
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)Altinity Ltd
 
MySQL Load Balancers - Maxscale, ProxySQL, HAProxy, MySQL Router & nginx - A ...
MySQL Load Balancers - Maxscale, ProxySQL, HAProxy, MySQL Router & nginx - A ...MySQL Load Balancers - Maxscale, ProxySQL, HAProxy, MySQL Router & nginx - A ...
MySQL Load Balancers - Maxscale, ProxySQL, HAProxy, MySQL Router & nginx - A ...Severalnines
 
Disaster Recovery with MySQL InnoDB ClusterSet - What is it and how do I use it?
Disaster Recovery with MySQL InnoDB ClusterSet - What is it and how do I use it?Disaster Recovery with MySQL InnoDB ClusterSet - What is it and how do I use it?
Disaster Recovery with MySQL InnoDB ClusterSet - What is it and how do I use it?Miguel Araújo
 
The consequences of sync_binlog != 1
The consequences of sync_binlog != 1The consequences of sync_binlog != 1
The consequences of sync_binlog != 1Jean-François Gagné
 
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...Jean-François Gagné
 
MySQL Performance for DevOps
MySQL Performance for DevOpsMySQL Performance for DevOps
MySQL Performance for DevOpsSveta Smirnova
 
ProxySQL Tutorial - PLAM 2016
ProxySQL Tutorial - PLAM 2016ProxySQL Tutorial - PLAM 2016
ProxySQL Tutorial - PLAM 2016Derek Downey
 
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best Practices
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best PracticesMySQL InnoDB Cluster - New Features in 8.0 Releases - Best Practices
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best PracticesKenny Gryp
 
MySQL GTID Concepts, Implementation and troubleshooting
MySQL GTID Concepts, Implementation and troubleshooting MySQL GTID Concepts, Implementation and troubleshooting
MySQL GTID Concepts, Implementation and troubleshooting Mydbops
 
Evolution of MySQL Parallel Replication
Evolution of MySQL Parallel Replication Evolution of MySQL Parallel Replication
Evolution of MySQL Parallel Replication Mydbops
 
PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs PGConf APAC
 
MySQL Replication Performance Tuning for Fun and Profit!
MySQL Replication Performance Tuning for Fun and Profit!MySQL Replication Performance Tuning for Fun and Profit!
MySQL Replication Performance Tuning for Fun and Profit!Vitor Oliveira
 
Maria db 이중화구성_고민하기
Maria db 이중화구성_고민하기Maria db 이중화구성_고민하기
Maria db 이중화구성_고민하기NeoClova
 
binary log と 2PC と Group Commit
binary log と 2PC と Group Commitbinary log と 2PC と Group Commit
binary log と 2PC と Group CommitTakanori Sejima
 
ProxySQL High Avalability and Configuration Management Overview
ProxySQL High Avalability and Configuration Management OverviewProxySQL High Avalability and Configuration Management Overview
ProxySQL High Avalability and Configuration Management OverviewRené Cannaò
 
How to set up orchestrator to manage thousands of MySQL servers
How to set up orchestrator to manage thousands of MySQL serversHow to set up orchestrator to manage thousands of MySQL servers
How to set up orchestrator to manage thousands of MySQL serversSimon J Mudd
 
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfJesmar Cannao'
 

Tendances (20)

MMUG18 - MySQL Failover and Orchestrator
MMUG18 - MySQL Failover and OrchestratorMMUG18 - MySQL Failover and Orchestrator
MMUG18 - MySQL Failover and Orchestrator
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
 
MySQL Load Balancers - Maxscale, ProxySQL, HAProxy, MySQL Router & nginx - A ...
MySQL Load Balancers - Maxscale, ProxySQL, HAProxy, MySQL Router & nginx - A ...MySQL Load Balancers - Maxscale, ProxySQL, HAProxy, MySQL Router & nginx - A ...
MySQL Load Balancers - Maxscale, ProxySQL, HAProxy, MySQL Router & nginx - A ...
 
Disaster Recovery with MySQL InnoDB ClusterSet - What is it and how do I use it?
Disaster Recovery with MySQL InnoDB ClusterSet - What is it and how do I use it?Disaster Recovery with MySQL InnoDB ClusterSet - What is it and how do I use it?
Disaster Recovery with MySQL InnoDB ClusterSet - What is it and how do I use it?
 
The consequences of sync_binlog != 1
The consequences of sync_binlog != 1The consequences of sync_binlog != 1
The consequences of sync_binlog != 1
 
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
 
MySQL Performance for DevOps
MySQL Performance for DevOpsMySQL Performance for DevOps
MySQL Performance for DevOps
 
ProxySQL Tutorial - PLAM 2016
ProxySQL Tutorial - PLAM 2016ProxySQL Tutorial - PLAM 2016
ProxySQL Tutorial - PLAM 2016
 
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best Practices
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best PracticesMySQL InnoDB Cluster - New Features in 8.0 Releases - Best Practices
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best Practices
 
MySQL GTID Concepts, Implementation and troubleshooting
MySQL GTID Concepts, Implementation and troubleshooting MySQL GTID Concepts, Implementation and troubleshooting
MySQL GTID Concepts, Implementation and troubleshooting
 
Query logging with proxysql
Query logging with proxysqlQuery logging with proxysql
Query logging with proxysql
 
Evolution of MySQL Parallel Replication
Evolution of MySQL Parallel Replication Evolution of MySQL Parallel Replication
Evolution of MySQL Parallel Replication
 
PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs
 
MySQL Replication Performance Tuning for Fun and Profit!
MySQL Replication Performance Tuning for Fun and Profit!MySQL Replication Performance Tuning for Fun and Profit!
MySQL Replication Performance Tuning for Fun and Profit!
 
Maria db 이중화구성_고민하기
Maria db 이중화구성_고민하기Maria db 이중화구성_고민하기
Maria db 이중화구성_고민하기
 
binary log と 2PC と Group Commit
binary log と 2PC と Group Commitbinary log と 2PC と Group Commit
binary log と 2PC と Group Commit
 
ProxySQL High Avalability and Configuration Management Overview
ProxySQL High Avalability and Configuration Management OverviewProxySQL High Avalability and Configuration Management Overview
ProxySQL High Avalability and Configuration Management Overview
 
How to set up orchestrator to manage thousands of MySQL servers
How to set up orchestrator to manage thousands of MySQL serversHow to set up orchestrator to manage thousands of MySQL servers
How to set up orchestrator to manage thousands of MySQL servers
 
Automated master failover
Automated master failoverAutomated master failover
Automated master failover
 
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
 

Similaire à MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)

MySQL Replication Evolution -- Confoo Montreal 2017
MySQL Replication Evolution -- Confoo Montreal 2017MySQL Replication Evolution -- Confoo Montreal 2017
MySQL Replication Evolution -- Confoo Montreal 2017Dave Stokes
 
Demystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyDemystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyJean-François Gagné
 
Demystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyDemystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyJean-François Gagné
 
ConFoo MySQL Replication Evolution : From Simple to Group Replication
ConFoo  MySQL Replication Evolution : From Simple to Group ReplicationConFoo  MySQL Replication Evolution : From Simple to Group Replication
ConFoo MySQL Replication Evolution : From Simple to Group ReplicationDave Stokes
 
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...Insight Technology, Inc.
 
MySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentMySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentJean-François Gagné
 
Exploring mysql cluster 7.4
Exploring mysql cluster 7.4Exploring mysql cluster 7.4
Exploring mysql cluster 7.4Ivan Ma
 
User Camp High Availability Presentation
User Camp High Availability PresentationUser Camp High Availability Presentation
User Camp High Availability Presentationsankalita chakraborty
 
MySQL Parallel Replication by Booking.com
MySQL Parallel Replication by Booking.comMySQL Parallel Replication by Booking.com
MySQL Parallel Replication by Booking.comJean-François Gagné
 
MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012Colin Charles
 
NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
NoSQL on MySQL - MySQL Document Store by Vadim TkachenkoNoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
NoSQL on MySQL - MySQL Document Store by Vadim TkachenkoData Con LA
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo
 
My sql 56_roadmap_april2012
My sql 56_roadmap_april2012My sql 56_roadmap_april2012
My sql 56_roadmap_april2012sqlhjalp
 
My sql 56_roadmap_april2012_zht2
My sql 56_roadmap_april2012_zht2My sql 56_roadmap_april2012_zht2
My sql 56_roadmap_april2012_zht2Ivan Tu
 
MySQL Tech Tour 2015 - 5.7 Whats new
MySQL Tech Tour 2015 - 5.7 Whats newMySQL Tech Tour 2015 - 5.7 Whats new
MySQL Tech Tour 2015 - 5.7 Whats newMark Swarbrick
 
Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014Michael Renner
 
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)Aurimas Mikalauskas
 
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...NETWAYS
 

Similaire à MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK) (20)

MySQL Replication Evolution -- Confoo Montreal 2017
MySQL Replication Evolution -- Confoo Montreal 2017MySQL Replication Evolution -- Confoo Montreal 2017
MySQL Replication Evolution -- Confoo Montreal 2017
 
Demystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyDemystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash Safety
 
Demystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyDemystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash Safety
 
ConFoo MySQL Replication Evolution : From Simple to Group Replication
ConFoo  MySQL Replication Evolution : From Simple to Group ReplicationConFoo  MySQL Replication Evolution : From Simple to Group Replication
ConFoo MySQL Replication Evolution : From Simple to Group Replication
 
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
 
MySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentMySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated Environment
 
Exploring mysql cluster 7.4
Exploring mysql cluster 7.4Exploring mysql cluster 7.4
Exploring mysql cluster 7.4
 
User Camp High Availability Presentation
User Camp High Availability PresentationUser Camp High Availability Presentation
User Camp High Availability Presentation
 
Oss4b - pxc introduction
Oss4b   - pxc introductionOss4b   - pxc introduction
Oss4b - pxc introduction
 
MySQL Parallel Replication by Booking.com
MySQL Parallel Replication by Booking.comMySQL Parallel Replication by Booking.com
MySQL Parallel Replication by Booking.com
 
MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012
 
NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
NoSQL on MySQL - MySQL Document Store by Vadim TkachenkoNoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
My sql 56_roadmap_april2012
My sql 56_roadmap_april2012My sql 56_roadmap_april2012
My sql 56_roadmap_april2012
 
My sql 56_roadmap_april2012_zht2
My sql 56_roadmap_april2012_zht2My sql 56_roadmap_april2012_zht2
My sql 56_roadmap_april2012_zht2
 
MySQL 5.7 what's new
MySQL 5.7 what's newMySQL 5.7 what's new
MySQL 5.7 what's new
 
MySQL Tech Tour 2015 - 5.7 Whats new
MySQL Tech Tour 2015 - 5.7 Whats newMySQL Tech Tour 2015 - 5.7 Whats new
MySQL Tech Tour 2015 - 5.7 Whats new
 
Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014
 
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
 
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...
 

Plus de Jean-François Gagné

Autopsy of a MySQL Automation Disaster
Autopsy of a MySQL Automation DisasterAutopsy of a MySQL Automation Disaster
Autopsy of a MySQL Automation DisasterJean-François Gagné
 
MySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentMySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentJean-François Gagné
 
MySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitationsMySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitationsJean-François Gagné
 
The two little bugs that almost brought down Booking.com
The two little bugs that almost brought down Booking.comThe two little bugs that almost brought down Booking.com
The two little bugs that almost brought down Booking.comJean-François Gagné
 
How Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lagHow Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lagJean-François Gagné
 
MySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitationsMySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
 
MySQL Parallel Replication: inventory, use-cases and limitations
MySQL Parallel Replication: inventory, use-cases and limitationsMySQL Parallel Replication: inventory, use-cases and limitations
MySQL Parallel Replication: inventory, use-cases and limitationsJean-François Gagné
 
Riding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication StreamRiding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication StreamJean-François Gagné
 

Plus de Jean-François Gagné (9)

Autopsy of a MySQL Automation Disaster
Autopsy of a MySQL Automation DisasterAutopsy of a MySQL Automation Disaster
Autopsy of a MySQL Automation Disaster
 
MySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentMySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated Environment
 
MySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitationsMySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitations
 
The two little bugs that almost brought down Booking.com
The two little bugs that almost brought down Booking.comThe two little bugs that almost brought down Booking.com
The two little bugs that almost brought down Booking.com
 
Autopsy of an automation disaster
Autopsy of an automation disasterAutopsy of an automation disaster
Autopsy of an automation disaster
 
How Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lagHow Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lag
 
MySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitationsMySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitations
 
MySQL Parallel Replication: inventory, use-cases and limitations
MySQL Parallel Replication: inventory, use-cases and limitationsMySQL Parallel Replication: inventory, use-cases and limitations
MySQL Parallel Replication: inventory, use-cases and limitations
 
Riding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication StreamRiding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication Stream
 

Dernier

2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAndrey Devyatkin
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?Igalia
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilV3cube
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slidevu2urc
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdflior mazor
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?Antenna Manufacturer Coco
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century educationjfdjdjcjdnsjd
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesBoston Institute of Analytics
 

Dernier (20)

2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation Strategies
 

MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)

  • 1. MySQL Parallel Replication: all the 5.7 and 8.0 details (LOGICAL_CLOCK) by Jean-François Gagné System and MySQL Expert at HubSpot Presented at Percona Live Austin 2022 jfg.mysql AT gmail DOT com Twitter: @jfg956
  • 2. Benchmark Preview (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) 2 Replication throughput (TPS) without and with parallel replication
  • 3. Terminology [1 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) MySQL Terminology Updates in 2020: https://dev.mysql.com/blog-archive/mysql-terminology-updates/ • master / slave à primary / replica • the master of a slave à the source of a replica This presentation uses above terminology, with below additions: • Multi-Threaded Slave (MTS) à Multi-Threaded Replication (MTR) • intermediary master à intermediary source or non-primary source (Exceptions for citing content from before the terminology change) 3
  • 4. Terminology [2 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) From MySQL 8.0.22 (not in 5.7), new commands were introduced: • START/STOP SLAVE à START/STOP REPLICA • And more… • Some missing: START REPLICA UNTIL SQL_AFTER_MTS_GAPS From 8.0.26 (not in 5.7 either), new variables were introduced: • log_slave_updates à log_replica_updates • slave_parallel_workers à replica_parallel_workers • And more… Because this talk applies to 5.7, and not everyone is on 8.0-latest, this presentation uses old command and variable names
  • 5. Summary (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) • Replication reminders • Why parallel replication is important (and complicated) • How to enable parallel replication and what is LOGICAL_CLOCK • How parallel replication works on replicas • How dependency tracking works on primaries • Benchmarks • Closing thoughts 5
  • 6. MySQL Replication (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) • The primary records transactions in a journal (binary logs), and replicas • downloads the journal and saves them locally in the relay logs (IO thread) • executes the relay logs on their local database (SQL thread) • could also write binlogs to be themselves a source (log-slave-updates)
  • 7. Other Replication Concepts: Durability (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Persistence / Durability (D of ACID): • Is there data lost after a crash ? (different types of crash: MySQL process (mysqld) or OS (Linux)) • Parameters controlling durability: sync_binlog and trx_commit (trx_commit is short for innodb_flush_logs_at_trx_commit) • High Durability (HD) or no data lost after crashes (sync_binlog = 1 and trx_commit = 1) (most expensive / slower because flush to disk after each trx) • No Durability (ND sometimes called low durability) (sync_binlog = 0 and trx_commit = 2) (faster, with data lost on OS crash, but no loss on mysqld crash) • (Data lost for mysqld crash in case trx_commit = 0 but not covered here) 7
  • 8. Other Replication Concepts: Group Commit (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Group Commit is an optimization making high durability faster • Before Group Commit (GC), each trx needed its disk flush • With GC, many trx can be persisted in the same flush à better write throughput on primaries with concurency • (GC was introduced in MariaDB xx and then in MySQL 5.6) • (Single threaded replication cannot Group Commit) • (MariaDB 10.0 introduced Replica Group Commit) 8
  • 9. From Single to Multi-Threaded Replication (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Without MTR, a single CPU is used for applying writes on replicas: • on the primary, many CPUs are used for writes (one per session) WO MTR, a single read IO can be “in flight” for writes on replicas: • this is a bottleneck, especially for high latency disks (cloud) • on primary, many IOs can be in flights for writes (one per session) WO MTR, no group commit on replicas: • on the p., a single flush can persist many trx in binlogs and redo logs • on replicas, flushing each trx is a bottleneck Multi-Threaded Replication is a solution to all above problems 9
  • 10. Challenges of Multi-Threaded Replication (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) But running transactions in parallel on replica is not trivial: • Updating a row cannot be run in parallel with the insert of this row • MTR implementation needs to keep data consistency Enabling MTR is also not straightforward: • Many parameters and complex tunning MTR also still has some rough edges: • In 5.7, START SLAVE UNTIL not supported with MTR (ok from 8.0.0) • Latest -1 Percona Server 5.7 crashes on STOP SLAVE with MTR (PS-8030: 5.7.27 to .36 affected and 8.0.17 to .21, Oracle MySQL not affected) • I have also seen deadlock with PS 5.7.36, and rumor of another crash (to be investigated, unknown if both in Percona Server and Oracle MySQL)
  • 11. Solutions for MTR Data Consistency (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) • MySQL 5.6: schema based parallel replication • MariaDB 10.0: group commit based parallel replication • MariaDB 10.1: optimistic parallel replication • Improvement on 10.0, probably the fastest solution, but high CPU cost • MySQL 5.7: same schema parallel replication (Logical Clock) • Not as fast as optimistic, but not wasteful in CPU • MySQL 8.0: Write-Set improvement (still Logical Clock) • Lot to say, including need for RBR and back-ported in 5.7.22 This talk is about Logical Clock in 5.7 and 8.0 11
  • 12. Solutions for MTR Data Consistency (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) • MySQL 5.6: schema based parallel replication • MariaDB 10.0: group commit based parallel replication • MariaDB 10.1: optimistic parallel replication • Improvement on 10.0, probably the fastest solution, but high CPU cost • MySQL 5.7: same schema parallel replication (Logical Clock) • Not as fast as optimistic, but not wasteful in CPU • MySQL 8.0: Write-Set improvement (still Logical Clock) • Lot to say, including need for RBR and back-ported in 5.7.22 This talk is about Logical Clock in 5.7 and 8.0 12
  • 13. Enabling Parallel Replication (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Set slave_parallel_type to LOGICAL_CLOCK • This is the new default in 8.0.27 (was DATABASE before (schema-based solution of 5.6), including in 5.7) (in 8.0.26, this variable is deprecated for dropping DATABASE support) And set slave_preserve_commit_order to ON • This is also the new default in 8.0.27(was OFF before, including in 5.7) (more about this parameter later in the talk) And set slave_parallel_workers to 2 or more • New default to 4 in 8.0.27 (was 0 before including in 5.7) These enable MTR, but are not sufficient for best throughput 13
  • 14. What is LOGICAL_CLOCK ? (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) From the MySQL Reference Manual for LOGICAL_CLOCK: • Transactions that are part of the same binary log group commit on a source are applied in parallel on a replica. • But this is not 100% exact (Bug#85977) When using LOGICAL_CLOCK, MTR is scheduling trx on replica using dependency tracking information from the binary logs: • We will name these information Intervals • By default, trx intervals are generated with Commit Timestamps (which is close, but not strictly the same as group commit) • Much better intervals can be generated with Write Set (and Write Set is completely different to group commit) 14
  • 15. Intervals (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Each trx is tagged with two numbers in the binlogs (starting with 5.7): • sequence_number: increasing id for each trx (not to confuse with GTID) • last_committed: id of the latest trx on which this trx depends The last_committed / sequence_number is the trx interval Here an example of intervals from a real system: 15 mysqlbinlog $f | awk '$11 ~ "last_committed"{print $1,$2,$11,$12}' [...] #170206 20:08:33 last_committed=6201 sequence_number=6203 #170206 20:08:33 last_committed=6203 sequence_number=6204 #170206 20:08:33 last_committed=6203 sequence_number=6205 #170206 20:08:33 last_committed=6203 sequence_number=6206 #170206 20:08:33 last_committed=6205 sequence_number=6207
  • 16. Parallel Scheduling on Replicas [1 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) 16 A transaction can be started when its last_committed and all previous transactions have been executed The intervals below mean that: • When transaction #6203 (and all previous) have been executed, transactions #6204, #6205 and #6206 can be started • When transaction #6205 (and all previous) have been executed, transaction #6207 can be started #170206 20:08:33 last_committed=6201 sequence_number=6203 #170206 20:08:33 last_committed=6203 sequence_number=6204 #170206 20:08:33 last_committed=6203 sequence_number=6205 #170206 20:08:33 last_committed=6203 sequence_number=6206 #170206 20:08:33 last_committed=6205 sequence_number=6207
  • 17. Parallel Scheduling on Replicas [2 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) 17 Trx #6204 and #6205 (and #6206) can be run in parallel but #6205 could complete before #6204 slave_preserve_commit_order (SPCO) does what you expect: • If SPCO is OFF, #6205 can commit before #6204 • If SPCO is ON , #6205 waits for #6204 completion before committing SPCO to OFF generates data states that never existed on the primary • Use with care (you probably want ON most of the time) #170206 20:08:33 last_committed=6201 sequence_number=6203 #170206 20:08:33 last_committed=6203 sequence_number=6204 #170206 20:08:33 last_committed=6203 sequence_number=6205 #170206 20:08:33 last_committed=6203 sequence_number=6206 #170206 20:08:33 last_committed=6205 sequence_number=6207
  • 18. Parallel Scheduling on Replicas [3 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) SPCO to OFF generates data states that never existed on the primary • Use with care (you probably want ON most of the time) Also, SPCO to OFF generates temporary gaps in GTIDs • For filling the gaps, use START SLAVE UNTIL SQL_AFTER_MTS_GAPS SPCO to ON needed log_slave_updates until 8.0.19 (Bug#75396) Before 5.7.33 and 8.0.23, SPCO to ON could deadlock (Bug#89247) • If replication gets stuck, kill -9 mysqld (no data lost unless trx_commit = 0) • Thanks Percona / Venkatesh for the fixing this, but… • But I have recently seen deadlocks in PS 5.7.36: not fully fixed 18
  • 19. Parallel Scheduling on Replicas [4 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) 19 When run in parallel, trx could fail to execute (below from the doc): • If replication fails to execute a trx because of an InnoDB deadlock or the trx's execution time exceeded innodb_lock_wait_timeout… Such failed trx will be retried (below more from the doc): • …it automatically retries slave_transaction_retries times before stopping with an error. The column COUNT_TRANSACTIONS_RETRIES from the the P_S table replication_applier_status indicates how many retries have happened, and the table repl._app._status_by_worker contains details about the retries
  • 20. Reminder: What is LOGICAL_CLOCK ? (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) From the MySQL Reference Manual for LOGICAL_CLOCK: • Transactions that are part of the same binary log group commit on a source are applied in parallel on a replica. • But this is not 100% exact (Bug#85977) When using LOGICAL_CLOCK, MTR is scheduling trx on replica using dependency tracking information from the binary logs: • We will name these information Intervals • By default, trx intervals are generated with Commit Timestamps (which is close, but not strictly the same as group commit) • Much better intervals can be generated with Write Set (and Write Set is completely different to group commit) 20
  • 21. Commit Timestamps [1 of 3] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Commit Timestamps (CTS) is the default for 5.7 and 8.0 (binlog_transaction_dependency_tracking = COMMIT_ORDER) It is a dependency tracking method exploiting parallelism on the source to generate the intervals: • While running a DML, the current trx notes the last committed trx (this is where last_committed from mysqlbinlog is coming from) (this is close to Group Commit, but not strictly the same) Without tunning, CTS is not producing efficient intervals (by default, MTR will not give much better replication speed) 21
  • 22. Commit Timestamps [2 of 3] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) In below, the update id=3 ran while the update id=1 was not yet committed: • Insert id=4 and update id=1 can run in parallel on a replica • (Note that nothing commit at the same time à no group commit) Session #1: CREATE TABLE t(id INT PRIMARY KEY, v INT); Session #1: INSERT INTO t VALUES (1,1),(2,2),(3,3); Session #2: BEGIN; UPDATE t SET v = 11 WHERE id = 1; Session #3: UPDATE t SET v = 33 WHERE id = 3; Session #1: INSERT INTO t VALUES (4,4); Session #2: COMMIT; #220410 11:21:47 last_committed=0 sequence_number=1 -- CREATE #220410 11:21:49 last_committed=1 sequence_number=2 –- INSERT id=1,2,3 #220410 11:21:55 last_committed=2 sequence_number=3 -- UPDATE id=3 #220410 11:21:59 last_committed=3 sequence_number=4 –- INSERT id=4 #220410 11:22:03 last_committed=2 sequence_number=5 -- UPDATE id=1 22
  • 23. Commit Timestamps [3 of 3] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) More binlog: note that the DML order is different from the commit order mysqlbinlog $f | awk '$11 ~ "last_committed"{print $1,$2,$11,$12} $10 ~ "_rows|Xid"{print $1,$2,$10}' #220410 11:21:47 last_committed=0 sequence_number=1 -- CREATE #220410 11:21:49 last_committed=1 sequence_number=2 -- INSERT id=1,2,3 #220410 11:21:49 Write_rows: #220410 11:21:49 Xid #220410 11:21:55 last_committed=2 sequence_number=3 -- UPDATE id=3 #220410 11:21:55 Update_rows: #220410 11:21:55 Xid #220410 11:21:59 last_committed=3 sequence_number=4 -- INSERT id=4 #220410 11:21:59 Write_rows: #220410 11:21:59 Xid #220410 11:22:03 last_committed=2 sequence_number=5 -- UPDATE id=1 #220410 11:21:52 Update_rows: #220410 11:22:03 Xid 23
  • 24. Tunning Commit Timestamp (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) If very few trx run in parallel, or if commit is quick (auto-commit, very fast disks or low durability), CTS do not identify much parallelism (low durability: sync_binlog != 1 and trx_commit != 1) A solution is to delay commit for having more transactions running in parallel, which leads to better interval generation: • binlog_group_commit_sync_delay • binlog_group_commit_sync_no_delay_count I call this slowing-down the primary to speed-up the replicas But performing this tunning just looking at replica speed is tedious24
  • 25. Intervals Quality (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) To ease the tunning, I came-up with a metric for interval quality: • the Average Modified Interval Length (AMIL) It is based on the following observation: • the larger the intervals are, the better MTR throughput will be All the details about the AMIL in http://jfg-mysql.blogspot.com/2017/02/metric-for-tuning-parallel-replication-mysql-5-7.html Computing the AMIL needs parsing the output of mysqlbinlog for extracting interval information, MySQL should make this easier: • Bug#85965: Expose, on the master, counters for monitoring // information quality • Bug#85966: Expose, on slaves, counters for monitoring // information quality 25
  • 26. Tunning Commit Timestamps with AMIL [1 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Monitoring the AMIL provides an immediate tunning feedback: • After increasing binlog_group_commit_sync_delay, the AMIL should increase (if not, increase more for better parallel replication, or rollback for not penalizing commit latency) But while doing this, also look at the commit-rate of the primary: • If the commit-rate noticeably drop, consider reverting the tuning (it is penalizing the application throughput) But penalizing throughput might be ok for solving replication lag: • lowering the primary throughput from 200 to 100 tps might be acceptable to increase replica throughput from 50 to 150 tps 26
  • 27. Tunning Commit Timestamps with AMIL [2 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) AMIL without and with tuning on four primaries: (speed-up the replicas by increasing binlog_group_commit_sync_delay) 27
  • 28. Other Commit Timestamp Tunning (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) 28 To alleviate drop in throughput caused by delay, increase the number of operations done in parallel in MySQL Also, more transactions run in parallel by the application leads to more parallelism identified by Commit Timestamps and better replication speed on replicas
  • 29. CTS and Non-Primary Sources (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Without MTR, CTS leads to no parallelism identification on replicas Similarly, even with MTR and compared to the primary, less parallelism is identified on replicas by Commit Timestamps: • MTR loses efficiency when replicating from a non-primary source Also, after promoting a replica as the new primary, its binlogs have smaller intervals than the one from the old primary: • Using these leads to less efficient MTR (eg: after restore backup) With SBR, no good solutions to that (except Binlog Servers) https://medium.com/booking-com-infrastructure/better-parallel-replication-for-mysql-14e2d7857813 29
  • 30. Write Set [1 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) If using RBR, a better dependency tracking method can be used: • Write Set was developed for Group Replication (in MySQL 5.7) • It can also be used for dependency tracking (introduced in 8.0) • It was back-ported in 5.7.22 (thanks Oracle for this back-port) The Write Set is a data structure used to generate intervals: • It stores a hash of all primary keys (and unique keys) modified by trx (tunable size with binlog_transaction_dependency_history_size) The intervals generated by Write Set on replicas are as good as on the primary (and better if the primary is using CTS) (Write Set can be used on a RBR replica of a SBR source) 30
  • 31. Write Set [2 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Enabling Write Set dependency tracking (on both primary and replicas): • Set BTDT to WRITESET (or WRITESET_SESSION) (BTDT for binlog_transaction_dependency_tracking) • This needs TWSE to XXHASH64 (or MURMUR32) (TWSE for transaction_write_set_extraction) • Which also needs binlog_format to ROW With BTDT to WRITESET, a second trx run by a single session does not depend on the first: • With BTDT to WRITESET, SPCO should be enabled on replica • Or set BTDT to WRITESET_SESSION on the primary (not a good setting for replicas, we will come back to this) 31
  • 32. Write Set [3 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) In below, updates id=3 and id=1, and insert id=4 can execute in parallel: • Without SPCO (and the update), insert id=4 could commit before id=1,2,3 SET GLOBAL binlog_transaction_dependency_tracking = WRITESET; Session #1: CREATE TABLE t(id INT PRIMARY KEY, v INT); Session #1: INSERT INTO t VALUES (1,1),(2,2),(3,3); Session #2: BEGIN; UPDATE t SET v = 11 WHERE id = 1; Session #3: UPDATE t SET v = 33 WHERE id = 3; Session #1: INSERT INTO t VALUES (4,4); Session #2: COMMIT; #220410 12:01:19 last_committed=0 sequence_number=1 -- CREATE #220410 12:01:21 last_committed=1 sequence_number=2 -- INSERT id=1,2,3 #220410 12:01:29 last_committed=2 sequence_number=3 -- UPDATE id=3 #220410 12:01:34 last_committed=1 sequence_number=4 -- INSERT id=4 #220410 12:01:38 last_committed=2 sequence_number=5 -- UPDATE id=1 32
  • 33. Barriers in Parallel Scheduling on Replicas (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) #220410 12:01:19 last_committed=0 sequence_number=1 -- CREATE #220410 12:01:21 last_committed=1 sequence_number=2 -- INSERT id=1,2,3 #220410 12:01:29 last_committed=2 sequence_number=3 -- UPDATE id=3 #220410 12:01:34 last_committed=1 sequence_number=4 -- INSERT id=4 #220410 12:01:38 last_committed=2 sequence_number=5 -- UPDATE id=1 33 In below, the update id=3 (seq_no 3) depends on the multi-value insert: • Parallel transaction scheduling blocks until the dependent trx completes A dependent transaction acts as a barrier in parallel scheduling: • Even if insert id=4 could be run at the same time as the multi-value insert, the update id=3 prevents this This might change in a future MySQL version (it is also the M in AMIL – Modified)
  • 34. Write Set [4 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) If running with slave_preserve_commit_order to OFF, WRITESET_SESSION avoid inconsistencies: insert id=4 runs after id=1,2,3 SET GLOBAL binlog_transaction_dependency_tracking = WRITESET_SESSION; Session #1: CREATE TABLE t(id INT PRIMARY KEY, v INT); Session #1: INSERT INTO t VALUES (1,1),(2,2),(3,3); Session #2: BEGIN; UPDATE t SET v = 11 WHERE id = 1; Session #3: UPDATE t SET v = 33 WHERE id = 3; Session #1: INSERT INTO t VALUES (4,4); Session #2: COMMIT; #220410 12:12:30 last_committed=0 sequence_number=1 -- CREATE #220410 12:12:32 last_committed=1 sequence_number=2 -- INSERT id=1,2,3 #220410 12:12:39 last_committed=2 sequence_number=3 -- UPDATE id=3 #220410 12:12:43 last_committed=2 sequence_number=4 -- INSERT id=4 #220410 12:12:47 last_committed=2 sequence_number=5 -- UPDATE id=1 34
  • 35. Write Set and Non-Primary Sources (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) We can enable Write Set on intermediary source: • Even on a single-threaded replica • But we should use BTDT to WRITESET because WRITESET_SESSION would create new dependencies (without MTR, all intervals of WRITESET_SESSION are of size one) • This solves the CTS interval degradation on intermediary source This was done to get the first (E*) benchmark results (in 2017): • Primary in 5.7 (pre .22), intermediary source in 8.0 with WRITESET, replica with MTR This is useful to “try” MTR before big migration: • SBR to RBR for Write Set, or 5.6 to 5.7 for MTR 35
  • 36. AMIL for Write Set [1 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) AMIL from a CTS primary and a Write Set intermediary source: • The primary does not need RBR, only the intermediary source does • Write Set AMIL is usually better than untuned and tunned CTS 36
  • 37. AMIL for Write Set [2 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Sometimes, combining delay with Write Set gives a better AMIL: • Discovered by accident, tunning opportunity?, more research needed (Write Set AMIL below 3x better with delay compared to without) 37
  • 38. More Tunning: Smaller Transaction Sizes (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Because transaction scheduling sometimes/often blocks (barriers and dependencies), some replica workers might be idle or waiting on another transaction completion: • In the worse case, many workers are waiting for a single big trx These big trx reduce the potential for speedups: • Avoiding big transactions in the application will make MTR faster Ironically, when commit was expensive, or to amortize round-trip to database, doing many things in a single (big) trx was / is the right thing to do, but this makes MTR run slower 38
  • 39. Recommended Settings for MTR Readiness (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Without RBR, CTS is the only available dependency tracking à not worth “preparing” for MTR (implement and tune when needed) With RBR and MySQL 8.0: • binlog_transaction_dependency_tracking to WRITESET • slave_parallel_type to LOGICAL_CLOCK (new default in 8.0.27) • slave_preserve_commit_order to ON (new default in 8.0.27) With RBR and MySQL 5.7, all above plus: • transaction_write_set_extraction to XXHASH64 (this is the default in 8.0) With these, faster repl. only needs updating slave_parallel_workers
  • 40. Thoughts on MySQL 8.0.27 new Defaults (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) In MySQL 8.0.27 we have: • slave_parallel_type to LOGICAL_CLOCK • slave_preserve_commit_order to ON (was OFF before) • and slave_parallel_workers to 4 in 8.0.27 (was 0 before) But intervals are still generated with Commit Timestamps: • Having 4 workers will probably not improve replication speed much • But this opens possibility for MTR rough-edges Enabling parallel replication by default might be a little early and when these bugs will be fixed, Write Set should also be enabled 40
  • 41. Benchmark (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Benchmarks done a few years ago and at HubSpot: • Used 0 to 256 threads (slave_parallel_workers) • E1 to E8: different environments / workloads (2017-2018) • F1 to F4: more environments / workloads (from HubSpot) • HD: High Durability: sync_binlog = 1 and trx_commit = 1 • ND: No Durability: sync_binlog = 0 and trx_commit = 2 • IS: Intermediary Source: with log_slave_updates • NO: No Order: slave_preserve_commit_order = OFF • (WO: with order: slave_preserve_commit_order = ON) • (RB: replica with binary logs: without log_slave_updates) 41
  • 42. Benchmark Presentation [1 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) 42 E5 IS-HD Single-Threaded: 6138 seconds E5 IS-ND Single-Threaded: 2238 seconds Going from HD to ND for E5 IS gives a speedup of 2.74
  • 43. Benchmark Presentation [2 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) All in a single graph: • HD et ND: previous speedups • HD vs ND single threaded: the speedup from HD to ND 43
  • 44. Old Benchmark Setup (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) The test environments involve three nodes: +---+ +---+ +---+ | A | -------> | B | -------> | C | +---+ +---+ +---+ • A is a production primary (MySQL 5.6 or 5.7: this was in 2017-2018) • Some of the primaries are SBR, some are RBR • B is a MySQL 8.0.3 Intermediary Source in RBR with Write Set (transaction_write_set_extraction = XXHASH64) (binlog_transaction_dependency_tracking = WRITESET) • C is a replica where timing is done 44
  • 45. Previous Benchmark Results [1 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
  • 46. Previous Benchmark Results [2 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Observations: • High variation of speedups, very workload dependent (probably possible to do better if spending time optimizing the application) • HD gives better speedups than ND (normal because group commit) • ND gives some interesting speedups, but do not expect 16x faster not possible to use all CPUs of modern hardware on replica for writes
  • 47. HubSpot Benchmark Setup (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) The test environments involve two nodes: +---+ +---+ | A | -------> | B | +---+ +---+ • A is a production primary (Percona Server 5.7.36 with Write Set) (transaction_write_set_extraction = XXHASH64) (binlog_transaction_dependency_tracking = WRITESET) • B is a replica where timing is done 47
  • 48. HubSpot Benchmarks (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Observations: • Higher HD speedups for F{1,2} than E* • Better HD vs ND single-threaded for F* than E* (F1 being much better) Something is different between E and F.
  • 49. Benchmark Discussion (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Previous was on local SSD, HubSpot is EBS in AWS • Disk-sync more expensive for HubSpot • Which explains better HD results • Also explains better HD vs ND single-threaded speedup • (Also, make benchmarking hard because AWS variance) (HubSpot is running production with ND + semi-sync) Also, HubSpot is running with the previously recommended settings: • Write Set enabled on primaries and replicas • slave_parallel_type to LOGICAL_CLOCK • slave_preserve_commit_order to ON à Ready for MTR by updating slave_parallel_workers 49
  • 50. Other Benchmarks Results (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) I also have results for: • WO, and NO vs WO • RB, and IS vs RB Too long to present everything, contact me if interested TL&DR: • NO is faster than WO • RB is faster than IS • Both highly variable depending on workload and durability 50
  • 51. Other Parallel Replication Subjects (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) MySQL Parallel Replication Simulator https://blog.koehntopp.info/2021/11/08/mysql-parallel-replication.html Replica Group Commit (Slave Group Commit in MariaDB) https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2 • Replicating trx sequentially, in many threads for Group Committing • Almost as good as ND single-threaded speed, with benefit of HD Optimistic Parallel Replication (in MariaDB) https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-4-more-benchmarks-in-production-49ee255043ab • Go beyond scheduling barriers on replica, achieving replication prefetching Facebook partial transaction parallel execution in MySQL 5.6 https://www.percona.com/live/e18/sessions/faster-mysql-replication-using-row-dependencies • Using Write Set on replicas for fine grain parallel statement excution 51
  • 52. Links [1 of 3] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Content related to this talk from the speaker: • A Metric for Tuning Parallel Replication in MySQL 5.7 https://jfg-mysql.blogspot.com/2017/02/metric-for-tuning-parallel-replication-mysql-5-7.html • Better Parallel Replication for MySQL (Binlog Server) https://medium.com/booking-com-infrastructure/better-parallel-replication-for-mysql-14e2d7857813 • An update on Write Set (parallel replication) bug fix in MySQL 8.0 https://jfg-mysql.blogspot.com/2018/01/an-update-on-write-set-parallel-replication-bug-fix-in-mysql-8-0.html Binlog Server content from the speaker: • MySQL Slave Scaling (and more) https://medium.com/booking-com-infrastructure/mysql-slave-scaling-and-more-a09d88713a20 • Abstracting Binlog Servers and MySQL Master Promotion without Reconfiguring all Slaves https://medium.com/booking-com-infrastructure/abstracting-binlog-servers-and-mysql-master-promotion-without-reconfiguring-all-slaves-44be1febc8a0 Other parallel replication content from the speaker: • Slave Group Commit https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2 • Optimistic Parallel Replication https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-4-more-benchmarks-in-production-49ee255043ab 52
  • 53. Links [2 of 3] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Related bugs (some fixed, some still open) for the full history: • Bug#75396: Allow slave_preserve_commit_order without log-slave-updates. • Bug#81840: Automatic Replication Recovery Does Not Handle Lost Events (applies to MTR) • Bug#85965: Expose, on the master, counters for monitoring // information quality. • Bug#85966: Expose, on slaves, counters for monitoring // information quality. • Bug#85977: The doc. for LOGICAL_CLOCK wrongly references Group Commit • Bug#89247: Deadlock with MTS when slave_preserve_commit_order = ON • PS-8030: Multithreaded replica crashes [at STOP SLAVE] inside rli::cannot_safely_rollback() • I think there is still a deadlock bug in 5.7 (at least in Percona Server 5.7.36), troubleshooting in progress at HubSpot • Also, there might be other bugs 53
  • 54. Links [3 of 3] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) And more related content: • Solving MySQL Replication Lag with LOGICAL_CLOCK and Calibrated Delay https://orangematter.solarwinds.com/2017/01/13/solving-mysql-replication-lag-with-logical_clock-and-calibrated-delay/ • How to Fix a Lagging MySQL Replication https://thoughts.t37.net/fixing-a-very-lagging-mysql-replication-db6eb5a6e15d • Estimating potential for MySQL 5.7 parallel replication https://www.percona.com/blog/2016/02/10/estimating-potential-for-mysql-5-7-parallel-replication/ • How Binary Logs (and Filesystems) Affect MySQL Performance https://www.percona.com/blog/2018/05/04/how-binary-logs-and-filesystems-affect-mysql-performance/ • A Dive Into MySQL Multi-Threaded Replication https://www.percona.com/blog/a-dive-into-mysql-multi-threaded-replication/ • MySQL Parallel Replication Simulator https://blog.koehntopp.info/2021/11/08/mysql-parallel-replication.html 54
  • 55. Thanks ! by Jean-François Gagné System and MySQL Expert at HubSpot Presented at Percona Live Austin 2022 jfg.mysql AT gmail DOT com Twitter: @jfg956