Presentation on ReplacingMergeTree by Robert Hodges of Altinity at the 14 December 2022 SF Bay Area ClickHouse Meetup (https://www.meetup.com/san-francisco-bay-area-clickhouse-meetup/events/289605843/)

1. © 2022 Altinity, Inc.
Adventures with the ClickHouse
ReplacingMergeTree Engine
Mirroring data from OLTP databases to ClickHouse
Robert Hodges & Altinity Engineering
1
14 December 2022

2. © 2022 Altinity, Inc.
Let’s make some introductions
ClickHouse support and services including Altinity.Cloud
Authors of Altinity Kubernetes Operator for ClickHouse
and other open source projects
Robert Hodges
Database geek with 30+ years
on DBMS systems. Day job:
Altinity CEO
Altinity Engineering
Database geeks with centuries
of experience in DBMS and
applications
2

3. © 2022 Altinity, Inc.
How do different database types arise?
3
eCommerce
Inventory management
and purchasing
vs
Funnel analysis and
fraud detection
Digital Marketing Campaign management vs Campaign evaluation
Software Defined
Networking
Network topology and
micro-segment
definition
vs
Access patterns
analysis
MySQL - OLTP ClickHouse - Analytics

4. © 2022 Altinity, Inc.
OLTP vs OLAP – Key difference is storage organization
4
ClickHouse
Read only selected columns
Rows minimally or not compressed Columns highly compressed
PostgreSQL, MySQL
Read all columns in row

5. © 2022 Altinity, Inc. 5
Some data just
needs a copy in
a column store
WordPress
dynamic site data
eCommerce
transactions
Ad bidding
transactions
Online auction
transactions
Chat messages Mobile
provisioning
requests
Credit card
transactions
Financial market
transactions

6. © 2022 Altinity, Inc.
Mirroring copies data and keeps it up-to-date
6
Initial Dump/Load
MySQL ClickHouse
OLTP App Analytic App
MySQL
Binlog Real-time Replication

7. © 2022 Altinity, Inc.
So…What’s the problem?
7
Mutating Rows

8. © 2022 Altinity, Inc.
© 2022 Altinity, Inc.
Basics of
ReplacingMergeTree
8

9. © 2022 Altinity, Inc.
Let’s consider a source table in MySQL
CREATE TABLE `film` (
`film_id` smallint unsigned NOT NULL AUTO_INCREMENT,
`title` varchar(128) NOT NULL,
. . .
`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`film_id`),
KEY `idx_title` (`title`),
. . .
) ENGINE=InnoDB
9
Primary key

10. © 2022 Altinity, Inc.
CREATE TABLE sakila.film (
`film_id` UInt16,
`title` String,
`description` Nullable(String),
`release_year` Nullable(String),
. . .
`last_update` DateTime,
`_version` UInt64 DEFAULT 0,
`_sign` Int8 DEFAULT 1
)
ENGINE = ReplacingMergeTree(_version)
ORDER BY language_id, studio_id, film_id
Create a ClickHouse table to contain the mirrored data
10

11. © 2022 Altinity, Inc.
How ReplacingMergeTree works
11
0
3
3
1001 . . . . . .
1001 . . . . . .
1001
INSERT
_version
+1
-1
+1
_sign
(Other data columns)
fiilm
_id
UPDATE
DELETE
language_id
5 1001 . . . . . .
-1
studio_id
Eventually
consistent
replacement
of rows

12. © 2022 Altinity, Inc.
INSERT INTO sakila.film VALUES
(1001,'Blade Runner','Best. Sci-fi. Film. Ever.',
'1982',1,NULL,6,'0.99',117,'20.99','PG',
'Deleted Scenes,Behind the Scenes',now()
,0,1)
SELECT title, release_year
FROM film WHERE film_id = 1001
┌─title────────┬─release_year─┐
│ Blade Runner │ 1982 │
└──────────────┴──────────────┘
Adding a row to RMT table
12

13. © 2022 Altinity, Inc.
INSERT INTO sakila.film VALUES (1001,'Blade Runner',
'Best. Sci-fi. Film. Ever.',...,3,-1),
(1001,'Blade Runner - Director''s Cut','Best. Sci-fi. Film.
Ever.',...,3,1)
SELECT title, release_year
FROM film WHERE film_id = 1001
┌─title─────────────────────────┬─release_year─┐
│ Blade Runner - Director's Cut │ 1982 │
└───────────────────────────────┴──────────────┘
┌─title────────┬─release_year─┐
│ Blade Runner │ 1982 │
└──────────────┴──────────────┘
Updating a row in the RMT table
13
Unmerged
rows!

14. © 2022 Altinity, Inc.
Rows are replaced when merges occur
14
0
3
3
1001 1 . . .
1001 1 . . .
1001 2
+1
-1
+1
Part
Part
Merged Part
3 1001 1
-1
3 1001 2
+1
X
Pro tip: never assume rows
will merge full
?

15. © 2022 Altinity, Inc.
SELECT film_id, title
FROM sakila.film FINAL
WHERE film_id = 1001
┌─title─────────────────────────┬─release_year─┐
│ Blade Runner - Director's Cut │ 1982 │
└───────────────────────────────┴──────────────┘
FINAL keyword merges data dynamically
15
Adds initial scan to
merge rows

16. © 2022 Altinity, Inc.
INSERT INTO sakila.film VALUES
(1001,'Blade Runner - Director''s Cut',
'Best. Sci-fi. Film. Ever.',...,5,-1)
SELECT title, release_year, _version, _sign
FROM sakila.film FINAL
WHERE film_id = 1001
┌─title─────────────────────────┬─release_year─┬─_version─┬─_sign─┐
│ Blade Runner - Director's Cut │ 1982 │ 5 │ -1 │
└───────────────────────────────┴──────────────┴──────────┴───────┘
Deleting a row in RMT table
16
Deleted row!

17. © 2022 Altinity, Inc.
CREATE ROW POLICY
sakila_film_rp ON sakila.film
FOR SELECT USING sign != -1 TO ALL
SELECT title, release_year, _version, _sign
FROM sakila.film FINAL
WHERE film_id = 1001
Ok.
0 rows in set. Elapsed: 0.005 sec.
Row policies prevent deleted rows from showing up
17
Predicate automatically
added to queries

18. © 2022 Altinity, Inc.
SELECT inventory_id, film_id, title
FROM sakila.inventory AS i
INNER JOIN sakila.film AS f ON i.film_id = f.film_id
WHERE film.film_id = 1001
┌─inventory_id─┬─film_id─┬─title─────────────────────────┐
│ 1 │ 1001 │ Blade Runner - Director's Cut │
│ 1 │ 1001 │ Blade Runner │
└──────────────┴─────────┴───────────────────────────────┘
JOINs are tricky with RMT
18
Right side table does
not have FINAL!

19. © 2022 Altinity, Inc.
SELECT inventory_id, f.film_id, title
FROM sakila.inventory AS i
INNER JOIN (
SELECT film_id, title FROM sakila.film FINAL
)
AS f ON i.film_id = f.film_id
WHERE f.film_id = 1001
┌─inventory_id─┬─film_id─┬─title─────────────────────────┐
│ 1 │ 1001 │ Blade Runner - Director's Cut │
└──────────────┴─────────┴───────────────────────────────┘
Use a subquery with FINAL for right hand side table
19
Not elegant, but we can
work with it

20. © 2022 Altinity, Inc.
© 2022 Altinity, Inc.
Performance
tips
20

21. © 2022 Altinity, Inc.
CREATE TABLE sakila.film (
`film_id` UInt16,
`title` String,
. . .
`_version` UInt64 DEFAULT 0,
`_sign` Int8 DEFAULT 1
)
ENGINE = ReplacingMergeTree(_version)
ORDER BY language_id, studio_id, film_id
ORDER BY is critical for MergeTree performance
21
Row key goes
on right
Other cols go
to left
Pro tip: Use PRIMARY KEY to
prefix a long ORDER BY

22. © 2022 Altinity, Inc.
Use care on updates when ORDER BY has > 1 column
INSERT INTO sakila.film VALUES
(1001,'Blade Runner','Best. Sci-fi. Film. Ever.',
'1982',1,NULL,6,'0.99',117,'20.99','PG',
'Deleted Scenes,Behind the Scenes',now()
,3,-1),
(1001,'Blade Runner - Director''s Cut',
'Best. Sci-fi. Film. Ever.',
'1982',2,NULL,6,'0.99',120,'20.99','PG',
'Deleted Scenes,Behind the Scenes',now()
,3,1)
22
Must delete row if
ORDER BY
columns change!

23. © 2022 Altinity, Inc.
CREATE TABLE sakila.film (
`film_id` UInt16,
`title` String,
. . .
`_version` UInt64 DEFAULT 0,
`_sign` Int8 DEFAULT 1
)
ENGINE = ReplacingMergeTree(_version)
PARTITION BY intDiv(film_id, 10000000)
ORDER BY language_id, studio_id, film_id
Partitioning is important for large tables
23
Choose a partition
key that keeps row
changes local to
single partitions

24. © 2022 Altinity, Inc.
SELECT release_year, count()
FROM sakila.film FINAL
GROUP BY release_year
ORDER BY release_year
SETTINGS
do_not_merge_across_partitions_select_final = 1
Restrict FINAL merge scope to single partitions
24
Run faster;
parallelize across
partitions
Note: Run ClickHouse
22.10+ to use this feature*
* https://github.com/ClickHouse/ClickHouse/issues/43296

25. © 2022 Altinity, Inc.
© 2022 Altinity, Inc.
Current work to
improve
ReplacingMergeTree
25

26. © 2022 Altinity, Inc.
Just update from my side: the param min_age_to_force_merge_on_partition_only
works
Alexandr DubovikovSETTINGS min_age_to_force_merge_seconds = 120,
min_age_to_force_merge_on_partition_only = true;
Alexandr Dubovikovit will merge all parts at once
26

27. © 2022 Altinity, Inc.
# https://github.com/ClickHouse/ClickHouse/pull/40945
SET force_select_final = 1
SELECT inventory_id, film_id, title
FROM sakila.inventory AS i
INNER JOIN sakila.film AS f
ON i.film_id = f.film_id
WHERE film.film_id = 1001
┌─inventory_id─┬─film_id─┬─title─────────────────────────┐
│ 1 │ 1001 │ Blade Runner - Director's Cut │
└──────────────┴─────────┴───────────────────────────────┘
Add implicit FINAL at query level (Altinity)
27
Add FINAL
automatically to table

28. © 2022 Altinity, Inc.
# https://github.com/ClickHouse/ClickHouse/pull/41005
CREATE TABLE sakila.film (
`film_id` UInt16,
. . .
`_version` UInt64 DEFAULT 0,
`_sign` UInt8 DEFAULT 1
)
ENGINE = ReplacingMergeTree(_version, _sign)
ORDER BY language_id, studio_id, film_id
Eliminate deleted rows automatically (ContentSquare)
28
Delete column
processed by
RMT engine

29. © 2022 Altinity, Inc.
© 2022 Altinity, Inc.
Wrap up
29

30. © 2022 Altinity, Inc.
Fully wired, continuous replication based on RMT
30
Table Engine(s)
Initial Dump/Load
MySQL ClickHouse
OLTP App Analytic App
MySQL
Binlog
Debezium
Altinity Sink
Connector
Kafka*
Event
Stream
*Including Pulsar and RedPanda
ReplacingMergeTree

31. © 2022 Altinity, Inc.
Where is the documentation?
ClickHouse official docs – https://clickhouse.com/docs/
Altinity Blog – https://altinity.com/blog/
Altinity Sink Connector for ClickHouse –
https://github.com/Altinity/clickhouse-sink-connector
Altinity Knowledge Base – https://kb.altinity.com/
31

32. © 2022 Altinity, Inc.
Thank you!
Questions?
rhodges at altinity dot com
https://altinity.com
32