1. Tracking Page Changes
for Your Database and Backups
Laurynas
Biveinis

2. 2
Agenda
• Incremental XtraBackup: performance
• Incremental XtraBackup with bitmaps:
performance
• Server overhead
• INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• Implementation
– Bitmap file format
– New server thread

3. 3
Incremental XtraBackup:
Performance
• Does time to backup depend on the % of changed data?
100%
90%
80%
70%
Backup Time
60%
50%
40%
30%
20%
10%
0%
0.10% 1.00% 10.00% 100.00%
Delta Size

4. 4
Incremental XtraBackup: How Data
Page Copying Works
Can we avoid reading
the old pages? Base Backup
LSN = 1000
table.ibd
LSN = 950 read
LSN = 960 read LSN
LSN = 1002 read > write
LSN = 1003 read 1000 write
LSN = 940 read ? Table.ibd.delta
LSN = 1010 read write

5. 5
Incremental XtraBackup: Can We
Avoid Reading the Old Pages?
• http://bit.ly/FBIncBackup

6. 6
Incremental XtraBackup: Can We
Avoid Reading the Old Pages?
• How do we know which pages to read then?
• Two ways to get the modification LSN of a page:
– It is written on the page, - or -
– We can figure it out from the redo log
• The log is cyclical, we must, in the server, save the
info before it is overwritten
• --innodb-track-changed-pages=TRUE
– Percona Server 5.1.67-14.4, 5.5.29-30.0, 5.6.
– Percona XtraBackup 2.1 – zero configuration!

7. 7
Incremental XtraBackup with
Changed Page Tracking
…
Changed pages between
Percona LSNs 980 and 1020: Base Backup
Server 1002, 1003, 1010 LSN = 1000
...
table.ibd
LSN = 950
LSN = 960 LSN
LSN = 1002 read > write
LSN = 1003 read 1000 write
LSN = 940 ? Table.ibd.delta
LSN = 1010 read write

8. 8
Incremental XtraBackup with Changed Page Tracking:
Performance
100%
90%
80%
70%
Backup Time
60%
Full Scan
50% Bitmap
40%
30%
20%
10%
0%
0.00% 0.01% 1.00% 100.00%
Delta Size

9. 9
Percona Server with Changed Page
Tracking: Server Overhead
• Nothing is ever free!
– But the price might be very well
acceptable
• Potential overhead #1: extra disk
space requirements
• Potential overhead #2: extra code
running in the server

10. 1
0
Percona Server with Changed Page
Tracking: Server Overhead
Log and bitmap file size comparison
800
Log bytes / bitmap byte
700
600
500
400
300
200
100
0
1 2 3 4 5 6 7 8
Bitmap file #

11. 1
1
Percona Server with Changed Page
Tracking: Server Overhead
• Impact on TPS and response time:
– It's a wash
– With
--innodb_flush_method=fdatasync

12. 1
2
Bitmap File Naming & Sizing
• ib_modified_log_<seq>_<LSN>.xdb
– <Seq>: 1, 2, 3, ...
– <LSN>: the server LSN at the file create
time
• Rotated on
– Server start
– innodb_max_bitmap_file_size

13. 1
3
Bitmap File Management
• PURGE CHANGED_PAGE_BITMAPS BEFORE <lsn>
– ib_1_8192.xdb
– ib_2_10000.xdb
– ib_3_20000.xdb
– Full backup taken, LSN = 22000
– PURGE C_P_B BEFORE 22000;
– ib_4_30000.xdb
– Incremental backup taken, LSN = 33000
– PURGE C_P_B BEFORE 33000;

14. 1
4
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• Percona Server can read the bitmaps too
SHOW CREATE TABLE INFORMATION_SCHEMA.INNODB_CHANGED_PAGES;
CREATE TABLE `INNODB_CHANGED_PAGES` (
`space_id` int(11) unsigned NOT NULL DEFAULT '0',
`page_id` int(11) unsigned NOT NULL DEFAULT '0',
`start_lsn` bigint(21) unsigned NOT NULL DEFAULT '0',
`end_lsn` bigint(21) unsigned NOT NULL DEFAULT '0'
)
• start_lsn and end_lsn are always at the checkpoint boundary
• Does not show the exact LSN of a change
• Does not show the number of changes for one page
• Does show the number of flushes for a page over the workload

15. 1
5
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
SELECT * FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES;
space_id page_id start_lsn end_lsn
0 0 8204 38470
0 1 8204 38470
5 0 8204 38470
5 3 8204 38470
0 1 38471 50000
5 3 38471 50000
5 3 50001 60000
• Don't query like that in production!
– It will read all the bitmaps you have. Gigabytes, terabytes, ...
– Add WHERE start_lsn > X AND end_lsn < Y (index condition pushdown
implemented for this case)

16. 1
6
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• Which tables are written to?
SELECT DISTINCT space_id FROM
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES WHERE ...;
space_id
0
10
SELECT DISTINCT t1.space_id AS space_id, t2.schema AS db,
t2.name AS tname
FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES AS t1,
INFORMATION_SCHEMA.INNODB_SYS_TABLES AS t2
WHERE t1.space_id = t2.space AND t1.start_lsn >...
space_id db tname
0 SYS_FOREIGN
0 SYS_FOREIGN_COLS
10 test foo

17. 1
7
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• What are the hottest tables?
SELECT space_id,
COUNT(space_id) AS number_of_flushes
FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
GROUP BY space_id
ORDER BY number_of_flushes DESC;
space_id number_of_flushes
0 65
10 5
11 4

18. 1
8
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• What are the hottest pages?
SELECT space_id, page_id,
COUNT(page_id) AS number_of_flushes
FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
GROUP BY space_id, page_id
HAVING number_of_flushes > 2
ORDER BY number_of_flushes DESC
LIMIT 8;
space_id page_id number_of_flushes
0 5 3
0 7 3
0 0 2
0 11 2
10 3 2
0 1 2
0 12 2
0 2 2

19. 1
9
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• For complex queries, copy data first
CREATE TEMPORARY TABLE icp (
space_id INT(11) NOT NULL,
page_id INT(11) NOT NULL,
start_lsn BIGINT(21) NOT NULL,
end_lsn BIGINT(21) NOT NULL,
INDEX page_id(space_id, page_id),
INDEX start_lsn(start_lsn),
INDEX end_lsn(end_lsn)) ENGINE=InnoDB;
INSERT INTO icp SELECT * FROM
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES WHERE
start_lsn > 8000;

20. 2
0
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• For complex queries, copy data first
EXPLAIN SELECT DISTINCT space_id FROM
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES;
id select_type table type possible_keys key key_len
ref rows Extra
1 SIMPLE INNODB_CHANGED_PAGES ALL NULL NULL NULL
NULL NULL Using temporary
EXPLAIN SELECT DISTINCT space_id FROM icp;
id select_type table type possible_keys key key_len
ref rows Extra
1 SIMPLE icp index NULL page_id 8 NULL 74 Using
index

21. 2
1
Implementation: File Format
A sequence of per-checkpoint varying number of data pages:
Data for checkpoint at LSN 9000
LSN 10000
LSN 10500
For each checkpoint: 4KB
space, start page space, start page space, start page
Each page contains a bitmap for the next 32480 pages in space starting
from start page

22. 2
2
Implementation: Server Side
• A new XtraDB thread
– 1. Wait for log checkpoint completed event
– 2. Read the log up to the checkpoint, write the bitmap
– 3. goto 1
• Little data sharing with the rest of XtraDB
– log_sys->mutex for:
● setting and getting LSNs;
● calculating log read offset from LSN.
• Little extra code for the query threads
– Unread log overwrite check
– Firing of the log checkpoint completed event
–

23. 2
3
Implementation: Things We Had to
Account For
• Maximum checkpoint age violation
– Destroys untracked log data
– Make effort to avoid, but in the end we
allow to overwrite it
– Responding server > fast backups
• Crash recovery
– Re-read the log if available

24. 2
4
Conclusions
• Percona Server together with Percona
XtraBackup:
• Enable faster incremental backups
• Enable more frequent incremental backups
• Does not hurt server operation, but have to
manage the bitmaps now
• New INFORMATION_SCHEMA table for gaining
insight into data change patterns
• Thank you! Questions?