In this session we will discuss selected areas of InnoDB and XtraDB 5.7 internals that are mostly related to buffer pool management and flushing, from a performance and scalability point of view. We will describe the motivation behind the features in Percona Server XtraDB 5.7 (buffer pool mutex split; multi-threaded LRU flusher; parallel doublewrite), given that MySQL InnoDB 5.7 has reimplemented many similar features that we have in XtraDB 5.6 -- especially for disk-bound RW workloads. The talk will be technical, and assumes you have familiarity with InnoDB internals.
2. XtraDB 5.7 Key Performance
Algorithms
• Focus on the buffer pool, flushing, the doublewrite
buffer
• Talk assumes familiarity, but feel free to interrupt
• What we learned
• What we did
• How we did it
3. InnoDB buffer pool
• Memory cache of disk data pages
• In-memory data pages accessible through several
data structures
• 1) Page hash for lookup
(space_id; page_id) ?
Hash array Data page lists
Fold
4. InnoDB buffer pool
• 2) flush list for dirty page management. Dirtying:
Clean page
INSERT INTO foo VALUES(bar)
Dirty page, LSN = 42
Dirty page, LSN = 25 Dirty page, LSN = 32 Dirty page, LSN = 42
Flush list tail:
5. InnoDB buffer pool
• 2) flush list for dirty page management. Flushing:
Dirty page, LSN = 5 Dirty page, LSN = 7 Dirty page, LSN = 12
Flush list head:
Flush up to LSN 10
Clean page
Clean page
Dirty page, LSN = 12
Flush list head:
6. InnoDB buffer pool
• 3) LRU list for deciding which pages to evict
• Preventing eviction for recently-used pages
(making them young):
Dirty pageClean page Dirty page Clean page Clean page
Page access
Dirty pageClean page Dirty page Clean pageClean page
7. InnoDB buffer pool
• 4) free list for having free space in the buffer pool
to read currently non-present pages. Reading:
Free page Free page Free page Free page Free page
Page read
Free page Free page Free page Free page
Clean page
8. InnoDB buffer pool
• 3/4) Evicting/flushing pages from the LRU list and
putting them on the free list:
Dirty pageClean page Dirty page Clean page Clean page
Free page Free page Free page Free page
Dirty page Dirty page Clean page Clean page
Free page Free page Free page Free page Free page
10. Buffer pool concurrency
flush list
LRU list
free listpage hash
misc.
buffer pool mutexflush list mutexpage hash latch
11. Buffer pool instances
flush list LRU list
free list
page hash
misc.
buffer pool mutexflush list mutexpage hash latch
buffer pool instance 0
flush list LRU list
free list
page hash
misc.
buffer pool mutexflush list mutexpage hash latch
buffer pool instance 1
12. Buffer pool instances
• Problem: some instances are cold and some are
hot
• “First the accesses to the buffer pools is in no way
evenly spread out.”
• http://bit.ly/bpsplit
• Six year-old quote, still relevant the same today
13. Concurrency in XtraDB
flush listpage hash
flush list mutexpage hash latch
LRU list
LRU list mutex
free list
free list mutex
misc
misc mutex /
atomics
Patch contributed to MySQL, and merged in 8.0.0
http://bugs.mysql.com/bug.php?id=75534
14. Concurrency solutions are
compatible
flush listpage hash
flush list mutexpage hash latch
LRU list
LRU list mutex
free list
free list mutex
misc
misc mutex /
atomics
buffer pool instance 0
buffer pool instance 1
flush listpage hash
flush list mutexpage hash latch
LRU list
LRU list mutex
free list
free list mutex
misc
misc mutex /
atomics
15. Buffer pool mutexes are so
5.5
Improvement
by the buffer
pool mutex
split
Improvement
by adaptive
flushing
16. 5.6+ changed things
• In 5.5 and earlier: reduce mutex contention by X%,
observe TPS increase by ~X%
• Changing flushing heuristics is driven by
performance stability, not necessarily by peak
performance
• Pre-release Percona Server 5.6: reduce mutex
contention by X%, observe TPS increase by ~0%
• What happened? InnoDB cleaner thread happened
17. Buffer pool / flushing
concurrency in 5.5
Time Master thread Query thread 1 Query thread 2
flush list flush
flush list flush
flush list flush
make page young
make page young
LRU list flush
make page young
LRU list flush
make page young
18. Buffer pool / flushing
concurrency in 5.6+
Time Cleaner thread Query thread 1 Query thread 2
flush list flush
flush list flush
flush list flush
make page young
make page young
LRU list flush
make page young
LRU list flush
make page young
LRU list flush
19. Buffer pool / flushing
concurrency in 5.6+
• In 5.6+, code-level changes to reduce locking
granularity are still important, but
• Increasing thread specialization means that…
• …flushing - including LRU - heuristics are very
important now
20. MySQL 5.7 multi-threaded
flushing
LRU instance #0 flush list instance #0
LRU instance #1 flush list instance #1
LRU instance #2 flush list instance #2
coordinator thread
worker thread #0
worker thread #1
time
0 s 1 s
LRU…
LRU…
LRU…
43. Summary: 5.7 story
• I/O-bound workloads: high demand for free pages,
provided by LRU batch flushing or single-page flushing
• Single-page flushes are bad, w/ and w/o doublewrite
• Removed it
• Made batch LRU flusher truly parallel
• Doublewrite buffer negates parallel flushing gains
• Made it parallel too