![Page 1: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/1.jpg)
XtraDB 5.6Key Performance Algorithms
Laurynas BiveinisAlexey Stroganov2014-04-02
![Page 2: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/2.jpg)
Agenda
• Background: InnoDB and XtraDB in 5.5• XtraDB 5.6 mk I: InnoDB with trx descriptors• XtraDB 5.6 mk II: 5.5 performance patches
revisited• XtraDB 5.6 mk III: flushing rework• XtraDB 5.6: the rest and benchmarks
2
![Page 3: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/3.jpg)
5.5
• InnoDB 5.5 & XtraDB 5.5
3
![Page 4: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/4.jpg)
InnoDB 5.5: Buffer Pool4
Flush list: …
LRU list: …
Free list: …
Page hash:
flush list mutex
buffer pool mutex
Accessed on:
Page dirtying. Master thread checkpoint age flush
Page access. Query thread flush to make space in buffer pool
Flush to make space in buffer pool
Page lookup
![Page 5: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/5.jpg)
MySQL solution for the contention5
Flush list 1: …
LRU list 1: …
Free list 1: …
Page hash 1:
flush list mutex 1
buffer pool mutex 1
Flush list 2: …
LRU list 2: …
Free list 2: …
Page hash 2:
flush list mutex 2
buffer pool mutex 2
![Page 6: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/6.jpg)
XtraDB BP mutex split solution6
Flush list: …
LRU list: …
Free list: …
Page hash:
flush list mutex
LRU list mutex
free list mutex
page hash latch
![Page 7: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/7.jpg)
The solutions may be combined7
Flush list: …
LRU list 1: …
Free list 1: …
Page hash 1:
Flush list: …
LRU list 2: …
Free list 2: …
Page hash 2:
![Page 8: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/8.jpg)
InnoDB 5.5: trx list8
1. TRX_ACTIVE
kernel_mutex:
2. TRX_ACTIVE
3. TRX_COMMITTED_IN_MEMORY
trx_sys->trx_list:
…
CreateReadView
listscan
view->trx_ids:
1
2
…
![Page 9: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/9.jpg)
XtraDB 5.5: trx descriptors9
kernel_mutex:
trx_sys->descriptorsCreateReadview memcpy
view->descriptors:
1 2 … 1 2
• http://bugs.mysql.com/bug.php?id=49169• http://bit.ly/pstrx1• http://bit.ly/pstrxdesc2
![Page 10: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/10.jpg)
InnoDB 5.5: AHI10
4
2 3
. 7
1 5 64 8 97
. .
t1
d
b c
. g
a e fd h jg
. .
t2
t1.3
t1.5
t1.7
t2.c
t2.g
btr_search_sys->hash_index
btr_search_latch:
![Page 11: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/11.jpg)
XtraDB 5.5: AHI partitions11
4
2 3
. 7
1 5 64 8 97
. .
t1
d
b c
. g
a e fd h jg
. .
t2
t1.3
t1.5
t1.7
t2.c
t2.g
btr_search_sys->hash_tables[0]
btr_search_latch[0]:
btr_search_latch[1]:
btr_search_sys->hash_tables[1]
![Page 12: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/12.jpg)
XtraDB 5.6 mk I
• Can we drop the scalability patches?– InnoDB 5.6 introduced cleaner thread, http://bit.ly/ibclean– What it does:
● Flush LRU tail, somehow, innodb_lru_scan_depth● Flush flush list, adaptively, innodb_io_capacity[_max]● Sleep until 1 s completes● Goto 1
– http://bit.ly/markexplainslru● We knew we could not drop the trx descriptor patch
12
![Page 13: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/13.jpg)
XtraDB 5.6 mk I: InnoDB / trx desc13
![Page 14: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/14.jpg)
XtraDB 5.6 mk I: InnoDB / trx desc14
![Page 15: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/15.jpg)
XtraDB 5.6 mk I: InnoDB / trx desc15
• Why XtraDB 5.6 mk I scales worse than XtraDB 5.5?– AHI contention
● OK, should have ported it too at once, no changes there in 5.6
● http://bugs.mysql.com/bug.php?id=62018– Buffer pool mutex contention
● No problem, we have a patch, right?
![Page 16: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/16.jpg)
XtraDB 5.6 mk II: 5.5 patches
• Ported the AHI partition patch– Improved it during the port.
http://bugs.mysql.com/bug.php?id=70216
– Worked as expected
• Ported the buffer pool mutex split–Attempted to improve it during the port too
– InnoDB 5.6 split off page hash latch
–And it did not help much/at all
16
![Page 17: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/17.jpg)
BP Access Concurrency Analysis, 5.517
master threadtime
flush list flush
flush list flush
flush list flush
query thread 1
Make page young
Make page young
LRU list flush
query thread 2
LRU list flush
Make page young
Make page young
![Page 18: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/18.jpg)
BP Access Concurrency Analysis, 5.618
cleaner threadtime
LRU list flush
flush list flush
LRU list flush
query thread 1
Make page young
Make page young
LRU list flush
query thread 2
LRU list flush
Make page young
Make page young
flush list flush
![Page 19: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/19.jpg)
XtraDB 5.6 mk III: flushing rework
• Issue 1: cleaner runs the flushing, but does the cleaner itself run?
• One of the reasons innodb_thread_concurrency helps...
19
100% CPU run time share
few query threads cleaner
many query threads c
![Page 20: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/20.jpg)
XtraDB 5.6 mk III: flushing rework
• Issue 1 solution: increase the cleaner priority
20
100% CPU run time share
few query threads cleaner many query threads cleaner
![Page 21: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/21.jpg)
XtraDB 5.6 mk III: flushing rework
• setpriority()• --innodb_sched_priority_cleaner=n
21
100% CPU run time share
few query threads cleaner many query threads cleaner
![Page 22: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/22.jpg)
XtraDB 5.6 mk III: flushing rework22
![Page 23: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/23.jpg)
XtraDB 5.6 mk III: flushing rework
• Issue 2: What happens in the case of an empty free list in some instance?
• Query thread:– Lock (buffer pool|free list) mutex– If free list empty, unlock the mutex, do
something (*), goto 1– * - query thread eviction / LRU flush attempt,
sleep before looping
23
![Page 24: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/24.jpg)
XtraDB 5.6 mk III: flushing rework
• Issue 2. Still no free page, so given enough query threads...– Thread 1 waiting for the mutex– Thread 2 has the mutex, checks
for empty free list– Thread n waiting for the mutex
24
![Page 25: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/25.jpg)
XtraDB 5.6 mk III: flushing rework
• Issue 2. Cleaner produced a free page!– Thread 1 waiting for the mutex– Thread 2 has the mutex, checks for
empty free list– Thread n waiting for the mutex– Cleaner thread waiting for the mutex
25
![Page 26: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/26.jpg)
XtraDB 5.6 mk III: flushing rework
• Issue 2. Cleaner produced a free page, so what?– Thread 2 unlocked the mutex– Thundering herd of 1, 3..n, and cleaner threads,
mutex_signal_object()– The chance that cleaner thread gets the mutex
is roughly 1 / n– No mutex, no free page, n likely to
increase
26
![Page 27: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/27.jpg)
XtraDB 5.6 mk III: flushing rework
• Issue 2. Solution: obviously, the cleaner must get the free list mutex before anybody else
• Adapted the sync array code• mutex->high_priority_event• mutex_enter_first()
27
![Page 28: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/28.jpg)
XtraDB 5.6 mk III: flushing rework
• Issue 1+2: what applies to cleaner, applies to other background threads as well– Under high concurrency, background
threads must have priority over query threads to keep the server in the steady state
– State is: checkpoint age, history length, ...
28
![Page 29: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/29.jpg)
XtraDB 5.6 mk III: flushing rework
• Issue 3. Why do query threads try to (LRU|flush list) flush anyway?
• Because the cleaner thread failed to do so in time
• Is it a good idea to increase mutex pressure if the cleaner thread is already in trouble?
29
![Page 30: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/30.jpg)
XtraDB 5.6 mk III: flushing rework
• Issue 3 solution: do not let query threads flush
• --innodb_empty_free_list_algorithm =backoff
• --innodb_foreground_preflush =exponential_backoff
• http://bit.ly/noqueryflush
30
![Page 31: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/31.jpg)
XtraDB 5.6 mk III: flushing rework31
![Page 32: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/32.jpg)
XtraDB 5.6 mk III: flushing rework
• Issue 4: why do we have empty free lists in the first place?• mysql -uroot -e'show engine innodb status\G'| grep Free • Free buffers 1142• Free buffers 1235• Free buffers 1288• Free buffers 318• Free buffers 279• Free buffers 938• Free buffers 0• Free buffers 406
32
• Free buffers 312
• Free buffers 687
• Free buffers 696
• Free buffers 910
• Free buffers 1029
• Free buffers 993
• Free buffers 537
![Page 33: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/33.jpg)
XtraDB 5.6 mk III: flushing rework
• Issue 4: buffer pool instance usage is not uniform!– Mikael Ronstrom: “First the accesses to the buffer pools
is in no way evenly spread out.”– http://bit.ly/bpsplit
● But 5.6 flushing does not recognize that!
– LRU/flush list flush request for x pages across y instances:
● Request x/y flush for each instance● Assuming uniformity where there is none
33
![Page 34: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/34.jpg)
XtraDB 5.6 mk III: flushing rework
• Current issue 4 solution attempt– “pseudoparallel” flushing: instead of issuing all
requests to each instance before advancing to the next one, issue requests across instances
– keep flushing an instance in a loop if its free list nearly empty
– innodb_free_list_lwm, default 100, UNIV_PERF_DEBUG tuning var
34
![Page 35: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/35.jpg)
XtraDB 5.6 mk III: flushing rework
• Proper issue 4 solution: parallel LRU flushers for each instance– Future work...– 5.7 MT flushing is a step to right
direction but it must be ensured that it does the right thing
35
![Page 36: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/36.jpg)
XtraDB 5.6 mk III: flushing rework
• Talking of parallel LRU flushers...• Busy cleaner thread:
36
flush list flush LRU list flush flush list flush LRU list flush ...
Empty free list danger here
Max checkpoint age danger here
![Page 37: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/37.jpg)
XtraDB 5.6 mk III: flushing rework
• Solution is separate flush list and LRU flush threads
37
flush list flush
LRU list flush
flush list flush
LRU list flush
flush list flush
![Page 38: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/38.jpg)
XtraDB 5.6: the rest
• Last but not least:– Adaptive flushing formula tweak to
maintain higher checkpoint age– Efficient log block checksums– http://bugs.mysql.com/bug.php?
id=70453
38
![Page 39: XtraDB 5.6: Key Performance Algorithms / PLMCE 2014](https://reader033.vdocument.in/reader033/viewer/2022052906/558aa0e4d8b42a996a8b4585/html5/thumbnails/39.jpg)
XtraDB 5.639