xtradb 5.6: key performance algorithms / plmce 2014

XtraDB 5.6Key Performance Algorithms

Laurynas BiveinisAlexey Stroganov2014-04-02

Agenda

• Background: InnoDB and XtraDB in 5.5• XtraDB 5.6 mk I: InnoDB with trx descriptors• XtraDB 5.6 mk II: 5.5 performance patches

revisited• XtraDB 5.6 mk III: flushing rework• XtraDB 5.6: the rest and benchmarks

2

5.5

• InnoDB 5.5 & XtraDB 5.5

3

InnoDB 5.5: Buffer Pool4

Flush list: …

LRU list: …

Free list: …

Page hash:

flush list mutex

buffer pool mutex

Accessed on:

Page dirtying. Master thread checkpoint age flush

Page access. Query thread flush to make space in buffer pool

Flush to make space in buffer pool

Page lookup

MySQL solution for the contention5

Flush list 1: …

LRU list 1: …

Free list 1: …

Page hash 1:

flush list mutex 1

buffer pool mutex 1

Flush list 2: …

LRU list 2: …

Free list 2: …

Page hash 2:

flush list mutex 2

buffer pool mutex 2

XtraDB BP mutex split solution6

Flush list: …

LRU list: …

Free list: …

Page hash:

flush list mutex

LRU list mutex

free list mutex

page hash latch

The solutions may be combined7

Flush list: …

LRU list 1: …

Free list 1: …

Page hash 1:

Flush list: …

LRU list 2: …

Free list 2: …

Page hash 2:

InnoDB 5.5: trx list8

1. TRX_ACTIVE

kernel_mutex:

2. TRX_ACTIVE

3. TRX_COMMITTED_IN_MEMORY

trx_sys->trx_list:

…

CreateReadView

listscan

view->trx_ids:

1

2

…

XtraDB 5.5: trx descriptors9

kernel_mutex:

trx_sys->descriptorsCreateReadview memcpy

view->descriptors:

1 2 … 1 2

• http://bugs.mysql.com/bug.php?id=49169• http://bit.ly/pstrx1• http://bit.ly/pstrxdesc2

http://bugs.mysql.com/bug.php?id=49169

http://bit.ly/pstrx1

InnoDB 5.5: AHI10

4

2 3

. 7

1 5 64 8 97

. .

t1

d

b c

. g

a e fd h jg

. .

t2

t1.3

t1.5

t1.7

t2.c

t2.g

btr_search_sys->hash_index

btr_search_latch:

XtraDB 5.5: AHI partitions11

4

2 3

. 7

1 5 64 8 97

. .

t1

d

b c

. g

a e fd h jg

. .

t2

t1.3

t1.5

t1.7

t2.c

t2.g

btr_search_sys->hash_tables[0]

btr_search_latch[0]:

btr_search_latch[1]:

btr_search_sys->hash_tables[1]

XtraDB 5.6 mk I

• Can we drop the scalability patches?– InnoDB 5.6 introduced cleaner thread, http://bit.ly/ibclean– What it does:

● Flush LRU tail, somehow, innodb_lru_scan_depth● Flush flush list, adaptively, innodb_io_capacity[_max]● Sleep until 1 s completes● Goto 1

– http://bit.ly/markexplainslru● We knew we could not drop the trx descriptor patch

12

http://bit.ly/ibclean

http://bit.ly/markexplainslru

XtraDB 5.6 mk I: InnoDB / trx desc13


• Why XtraDB 5.6 mk I scales worse than XtraDB 5.5?– AHI contention

● OK, should have ported it too at once, no changes there in 5.6

● http://bugs.mysql.com/bug.php?id=62018– Buffer pool mutex contention

● No problem, we have a patch, right?

XtraDB 5.6 mk II: 5.5 patches

• Ported the AHI partition patch– Improved it during the port.

http://bugs.mysql.com/bug.php?id=70216

– Worked as expected

• Ported the buffer pool mutex split–Attempted to improve it during the port too

– InnoDB 5.6 split off page hash latch

–And it did not help much/at all

16

BP Access Concurrency Analysis, 5.517

master threadtime

flush list flush

flush list flush

flush list flush

query thread 1

Make page young

Make page young

LRU list flush

query thread 2

LRU list flush

Make page young

Make page young

BP Access Concurrency Analysis, 5.618

cleaner threadtime

LRU list flush

flush list flush

LRU list flush

query thread 1

Make page young

Make page young

LRU list flush

query thread 2

LRU list flush

Make page young

Make page young

flush list flush

XtraDB 5.6 mk III: flushing rework

• Issue 1: cleaner runs the flushing, but does the cleaner itself run?

• One of the reasons innodb_thread_concurrency helps...

19

100% CPU run time share

few query threads cleaner

many query threads c


• Issue 1 solution: increase the cleaner priority

20


few query threads cleaner many query threads cleaner


• setpriority()• --innodb_sched_priority_cleaner=n

21


few query threads cleaner many query threads cleaner

XtraDB 5.6 mk III: flushing rework22


• Issue 2: What happens in the case of an empty free list in some instance?

• Query thread:– Lock (buffer pool|free list) mutex– If free list empty, unlock the mutex, do

something (*), goto 1– * - query thread eviction / LRU flush attempt,

sleep before looping

23


• Issue 2. Still no free page, so given enough query threads...– Thread 1 waiting for the mutex– Thread 2 has the mutex, checks

for empty free list– Thread n waiting for the mutex

24


• Issue 2. Cleaner produced a free page!– Thread 1 waiting for the mutex– Thread 2 has the mutex, checks for

empty free list– Thread n waiting for the mutex– Cleaner thread waiting for the mutex

25


• Issue 2. Cleaner produced a free page, so what?– Thread 2 unlocked the mutex– Thundering herd of 1, 3..n, and cleaner threads,

mutex_signal_object()– The chance that cleaner thread gets the mutex

is roughly 1 / n– No mutex, no free page, n likely to

increase

26


• Issue 2. Solution: obviously, the cleaner must get the free list mutex before anybody else

• Adapted the sync array code• mutex->high_priority_event• mutex_enter_first()

27


• Issue 1+2: what applies to cleaner, applies to other background threads as well– Under high concurrency, background

threads must have priority over query threads to keep the server in the steady state

– State is: checkpoint age, history length, ...

28


• Issue 3. Why do query threads try to (LRU|flush list) flush anyway?

• Because the cleaner thread failed to do so in time

• Is it a good idea to increase mutex pressure if the cleaner thread is already in trouble?

29


• Issue 3 solution: do not let query threads flush

• --innodb_empty_free_list_algorithm =backoff

• --innodb_foreground_preflush =exponential_backoff

• http://bit.ly/noqueryflush

30

http://bit.ly/noqueryflush

XtraDB 5.6 mk III: flushing rework31


• Issue 4: why do we have empty free lists in the first place?• mysql -uroot -e'show engine innodb status\G'| grep Free • Free buffers 1142• Free buffers 1235• Free buffers 1288• Free buffers 318• Free buffers 279• Free buffers 938• Free buffers 0• Free buffers 406

32

• Free buffers 312








• Issue 4: buffer pool instance usage is not uniform!– Mikael Ronstrom: “First the accesses to the buffer pools

is in no way evenly spread out.”– http://bit.ly/bpsplit

● But 5.6 flushing does not recognize that!

– LRU/flush list flush request for x pages across y instances:

● Request x/y flush for each instance● Assuming uniformity where there is none

33

http://bit.ly/bpsplit


• Current issue 4 solution attempt– “pseudoparallel” flushing: instead of issuing all

requests to each instance before advancing to the next one, issue requests across instances

– keep flushing an instance in a loop if its free list nearly empty

– innodb_free_list_lwm, default 100, UNIV_PERF_DEBUG tuning var

34


• Proper issue 4 solution: parallel LRU flushers for each instance– Future work...– 5.7 MT flushing is a step to right

direction but it must be ensured that it does the right thing

35


• Talking of parallel LRU flushers...• Busy cleaner thread:

36

flush list flush LRU list flush flush list flush LRU list flush ...

Empty free list danger here

Max checkpoint age danger here


• Solution is separate flush list and LRU flush threads

37

flush list flush

LRU list flush

flush list flush

LRU list flush

flush list flush

XtraDB 5.6: the rest

• Last but not least:– Adaptive flushing formula tweak to

maintain higher checkpoint age– Efficient log block checksums– http://bugs.mysql.com/bug.php?

id=70453

38

XtraDB 5.639

xtradb 5.6: key performance algorithms / plmce 2014

Software

trx list

flush list flush query

depth flush flush list

mutex thread

query thread flush

mutex cleaner thread

free list thread n

master threadtime flush