the new replication features in mysql 8 - percona · since epoch, per transaction. • ... issuing...

69
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. The New Replication Features in MySQL 8 Luís Soares ([email protected]) Principal Software Engineer, MySQL Replication Lead Tuesday, 25th April 2017 Percona Live, CA, US 1

Upload: phamngoc

Post on 02-Jul-2018

225 views

Category:

Documents


0 download

TRANSCRIPT

Copyright © 2017, Oracle and/or its affiliates. All rights reserved.

The New Replication Features in MySQL 8

Luís Soares ([email protected])Principal Software Engineer, MySQL Replication Lead

Tuesday, 25th April 2017 Percona Live, CA, US 1

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

2Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Program Agenda

Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Background

Features

Roadmap

Conclusion

Program Agenda

Tuesday, 25th April 2017 Percona Live, CA, US 4

1

2

3

4

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Background

Tuesday, 25th April 2017 Percona Live, CA, US

1

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Database Replication

6Tuesday, 25th April 2017 Percona Live, CA, US

ServerB

ServerA

App

INSERT ...

INSERT ... INSERT ...

Replication“The process of generating and reproducing multiple copies of data at one or more sites.”,Database Systems: A Practical Approach to Design, Implementation, and Management, Thomas M. Connolly, Carolyn E. Begg, Third Edition, 2002.

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

MySQL Database Replication: Overview

7Tuesday, 25th April 2017 Percona Live, CA, US

INSERT ...Server

B

binary log

INSERT ...

relay log

INSERT ...Server

A

binary log

App

ReceiverMeta-data

Update

ApplierMeta-data

Update

INSERT ...

Comm. Framework

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

MySQL Database Replication: Overview

8Tuesday, 25th April 2017 Percona Live, CA, US

INSERT ...Server

B

binary log

INSERT ...

relay log

INSERT ...Server

A

binary log

App

ReceiverMeta-data

Update

ApplierMeta-data

Update

INSERT ...

Replicationstream/log Threaded

applier

Send, Receive,ACK, NACK,

Heartbeating,...

Comm. Framework

Capturestatements

or datachanges.

Persistentreplicationstream/log

buffer

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

MySQL Database Replication: Some Notes

• Logical log that records the changes on the master.

• Statement or Row based format (may be intermixed).

• Each transaction is split into groups of events.

• Control events: Rotate, Format Description, Gtid, ...

The Binary Log

9Tuesday, 25th April 2017 Percona Live, CA, US

Layout of the BinaryLog (i.e., the replication stream).

BEGIN ...E1 E2 COMMIT BEGIN ...E1 E2 COMMITGTID GTID

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Coordination Between Servers

MySQL Database Replication: Some Notes

10Tuesday, 25th April 2017 Percona Live, CA, US

A B Since 3.23

A B

semi-synchronous (plugin)

A CB

group replication (plugin)

transactions

transactions

Since 5.5

Since 5.7.17

asynchronous (native)

transactions, membership, coordination

acks

And now in MySQL 8 as of 8.0.1

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Features

Tuesday, 25th April 2017 Percona Live, CA, US

2

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Features

Tuesday, 25th April 2017 Percona Live, CA, US

2.1 Metadata

2

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

New Replication Timestamps in the Binary Log

• As a MySQL user I want to deploy a chained replication setup and measure the replication delay with respect to the immediate master and to the original master, so I can have an indication of where delays are introduced in my topology.

• As a MySQL user I want to measure replication delay with microseconds precision so I can read delays with a more accurate precision.

• As a MySQL user I want to deploy a replication chain across different timezones so that I have geographically distant replication and still having accurate delays reported across the different timezones.

• As a MySQL DBA I want to setup delayed replication in a chained replication scenario that works with respect to the immediate master and not the original master.

User Stories

13Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

New Replication Timestamps in the Binary Log

14Tuesday, 25th April 2017 Percona Live, CA, US

A

App

INSERT ...

B

C

INSERT ...

INSERT ...

Original Master(oct=t, ict=t)

Immediate Master(oct=t, ict=t+N)

Slave gets two commit Timestamps

Two new timestamps, in microseconds since epoch, per transaction.

• original commit timestamp (OCT)

– when the transaction was written to the binary log on the original master

• immediate commit timestamp (ICT)

– when the transaction was written to the binary log (committed) of the immediate master

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

New Replication Timestamps in the Binary Log

• mysqlbinlog: displays the new timestamps in two formats:

– microseconds since epoch

– TIMESTAMP format in the user timezone (for better readability)

• Time stamps supersede seconds_behind_master:– Allows calculation of lag relative to original and immediate masters

– Sub-second precision and timezone handling

15Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Additional Table Metadata in ROW-based Binary Logs

• As a developer I want to mine ROW based MySQL replication logs so that I am able to write applications integrate the data into other tools, without having to search for metadata elsewhere other than the binary log itself.

User Stories

16Tuesday, 25th April 2017 Percona Live, CA, US

Table_map event• types, primary key, …• sign, charset, …

Write_rowsCommit

ServerA

binary log

App

INSERT ...

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Additional Table Metadata in ROW-based Binary Logs

• Signedness of numeric columns

• Collation number of character columns

• Geometry Types

• Column Names

• String Value of SET/ENUM columns

• Primary Key

Tuesday, 25th April 2017 Percona Live, CA, US

Not logged by default.

master> SET @@GLOBAL.binlog_row_metadata = FULL;Query OK, 0 rows affected (0,00 sec)

master> SET @@GLOBAL.binlog_row_metadata = MINIMAL; -- defaultQuery OK, 0 rows affected (0,00 sec)

This additional metadata is

unconditionally logged.

17

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Additional Table Metadata in ROW-based Binary Logs

• Signedness of numeric columns

• Collation number of character columns

• Geometry Types

• Column Names

• String Value of SET/ENUM columns

• Primary Key

Tuesday, 25th April 2017 Percona Live, CA, US

Not logged by default.

This additional metadata is

unconditionally logged.

$ mysqlbinlog --print-table-metadata master-bin.000001[…]# Columns(`int_col` INT UNSIGNED, `char_col` CHAR(10) CHARSET utf8 COLLATION utf8 general_ci)

18

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Features

Tuesday, 25th April 2017 Percona Live, CA, US

Operations2.2

2

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Multi-Source Replication Filters

• As a MySQL DBA I want to selectively replicate from multiple sources into a single replica, so that I am able to specify that a common partition of the data is only replicated from a specific source.

User Stories

20Tuesday, 25th April 2017 Percona Live, CA, US

A

B

C

replicate_do_db=“db1,db2”Replicate_rewrite_db=“db1->a_db1”

replicate_do_db=“db3,db4”Replicate_rewrite_db=“db1->c_db1”

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Multi-Source Replication Filters

• Different replication channels may have different filter rules.

• Can be configured at server start up or dynamically in a session.

• Monitor filter activity through performance schema (where, which, what, when, how many times and more).

21Tuesday, 25th April 2017 Percona Live, CA, US

slave> CHANGE REPLICATION FILTER REPLICATE_DO_DB=(db1) FOR CHANNEL 'channel_1';Query OK, 0 rows affected (0,00 sec)

slave> CHANGE REPLICATION FILTER REPLICATE_IGNORE_TABLE=(db6.t1) FOR CHANNEL 'channel_6';Query OK, 0 rows affected (0,00 sec)

slave> CHANGE REPLICATION FILTER REPLICATE_REWRITE_DB=((db4,db44)) FOR CHANNEL 'channel_4';Query OK, 0 rows affected (0,00 sec)

slave> RESET SLAVE ALL FOR CHANNEL 'channel_1'; -- also removes filters for channel_1Query OK, 0 rows affected (0,00 sec)

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Binary Log Expiration Period in Seconds

• As a MySQL DBA I want to be able to set the binary log files retention policies to a value smaller than one day, so I can lower the storage footprint of my volatile binary logs

• As a MySQL DBA I want to be able to set the binary log files retention policy to one and a half days, so I can meet two requirements of the infrastructure I maintain [ (a) keep below the maximum storage quota assigned to my system; (b) keep the last 24 hours of binary logs always around].

User Stories

22Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Binary Log Expiration Period in Seconds

• Binary log files expiration can be expressed in seconds instead of days.

• Even though you can still express the expiration period in days you can also add sub-day precision by setting the amount of seconds as well.

23Tuesday, 25th April 2017 Percona Live, CA, US

slave> SET @@GLOBAL.binlog_expire_logs_seconds = 3600; -- one hour of logs Query OK, 0 rows affected (0,00 sec)

slave> SELECT @@global.binlog_expire_logs_seconds;+-------------------------------------+| @@global.binlog_expire_logs_seconds |+-------------------------------------+| 3600 |+-------------------------------------+1 row in set (0,00 sec)

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Specify Next Log Index at RESET MASTER

• As a MySQL DBA I want to be able to specify the index of the fresh binary log file when issuing RESET MASTER command, so that I simplify my failover scripts by replacing FLUSH BINARY LOGS and PURGE BINARY LOGS TO command with a single RESET MASTER invocation.

User Stories

24Tuesday, 25th April 2017 Percona Live, CA, US

A A

master-bin.000021master-bin.000022

RESET MASTER

master-bin.000001

A A

master-bin.000021master-bin.000022

RESET MASTER TO 150908

master-bin.150908

Thank you Daniël van Eeden for contributing!

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Specify Next Log Index at RESET MASTER

• RESET MASTER command takes an argument that specifies the index of the new binary log file.

• Simplifies failover procedures by replacing FLUSH BINARY LOGS + PURGE BINARY LOGS TO with a single 'RESET MASTER' command.

25Tuesday, 25th April 2017 Percona Live, CA, US

slave> RESET MASTER TO 150908;Query OK, 0 rows affected (0,00 sec)

slave> SHOW BINARY LOGS;+------------------+-----------+| Log_name | File_size |+------------------+-----------+| slave-bin.150908 | 154 |+------------------+-----------+1 row in set (0,00 sec)

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Global Transaction Identifiers Handling

• As a MySQL DBA I want to be able to set GTID_PURGED even if GTID_EXECUTED is not empty, so I can restore a backup on an existing server without loosing GTID information or binary logs.

User Stories

26Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Global Transaction Identifiers Handling

27Tuesday, 25th April 2017 Percona Live, CA, US

master> SELECT @@GLOBAL.GTID_EXECUTED;+-------------------------------------------+| @@GLOBAL.GTID_EXECUTED |+-------------------------------------------+| 4a05c7cf-241e-11e7-94b0-28b2bd168d07:1-13 |+-------------------------------------------+1 row in set (0,00 sec)

mysql> SET GLOBAL GTID_PURGED='4a05c7cf-241e-11e7-94b0-28b2bd168d0:14-20';Query OK, 0 rows affected (0,00 sec)

mysql> SET GLOBAL GTID_PURGED='+4a05c7cf-241e-11e7-94b0-28b2bd168d07:30-32';Query OK, 0 rows affected (0,01 sec)

mysql> SELECT @@GLOBAL.GTID_EXECUTED;+-------------------------------------------------+| @@GLOBAL.GTID_EXECUTED |+-------------------------------------------------+| 4a05c7cf-241e-11e7-94b0-28b2bd168d07:1-20:30-32 |+-------------------------------------------------+1 row in set (0,00 sec)

mysql>

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Monitoring

Features

Tuesday, 25th April 2017 Percona Live, CA, US

2.3

2

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Multi-Source Replication Filters In Performance Schema

• As a MySQL DBA I want to know since when a filter has become active.

• As a MySQL DBA I want to know how many times has a replication filter been applied to the incoming replication load.

• As a MySQL DBA I want to know how was the filter configured in the first place, so I am able to fully understand the deployment of my filtering infrastructure.

• As a MySQL DBA I want a regular SQL language driven interface to access my filtering information.

User Stories

29Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Multi-Source Replication Filters In Performance Schema

• Monitor filter activity through performance schema (where, which, what, when, how many times and more).

• Two tables, replication_applier_filters and replication_global_applier_filters, displaying per-channel and global filters respectively.

30Tuesday, 25th April 2017 Percona Live, CA, US

slave> SELECT CHANNEL_NAME, FILTER_NAME, FILTER_RULE, ACTIVE_SINCE, COUNTER FROM performance_schema.replication_applier_filters WHERE COUNTER > 0;+--------------+-----------------------------+-------------+----------------------------+---------+| CHANNEL_NAME | FILTER_NAME | FILTER_RULE | ACTIVE_SINCE | COUNTER |+--------------+-----------------------------+-------------+----------------------------+---------+| channel_5 | REPLICATE_DO_TABLE | db5.t1 | 2017-04-13 19:03:44.509472 | 2 || channel_6 | REPLICATE_IGNORE_TABLE | db6.t1 | 2017-04-13 19:03:44.509639 | 2 || channel_4 | REPLICATE_REWRITE_DB | (db4,db44) | 2017-04-13 19:03:44.509334 | 1 || channel_3 | REPLICATE_WILD_IGNORE_TABLE | db%.t1% | 2017-04-13 19:03:44.509198 | 5 |+--------------+-----------------------------+-------------+----------------------------+---------+4 rows in set (0,01 sec)

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Monitor Applier Lag in Performance Schema

• As a MySQL DBA I want to be able to observe how much time a transaction takes to be processed once it arrives at a replica, so I am able to fine tune/size the infrastructure such as disk caches, memory buffers on the replica.

• As a MySQL DBA I want to be able to read timing metrics for each stage a transaction goes through once it arrives at the slave, so I know, for instance, whether there is contention on the applier or on the relay log.

• As a MySQL DBA I want a more reliable hint of replication lag, other than seconds_behind_master, so I am able to get more accurate reports of potential replication lag.

• As a MySQL DBA I want an SQL language driven interface to replication lag related information.

User Stories

31Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

• Performance schema tables extended: new fields to help detecting anddiagnosing lag at several points in the replication stream:

Monitor Applier Lag in Performance Schema

32Tuesday, 25th April 2017 Percona Live, CA, US

ServerB

binary logrelay log

ApplierThreads

Coordinatorthread

Receiverthread

replication_connection_status

replication_applier_status_by_coordinator

replication_applier_status_by_worker

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Monitor Applier Lag in Performance Schema

• Nine additional fields with information about:

– last transaction the corresponding thread processed

– the transaction that thread is currently processing

• This information comprises:– the transaction’s global identifier

– its original and immediate commit timestamps

– the time the thread started processing it

– for the last processed transaction, the time the thread finished processing it.

33Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Monitor Applier Lag in Performance Schema

• Snippet of the output related to the applier stats.

34Tuesday, 25th April 2017 Percona Live, CA, US

slave> select * from replication_applier_status_by_worker\G*************************** 1. row ***************************

CHANNEL_NAME: WORKER_ID: 0THREAD_ID: 33

SERVICE_STATE: ONLAST_ERROR_NUMBER: 0LAST_ERROR_MESSAGE:

LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00.000000LAST_APPLIED_TRANSACTION: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1

LAST_APPLIED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 2017-04-17 14:36:50.462643LAST_APPLIED_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 2017-04-17 14:36:50.462643

LAST_APPLIED_TRANSACTION_START_APPLY_TIMESTAMP: 2017-04-17 14:36:50.479478LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP: 2017-04-17 14:36:50.705676

APPLYING_TRANSACTION: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:2APPLYING_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 2017-04-17 14:36:50.917499APPLYING_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 2017-04-17 14:36:50.917499

APPLYING_TRANSACTION_START_APPLY_TIMESTAMP: 2017-04-17 14:36:50.9323711 row in set (0,38 sec)

slave>

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Replication Applier Progress in Performance Schema

• As a MySQL DBA I want to monitor progress of the row-based replication applier so that I am able to understand whether a long running statement was executed on the master.

• As a MySQL DBA I want to be able to read the status of the applier threads while applying a ROW event, so that I am able to report this in the monitoring tool or just understand what is happening in my system.

User Stories

35Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Replication Applier Progress in Performance Schema

• Rows_query_log_event contents is reported in the processlist_info column in the threads table.

• Three new thread stages were introduced to specify when the applier thread is handling Write, Delete or Update ROW operations.

• Applier threads report progress through the performance schema stages framework.

• Join with other performance schema tables for additional, consolidated, information in a single view.

– For example, join on thread_id against performance_schema.threads table.

36Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Replication Applier Progress in Performance Schema

37Tuesday, 25th April 2017 Percona Live, CA, US

slave> SELECT thread_id, name, processlist_command, processlist_state, processlist_info FROM performance_schema.threads WHERE NAME LIKE '%slave_sql'\G*************************** 1. row ***************************

thread_id: 38name: thread/sql/slave_sql

processlist_command: Queryprocesslist_state: Applying batch of row changes (write)processlist_info: INSERT INTO t1 VALUES (1), (2), (3)

1 row in set (0,00 sec)

slave> SELECT THREAD_ID,WORK_COMPLETED,WORK_ESTIMATED FROM performance_schema.events_stages_currentWHERE EVENT_NAME LIKE 'stage/sql/Applying%';+-----------+----------------+----------------+| THREAD_ID | WORK_COMPLETED | WORK_ESTIMATED |+-----------+----------------+----------------+| 38 | 1 | 3 |+-----------+----------------+----------------+1 row in set (0,00 sec)

slave>

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Performance

Features

Tuesday, 25th April 2017 Percona Live, CA, US

2.4

2

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Faster and More Efficient Replication Pipeline

• As a MySQL user I want my replication lag to be as small as possible, so that my replication infrastructure is able to move data around as quickly as possible.

User Stories

39Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Faster and More Efficient Replication Pipeline

• Remove I/O operations from the critical execution path.

– Locks are held for less duration of time

– Threads wait less time for getting access to the relay log

• This translates into higher applier throughput.

40Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Faster and More Efficient Replication Pipeline

41Tuesday, 25th April 2017 Percona Live, CA, US

start

stopped?

“hot” log?

stop

lock

unlock

EOF?

unlock

waitnotification

start

stopped?

lock

notifyupdates

stop

unlock

read from net

queue

read next

apply/buffer

Applier (SQL Thread/MTS Coordinator)

Connection (I/O Thread)

yes

no

yes

yes

yes

no

no

no

read next

arbitration zoneI/O operation

arbitration zoneI/O operation

Bef

ore

MyS

QL

8.0

.1

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Faster and More Efficient Replication Pipeline

42Tuesday, 25th April 2017 Percona Live, CA, US

start

stopped?

“hot” log?

stop

unlock

updatedbefore

locking?

start

stopped?

lock

update last_posand notify

stop

unlock

read from net

read next

apply/buffer

Applier (SQL Thread/MTS Coordinator)

Connection (I/O Thread)

yes

no

yes

yes

no

no

no

arbitration zoneI/O operation

arbitration zoneI/O operation

queue pos < last_pos?yes

lock

Waitnotification

unlock

no

yesMyS

QL

8.0

.1 a

nd

Aft

er

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Faster and More Efficient Replication Pipeline

43Tuesday, 25th April 2017 Percona Live, CA, US

0

10

20

30

40

50

60

70

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

16 threads 64 threads 16 threads 64 threads

MyS

QL

8.0

.1 G

ain

(%

)

Tran

sact

ion

s /

seco

nd

ap

plie

d o

n t

he

re

plic

a

Sysbench Applier Throughput: Sysbench Update Indexed(RBR, durability and applier threads)

MySQL 8.0.0 MySQL 8.0.1 Gain (%)

Durable Binary Log Settings Non-durable Binary Log Settings

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

More Efficient Applier: Write set Based Transaction Dependencies

• (Again) As a MySQL user I want my replication lag to be as small as possible, so that my replication infrastructure is able to move data around as fast as possible.

• As a MySQL DBA while connecting a slave or a group replication member to a topology, I want it to apply as many transactions in parallel, even on low concurrency worklogs or even single threaded workloads so that it catches up quickly.

• As a MySQL DBA I want to deploy chained replication without loosing applier parallelism down the chain so that I have my topology replicating at full speed.

User Stories

44Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

More Efficient Applier: Write set Based Transaction Dependencies

• Transaction write set: set of items modified by the transaction execution.

• Write sets as a tool to track dependencies between transactions.

• Corollary: Any two transactions exhibiting non-intersecting write sets can be applied in parallel, if and only if, their write set can be accurately determined.

• Write set based parallelization builds on write set extraction and the logical clock infrastructure:

– Builds on the same infrastructure group replication and regular logical clock parallelization scheme.

– Parallelization window is expressed using the same time stamps in the binary log: last_committed and sequence_number.

45Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

More Efficient Applier: Write set Based Transaction Dependencies

• Dependency tracking can be changed dynamically:

• Write set history size can be changed dynamically:

46Tuesday, 25th April 2017 Percona Live, CA, US

master> SET @@GLOBAL.binlog_transaction_dependency_tracking=WRITESET;Query OK, 0 rows affected (0,00 sec)

master> SET @@GLOBAL.binlog_transaction_dependency_tracking=WRITESET_SESSION;Query OK, 0 rows affected (0,00 sec)

master> SET @@GLOBAL.binlog_transaction_dependency_tracking=COMMIT_ORDER; -- defaultQuery OK, 0 rows affected (0,00 sec)

master> SET @@GLOBAL.binlog_transaction_dependency_history_size= 1000;Query OK, 0 rows affected (0,00 sec)

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

More Efficient Applier: Write set Based Transaction Dependencies

47Tuesday, 25th April 2017 Percona Live, CA, US

step 1 step n step n+1 step n+2 step n+3 step n+4

T1Session

A

T2

Session B

T3Session

B

T4

Session A

T5Session

C

T6Session

C

T7Session

C

T8

Session B

time

dat

abas

e a

dd

ress

sp

ace

COMMIT ORDER DEPENDENCY TRACKING

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

More Efficient Applier: Write set Based Transaction Dependencies

48Tuesday, 25th April 2017 Percona Live, CA, US

step n step n+1 step n+2 step n+3 step n+4

T1Session

A

T2

Session B

T3Session

B

T4

Session A

T5Session

C

T6Session

C

T7Session

CT8

Session B

time

dat

abas

e a

dd

ress

sp

ace

step 1

Write set history size

WRITE SET DEPENDENCY TRACKING

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

More Efficient Applier: Write set Based Transaction Dependencies

49Tuesday, 25th April 2017 Percona Live, CA, US

step 1 step n step n+1 step n+2 step n+3 step n+4

T1Session

A

T2

Session B

T3Session

B

T4

Session A

T5Session

C

T6Session

C

T7Session

C

T8

Session B

time

dat

abas

e a

dd

ress

sp

ace

WRITE SET SESSION DEPENDENCY TRACKING

Write set history size

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

More Efficient Applier: Write set Based Transaction Dependencies

• Write set requirements

– Table must have Primary Keys / Unique Keys

– Write set extraction must be turned on.

• Limitations (falls back to commit order)– Foreign Keys constraints unsupported

– DDL do not have a write set (yet)

– Empty transactions or those that have partial write sets

• Warning– Applying transactions in parallel may result in a different commit history than that on

the originating server. slave_preserve_commit_order may be useful in those cases that commit order is required.

50Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

More Efficient Applier: Write set Based Transaction Dependencies

51Tuesday, 25th April 2017 Percona Live, CA, US

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

50000

1 2 4 8 16 32 64 128 256

Up

dat

es/

seco

nd

Ap

plie

s o

n t

he

Re

plic

a

Number of Clients on the Master

Applier Throughput: Sysbench Update Index

COMMIT_ORDER WRITESET WRITESET_SESSION

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

More Efficient Applier: Write set Based Transaction Dependencies

• The chart almost speaks for itself, but lets spell out some of the take aways:

– WRITESET dependency tracking allows applying a single threaded workload in parallel.

– WRITESET dependency delivers the best throughput of the three dependency trackers, at any concurrency level.

– WRITESET_SESSION depends on the number of clients on the master, but still it delivers higher throughput at lower number of clients than that of COMMIT_ORDER.

• Enhances Group Replication recovery – recovery load can also be applied with increased parallelism.

52Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Resilience and Reliability

Features

Tuesday, 25th April 2017 Percona Live, CA, US

2.5

2

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Resilience to Crash and Recovery on DDL Execution

• As a MySQL DBA I want to be able to recover a server that crashes while DDL was executing so that I am able to bring back the server with data dictionary in sync with the binary log.

• As a MySQL DBA I want to be able to recover a crashed replica, that crashed while DDL was executing and put it back into the replication topology and let it catch up, without having to manually recovery positions, GTIDs or DDL statements.

User Stories

54Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

DDL

DDL

Resilience to Crash and Recovery on DDL Execution

55Tuesday, 25th April 2017 Percona Live, CA, US

time

Bin

ary

Log

Inn

oD

B

PrepareFlush

Crash

DDL

DDL

Recovery start

Binary Log and InnoDBin sync after recovering from a crash while executing a DDL operation. (*)

Recovery ended

(*) This is valid for DDL operations that are atomic under the new data dictionary in MySQL 8.

Master

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

DDL

DDL + RPL positions / GTIDS

Resilience to Crash and Recovery on DDL Execution

56Tuesday, 25th April 2017 Percona Live, CA, US

time

Bin

ary

Log

Inn

oD

B

PrepareFlush

Crash

DDL + RPL positions / GTIDS

DDL

Recovery start

Binary Log and InnoDBin sync after recovering from a crash while executing a DDL operation. (*)

Recovery ended

(*) This is valid for DDL operations that are atomic under the new data dictionary in MySQL 8.

SlaveApplier

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Functionality

Features

Tuesday, 25th April 2017 Percona Live, CA, US

2.6

2

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Transaction SAVEPOINT support with Transaction Write Set Extraction

• As a MySQL user I want to use “SAVEPOINT” and “ROLLBACK TO” when transaction write-set extraction is enabled.

User Stories

58Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Transaction SAVEPOINT support with Transaction Write Set Extraction

59Tuesday, 25th April 2017 Percona Live, CA, US

mysql> SET GLOBAL TRANSACTION_WRITE_SET_EXTRACTION=XXHASH64;Query OK, 0 rows affected (0,00 sec)

mysql> BEGIN;Query OK, 0 rows affected (0,00 sec)

mysql> INSERT INTO t1 VALUES(1);Query OK, 1 row affected (0,00 sec)

mysql> SAVEPOINT S1;Query OK, 0 rows affected (0,00 sec)

mysql> INSERT INTO t1 VALUES(2);Query OK, 1 row affected (0,00 sec)

mysql> ROLLBACK TO S1;Query OK, 0 rows affected (0,00 sec)

mysql> RELEASE SAVEPOINT S1;Query OK, 0 rows affected (0,00 sec)

mysql> COMMIT;Query OK, 0 rows affected (0,00 sec)

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Group Replication in MySQL 8

60Tuesday, 25th April 2017 Percona Live, CA, US

M M M M M

Replication Group

Clients

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Group Replication in MySQL 8

• What is MySQL Group Replication?

“Single primary/Multi-master update everywhere replication plugin for MySQL with built-in automatic distributed recovery, conflict detection and group membership.”

• What does the MySQL Group Replication plugin do for the user?– Removes the need for handling server fail-over.

– Provides fault tolerance.

– Enables update everywhere setups.

– Automates group reconfiguration (handling of crashes, failures, re-connects).

– Provides a highly available replicated database.

61Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Group Replication in MySQL 8

• There is a whole session on the topic on Thursday 27th April 2017:

– Everything You Need to Know About MySQL Group Replication• By Nuno Carvalho and Luís Soares

• 1:50pm – 2:40 pm

• https://www.percona.com/live/17/sessions/everything-you-need-know-about-mysql-group-replication

• Come and learn more about it.

62Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Conclusion

Tuesday, 25th April 2017 Percona Live, CA, US

3

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

GR 0.8.0 labs (beta)performance enhancementsreplication stream compressionSSL supportIP whitelistingread-only modeerror log enhancementssplit brain handling

The Road to Group Replication in MySQL 8 and InnoDB Clusters

64Tuesday, 25th April 2017 Percona Live, CA, US

MySQL 5.6.10

MySQL 5.7.9lifecycle interfacesP_S tables for GRServer side changes

GR 0.2.0 labsHello world!

GR 0.3.0 labssupport for corosync 2.x

GR 0.4.0 labsversion handling

GR 0.5.0 labsauto-inc fields handlingrecovery enhancements

GR 0.6.0 labsmulti-platform supportPaxos-based consensus

GR 0.7.0 labsbug fixes

GR 0.9.0 labs (RC)multi-threaded appliersupport

single primary mode

GR is GA inMySQL Server5.7.17

...

GR is released withMySQL 8.0.1 andInnoDB Cluster GAs

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

MySQL InnoDB Cluster: The End Goal

65Tuesday, 25th April 2017 Percona Live, CA, US

S1 S2 S3 S4 S…

M

M M

MySQL Connector

Application

MySQL Router

MySQL Connector

Application

MySQL Router

MySQL Shell

HA

Rep

licaS

et1

S1 S2 S3 S4 S…

M

M M

MySQL Connector

Application

MySQL Router

HA

Rep

licaS

et 2

Rep

licaS

et 3

MySQL Connector

Application

MySQL Router

S1 S2 S3 S4

M

M M

HA

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Conclusion

Tuesday, 25th April 2017 Percona Live, CA, US

4

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Conclusion

MySQL 8.0.1 is out and with lots of interesting features across the board.

• Performance improvements.

• More replication instrumentation.

• Operations driven enhancements.

• New functionality!

And the first 8.0 release with Group Replication.

67Tuesday, 25th April 2017 Percona Live, CA, US

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Where to go from here?

• Packages

– http://www.mysql.com/downloads/

• Documentation– http://dev.mysql.com/doc/refman/5.7/en/group-replication.html

• Blogs from the Engineers (news, technical information, and much more)

– http://mysqlhighavailability.com

68Tuesday, 25th April 2017 Percona Live, CA, US