the new replication features in mysql 8 - percona · since epoch, per transaction. • ... issuing...
TRANSCRIPT
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
The New Replication Features in MySQL 8
Luís Soares ([email protected])Principal Software Engineer, MySQL Replication Lead
Tuesday, 25th April 2017 Percona Live, CA, US 1
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
2Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Background
Features
Roadmap
Conclusion
Program Agenda
Tuesday, 25th April 2017 Percona Live, CA, US 4
1
2
3
4
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Background
Tuesday, 25th April 2017 Percona Live, CA, US
1
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Database Replication
6Tuesday, 25th April 2017 Percona Live, CA, US
ServerB
ServerA
App
INSERT ...
INSERT ... INSERT ...
Replication“The process of generating and reproducing multiple copies of data at one or more sites.”,Database Systems: A Practical Approach to Design, Implementation, and Management, Thomas M. Connolly, Carolyn E. Begg, Third Edition, 2002.
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL Database Replication: Overview
7Tuesday, 25th April 2017 Percona Live, CA, US
INSERT ...Server
B
binary log
INSERT ...
relay log
INSERT ...Server
A
binary log
App
ReceiverMeta-data
Update
ApplierMeta-data
Update
INSERT ...
Comm. Framework
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL Database Replication: Overview
8Tuesday, 25th April 2017 Percona Live, CA, US
INSERT ...Server
B
binary log
INSERT ...
relay log
INSERT ...Server
A
binary log
App
ReceiverMeta-data
Update
ApplierMeta-data
Update
INSERT ...
Replicationstream/log Threaded
applier
Send, Receive,ACK, NACK,
Heartbeating,...
Comm. Framework
Capturestatements
or datachanges.
Persistentreplicationstream/log
buffer
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL Database Replication: Some Notes
• Logical log that records the changes on the master.
• Statement or Row based format (may be intermixed).
• Each transaction is split into groups of events.
• Control events: Rotate, Format Description, Gtid, ...
The Binary Log
9Tuesday, 25th April 2017 Percona Live, CA, US
Layout of the BinaryLog (i.e., the replication stream).
BEGIN ...E1 E2 COMMIT BEGIN ...E1 E2 COMMITGTID GTID
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Coordination Between Servers
MySQL Database Replication: Some Notes
10Tuesday, 25th April 2017 Percona Live, CA, US
A B Since 3.23
A B
semi-synchronous (plugin)
A CB
group replication (plugin)
transactions
transactions
Since 5.5
Since 5.7.17
asynchronous (native)
transactions, membership, coordination
acks
And now in MySQL 8 as of 8.0.1
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Features
Tuesday, 25th April 2017 Percona Live, CA, US
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Features
Tuesday, 25th April 2017 Percona Live, CA, US
2.1 Metadata
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
New Replication Timestamps in the Binary Log
• As a MySQL user I want to deploy a chained replication setup and measure the replication delay with respect to the immediate master and to the original master, so I can have an indication of where delays are introduced in my topology.
• As a MySQL user I want to measure replication delay with microseconds precision so I can read delays with a more accurate precision.
• As a MySQL user I want to deploy a replication chain across different timezones so that I have geographically distant replication and still having accurate delays reported across the different timezones.
• As a MySQL DBA I want to setup delayed replication in a chained replication scenario that works with respect to the immediate master and not the original master.
User Stories
13Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
New Replication Timestamps in the Binary Log
14Tuesday, 25th April 2017 Percona Live, CA, US
A
App
INSERT ...
B
C
INSERT ...
INSERT ...
Original Master(oct=t, ict=t)
Immediate Master(oct=t, ict=t+N)
Slave gets two commit Timestamps
Two new timestamps, in microseconds since epoch, per transaction.
• original commit timestamp (OCT)
– when the transaction was written to the binary log on the original master
• immediate commit timestamp (ICT)
– when the transaction was written to the binary log (committed) of the immediate master
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
New Replication Timestamps in the Binary Log
• mysqlbinlog: displays the new timestamps in two formats:
– microseconds since epoch
– TIMESTAMP format in the user timezone (for better readability)
• Time stamps supersede seconds_behind_master:– Allows calculation of lag relative to original and immediate masters
– Sub-second precision and timezone handling
15Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Additional Table Metadata in ROW-based Binary Logs
• As a developer I want to mine ROW based MySQL replication logs so that I am able to write applications integrate the data into other tools, without having to search for metadata elsewhere other than the binary log itself.
User Stories
16Tuesday, 25th April 2017 Percona Live, CA, US
Table_map event• types, primary key, …• sign, charset, …
Write_rowsCommit
ServerA
binary log
App
INSERT ...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Additional Table Metadata in ROW-based Binary Logs
• Signedness of numeric columns
• Collation number of character columns
• Geometry Types
• Column Names
• String Value of SET/ENUM columns
• Primary Key
Tuesday, 25th April 2017 Percona Live, CA, US
Not logged by default.
master> SET @@GLOBAL.binlog_row_metadata = FULL;Query OK, 0 rows affected (0,00 sec)
master> SET @@GLOBAL.binlog_row_metadata = MINIMAL; -- defaultQuery OK, 0 rows affected (0,00 sec)
This additional metadata is
unconditionally logged.
17
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Additional Table Metadata in ROW-based Binary Logs
• Signedness of numeric columns
• Collation number of character columns
• Geometry Types
• Column Names
• String Value of SET/ENUM columns
• Primary Key
Tuesday, 25th April 2017 Percona Live, CA, US
Not logged by default.
This additional metadata is
unconditionally logged.
$ mysqlbinlog --print-table-metadata master-bin.000001[…]# Columns(`int_col` INT UNSIGNED, `char_col` CHAR(10) CHARSET utf8 COLLATION utf8 general_ci)
18
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Features
Tuesday, 25th April 2017 Percona Live, CA, US
Operations2.2
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Multi-Source Replication Filters
• As a MySQL DBA I want to selectively replicate from multiple sources into a single replica, so that I am able to specify that a common partition of the data is only replicated from a specific source.
User Stories
20Tuesday, 25th April 2017 Percona Live, CA, US
A
B
C
replicate_do_db=“db1,db2”Replicate_rewrite_db=“db1->a_db1”
replicate_do_db=“db3,db4”Replicate_rewrite_db=“db1->c_db1”
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Multi-Source Replication Filters
• Different replication channels may have different filter rules.
• Can be configured at server start up or dynamically in a session.
• Monitor filter activity through performance schema (where, which, what, when, how many times and more).
21Tuesday, 25th April 2017 Percona Live, CA, US
slave> CHANGE REPLICATION FILTER REPLICATE_DO_DB=(db1) FOR CHANNEL 'channel_1';Query OK, 0 rows affected (0,00 sec)
slave> CHANGE REPLICATION FILTER REPLICATE_IGNORE_TABLE=(db6.t1) FOR CHANNEL 'channel_6';Query OK, 0 rows affected (0,00 sec)
slave> CHANGE REPLICATION FILTER REPLICATE_REWRITE_DB=((db4,db44)) FOR CHANNEL 'channel_4';Query OK, 0 rows affected (0,00 sec)
slave> RESET SLAVE ALL FOR CHANNEL 'channel_1'; -- also removes filters for channel_1Query OK, 0 rows affected (0,00 sec)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Binary Log Expiration Period in Seconds
• As a MySQL DBA I want to be able to set the binary log files retention policies to a value smaller than one day, so I can lower the storage footprint of my volatile binary logs
• As a MySQL DBA I want to be able to set the binary log files retention policy to one and a half days, so I can meet two requirements of the infrastructure I maintain [ (a) keep below the maximum storage quota assigned to my system; (b) keep the last 24 hours of binary logs always around].
User Stories
22Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Binary Log Expiration Period in Seconds
• Binary log files expiration can be expressed in seconds instead of days.
• Even though you can still express the expiration period in days you can also add sub-day precision by setting the amount of seconds as well.
23Tuesday, 25th April 2017 Percona Live, CA, US
slave> SET @@GLOBAL.binlog_expire_logs_seconds = 3600; -- one hour of logs Query OK, 0 rows affected (0,00 sec)
slave> SELECT @@global.binlog_expire_logs_seconds;+-------------------------------------+| @@global.binlog_expire_logs_seconds |+-------------------------------------+| 3600 |+-------------------------------------+1 row in set (0,00 sec)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Specify Next Log Index at RESET MASTER
• As a MySQL DBA I want to be able to specify the index of the fresh binary log file when issuing RESET MASTER command, so that I simplify my failover scripts by replacing FLUSH BINARY LOGS and PURGE BINARY LOGS TO command with a single RESET MASTER invocation.
User Stories
24Tuesday, 25th April 2017 Percona Live, CA, US
A A
master-bin.000021master-bin.000022
RESET MASTER
master-bin.000001
A A
master-bin.000021master-bin.000022
RESET MASTER TO 150908
master-bin.150908
Thank you Daniël van Eeden for contributing!
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Specify Next Log Index at RESET MASTER
• RESET MASTER command takes an argument that specifies the index of the new binary log file.
• Simplifies failover procedures by replacing FLUSH BINARY LOGS + PURGE BINARY LOGS TO with a single 'RESET MASTER' command.
25Tuesday, 25th April 2017 Percona Live, CA, US
slave> RESET MASTER TO 150908;Query OK, 0 rows affected (0,00 sec)
slave> SHOW BINARY LOGS;+------------------+-----------+| Log_name | File_size |+------------------+-----------+| slave-bin.150908 | 154 |+------------------+-----------+1 row in set (0,00 sec)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Global Transaction Identifiers Handling
• As a MySQL DBA I want to be able to set GTID_PURGED even if GTID_EXECUTED is not empty, so I can restore a backup on an existing server without loosing GTID information or binary logs.
User Stories
26Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Global Transaction Identifiers Handling
27Tuesday, 25th April 2017 Percona Live, CA, US
master> SELECT @@GLOBAL.GTID_EXECUTED;+-------------------------------------------+| @@GLOBAL.GTID_EXECUTED |+-------------------------------------------+| 4a05c7cf-241e-11e7-94b0-28b2bd168d07:1-13 |+-------------------------------------------+1 row in set (0,00 sec)
mysql> SET GLOBAL GTID_PURGED='4a05c7cf-241e-11e7-94b0-28b2bd168d0:14-20';Query OK, 0 rows affected (0,00 sec)
mysql> SET GLOBAL GTID_PURGED='+4a05c7cf-241e-11e7-94b0-28b2bd168d07:30-32';Query OK, 0 rows affected (0,01 sec)
mysql> SELECT @@GLOBAL.GTID_EXECUTED;+-------------------------------------------------+| @@GLOBAL.GTID_EXECUTED |+-------------------------------------------------+| 4a05c7cf-241e-11e7-94b0-28b2bd168d07:1-20:30-32 |+-------------------------------------------------+1 row in set (0,00 sec)
mysql>
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Monitoring
Features
Tuesday, 25th April 2017 Percona Live, CA, US
2.3
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Multi-Source Replication Filters In Performance Schema
• As a MySQL DBA I want to know since when a filter has become active.
• As a MySQL DBA I want to know how many times has a replication filter been applied to the incoming replication load.
• As a MySQL DBA I want to know how was the filter configured in the first place, so I am able to fully understand the deployment of my filtering infrastructure.
• As a MySQL DBA I want a regular SQL language driven interface to access my filtering information.
User Stories
29Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Multi-Source Replication Filters In Performance Schema
• Monitor filter activity through performance schema (where, which, what, when, how many times and more).
• Two tables, replication_applier_filters and replication_global_applier_filters, displaying per-channel and global filters respectively.
30Tuesday, 25th April 2017 Percona Live, CA, US
slave> SELECT CHANNEL_NAME, FILTER_NAME, FILTER_RULE, ACTIVE_SINCE, COUNTER FROM performance_schema.replication_applier_filters WHERE COUNTER > 0;+--------------+-----------------------------+-------------+----------------------------+---------+| CHANNEL_NAME | FILTER_NAME | FILTER_RULE | ACTIVE_SINCE | COUNTER |+--------------+-----------------------------+-------------+----------------------------+---------+| channel_5 | REPLICATE_DO_TABLE | db5.t1 | 2017-04-13 19:03:44.509472 | 2 || channel_6 | REPLICATE_IGNORE_TABLE | db6.t1 | 2017-04-13 19:03:44.509639 | 2 || channel_4 | REPLICATE_REWRITE_DB | (db4,db44) | 2017-04-13 19:03:44.509334 | 1 || channel_3 | REPLICATE_WILD_IGNORE_TABLE | db%.t1% | 2017-04-13 19:03:44.509198 | 5 |+--------------+-----------------------------+-------------+----------------------------+---------+4 rows in set (0,01 sec)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Monitor Applier Lag in Performance Schema
• As a MySQL DBA I want to be able to observe how much time a transaction takes to be processed once it arrives at a replica, so I am able to fine tune/size the infrastructure such as disk caches, memory buffers on the replica.
• As a MySQL DBA I want to be able to read timing metrics for each stage a transaction goes through once it arrives at the slave, so I know, for instance, whether there is contention on the applier or on the relay log.
• As a MySQL DBA I want a more reliable hint of replication lag, other than seconds_behind_master, so I am able to get more accurate reports of potential replication lag.
• As a MySQL DBA I want an SQL language driven interface to replication lag related information.
User Stories
31Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Performance schema tables extended: new fields to help detecting anddiagnosing lag at several points in the replication stream:
Monitor Applier Lag in Performance Schema
32Tuesday, 25th April 2017 Percona Live, CA, US
ServerB
binary logrelay log
ApplierThreads
Coordinatorthread
Receiverthread
replication_connection_status
replication_applier_status_by_coordinator
replication_applier_status_by_worker
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Monitor Applier Lag in Performance Schema
• Nine additional fields with information about:
– last transaction the corresponding thread processed
– the transaction that thread is currently processing
• This information comprises:– the transaction’s global identifier
– its original and immediate commit timestamps
– the time the thread started processing it
– for the last processed transaction, the time the thread finished processing it.
33Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Monitor Applier Lag in Performance Schema
• Snippet of the output related to the applier stats.
34Tuesday, 25th April 2017 Percona Live, CA, US
slave> select * from replication_applier_status_by_worker\G*************************** 1. row ***************************
CHANNEL_NAME: WORKER_ID: 0THREAD_ID: 33
SERVICE_STATE: ONLAST_ERROR_NUMBER: 0LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00.000000LAST_APPLIED_TRANSACTION: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1
LAST_APPLIED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 2017-04-17 14:36:50.462643LAST_APPLIED_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 2017-04-17 14:36:50.462643
LAST_APPLIED_TRANSACTION_START_APPLY_TIMESTAMP: 2017-04-17 14:36:50.479478LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP: 2017-04-17 14:36:50.705676
APPLYING_TRANSACTION: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:2APPLYING_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 2017-04-17 14:36:50.917499APPLYING_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 2017-04-17 14:36:50.917499
APPLYING_TRANSACTION_START_APPLY_TIMESTAMP: 2017-04-17 14:36:50.9323711 row in set (0,38 sec)
slave>
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Replication Applier Progress in Performance Schema
• As a MySQL DBA I want to monitor progress of the row-based replication applier so that I am able to understand whether a long running statement was executed on the master.
• As a MySQL DBA I want to be able to read the status of the applier threads while applying a ROW event, so that I am able to report this in the monitoring tool or just understand what is happening in my system.
User Stories
35Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Replication Applier Progress in Performance Schema
• Rows_query_log_event contents is reported in the processlist_info column in the threads table.
• Three new thread stages were introduced to specify when the applier thread is handling Write, Delete or Update ROW operations.
• Applier threads report progress through the performance schema stages framework.
• Join with other performance schema tables for additional, consolidated, information in a single view.
– For example, join on thread_id against performance_schema.threads table.
36Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Replication Applier Progress in Performance Schema
37Tuesday, 25th April 2017 Percona Live, CA, US
slave> SELECT thread_id, name, processlist_command, processlist_state, processlist_info FROM performance_schema.threads WHERE NAME LIKE '%slave_sql'\G*************************** 1. row ***************************
thread_id: 38name: thread/sql/slave_sql
processlist_command: Queryprocesslist_state: Applying batch of row changes (write)processlist_info: INSERT INTO t1 VALUES (1), (2), (3)
1 row in set (0,00 sec)
slave> SELECT THREAD_ID,WORK_COMPLETED,WORK_ESTIMATED FROM performance_schema.events_stages_currentWHERE EVENT_NAME LIKE 'stage/sql/Applying%';+-----------+----------------+----------------+| THREAD_ID | WORK_COMPLETED | WORK_ESTIMATED |+-----------+----------------+----------------+| 38 | 1 | 3 |+-----------+----------------+----------------+1 row in set (0,00 sec)
slave>
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Performance
Features
Tuesday, 25th April 2017 Percona Live, CA, US
2.4
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Faster and More Efficient Replication Pipeline
• As a MySQL user I want my replication lag to be as small as possible, so that my replication infrastructure is able to move data around as quickly as possible.
User Stories
39Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Faster and More Efficient Replication Pipeline
• Remove I/O operations from the critical execution path.
– Locks are held for less duration of time
– Threads wait less time for getting access to the relay log
• This translates into higher applier throughput.
40Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Faster and More Efficient Replication Pipeline
41Tuesday, 25th April 2017 Percona Live, CA, US
start
stopped?
“hot” log?
stop
lock
unlock
EOF?
unlock
waitnotification
start
stopped?
lock
notifyupdates
stop
unlock
read from net
queue
read next
apply/buffer
Applier (SQL Thread/MTS Coordinator)
Connection (I/O Thread)
yes
no
yes
yes
yes
no
no
no
read next
arbitration zoneI/O operation
arbitration zoneI/O operation
Bef
ore
MyS
QL
8.0
.1
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Faster and More Efficient Replication Pipeline
42Tuesday, 25th April 2017 Percona Live, CA, US
start
stopped?
“hot” log?
stop
unlock
updatedbefore
locking?
start
stopped?
lock
update last_posand notify
stop
unlock
read from net
read next
apply/buffer
Applier (SQL Thread/MTS Coordinator)
Connection (I/O Thread)
yes
no
yes
yes
no
no
no
arbitration zoneI/O operation
arbitration zoneI/O operation
queue pos < last_pos?yes
lock
Waitnotification
unlock
no
yesMyS
QL
8.0
.1 a
nd
Aft
er
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Faster and More Efficient Replication Pipeline
43Tuesday, 25th April 2017 Percona Live, CA, US
0
10
20
30
40
50
60
70
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
16 threads 64 threads 16 threads 64 threads
MyS
QL
8.0
.1 G
ain
(%
)
Tran
sact
ion
s /
seco
nd
ap
plie
d o
n t
he
re
plic
a
Sysbench Applier Throughput: Sysbench Update Indexed(RBR, durability and applier threads)
MySQL 8.0.0 MySQL 8.0.1 Gain (%)
Durable Binary Log Settings Non-durable Binary Log Settings
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
More Efficient Applier: Write set Based Transaction Dependencies
• (Again) As a MySQL user I want my replication lag to be as small as possible, so that my replication infrastructure is able to move data around as fast as possible.
• As a MySQL DBA while connecting a slave or a group replication member to a topology, I want it to apply as many transactions in parallel, even on low concurrency worklogs or even single threaded workloads so that it catches up quickly.
• As a MySQL DBA I want to deploy chained replication without loosing applier parallelism down the chain so that I have my topology replicating at full speed.
User Stories
44Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
More Efficient Applier: Write set Based Transaction Dependencies
• Transaction write set: set of items modified by the transaction execution.
• Write sets as a tool to track dependencies between transactions.
• Corollary: Any two transactions exhibiting non-intersecting write sets can be applied in parallel, if and only if, their write set can be accurately determined.
• Write set based parallelization builds on write set extraction and the logical clock infrastructure:
– Builds on the same infrastructure group replication and regular logical clock parallelization scheme.
– Parallelization window is expressed using the same time stamps in the binary log: last_committed and sequence_number.
45Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
More Efficient Applier: Write set Based Transaction Dependencies
• Dependency tracking can be changed dynamically:
• Write set history size can be changed dynamically:
46Tuesday, 25th April 2017 Percona Live, CA, US
master> SET @@GLOBAL.binlog_transaction_dependency_tracking=WRITESET;Query OK, 0 rows affected (0,00 sec)
master> SET @@GLOBAL.binlog_transaction_dependency_tracking=WRITESET_SESSION;Query OK, 0 rows affected (0,00 sec)
master> SET @@GLOBAL.binlog_transaction_dependency_tracking=COMMIT_ORDER; -- defaultQuery OK, 0 rows affected (0,00 sec)
master> SET @@GLOBAL.binlog_transaction_dependency_history_size= 1000;Query OK, 0 rows affected (0,00 sec)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
More Efficient Applier: Write set Based Transaction Dependencies
47Tuesday, 25th April 2017 Percona Live, CA, US
step 1 step n step n+1 step n+2 step n+3 step n+4
T1Session
A
T2
Session B
T3Session
B
T4
Session A
T5Session
C
T6Session
C
T7Session
C
T8
Session B
time
dat
abas
e a
dd
ress
sp
ace
COMMIT ORDER DEPENDENCY TRACKING
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
More Efficient Applier: Write set Based Transaction Dependencies
48Tuesday, 25th April 2017 Percona Live, CA, US
step n step n+1 step n+2 step n+3 step n+4
T1Session
A
T2
Session B
T3Session
B
T4
Session A
T5Session
C
T6Session
C
T7Session
CT8
Session B
time
dat
abas
e a
dd
ress
sp
ace
step 1
Write set history size
WRITE SET DEPENDENCY TRACKING
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
More Efficient Applier: Write set Based Transaction Dependencies
49Tuesday, 25th April 2017 Percona Live, CA, US
step 1 step n step n+1 step n+2 step n+3 step n+4
T1Session
A
T2
Session B
T3Session
B
T4
Session A
T5Session
C
T6Session
C
T7Session
C
T8
Session B
time
dat
abas
e a
dd
ress
sp
ace
WRITE SET SESSION DEPENDENCY TRACKING
Write set history size
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
More Efficient Applier: Write set Based Transaction Dependencies
• Write set requirements
– Table must have Primary Keys / Unique Keys
– Write set extraction must be turned on.
• Limitations (falls back to commit order)– Foreign Keys constraints unsupported
– DDL do not have a write set (yet)
– Empty transactions or those that have partial write sets
• Warning– Applying transactions in parallel may result in a different commit history than that on
the originating server. slave_preserve_commit_order may be useful in those cases that commit order is required.
50Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
More Efficient Applier: Write set Based Transaction Dependencies
51Tuesday, 25th April 2017 Percona Live, CA, US
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
50000
1 2 4 8 16 32 64 128 256
Up
dat
es/
seco
nd
Ap
plie
s o
n t
he
Re
plic
a
Number of Clients on the Master
Applier Throughput: Sysbench Update Index
COMMIT_ORDER WRITESET WRITESET_SESSION
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
More Efficient Applier: Write set Based Transaction Dependencies
• The chart almost speaks for itself, but lets spell out some of the take aways:
– WRITESET dependency tracking allows applying a single threaded workload in parallel.
– WRITESET dependency delivers the best throughput of the three dependency trackers, at any concurrency level.
– WRITESET_SESSION depends on the number of clients on the master, but still it delivers higher throughput at lower number of clients than that of COMMIT_ORDER.
• Enhances Group Replication recovery – recovery load can also be applied with increased parallelism.
52Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Resilience and Reliability
Features
Tuesday, 25th April 2017 Percona Live, CA, US
2.5
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Resilience to Crash and Recovery on DDL Execution
• As a MySQL DBA I want to be able to recover a server that crashes while DDL was executing so that I am able to bring back the server with data dictionary in sync with the binary log.
• As a MySQL DBA I want to be able to recover a crashed replica, that crashed while DDL was executing and put it back into the replication topology and let it catch up, without having to manually recovery positions, GTIDs or DDL statements.
User Stories
54Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
DDL
DDL
Resilience to Crash and Recovery on DDL Execution
55Tuesday, 25th April 2017 Percona Live, CA, US
time
Bin
ary
Log
Inn
oD
B
PrepareFlush
Crash
DDL
DDL
Recovery start
Binary Log and InnoDBin sync after recovering from a crash while executing a DDL operation. (*)
Recovery ended
(*) This is valid for DDL operations that are atomic under the new data dictionary in MySQL 8.
Master
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
DDL
DDL + RPL positions / GTIDS
Resilience to Crash and Recovery on DDL Execution
56Tuesday, 25th April 2017 Percona Live, CA, US
time
Bin
ary
Log
Inn
oD
B
PrepareFlush
Crash
DDL + RPL positions / GTIDS
DDL
Recovery start
Binary Log and InnoDBin sync after recovering from a crash while executing a DDL operation. (*)
Recovery ended
(*) This is valid for DDL operations that are atomic under the new data dictionary in MySQL 8.
SlaveApplier
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Functionality
Features
Tuesday, 25th April 2017 Percona Live, CA, US
2.6
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transaction SAVEPOINT support with Transaction Write Set Extraction
• As a MySQL user I want to use “SAVEPOINT” and “ROLLBACK TO” when transaction write-set extraction is enabled.
User Stories
58Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transaction SAVEPOINT support with Transaction Write Set Extraction
59Tuesday, 25th April 2017 Percona Live, CA, US
mysql> SET GLOBAL TRANSACTION_WRITE_SET_EXTRACTION=XXHASH64;Query OK, 0 rows affected (0,00 sec)
mysql> BEGIN;Query OK, 0 rows affected (0,00 sec)
mysql> INSERT INTO t1 VALUES(1);Query OK, 1 row affected (0,00 sec)
mysql> SAVEPOINT S1;Query OK, 0 rows affected (0,00 sec)
mysql> INSERT INTO t1 VALUES(2);Query OK, 1 row affected (0,00 sec)
mysql> ROLLBACK TO S1;Query OK, 0 rows affected (0,00 sec)
mysql> RELEASE SAVEPOINT S1;Query OK, 0 rows affected (0,00 sec)
mysql> COMMIT;Query OK, 0 rows affected (0,00 sec)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Group Replication in MySQL 8
60Tuesday, 25th April 2017 Percona Live, CA, US
M M M M M
Replication Group
Clients
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Group Replication in MySQL 8
• What is MySQL Group Replication?
“Single primary/Multi-master update everywhere replication plugin for MySQL with built-in automatic distributed recovery, conflict detection and group membership.”
• What does the MySQL Group Replication plugin do for the user?– Removes the need for handling server fail-over.
– Provides fault tolerance.
– Enables update everywhere setups.
– Automates group reconfiguration (handling of crashes, failures, re-connects).
– Provides a highly available replicated database.
61Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Group Replication in MySQL 8
• There is a whole session on the topic on Thursday 27th April 2017:
– Everything You Need to Know About MySQL Group Replication• By Nuno Carvalho and Luís Soares
• 1:50pm – 2:40 pm
• https://www.percona.com/live/17/sessions/everything-you-need-know-about-mysql-group-replication
• Come and learn more about it.
62Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Conclusion
Tuesday, 25th April 2017 Percona Live, CA, US
3
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
GR 0.8.0 labs (beta)performance enhancementsreplication stream compressionSSL supportIP whitelistingread-only modeerror log enhancementssplit brain handling
The Road to Group Replication in MySQL 8 and InnoDB Clusters
64Tuesday, 25th April 2017 Percona Live, CA, US
MySQL 5.6.10
MySQL 5.7.9lifecycle interfacesP_S tables for GRServer side changes
GR 0.2.0 labsHello world!
GR 0.3.0 labssupport for corosync 2.x
GR 0.4.0 labsversion handling
GR 0.5.0 labsauto-inc fields handlingrecovery enhancements
GR 0.6.0 labsmulti-platform supportPaxos-based consensus
GR 0.7.0 labsbug fixes
GR 0.9.0 labs (RC)multi-threaded appliersupport
single primary mode
GR is GA inMySQL Server5.7.17
...
GR is released withMySQL 8.0.1 andInnoDB Cluster GAs
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL InnoDB Cluster: The End Goal
65Tuesday, 25th April 2017 Percona Live, CA, US
S1 S2 S3 S4 S…
M
M M
MySQL Connector
Application
MySQL Router
MySQL Connector
Application
MySQL Router
MySQL Shell
HA
Rep
licaS
et1
S1 S2 S3 S4 S…
M
M M
MySQL Connector
Application
MySQL Router
HA
Rep
licaS
et 2
Rep
licaS
et 3
MySQL Connector
Application
MySQL Router
S1 S2 S3 S4
M
M M
HA
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Conclusion
Tuesday, 25th April 2017 Percona Live, CA, US
4
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Conclusion
MySQL 8.0.1 is out and with lots of interesting features across the board.
• Performance improvements.
• More replication instrumentation.
• Operations driven enhancements.
• New functionality!
And the first 8.0 release with Group Replication.
67Tuesday, 25th April 2017 Percona Live, CA, US
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Where to go from here?
• Packages
– http://www.mysql.com/downloads/
• Documentation– http://dev.mysql.com/doc/refman/5.7/en/group-replication.html
• Blogs from the Engineers (news, technical information, and much more)
– http://mysqlhighavailability.com
68Tuesday, 25th April 2017 Percona Live, CA, US