instance provisioning in mysql innodb cluster

Copyright © 2019, Oracle and/or its affiliates. All rights reserved.

Instance Provisioning in MySQL InnoDB Cluster

Luís SoaresSoftware Development DirectorMySQL Replication

18th September, 2019 Oracle Code One 1

Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposesonly, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,or functionality, and should not be relied upon in making purchasing decisions. The development, release,timing, and pricing of any features or functionality described for Oracle’s products may change and remainsat the sole discretion of Oracle Corporation.

Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions andprospects are “forward-looking statements” and are subject to material risks and uncertainties. A detaileddiscussion of these factors and other risks that affect our business is contained in Oracle’s Securities andExchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q underthe heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s websiteat http://www.oracle.com/investor. All information in this presentation is current as of September 2019 andOracle undertakes no duty to update any statement in light of new information or future events.


http://www.oracle.com/investor


Agenda



Introduction

Use Cases

Under-the-hood

Conclusion

Agenda

18th September, 2019 Oracle Code One

1

2

3

4

4


Introduction


1

5


MySQL InnoDB Cluster


VisionA single product – MySQL – with high availability and scaling features baked in providing an integrated end-to-end

solution that is easy to use. Enter MySQL InnoDB Cluster


MySQL Group Replication

• Cluster is a set of MySQL servers working together.

– Relying on Paxos for distributed coordination and consistent message passing

• Replicated State Machine – shared nothing approach.

– Each MySQL instance has the same set of data.

– Group Replication implements protocols for distributed coordination.

• Recover, Execute, Replicate, Fence

– Group Replication provides automatic recovery, replication, membership, transaction execution and fencing.

– Removes the orchestration burden from the user.

A Group of MySQL Servers Cooperating Together



Adding Replicas to the Cluster

• Adding a new replica requires the user to ... do nothing as of MySQL 8.0.17!

– Well... The user still has to trigger the add replica action, MySQL does not read minds... Yet!

– Distributed recovery will take care of instance provisioning and catching up, by coordinating with the rest of the cluster.

– Distributed recovery = online provisioning + online catch up

– Before MySQL 8.0.17, user had to provision the instance prior to adding it to the cluster.

• The new clone plugin for MySQL is instrumental in this story.

Expanding the Cluster



InnoDB Clone Plugin

• Performs physical snapshots of databases

– Really fast!

• Transfers the snapshots over the network to provision servers

– Can be used remotely!

• Replaces all the server data on the receiver

• Shipped with the Server. In the server.– No external tools needed.

– And can be used outside of InnoDB Cluster / Group Replication too!

Fundamental piece of infrastructure for automated provisioning


mysql> CLONE INSTANCE FROM "user"@donor_host:9999 IDENTIFIED BY "password" …

Shipped with MySQL 8.0.17


Adding Replicas to the Cluster


Nice! As of MySQL 8.0.17, I do not need to backup one

of the members in the group and restore the image in the new member before adding

it to the cluster! I can just add it and the system will

work the details out for me automatically.

Primary SecondaryEmpty Instance

Add Replica


Adding Replicas to the Cluster (Demo)

• Cluster with a single server (3310)

• Empty server (3320) is added.

• 3310 has purged its binlogs.

• 3320 will recover using clone, for the initial snapshot, and then binary logs.

• 3310 state is copied to 3320.

• 3320 restarts and joins the cluster.



Use Cases


2

13


Empty Server Joins – Initial Provisioning

• Server joins the group. Has no data at all.

• Server clones one of the members (in this case the secondary is the donor).

• Server installs the snapshot, restarts and finishes the synchronization using binary logs.


Primary SecondaryEmpty InstanceRecoveringBinary LogsDatabase Snapshot


Old Server Joins – Not Enough Binary Logs in the Cluster

• Server joins the group. Has old data in it.

• No donor in the group with enough binary logs to donate the missing state.

• Server clones donor, replaces its local data, restarts and fetches from donor missing binary logs to apply on top of the cloned state.


Primary SecondaryOld MemberRecoveringBinary LogsDatabase Snapshot

Joiner is missing 2000 transactions and potential donors only have 1000 in their binary logs... Clone first and then resume catch-up.


Old Server Joins – Too Much Binary Log to Apply

• Server joins the group. Has old data in it.

• Data is too old. Binary log based gap is too large. Faster if we clone instead.

• Server clones donor, replaces its local data, restarts and fetches from donor missing binary logs to apply on top of the cloned state.


joiner has: group_replication_clone_threshold = 1000But it is missing 2000 transactions ... Clone first and then resume catch-up.



Diverged Server Joins

• Diverged server joins the group. Has different data in it.

• Clone as a resync operation.




Diverged Server Joins



Summary

• Distributed recovery = initial provisioning (if needed) + catch-up

• Both initial provisioning and catch-up are automatic and online.

• No need for the DBA/Operator to do log positions math.

– This is handled automatically by the distributed recovery procedure together with the clone operation.



Requirements and Limits


4

20


Requirements (By Design)

• Clone plugin installed and loaded (shipped with MySQL 8.0.17)

• Donor and joiner (who is taking the clone) must have the same MySQL version.

• Both donor and joiner have to run on the same operating system.

• Clone RPC is done over the MySQL Protocol (not X protocol)

• MySQL server system variables and configurations are not cloned.

– The recipient retains its system variables and configuration settings.



Limits

• Concurrent DDL with cloning operation is not permitted

• The clone plugin does not support cloning of binary logs.

• The clone plugin only clones data stored in InnoDB.

• Connecting to the donor through a MySQL Router is not supported.

• More details:

– https://dev.mysql.com/doc/refman/8.0/en/clone-plugin-limitations.html



Under-the-Hood


3

23


Technologies


Clone Plugin(Data Snapshots)

Group Replication(Orchestration)

InnoDB Cluster(Ease of Use)


The Clone Plugin

• Primarily motivated to make provisioning of Replicas for HA purposes extremely easy

– MySQL client and SQL commands to provision new instances

– Observe progress through an SQL interface

– Ensures that data and replication positions are consistent



The Clone Plugin

• CLONE command makes the current server (recipient) a clone of a remote one (donor).

• CLONE is destructive. The recipient’s data is wiped out.


Network

Donor Recipient

CLONESQLStatement

Network


The Clone Plugin: ExampleDonor


mysql> # Install the plugin, create the clone user and grant it required privilegesmysql> INSTALL PLUGIN CLONE SONAME "mysql_clone.so"; mysql> CREATE USER clone_user IDENTIFIED BY "clone_password";mysql> GRANT BACKUP_ADMIN ON *.* to clone_user;

mysql> # The following commands are issued to allow observing clone mysql> GRANT SELECT ON performance_schema.* TO clone_user;mysql> GRANT EXECUTE ON *.* to clone_user;


The Clone Plugin: ExampleRecipient


mysql> # Install the clone plugin, set a whitelist for valid donors, give the user mysql> # privileges to issue the clone commandmysql> INSTALL PLUGIN CLONE SONAME "mysql_clone.so";mysql> SET GLOBAL clone_valid_donor_list = "donor.host.com:3306"; mysql> CREATE USER clone_user IDENTIFIED BY "clone_password";mysql> GRANT CLONE_ADMIN ON *.* to clone_user;

mysql> # The following commands are issued to allow observing clone mysql> GRANT SELECT ON performance_schema.* TO clone_user; mysql> GRANT EXECUTE ON *.* to clone_user;


The Clone Plugin: Example

• Recipient reaches out to the donor through the MySQL Protocol port.

• Recipient shall authenticate using the credentials provided.

• ... And that’s it. After the clone operation finishes, recipient is a clone of the donor.

Cloning


# the following is issued on the recipient.mysql> CLONE INSTANCE

-> FROM [email protected]:3306-> IDENTIFIED BY "clone_password";


The Clone Plugin: Example

• Clone restarts the recipient automatically.

• Recipient requires a monitor process to detect shutdown and start the mysqld daemon again.

– If no monitoring process exists (and is running), clone plugin shall report an error and shutdown the server.

– Error does not mean clone failure, but that the user needs to start recipient manually.

• Group Replication automatically adjusts clone_valid_donor_list to match the group membership.

Notes



Group Replication

• Automate Donor discovery

– Uses cluster membership service to pick donor

• Automate Recipient operations

– START GROUP_REPLICATION will handle Clone operations under-the-hood

– Seamless hand-off from provisioning step to binary log catch-up

• One Recovery (Protocol) to rule them all

– Same recovery protocol that joiners run when they join the cluster

– Recovery protocol was extended with clone support

Clone Integration



Group Replication


START GROUP_REPLICATION

• Membership notifications• Cluster Handshake• Cluster Metadata Exchange• Decide recovery strategy• If clone, override whitelist• If clone, clone and restart• Join cluster again• Catch up using binary logs

Primary SecondaryEmpty Instance


InnoDB Cluster

• Interactive multi-language console interface that supports development and administration for the MySQL Server:

– Supports JavaScript, Python and SQL

– Interactive and batch operations (scripting)

– Supports Document and Relational models (X and MySQL protocols)

– Scriptable “DevOps” APIS

– Unified interface for developers and DBAs

MySQL Shell



InnoDB Cluster

• Creation and Management of InnoDB clusters

• Available in both JavaScript and Python; and command-line

• Simple and straight-forward

• Flexible, powerful and secure!

• Automates configuration and setup.

– And now with improved automation, checks and procedures to adding servers to the group.

MySQL AdminAPI



Adding an Instance – may not use clone at all

InnoDB Cluster



Adding an Instance – clone is required in some cases – prompt user

InnoDB Cluster



InnoDB Cluster

• Built-in support for full instance provisioning!

• AdminAPI transparently:

– Setups the cluster to be ready for clone usage (plugin setup, user grants, etc.)

– Detects every possible scenario and guides the user accordingly to the best provisioning method

– Still allowing manual intervention to choose the provisioning method

Summary


MySQL 8.0.17


Performance

• Clone transfers the full database and the slave will needs to be completely reset in the process.

– Slave is offline during this process.

– May take longer than using binary logs...

• Binary logs contains the differences only– Slave is online during this process, but application reads stale data (or does not read

at all).

– May take longer than using clone...

• Catch-up only with binary logs or Clone and Catch-up with the tail?

– May be more effective to do a full server re-clone from a donor instead a letting the server catch up with the master and vice-versa.



Performance

• Time to apply TPCC workload on the slave (binary logs ) compared to the time it takes to transfer the full 100GB of the dataset (clone).

– From 0 to 10GB of changes

– Local clone

– Clone latency increases with

network, throughput is not as

impacted as long as network is

not the bottleneck.


0 1 2 3 4 5 6 7 8 9 10

0

4

8

12

16

20

Database State Transfer Throughput(TPCC Workload, 100GB Database, local copy)

replication time (slave available) clone time (slave unavailable)

Size of the changes in the binlog (GB)

Tim

e t

o p

roce

ss (

min

ute

s)


Conclusion


5

40


Conclusion

• MySQL 8.0.17 ships with the clone plugin

– Extremely useful for provisioning new replciation instances

• Group Replication supports clone in its distributed recovery protocol

– Orchestrates clone operations together with the membership service

– Discovers potential donors (and picks one)

– Retries failed clones

• InnoDB cluster provides an AdminAPI over these components, making everything easier to use

– Shell as the tool to drive everything in InnoDB Cluster

– Do everything from the shell command line. No need for SSH.



Where to go from here?

• Packages

– http://www.mysql.com/downloads/

• Documentation

– https://dev.mysql.com/doc/refman/8.0/en/

• Blogs from the Engineers (news, technical information, and much more)– https://mysqlhighavailability.com/mysql-innodb-cluster-automatic-node-provisioning

– https://mysqlhighavailability.com/a-breakthrough-in-usability-automatic-node-provisioning

– https://mysqlhighavailability.com/automatic-provisioning-in-group-replication

– https://mysqlserverteam.com/clone-create-mysql-instance-replica


http://www.mysql.com/downloads/

https://dev.mysql.com/doc/refman/8.0/en/

https://mysqlhighavailability.com/mysql-innodb-cluster-automatic-node-provisioning

https://mysqlhighavailability.com/a-breakthrough-in-usability-automatic-node-provisioning

https://mysqlhighavailability.com/automatic-provisioning-in-group-replication

https://mysqlserverteam.com/clone-create-mysql-instance-replica


Safe Harbor Statement

The preceding is intended to outline our general product direction. It is intended for information purposesonly, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,or functionality, and should not be relied upon in making purchasing decisions. The development, release,timing, and pricing of any features or functionality described for Oracle’s products may change and remainsat the sole discretion of Oracle Corporation.

Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions andprospects are “forward-looking statements” and are subject to material risks and uncertainties. A detaileddiscussion of these factors and other risks that affect our business is contained in Oracle’s Securities andExchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q underthe heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s websiteat http://www.oracle.com/investor. All information in this presentation is current as of September 2019 andOracle undertakes no duty to update any statement in light of new information or future events.


http://www.oracle.com/investor

instance provisioning in mysql innodb cluster

Documents