mysql replication update -- zendcon 2016
TRANSCRIPT
![Page 1: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/1.jpg)
MySQL Replication UpdateZendcon 2016Dave Stokes
[email protected]@stokerslideshare.net/davidmstokesElephantanddolphin.blogger.com & opensourcedba.wordpress.com
1
![Page 2: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/2.jpg)
Safe Harbor 2
THE FOLLOWING IS INTENDED TO OUTLINE OUR GENERAL PRODUCT DIRECTION. IT IS INTENDED FOR INFORMATION PURPOSES ONLY, AND MAY NOT BE INCORPORATED INTO ANY CONTRACT. IT IS NOT A COMMITMENT TO DELIVER ANY MATERIAL, CODE, OR FUNCTIONALITY, AND SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. THE DEVELOPMENT, RELEASE, AND TIMING OF ANY FEATURES OR FUNCTIONALITY DESCRIBED FOR ORACLE'S PRODUCTS REMAINS AT THE SOLE DISCRETION OF ORACLE.
![Page 3: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/3.jpg)
Quick MySQL Catch Up
21 Years OldMySQL has been part of Oracle’s family of databases for six years.
MySQL 8MySQl 5.7 is the current release but the next version will be MySQL 8. Big feature is real time data dictionary
Group ReplicationActive master-master replication.
JSONA new native JSON datatype to store documents in a column of a table
Document StoreProgrammers not know SQL but need a database? X Devapi allows them to use RDMS from language of choice
EncryptionUse Oracle Key Vault to encrypt your data at rest.
3
![Page 4: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/4.jpg)
ReplicationPast
Present
Future
4
![Page 5: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/5.jpg)
Basic Premise Behind Replication1. Copy all the data on one server and copy onto another.
2. Setup Replication.
3. Log any changes on the first server to a log file, apply that logfile to the second server.
5
![Page 6: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/6.jpg)
Imagine Your Data as a Painting:Make copy of Mona Lisa
6
![Page 7: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/7.jpg)
Make copy of changes to Mona Lisa:Make copy of your data
7
![Page 8: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/8.jpg)
Keep making copy of changes to Mona Lisa:Apply changes made on master to slave
8
![Page 9: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/9.jpg)
Architecture 10,000’
Binary Log on master
Slave Attaches
I/O Thread Grabs Data
Data copied to slave
Applier thread changes slave’s data
9
![Page 10: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/10.jpg)
Graphic overview of replication
10
![Page 11: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/11.jpg)
But what are we replicating?
11
![Page 12: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/12.jpg)
Replicating changes to tables and data
SBR: When using statement-based binary logging, the master writes SQL statements to the binary log. Replication of the master to the slave works by executing the SQL statements on the slave. This is called statement-based replication (often abbreviated as SBR), which corresponds to the standard MySQL statement-based binary logging format. Replication capabilities in MySQL version 5.1.4 and earlier used this format exclusively.
What is sent: Structured Query Language (SQL), your query
RBR: When using row-based logging, the master writes events to the binary log that indicate how individual table rows are changed. Replication of the master to the slave works by copying the events representing the changes to the table rows to the slave. This is called row-based replication (often abbreviated as RBR).
What is sent: The Delta, the results of your query
12
![Page 13: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/13.jpg)
Or B
oth MBR: You can also configure MySQL to use a mix of both
statement-based and row-based logging, depending on which is most appropriate for the change to be logged. This is called mixed-format logging. When using mixed-format logging, a statement-based log is used by default. Depending on certain statements, and also the storage engine being used, the log is automatically switched to row-based in particular cases. Replication using the mixed format is often referred to as mixed-based replication or mixed-format replication.
13
![Page 14: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/14.jpg)
Notes for the future
With MySQL.5.6 RBR started sending over only the primary key and the changed data (not sending unchanged data) which can drastically cut the amount of data sent to slave servers. This can be huge!!
Many future products will work better with RBR as it more deterministic. So plan accordingly.
14
![Page 15: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/15.jpg)
Threading
Before 5.6 MySQL Replication is SINGLE threaded – Airline boarding example
MySQL 5.6 is multi-threaded at the database level
MySQL 5.7 is multi-threaded at the table level
15
![Page 16: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/16.jpg)
Synchronicity
16
![Page 17: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/17.jpg)
Async and Semi Synchronous
Semi Synchronous replication -- a commit performed on the master blocks before returning to the session that performed the transaction until at least one slave acknowledges that it has received and logged the events for the transaction (Note not written to table, just recorded for future).
Asynchronous replication -- slave servers retrieve data, master unaware of slave’s consumption.
17
![Page 18: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/18.jpg)
18
Topo
logi
es
![Page 19: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/19.jpg)
Topologies -- Before 5.7
19
![Page 20: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/20.jpg)
Common - Read / Write Split
20
![Page 21: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/21.jpg)
Multi-source Replication -- MySQL 5.7
21
![Page 22: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/22.jpg)
Group Replcation -- Labs.MySQL.Com for evaluation
22
![Page 23: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/23.jpg)
Multi-Master
Lots of folks want TWO active masters and this can be done but not recommended, You need to have a sharp crew and plan for conflicts.
Not generally recommended before Group Replication
23
![Page 24: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/24.jpg)
Multi-Master MySQL Cluster
You can run active-active master-master with MySQL Cluster, even between data centers.
This can be very expensive and MySQl Cluster is not a full featured version of the MySQL Server.
24
![Page 25: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/25.jpg)
Galera Cluster
Layer separate from MySQL that is mainly for high availability (not high performance).
Claims to have snapshot isolation on transactions but watch out for ‘first committer wins’ and prepare for rollbacks.
Not low latency
25
![Page 26: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/26.jpg)
How to setup MySQL
Replication26
![Page 27: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/27.jpg)
Two types of replication w/ & w/o GTIDs
A global transaction identifier (GTID) is a unique identifier created and associated with each transaction committed on the server of origin (master). This identifier is unique not only to the server on which it originated, but is unique across all servers in a given replication setup. There is a 1-to-1 mapping between all transactions and all GTIDs.
3E11FA47-71CA-11E1-9E33-C80AA9429562:1-5
Note the 1-5 is a group of transactions
27
![Page 28: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/28.jpg)
Binlog & ID
Enable Binary Log on Master, Unique ID number
Create User
Create user for replication
Binlog Offset
Get Master’s Binary Log coordinates
Snapshot
Snapshot data, copy onto slave
SLAVE running
CHANGE MASTER command and START SLAVE
Before GTIDs (MySQL 5.5 and before) 28
![Page 29: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/29.jpg)
Enable Binary Log & Unique ID on Master
Edit my.cnf file [mysqld]
log-bin=mysql-bin
server-id=1
29
![Page 30: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/30.jpg)
Create replication user
mysql> CREATE USER 'repl'@'%.mydomain.com' IDENTIFIED BY
'slavepass';
mysql> GRANT REPLICATION SLAVE ON *.* TO
'repl'@'%.mydomain.com';
30
![Page 31: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/31.jpg)
Get binary log position
mysql> FLUSH TABLES WITH READ LOCK;
mysql > SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000003 | 73 | test | manual,mysql |
+------------------+----------+--------------+------------------+
31
![Page 32: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/32.jpg)
Unlock tables mysql> UNLOCK TABLES;
32
![Page 33: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/33.jpg)
Config slave & load data
No config thee slave server. Remember server-id must be unique
[mysqld]server-id=2
shell> mysql -h master < fulldb.dump
33
![Page 34: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/34.jpg)
Change Master & Start Slave
mysql> CHANGE MASTER TO
-> MASTER_HOST='master_host_name',
-> MASTER_USER='replication_user_name',
-> MASTER_PASSWORD='replication_password',
-> MASTER_LOG_FILE='recorded_log_file_name',
-> MASTER_LOG_POS=recorded_log_position;
mysql> START SLAVE;
34
![Page 35: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/35.jpg)
Binlog & ID
Enable Binary Log on Master, Unique ID number
Create User
Create user for replication
Snapshot
Snapshot data, copy onto slave
SLAVErunning
CHANGE MASTER command and START SLAVE
SLAVE running
GTIDs make it much easier to automate many functions
With GTIDs (MySQL 5.6 & Later) 35
![Page 36: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/36.jpg)
Replication Setup with GTIDs
mysql>mysqladmin -uusername -p
shutdown
shell> mysqld --gtid-mode=ON
--enforce-gtid-consistency &
mysql> CHANGE MASTER TO
> MASTER_HOST = host,
> MASTER_PORT = port,
> MASTER_USER = user,
> MASTER_PASSWORD = password,
> MASTER_AUTO_POSITION = 1;
mysql> START SLAVE;
36
![Page 37: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/37.jpg)
30 Minutes!!!
Can’t Cover all of MySQL Replication in ~60 minutes!!!!! 37
![Page 38: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/38.jpg)
https://github.com/datacharmer
Giuseppe Maxia has lot of great code for getting used to replication -- see ~/mysql-replication-sample
38
Great Place to Find examples & MySQL Sandbox
![Page 39: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/39.jpg)
MySQL Utilities
FREE Scripts written in Python, used with MySQL Workbench or stand alone
1. Copy, diff databases
2. Disk usage, grants, copy users
3. Search for processed and kill ‘em
4. Setup replication and failover
39
![Page 40: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/40.jpg)
Mysqlrplcheck -- check replication setup
shell> mysqlrplcheck --master=root@host1:3310 --slave=root@host2:3311# master on host1: ... connected.# slave on host2: ... connected.Test Description Status------------------------------------------------------------------------Checking for binary logging on master [pass]Are there binlog exceptions? [pass]Replication user exists? [pass]Checking server_id values [pass]Is slave connected to master? [pass]Check master information file [pass]Checking InnoDB compatibility [pass]Checking storage engines compatibility [pass]Checking lower_case_table_names settings [pass]Checking slave delay (seconds behind master) [pass]# ...done. 40
![Page 41: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/41.jpg)
Mysqlrplcheck -- replication checkershell> mysqlrplsync --master=user:pass@localhost:3310 \ --slaves=rpl:pass@localhost:3311,rpl:pass@localhost:3312## GTID differences between Master and Slaves:# - Slave 'localhost@3311' is 15 transactions behind Master.# - Slave 'localhost@3312' is 12 transactions behind Master.## Checking data consistency.## Using Master 'localhost@3310' as base server for comparison.# Checking 'test_rplsync_db' database...# - Checking 't0' table data...# [OK] `test_rplsync_db`.`t0` checksum for server 'localhost@3311'.# [OK] `test_rplsync_db`.`t0` checksum for server 'localhost@3312'.# - Checking 't1' table data...# [OK] `test_rplsync_db`.`t1` checksum for server 'localhost@3311'.# [OK] `test_rplsync_db`.`t1` checksum for server 'localhost@3312'.# Checking 'test_db' database... 41
![Page 42: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/42.jpg)
Mysqlslavetrx -- skip bad transactions
shell> mysqlslavetrx --gtid-set=af6b22ee-7b0b-11e4-aa8d-606720440b68:7-9 \ --slaves=user:pass@localhost:3311,user:pass@localhost:3312WARNING: Using a password on the command line interface can be insecure.## GTID set to be skipped for each server:# - localhost@3311: af6b22ee-7b0b-11e4-aa8d-606720440b68:7-9# - localhost@3312: af6b22ee-7b0b-11e4-aa8d-606720440b68:7-9## Injecting empty transactions for 'localhost:3311'...# Injecting empty transactions for 'localhost:3312'...##...done.
42
![Page 43: MySQL Replication Update -- Zendcon 2016](https://reader030.vdocument.in/reader030/viewer/2022020213/58a5e2b91a28abd14d8b6445/html5/thumbnails/43.jpg)
MySQL Replication Update
Q/AZendcon 2016Dave Stokes
[email protected] https://legacy.joind.in/talk/view/[email protected]/davidmstokes Elephantanddolphin.blogger.com & opensourcedba.wordpress.com
43