mysql highav availability
TRANSCRIPT
![Page 2: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/2.jpg)
About MeBaruch osoveskiy
• Senior Consultant @ brillix.co.il
• Linux-Unix SYSADMIN form 1997
• DBA on Oracle and MySQL Since 2000
• Working with unstructured Data (“big data”) since 2000 like text and spatial images etc..
https://github.com/barucho
http://www.slideshare.net/baruchosoveskiy
![Page 3: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/3.jpg)
• In the end will be Q&A slide –
you can ask question I will try to answer
• The Demo will be in github :
https://github.com/barucho/MySQLHA2016.git
• You can send me question :
You can find more presentations on
http://www.slideshare.net/baruchosoveskiy
• I write in ildba.co.il (not so often)
![Page 4: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/4.jpg)
Some of our Customers
![Page 5: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/5.jpg)
High Availability
![Page 6: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/6.jpg)
The Problem
![Page 7: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/7.jpg)
Redundancy
▪ multiple copies of data in event of a disaster or bad DML/DDL run in Production
Scaling
▪ to increase throughput (more network, disk, cpu)
High Availability
▪ get databases back online quickly
High Availability , Redundancy, Scaling ?
![Page 8: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/8.jpg)
CAP …
Consistency▪ All nodes see the same data at the same time
• Availability▪ Every request receives a response
about whether it succeeded or not (COMMIT ?)
• Partition Tolerance▪ The system continues to operate
despite arbitrary partitioning due to
network failures
![Page 9: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/9.jpg)
What is my expected UP-TIME (9`s)?
![Page 10: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/10.jpg)
Manual Failover
▪ human usually can make a better decision as to whether failover is necessary.
▪ Avoid ping-ping between nodes!
▪ easy to configure
Automatic Failover
▪ More 9’s due to minimized outages.
▪ No need to wait on a DBA/ops.
▪ Very Complicate to configure (need cluser/scripts)
Automatic or Manual Failover?
![Page 11: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/11.jpg)
The Solutions
![Page 12: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/12.jpg)
Galera Cluster High Availability, Scaling-improve read performance
• Active Active MySQL* cluster • On Commit Changes are collected into a write
set.• Write set is set to all nodes for certification.• PKs are used to determine if the write set can be
applied.• If certification fails, the write set is dropped and
the transaction is rolled back.• All nodes will reach the same transaction
eventually.
![Page 13: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/13.jpg)
Corosync-pacemaker DRDB active/passiveHigh Availability
• Corosync/pacemaker cluster or a co-ordinatingthe startup and recovery of inter-related services across a set of machines.
• DRDB shared storage systems by networked mirroring
• Only one mysqld process can run at a time • No active slave.• DRDB can have performance issues in high IO
load • Complicate to Configure
![Page 14: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/14.jpg)
Master Slave
DRDB
corosync
pacemaker
DRDB
MySQL MySQL
corosync pacemaker DRDB active/passive MySQL
quorum
![Page 15: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/15.jpg)
Master Slave
DRDB
corosync
pacemaker
DRDB
MySQL MySQL
corosync pacemaker DRDB active/passive MySQL
quorumMaster
![Page 16: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/16.jpg)
Master Slave
DRDB
corosync
pacemaker
DRDB
MySQLMySQL
corosync pacemaker DRDB active/passive MySQL
quorum
MySQL
![Page 17: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/17.jpg)
• One or more mysqld connected to one or more data nodes
• The data on the data node is shard and replicated
• Simple and automatic sharding and replication
• To use MySQL cluster you have to use NDB storage engine.
MySQL Cluster / NDB / shardingHigh Availability, Scaling
![Page 18: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/18.jpg)
mysqld
MySQL Cluster / NDB / sharding
mysqld mysqld mysqld
A1
B2
B1
A2
NDB NDB NDB
NDB API
SQL
REST API
![Page 19: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/19.jpg)
February 2015: MySQL Cluster 7.4 - 200 Million NoSQL QPS
MySQL Cluster 7.4 delivers massively concurrent NoSQL access –
200 Million reads per second using the FlexAsync benchmark.
This was achieved with 32 (out of a maximum 48) data nodes,
each running on a server with 2x Intel Haswell E5-2697 v3 CPUs.https://www.mysql.com/why-mysql/benchmarks/mysql-cluster/
![Page 20: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/20.jpg)
INNODB vs NDB
Feature InnoDB 1.1 NDB 7.5
Transactions All standard types READ COMMITTED
MVCC Yes No
Data Compression Yes No
Large Row Support (> 14K) Supported for VARBINARY, VARCHAR, BLOB, and TEXT columns
Supported for BLOB and TEXT columns only
High Availability (HA) Requires additional software Yes (Designed for 99.999% uptime)
Time for Node Failure Recovery 30 seconds or longer Typically < 1 second
NoSQL Access to Storage Engine Yes Yes
![Page 21: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/21.jpg)
MySQL GTID Replication
![Page 22: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/22.jpg)
What is MySQL Replication
Master SlaveNETWORK
![Page 23: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/23.jpg)
What is MySQL Replication
Master SlaveNETWORKSlave
SlaveSlave
Slave
![Page 24: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/24.jpg)
Why MySQL Replication
• High availability – can promote slave to master
• Scale Out – add multiple read slaves • Analytics – can run Analytics on live data.• Backup – run cold backup on slave.• For DEV and Pre-Prod
![Page 25: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/25.jpg)
How MySQL Replication Works
Replication Formats• SBR - Statement Based Replication
• Nondeterministic function problem • RBR - Row Based Replication
• High network usage • Higher IO usage
• Mixed Mode
![Page 26: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/26.jpg)
How MySQL Replication Works
Not safe Nondeterministic functionFOUND_ROWS(), GET_LOCK(), IS_FREE_LOCK(), IS_USED_LOCK(), LOAD_FILE(), MASTER_POS_WAIT(), RELEASE_LOCK(), ROW_COUNT(), SESSION_USER(), SLEEP(), SYSDATE(), SYSTEM_USER(), USER(), UUID(), and UUID_SHORT().
![Page 27: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/27.jpg)
How MySQL Replication Works
Where the Changes Stored • Binary Log – Master (blue-bin.000002) • Relay Log – Slave (white-relay-bin.000003)
![Page 28: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/28.jpg)
How MySQL Replication Works
Replication Threads• Master Server
• Binlog Dump Thread• Slave Server
• IO Thread• SQL Thread
![Page 29: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/29.jpg)
How MySQL Replication Works
Method of Sending the Data (Replication Type)• Asynchronous Replication• Semi-Synchronous Replication (5.5 )• Group Replication (plugin)
![Page 30: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/30.jpg)
How MySQL Replication Works
Master
TRX1TRX2TRX3
.
Slave
1
Binarylog
DML+
COMMIT
Binlog Dump Thread
![Page 31: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/31.jpg)
How MySQL Replication Works
Master
TRX1TRX2TRX3
.
Slave
1
2
Binarylog
IO Thread
DML+
COMMIT
![Page 32: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/32.jpg)
How MySQL Replication Works
Master
TRX1TRX2TRX3
.
Slave
TRX1TRX2TRX3
.
1
2
3
Binarylog
RelayLog
IO Thread
DML+
COMMIT
![Page 33: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/33.jpg)
How MySQL Replication Works
Master
TRX1TRX2TRX3
.
Slave
TRX1TRX2TRX3
.
1
2
34
Binarylog
RelayLog
IO Thread
SQL Thread
DML+
COMMIT
![Page 34: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/34.jpg)
How MySQL Replication Works
Master
TRX1TRX2TRX3
.
Slave
TRX1TRX2TRX3
.
1
2
34
5
Binarylog
RelayLog
IO Thread
SQL Thread
DML+
COMMIT
![Page 35: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/35.jpg)
MySQL 5.6 Replication
What's New
• Multi threaded Slave• Relay log recovery • Binary Log Group Commit• Crash-safe Replication – Binlog Checksums• Global Transaction ID
You can read more in:http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html
![Page 36: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/36.jpg)
Multi-Threaded Slaves
Replication performance is improved by using multiple execution threads to apply replication events
slave-parallel-workers=2
You can read more in:http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html
![Page 37: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/37.jpg)
Binary Log Group Commit
• Commit a group of transactions to the binary log.• Ensures durability with good performance.• Designed to handle seamlessly temporary peeks on the
workload.
enable by default in 5.6
You can read more in:http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html
![Page 38: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/38.jpg)
Relay log recovery
• slave can automatically recover from a failure and resume replicating DML updates
• the slave can roll back replication to the last successfully committed transaction
relay-log-recovery =1
You can read more in:http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html
And in Bug #13893363
![Page 39: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/39.jpg)
Global Transaction ID
![Page 40: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/40.jpg)
Position (pre 5.6)
• Transactions are not guaranteed to execute in the order of the Binary logs
• The show slave status is not accurate
![Page 41: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/41.jpg)
Position
Master
Slave1
Slave2
BinlogFile:mysql-bin.0010Position:1238574
BinlogFile:mysql-bin.0020Position:1043
BinlogFile:mysql-bin.0140Position:8373
![Page 42: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/42.jpg)
Position - What's the problem
• The same event can have different Binary log file and position on different slaves
• Hard to find inconsistency in replication• Very problematic to reconfigure replication
![Page 43: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/43.jpg)
Global Transaction ID
• Unique identifier of a transaction across all servers in replication topology
The GTID look like :
GTID = source_id:transaction_idWhere transaction_id = transaction_start-transaction_stop
eg :3E11FA47-71CA-11E1-9E33-C80AA9429562:1-5
![Page 44: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/44.jpg)
Global Transaction ID
Master
Slave1
Slave2
BinlogFile:mysql-bin.00103E11FA47-71CA-11E1-9E33-C80AA9429562:1-5
BinlogFile:mysql-bin.00203E11FA47-71CA-11E1-9E33-C80AA9429562:1-5
BinlogFile:mysql-bin.01403E11FA47-71CA-11E1-9E33-C80AA9429562:1-5
![Page 45: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/45.jpg)
Global Transaction IDHow to enable GUID on MySQL replication• Enable read only mode on Slaves
set global read_only=1• Compare the binlog position on master and slave
show slave status show master status
• Change configuration on all servers and restart bin-logguid_mode=ONEnforce-guid-consistency
• Initialize GUID on all servers change master to MASTER_AUTO_POSITION = 1;
• Disable read only mode on Slaves
![Page 46: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/46.jpg)
Replication Setupmaster.my.cnf
[mysqld]...server-id=1report-host=bluebinlog-format=MIXEDlog-binlog-slave-updates=truegtid-mode=on enforce-gtid-consistency=true master-info-repository=TABLErelay-log-info-repository=TABLEsync-master-info=1binlog-checksum=CRC32master-verify-checksum=1
![Page 47: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/47.jpg)
Replication Setupslave.my.cnf[mysqld]...server-id=2report-host=whitebinlog-format=MIXEDlog-slave-updates=truelog-bingtid-mode=on enforce-gtid-consistency=true master-info-repository=TABLErelay-log-info-repository=TABLEsync-master-info=1slave-parallel-workers=2binlog-checksum=CRC32master-verify-checksum=1slave-sql-verify-checksum=1binlog-rows-query-log_events=1
![Page 48: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/48.jpg)
MySQL Replication Types
![Page 49: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/49.jpg)
Group Replication
• The replication group is a set of servers that interact with each other through message passing
• Each server in the group may execute transactions independently
• RW transaction only commits after that operation is coordinated within the group
• when a transaction is ready to commit, the server atomically broadcasts the write values (row events) and write set (unique identifiers of the rows that were updated). This establishes a global total order for that transaction
![Page 50: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/50.jpg)
Group Replication
• A highly available distributed MySQL database service
• Removes the need for manually handling server fail-over
• Provides distributed fault tolerance
• Enables Active/Active update anywhere setups
• Automates reconfiguration (adding/removing nodes, crashes, failures)
• Automatically detects and handles conflicts
![Page 51: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/51.jpg)
• Automated master monitoring and failover.
• MHA consists of MHA Manager and MHA Node packages. MHA Manager runs on a manager server, and MHA Node runs on each MySQL server
• MHA is a free opensource tool
• Can be configured to power off old Master to avoid split brain.
MySQL Replication and MHA
![Page 52: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/52.jpg)
MaxScale (open source database proxy)
• Open Source; Developed by mariadb
• “simple” database proxy
• Provides plugins
• Content aware
![Page 53: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/53.jpg)
MaxScale (open source database proxy)Redundancy, High Availability, Scaling (read)
Master Slave
MaxScleMaxScle
MaxScle
Client
Read Write Read Only
SlaveSlave
MySQL ReplicationSlave
Delay Replication
![Page 54: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/54.jpg)
MySQL Router
• Open Source; Developed by Oracle MySQL HA
• Simple layer 4 packet router • No packet inspection
• Have Framework for plugins
• Natively integrated with MySQL Fabric
• Support MySQL InnoDB Cluster (Lab)
![Page 55: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/55.jpg)
MySQL Router - plugins
• Routing • First available • Round robin
• Fabric cache
• Logger
![Page 56: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/56.jpg)
fa fa
MySQL Router
Slave
MySQL Router
Client
Read Write Read Only
SlaveMySQL
Replication
Fabric repository
RO HA GROUP
Metadata
RW HA GROUP
![Page 57: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/57.jpg)
More database routers/Load balancer
• HAProxy
• F5 BigIP
• ProxySQL
• ELB (AWS)
![Page 58: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/58.jpg)
MySQL INNODB Cluster
MySQL InnoDB Cluster: Vision “A single product — MySQL — with high availability and scaling features baked in; providing an integrated end-to-end solution that is easy to use.”
Redundancy, High Availability, Scaling
![Page 59: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/59.jpg)
MySQL INNODB Cluster
A minimum of three instances are required to create an InnoDB
cluster that is tolerant to the failure of one instance.
![Page 60: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/60.jpg)
MySQL INNODB Cluster
![Page 61: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/61.jpg)
MySQL Replication demo
![Page 62: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/62.jpg)
3306 3307
MaxScle
Client
Read Write Read Only
3308MySQL
Replication
![Page 63: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/63.jpg)
Q&A
![Page 64: MySQL highav Availability](https://reader035.vdocument.in/reader035/viewer/2022062302/58a5e3011a28abd14d8b64ed/html5/thumbnails/64.jpg)
Thank [email protected]