mysql high availability with replication new features

2 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, 8 th November, 2014. |

Open Source India 2014

MySQL High Availability with Replication New Features

Shivji Kumar Jha ([email protected])Manish Kumar ([email protected])Software Developer, MySQL Replication Team

mailto:[email protected]

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, 8 th November, 2014. |


Program Agenda

Background

Performance

Usability

Flexibility

Consistency

Extras

0

1

2

3

4

5


Background: Replication Components

Insert...

Insert...

B

binary log

Insert...

relay log

Insert...

A

binary log

ClientSenderthread

Receiverthread

ApplierThreads

Network


Read scale-out

M S

S

S

S

M

write clients read clientsread clients

write clients

Morereads?More

slaves!

Background: What is Replication Used For?


M M?

write clients write clients

Morewrites?More

Masters?

What about write scale-out?

M?

M?

MySQL Fabric helps sharding your data with MySQL...


Redundancy: If master crashes, promote slave to master

C

B

A


C

B

ACrash

C

B

A

B is thenew master

Uh Oh! Whew!


On-line Backup and Reporting


M

S

write clients

business intelligent client apps reporting client apps big queries client apps


CB

BAAC

Image fromwww.ginkgomaps.com



Program Agenda

Background

Performance

Usability

Flexibility

Consistency

Extras

1

2

3

4

5

6


Performance

— Higher Master Throughput. Improved concurrency between Sender and user threads, More Sender thread enhancements...

— Higher Slave Throughout. MySQL- 5.7.2: Multi-Threaded Slave (MTS) Timestamp based Applier. MySQL Labs: Enhanced MTS Timestamp based Applier. Options to Fine Tune Binlog Group Commit and Multi-Threaded Applier


Insert...

Insert...

B

binary log

Insert...

relay log

Insert...

A

binary log

Client

Network

Improved concurrency b/w Sender and user threads

Senderthread

Receiverthread

ApplierThreads


Sender Thread Reads Binary Log User Thread Writes to Binary Log

Sender Thread Reads Binary Log

User Thread Writes to Binary Log

Time

pre 5.7.2

5.7.2+

Concurrent reads by the sender thread with ongoing writes from user threads.

— Sender thread does not block user sessions more than necessary.— Higher throughput for both sender threads and user sessions.

Improved concurrency b/w Sender and user threads


Insert...

Insert...

B

binary log

Insert...

relay log

Insert...

A

binary log

Client

Network

Senderthread

Receiverthread

ApplierThreads

More Sender Thread Enhancements



Send buffer is not allocated and freed every time an event is sent.

When sender threads require larger send buffer → buffer grows as needed.

When buffer is larger than needed→ buffer shrinks dynamically.

Together with the sender thread enhancements released on MySQL 5.7.2:— increase master scalability;— Master, with dump threads attached, copes better with peak loads;— Separate semisync ACK receiver thread (details later!)



Ad hoc microbenchmark using mysqlslap

1M queries, concurrency=200, commit=1

N slaves attached (fake slaves using remote mysqlbinlogs)

48 cores HT / 512 GB RAM / HDD

Small number of slaves: no difference between 5.6 and 5.7. Larger number of slaves: 5.7 is able to sustain the same throughput while

5.6 is not.

0 1 5 10 20 30 40 50 1000

2000

4000

6000

8000

10000

12000

14000

16000

Master's Throughput

MySQL 5.6.16

MySQL 5.7.4

Number of Slaves

Qp

S


Up to 5x more throughput with 10 sysbench schemas.

Changes to each schema applied and committed independently by worker threads (both in SBR and RBR).

This was the case when we released MySQL 5.6 GA.

0 1 2 3 4 5 6 10 240

50

100

150

200

250

300

Slave Throughput

Number Worker Threads

Qp

SHigher Slave Throughput (MySQL-5.6)


“It's the bee's knees! We've been using it at VividCortex in production and it works great.”

– Baron Schwartz, CEO, VividCortex

https://vividcortex.com/blog/2014/03/17/multi-threaded-replication-in-mysql/

Higher Slave Throughput (MySQL-5.6)

https://vividcortex.com/blog/2014/03/17/multi-threaded-replication-in-mysql/


Insert...

Insert...

B

binary log

Insert...

relay log

Insert...

A

binary log

Client

Network

Senderthread

Receiverthread

ApplierThreads

Higher Slave Throughput (MySQL-5.7)


Higher Slave Throughput

Timestamp based Multi-threaded Slave (MTS) Applier● Leverage parallelization information obtained from the execution on the

master.● Transactions that prepare on the same “version” of the database, are

assigned the same timestamp.

● Meanwhile, at the slave:● Transactions with the same timestamp can be executed in parallel;● Concurrent transactions commit independently, thus no waiting involved.


Higher Slave Throughput

Supports statement-based or row-based formats.

Scheduling policy controlled through:

— logical_clock - means schedule based on the prepare timestamp.— database - the scheduling policy from 5.6 (concurrency control done per

database).

Work to improve slave scalability continues, does not stop here.

mysql> SET slave_parallel_type= [logical_clock|database]


Insert...

Insert...

B

binary log

Insert...

relay log

Insert...

A

binary log

Client

Network

Senderthread

Receiverthread

ApplierThreads

Enhanced Scheduling Policy (MTS)


Enhanced Scheduling Policy (MTS)

Up to 6X performance improvement vs single threaded slave without any changes to the application

Sysbench / oltp test / 48 clients / 10M rows

SSD / 48 core HT / 512 GB RAM

GTIDs / Crash-safe

Numbers from mysql-5.7.5-labs-previewTO BE UPDATED

0 4 16 50 2000

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Transactions per Second (TPS)

TPS while replaying 300 seconds of accumulated load from the master.

Baseline

100 clients – 5.7.5-labs

200 clients – 5.7.5-labs

Number of Worker Threads

Tra

nsa

ctio

ns

Pe

r S

eco

nd


Insert...

Insert...

B

binary log

Insert...

relay log

Insert...

A

binary log

Client

Network

BGC tuning options – MTS Enhancers

Senderthread

Receiverthread

ApplierThreads


Sysbench / oltp test / 48 clients / 10M rows

SSD / 48 core HT / 512 GB RAM

GTIDs / Crash-safe

Numbers from mysql-5.7.4-labs-mts


0 5 10 15 20 25 30 35 40 45 500

500

1000

1500

2000

2500

3000

3500

4000

4500

Transactions Per Second

small commit delay

large commit delay

Number of Slave Worker Threads

TP

S



By tuning BGC on the master, we get more parallelism on the slave.

Tuning means more transactions preparing together (no magic formula – highly dependent on hardware and workload).

Ongoing work on improving MTS and on the analysis of experimental results.

6X slave throughput – large commit delay 3X slave throughput – small commit delay

0 5 10 15 20 25 30 35 40 45 500

500

1000

1500

2000

2500

3000

3500

4000

4500

Transactions Per Second

small commit delay

large commit delay

Number of Slave Worker Threads

TP

S


Program Agenda

Background

Performance

Usability

Flexibility

Consistency

Extras

1

2

3

4

5

6


Insert...

Insert...

B

binary log

Insert...

relay log

Insert...

A

binary log

Client

Network

Better Replication Monitoring

Senderthread

Receiverthread

ApplierThreads



Performance Schema Replication Tables

● Access monitoring information through an SQL interface.

● Write stored functions or procedures with input from replication internals.

● Logically unrelated information into different places.

● Flexible and easier to extend and adapt as new feature get into the server.

● More user friendly names identifying the monitoring fields.



Connection Configuration

Connection Status

Execution Configuration

Execution Status

Slave Status

Applier / CoordinatorStatus

WorkersStatus


Program Agenda

Background

Performance

Usability

Flexibility

Consistency

Extras

1

2

3

4

5

6


Insert...

Insert...

B

binary log

Insert...

relay log

Insert...

A

binary log

Client

Network

Senderthread

Receiverthread

ApplierThreads

Semi-sync Replication – Wait for Multiple ACKs


Semi-sync Replication – Wait for Multiple ACKs

Master

Slave 2 relaylog

T1: COMMIT

ACK

Time

relaylogSlave 1ACK

T1: COMMITsucceeds

Master does not commit transaction until it receives N ACKs from N slaves. Dynamically settable:

mysql> SET GLOBAL rpl_semi_sync_master_wait_for_slave_count= N

mysql> SET GLOBAL rpl_semi_sync_master_wait_for_slave_count= 2


Insert...

B

Insert...

relay log

Insert...

A

binary log

Client

Network

Storing GTIDs In a Table

Senderthread

Receiverthread

ApplierThreads

gtid_executed gtid_executed



Use Cases

Slaves can use GTIDs with binary logs disabled.— Slaves that are never candidates to become a master can still use GTIDs

for auto positioning.

GTIDs are more resilient to unexpected deletion of binary logs.— If the DBA decides to delete all binlogs but the most recent one, GTID

history is still preserved.



Mechanics GTIDs are saved into a mysql system table.

CREATE TABLE gtid_executed( source_uuid CHAR(36) NOT NULL, interval_start BIGINT NOT NULL, interval_end BIGINT NOT NULL, PRIMARY KEY(source_uuid, interval_start) );



Mechanics

If binary log is enabled, gtids are first stored on the binary log and then copied to the table on binlog rotation or shutdown.

If binary log is disabled, gtids are stored in the table on commit.


Insert...

Insert...

B

binary log

Insert...

relay log

Insert...

A

binary log

Client

Network

Preserve Commit Order (MTS)

Senderthread

Receiverthread

ApplierThreads


Preserve Commit Order (MTS)

Multi-threaded Applier May be Instructed to Preserve Commit Order.

— Applier Threads will commit in the same order transactions appear in the relay log.

— Order is enforced at the expense of performance degradation.

— This is controllable through the new global option/variable: --slave-preserve-commit-order. Default is FALSE.

mysql> SET GLOBAL slave_preserve_commit_order= [TRUE|FALSE]


Insert...

Insert...

B

binary log

Insert...

relay log

Insert...

A

binary log

Client

Network

Senderthread

Receiverthread

ApplierThreads

Redirect to new Master without interrupting the Applier



Change master to B from A without stopping the applier threads.

C

B

A

C

B

ACrash

C

B

A

B is thenew master

Uh Oh! Whew!



C

B

A

C

B

ACrash

C

B

A

DBA did not need C to stop applying its relay log to change C's master from A to B.

Uh Oh! Whew!

Change master to M2 from M1 without stopping the applier threads.



More “onlineness” during fail-over:

mysql> STOP SLAVE IO_TRHEAD;mysql> CHANGE MASTER TO MASTER_HOST='master2', …;mysql> START SLAVE IO_THREAD;

Stopping, changing master and restarting the receiver thread is all done while the applier thread is running.

Change applier properties while the receiver thread is working:

mysql> STOP SLAVE SQL_TRHEAD;mysql> CHANGE MASTER TO MASTER_DELAY=3600, …;mysql> START SLAVE SQL_THREAD;


Insert...

Insert...

B

binary log

Insert...

relay log

Insert...

A

binary log

Client

Network

Dynamic Slave Replication Filters

Senderthread

Receiverthread

ApplierThreads


Dynamic Slave Replication Filters

mysql> CHANGE REPLICATION FILTER REPLICATE_DO_DB= (db1, db2)

Change Slave's Replication Filters dynamically.

— No need to stop and restart slave for establishing new replication filtering rules.

— All slave filters are supported.


Multi-Source Replication

S

M M

Slave can have more than one master.

Feature preview based on 5.7 is availableon labs.mysql.com .

M M

The need for gathering data in a central server: Integrated backup; Complex queries for analytics purposes;



A server (slave) can replicate from multiple sources (masters).

Multiple channels (channel: connection thread, relay log, applier threads) that can be stopped started individually.

Integrated with Multi-threaded Slave: each channel has its own multi-threaded applier set of threads.

Integrated with the new P_S tables.— replication_execute_status_by_coordinator shows multiple entries, one per

channel/source applier.— replication_connection_status shows multiple entries, one per connection to

different sources.



Integrated with GTIDs.

Integrated with crash-safe tables.— Progress state is stored in these tables for recoverability purposes.

Works with semisync replication.— Working on further improving the integration.

Virtually no limit on the number of sources (capped to 256, but can be changed if server binary is rebuilt).

Able to manage each source separately.


Program Agenda

Background

Performance

Usability

Flexibility

Consistency

Extras

0

1

2

3

4

5


Insert...

Insert...

B

binary log

Insert...

relay log

Insert...

A

binary log

Client

Network

Loss-less Semi-sync Replication

Senderthread

Receiverthread

ApplierThreads


Loss-less Semi-sync Replication Master waits for slave's ACK before committing (as opposed to: master waits

for slave's ACK after committing).

— Therefore, concurrent transactions do not externalize changes while waiting for ACK.

Should a master fail, then any transaction that it may have externalized is also persisted on a slave.

User can choose between the original semisync behavior and the new one.

mysql> SET rpl_semi_sync_master_wait_point= [AFTER_SYNC|AFTER_COMMIT]


Loss-less Semi-sync Replication

Master

Slave

execute binlog

relaylog

commitprepare

T1: INSERT INTO t1 VALUES (1000)

ACK

Time

T1

T2: SELECT * FROM t1;

T2

empty set


Insert...

Insert...

B

binary log

Insert...

relay log

Insert...

A

binary log

Client

Network

Semi-sync Replication – ACK Receiver Thread

Senderthread

Receiverthread

ApplierThreads

ACKReceiver

thread


Semi-sync Replication – ACK Receiver Thread Consecutive transactions do not block each other while waiting for ACKs

— Transaction t1 and t2 are sent immediately to the slave by the sender thread— ACKs are received only by a special thread— Transaction t2 does not include t1 round trip in its semisync overall latency

Thread starts when semisync is activated

Thread stops when semisync is deactivated

mysql> SET GLOBAL rpl_semi_master_enabled= ON

mysql> SET GLOBAL rpl_semi_master_enabled= ON


“Semi-sync in MySQL 5.7 looks like a great improvement. [...] With MySQL 5.7, I’d be much more apt to recommend semi-sync as an option than in previous releases. Thanks to Oracle for investing here.”

– Jay Janssen, MySQL Consulting Manager, Percona Inc

Semi-sync Replication in MySQL 5.7

http://www.percona.com/blog/2014/05/27/semi-sync-replication-performance-mysql-5-7-4-dmr/

http://www.percona.com/blog/2014/05/27/semi-sync-replication-performance-mysql-5-7-4-dmr/


Introducing MySQL Group Replication

Multi-master update everywhere with conflict resolution.

Automatic distributed member recovery— When a server joins the group, it gets all the missing transactions before actually

start serving requests.


M M M M M

ReplicationPlugin

API

MySQLServer

Group Comm.System (Corosync)Group Comm.

System (Corosync)

Group-Based Replication Plugin

M M M M M

ReplicationPlugin

API

MySQLServer

Group Comm.System (Corosync)



M M M M M

Update ...

Application issues an update.



M M M M M

ReplicationEvents


Master executes and multi/broadcasts the update to the other servers.



M M M M M



Servers receive same transaction in the same order and check for conflicts.



M M M M M




All servers, independently, decide to commit the transaction – no conflicts.






All servers, independently, decide to commit the transaction – no conflicts.

Originating master replies to the application.

M M M M M



MySQL Group Replication: Use Cases

● Automatic group membership management and failure detection - No need for server fail-over

● No single point of failure - Automatic reconfiguration


Under the hood

● Each server can belong to a group.

● The Group is identified by a name which is the Group UUID string, so group behaves like a single server.

● The Group UUID is used to log transactions.

● Whenever a server joins or leaves, it will start a new stable period in time on which servers can agree on distributed decisions. A special message called View Message is sent to each group member notifying that.

● Server join/leave can be issue manually or by scripts by DBA.

● Server leave can be caused also by node failures.


Current used group communication toolkit is Corosync. It provides:— Virtual synchrony guarantees for creating replicated state machines.

Replicated state machine simple definition:— If a set of processes with the same initial state receives the same set of deterministic

messages at the same order, them will reach the same state.

All write operations are sent to all group members for certification.

Read operations are only local.

Under the hood


s1

s2

UPDATEdbv: 1

dbv: 2

s3dbv: 2

dbv: 2COMMIT

COMMIT

COMMIT

cv: 11 >= 1

OK

1 >= 1

OK

1 >= 1

OK

cv: 1

cv: 1

dbv: 1

dbv: 1

cv: 2

cv: 2

cv: 2

s ← serverv ← snapshot versionsn ← certifier sequence number (write set version)

Certification - Positive


s1

s2

T1: UPDATEdbv: 1

dbv: 2

s3dbv: 2

dbv: 2T1: COMMIT

T1: COMMIT

T1: COMMIT

cv: 1

cv: 1

cv: 1

1 >= 1

OK

1 >= 1

OK

1 >= 1

OK

T2: UPDATE cv: 2

cv: 2

cv: 2

1 >= 2

KO

1 >= 2

KO

1 >= 2

KO

dbv: 1

dbv: 1

s ← serverv ← snapshot versionsn ← certifier sequence number (write set version)

Certification - Negative


Automated distributed member recovery

R1

R2

R3

R4

GCS

R1

R2

R3

R4

R4 joins the group.

GCS



R1

R2

R3

R4

R1 is R4 Donor

GCS

R1

R2

R3

R4

GCS



R1

R2

R3

R4

GCS

R1

R2

R3

R4

GCS



R1

R2

R3

R4

GCS

R1

R2

R3

R4

R4 has caught upwith the view!

GCS


R1

R2

R3

R4

GCS

R1

R2

R3

R4

GCS

Execute bufferedtransactions from the view.



– REPLICATION_NODE_STATUS● Statistics regarding Applier and Certifier module.

– REPLICATION_CONNECTION_NODES● Information regarding different nodes in the group and its

connection details

– REPLICATION_CONNECTION_STATUS● Extended table to give details about the group connection status.

PERFORMANCE SCHEMA TABLE


node1> SELECT * FROM performance_schema.replication_node_status\G *************************** 1. row *************************** GROUP_NAME: 8a84f397-aaa4-18df-89ab-c70aa9823561 NODE_ID: 2af09cb1-3e46-11e4-9e45-ecf4bb227f3b TRANSACTIONS_IN_QUEUE: 0 CERTIFIED_TRANSACTIONS: 12 POSITIVELY_CERTIFIED: 7 NEGATIVELY_CERTIFIED: 5 CERTIFICATION_DB_SIZE: 6 STABLE_SET: 8a84f397-aaa4-18df-89ab-c70aa9823561:1-7 LAST_CERTIFIED_TRANSACTION: 8a84f397-aaa4-18df-89ab-c70aa9823561:7 APPLIER_MODULE_STATUS: ON

Replication Monitoring: replication_node_status


node1> SELECT * FROM performance_schema.replication_connection_nodes\G *************************** 1. row *************************** GROUP_NAME 8a94f357-aab4-11df-86ab-c80aa9429573 NODE_ID 597dbb72-3e2c-11e4-9d9d-ecf4bb227f3b NODE_HOST nightfury NODE_PORT 13000 NODE_STATE ONLINE *************************** 2. row *************************** GROUP_NAME 8a94f357-aab4-11df-86ab-c80aa9429573 NODE_ID 59efb8a1-3e2c-11e4-9d9d-ecf4bb227f3b NODE_HOST nightfury NODE_PORT 13001 NODE_STATE ONLINE *************************** 3. row *************************** GROUP_NAME 8a94f357-aab4-11df-86ab-c80aa9429573 NODE_ID 5a706f6b-3e2c-11e4-9d9d-ecf4bb227f3b NODE_HOST nightfury NODE_PORT 13002 NODE_STATE RECOVERING




node1> SELECT * FROM performance_schema.replication_connection_status\G *************************** 1. row *************************** GROUP_NAME: 8a94f357-aab4-11df-86ab-c80aa9429563 SOURCE_UUID: 8a94f357-aab4-11df-86ab-c80aa9429563 THREAD_ID: NULL SERVICE_STATE: ON COUNT_RECEIVED_HEARTBEATS: 0 LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00 RECEIVED_TRANSACTION_SET: LAST_ERROR_NUMBER: 0 LAST_ERROR_MESSAGE: LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00 TOTAL_MESSAGES_RECEIVED: 9 TOTAL_MESSAGES_SENT: 5 TOTAL_BYTES_RECEIVED: 1891 TOTAL_BYTES_SENT: 1065 LAST_MESSAGE_TIMESTAMP: 2014-09-23 09:20:26 MAX_MESSAGE_LENGTH: 358 MIN_MESSAGE_LENGTH: 51 VIEW_ID: 2 NUMBER_OF_NODES: 2


Only transactional engines are supported (currently only InnoDB).

All tables must have primary key

GTID must be enabled.

Only deterministic DDL is allowed.

Optimistic execution: transactions may abort on COMMIT.

Limitations


Program Agenda

Background

Performance

Usability

Flexibility

Consistency

Extras

0

1

2

3

4

5


References Read a feature preview of MySQL Group replication on Luis' blog:

http://mysqlhighavailability.com/mysql-group-replication-hello-world/

Read about how to get started with MySQL Group replication on Pedro's blog:http://mysqlhighavailability.com/getting-started-with-mysql-group-replication/

Read about Transaction life cycle in MySQL Group replication on Nuno's blog:http://mysqlhighavailability.com/mysql-group-replication-transaction-life-cycle-explained/

Read about Distributed recovery in MySQL Group Replication on Pedro's blog:http://mysqlhighavailability.com/distributed-recovery-behind-the-scenes/

Read about Monitoring for MySQL Group Replication on Manish's blog:http://mysqlhighavailability.com/mysql-group-replication-monitoring/

Read about Corosync and Group Communication details on Tiago's blog:http://mysqlhighavailability.com/group-communication-behind-the-scenes/http://mysqlhighavailability.com/mysql-group-replication-a-small-corosync-guide/

Read about Testing of MySQL Group Replication on Manish's blog:http://mysqlhighavailability.com/mysql-group-replication-testing/

http://mysqlhighavailability.com/group-communication-behind-the-scenes/


References: Read about the 5.7 DMRs Replication Features

Read more about P_S tables for replication on Shiv's blog:http://shivjijha.blogspot.com/2013/09/Monitoring-Replication-with-the-NEW-performance-schema-tables.html

Read more about Intra-Schema Multi-threaded Slave on Rohit's blog:http://geek.rohitkalhans.com/2013/09/enhancedMTS-deepdive.html

◾ Read more about Semisync enhancements on Libing's blog:http://my-replication-life.blogspot.com/2013/09/dump-thread-enhancement.html

http://my-replication-life.blogspot.pt/2013/09/loss-less-semi-synchronous-replication.html

http://shivjijha.blogspot.com/2013/09/Monitoring-Replication-with-the-NEW-performance-schema-tables.html

http://geek.rohitkalhans.com/2013/09/enhancedMTS-deepdive.html

http://my-replication-life.blogspot.com/2013/09/dump-thread-enhancement.html

http://my-replication-life.blogspot.pt/2013/09/loss-less-semi-synchronous-replication.html


◾ Read more about mysqlbinlog idempotent mode on Rohit's blog:http://binlogtorelaylog.blogspot.com/2013/05/mysqlbinlog-idemmpotent-mode.html

◾ Read more about mysqlbinlog –rewrite-db option on Manish's blog:http://manishthe.blogspot.com/2013/05/introduction-with-wonderful-and-best.html


http://binlogtorelaylog.blogspot.com/2013/05/mysqlbinlog-idemmpotent-mode.html

http://manishthe.blogspot.com/2013/05/introduction-with-wonderful-and-best.html


Read more on Dynamic Replication Filters on Venkat's blog:http://my-s-q-l.blogspot.com/2013/12/Making-MySQL-Slave-Replication-Filters-Dynamic.html

Read more on Semi-sync Multiple ACKs on Libing's blog:http://my-replication-life.blogspot.com/2013/12/enforced-semi-synchronous-replication.html

Read more on SSL options for mysqlbinlog on João's blog:http://jmysqlrep.blogspot.com/2013/12/mysql-57-mysqlbinlog-now-supports-ssl.html

Read more on Semisync ACK thread receiver on Libing's blog:http://my-replication-life.blogspot.com/2014/03/faster-semisync-replication.html


http://my-s-q-l.blogspot.com/2013/12/Making-MySQL-Slave-Replication-Filters-Dynamic.html

http://my-replication-life.blogspot.com/2013/12/enforced-semi-synchronous-replication.html

http://jmysqlrep.blogspot.com/2013/12/mysql-57-mysqlbinlog-now-supports-ssl.html

http://my-replication-life.blogspot.com/2014/03/faster-semisync-replication.html


Read more on Flexible Change Master on Shiv's blog:http://work.shivjijha.com/2014/04/change-master-without-stopping-slave.html


http://work.shivjijha.com/2014/04/change-master-without-stopping-slave.html


Read more about Multi-source replication on Rith's blog:http://on-mysql-replication.blogspot.com/2013/09/feature-preview-mysql-multi-source-replication.html

Read a lot more on : mysqlhighavailability.com


http://on-mysql-replication.blogspot.com/2013/09/feature-preview-mysql-multi-source-replication.html

mysql high availability with replication new features

Technology