mysql user camp: multi-threaded slaves

MySQL Replication and

Multi-threaded Slaves

Shivji Kumar Jha, Software Developer,MySQL Replication Team

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | `2

Safe Harbour Statement

The following is intended to outline our general product direction. It is

intended for information purposes only, and may not be incorporated into any

contract.

It is not a commitment to deliver any material, code, or functionality, and

should not be relied upon in making purchasing decisions. The development,

release, and timing of any features or functionality described for Oracle’s

products remains at the sole discretion of Oracle.

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |`3

Agenda Why Replication?

Replication Internals- A Simple Introduction

Why Multi-threaded Slaves (MTS)?

Different policies for Multi-threading Slaves

Keeping track of replication threads


Replication: Copy Changes Master → Slave

MySQL Master Server

– Changes data

– Sends changes to slave

MySQL Slave Server– Receives changes from master

– Applies received changes to database

M S


Why Replication? – Scalability

Read scale-out

M S

write clients read clients



Read scale-out

M S


Morereads?



Read scale-out

M S


Morereads?More

slaves!



Read scale-out

M SS

S

S

M

write clients read clientsread clients

write clients

Morereads?More

slaves!


Why Replication? – Redundancy

If master crashes, promote slave to master

C

B

A




C

B

ACrash




C

B

A

B is thenew master


Why Replication? – Disaster recovery

CB

BAAC

Image fromwww.ginkgomaps.com


But how do you copy changes to slave?


But how do you copy changes to slave?

Are there LOGS floating around?


Introducing replication LOGS...


Replication Logs

MySQL Master Server

– Changes data (Writes)

– Saves changes in Binary log

MySQL Slave Server– Copies changes to Relay log

– Reads Relay log

– Applies changes.

M

S

Binarylog

Relaylog


All Changes Written to Binary Log

A

binary log

Client



A

binary log

Clientcreate table t (a int);

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | `19


create...

A

binary log

Client

Table t

create table t (a int);



create...

A

binary log

Client

Table t

create table t (a int);insert into t values (1);



create...insert...A

binary log

Client

Table t1

create table t (a int);insert into t values (1);


Slave Initiates Replication

B

relay log

A

binary log

Client

1. Slave sendsrequest to start replication

to master

2. Master sendsstream of replication data

to slave


create...

A

binary log

Client

B

relay log

Slave Copies, Applies Same Changes


Table t

create...

A

binary log

Client create...

B

relay log



create...


Table t Table t

create...

A

binary log

Client

B

relay log

create...


create...insert...

Table t1

create...insert...A

binary log

Client

Table t

B

relay log

create...



create...insert...

Table t1

create...insert...A

binary log

Client

Table t

B

relay log

create...insert...

1



OK makes sense !


But I see clients applying changes at Master, who applies at Slave?


But I see clients applying changes at Master, who applies at Slave?

Do you have some internal replication threads?


Introducing replication THREADS...


create...insert...

Replication Threads (4.0-5.5)

Bcreate...insert...A

binary log

Client

create...insert...

relay log

IO THREAD SQL THREAD

Writes into relay log at slaveReads from relay log.

Applies transactions at slave

network


insert...


Bcreate...insert...Insert...

A

binary log

Client

create...insert...Insert...

relay log

IO THREAD SQL THREADnetwork

ClientClientMore clients on master...Create... insert...


insert...



A

binary log

Client


relay log



Slave starts to lag behind master


insert...



A

binary log

Client


relay log



Table t123

Table t12



Lets Parallelize slave as well...


Lets Parallelize slave as well...

Introducing parallelization byDatabase


create...

Replication Threads (5.6+)

B

create(db1)insert(db2)insert(db3)

A

binary log

Client

create(db1)insert(db2)insert(db3)relay log

IO THREAD

COORDINATOR THREAD

Reads & assigns to worker(s)

db1

network

wORKERS

Worker threads applyconcurrently on diff dbs

db2

db3

ClientClient

insert... insert...

create..

insert...

insert..



Setting up Parallelization by database

mysql> STOP SLAVE;

mysql> SET GLOBAL slave_parallel_workers=1;

mysql> START SLAVE;



More databases? More workers!

mysql> STOP SLAVE;


mysql> START SLAVE;


What if I have aa loaded

Database?

Image credits: http://www.easetechnology.co.uk/ask-an-expert/


Lets Parallelize slave somewhat similar to master


Introducing parallelization byMaster concurrency

Lets Parallelize slave somewhat similar to master

5.7.2


insert...



A

binary log

Client


relay log



Table t123

Table t12REVISITING THE OLD SLIDE !!!



create...


B


A

binary log

Client

create...insert...Insert...relay log

IO THREAD

COORDINATOR THREAD

db1

network

wORKERS

Worker threads applyconcurrently

db2

ClientClient

insert... insert...

capture parallelization info on master

Reads & assigns to worker(s)


create...


B


A

binary log

Client


IO THREAD

COORDINATOR THREAD

db1

network

wORKERS


db2

ClientClient

insert... insert... Reads & assigns to worker(s)

Save parallelization info in binary log


create...


B


A

binary log

Client


IO THREAD

COORDINATOR THREAD

db1

network

wORKERS


db2

ClientClient


Send parallelization info to slave


create...


B


A

binary log

Client


IO THREAD

COORDINATOR THREAD

db1

network

wORKERS


db2

ClientClient


Use parallelization info to assign


Leverage parallelization information obtained from the execution on the master:— Transactions that reach the prepare phase on the same data snapshot

are non-contending;— Write to the binary log on which snapshot id each transaction prepared;

This identifies the commit parent of each transaction.— The commit parent is stepped every time transactions commit.

Meanwhile, at the slave:— Transactions with the same commit parent can be executed in parallel;— Commit sequence at the slave may not be the same as that on the

master, but it is still a correct execution history.



Higher Slave Throughput: Intra-Schema MTS

T1

T2

T3

Time

Commit

Prepare

Concurrent Execution HistoryConcurrent Execution History on the Masteron the Master

commit_parent++

Execution

Prepare

Commit


Higher Slave Throughput: Intra-Schema MTS

T1

T2

T3

Time

Commit

Prepare

Concurrent Execution HistoryConcurrent Execution History on the Masteron the Master

commit_parent++

Execution

Prepare

Commit

Parallelon the Slave.

Not parallel on the slave.



mysql> STOP SLAVE;


mysql> START SLAVE;

mysql> SET GLOBAL slave_parallel_type=[ 'logical_clock' | 'database' ];

Setting up Parallelization by master concurrency

database=>parallelize by databaselogical_clock=>parallelize by master concurrency


Too many threads here and there!

How do I monitor?

Knew that was coming! Check out replication

Performance_Schema


Performance Schema (5.5+)

Inspect internal execution of the server at runtime Exposed within performance_schema database Records various run time statistics via in-built instrumentation points Tracks latency for various events:

File I/O, Mutexes, Read/Write Locks, Table I/O, Table Locks, Stages, Statements, Idle etc..

Its a long growing list


connection execution

configuration status

Coordinator thread / SQL thread Worker thread(s)

replication

Replication Performance Schema Tables (5.7+)

configuration status

5.7.2


The REPLICATION P_S Tables

We have six performance schema tables for replication: (MySQL-5.7.2):

replication_connection_configuration replication_connection_status (IO thread status) replication_execute_configuration replication_execute_status replication_execute_status_by_coordinator replication_execute_status_by_worker

Tables for monitoring MTS

5.7.2


Monitoring The Coordinator Thread

mysql> select * from performance_schema.replication_execute_status_by_coordinator\G

*************************** 1. row ***************************

THREAD_ID: 13

SERVICE_STATE: ON

LAST_ERROR_NUMBER: 0

LAST_ERROR_MESSAGE:

LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00

No error in thread

5.7.2


Monitoring The Coordinator Thread

mysql> select * from performance_schema.replication_execute_status_by_coordinator\G

*************************** 1. row ***************************

THREAD_ID: 13

SERVICE_STATE: ON

LAST_ERROR_NUMBER: 1146

LAST_ERROR_MESSAGE:...'Table 'test.t' doesn't exist'...

LAST_ERROR_TIMESTAMP: 2013-11-04 13:37:23

Oops! There was an error

5.7.2


mysql> select * from performance_schema.replication_execute_status_by_worker\G

*************************** 1. row *************************** WORKER_ID: 0 THREAD_ID: 16 SERVICE_STATE: ONLAST_SEEN_TRANSACTION: 2b3e03a6-453f-11e3-8668-0021ccb3570d:3 LAST_ERROR_NUMBER: 0 LAST_ERROR_MESSAGE: LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00

*************************** 2. row *************************** WORKER_ID: 1 THREAD_ID: 17 SERVICE_STATE: ON ...

Monitoring Worker Threads

mysql> SET GLOBAL slave_parallel_workers= 2;

one row per worker thread

5.7.2







The newest transaction this worker is aware of

5.7.2







No error in this worker thread

5.7.2


We Want Your

Feedback

MySQL-5.6 is our latest GA release.

A lot more to explore in MySQL-5.6, keep

reading...

MySQL-5.7 is our current development

branch.

We want your valuable feedback.

Suggest Features, report bugs, contribute

patches.

Help make MySQL-5.7 even better!


Summary

Introduction to MySQL Replication

Replication Logs

Replication Threads

Multi-threaded Slave (MTS)– Need for Parallelization– Parallelize by database– Parallelize by master concurrency

Keeping track of replication threads– Performance Schema– Replication info. in Performance Schema


Read More About

MySQLReplication

Find MySQL Official Documentation at

http://dev.mysql.com/doc/refman/5.7/en/replication.html

http://dev.mysql.com/doc/refman/5.7/en/replication.html


Read More About Multi-

Threaded Slaves

Parallelization by database - Luis's blog - Andrei's blog

Parallelization by master concurrency - Rohit's blog

Replication Performance Schema - Official MySQL documentation - Shiv's blog

http://d2-systems.blogspot.in/2011/04/mysql-56x-feature-preview-multi.html

http://andreithedolphin.blogspot.in/2012/10/parallel-slave-in-mysql-replication.html

http://geek.rohitkalhans.com/2013/09/enhancedMTS-deepdive.html

http://dev.mysql.com/doc/refman/5.7/en/performance-schema-replication-tables.html

http://shivjijha.blogspot.in/2013/09/Introducing-the-performance-schema-tables-to-monitor-MySQL-replication.html


Questions!Email: [email protected] Blog: shivjijha.blogspot.in

http://shivjijha.blogspot.in/

mysql user camp: multi-threaded slaves

Technology