scaling your web app with mysql replication

Scaling your web app with MySQL replication

Giuseppe MaxiaMySQL Community Team Lead

1This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.

1Thursday, 30 September 2010

2

Table of contents

Why replication

How to set replication

Using replication

Leveraging replication

From single server to

replication

Adding a slave

Replacing a slave

Replacing the master

Managing replication


3

Table of contents

Why replication


Using replication



replication

Adding a slave

Replacing a slave




a simple web application

scheme

4

database server

web server

clients

r/w requests


database server

web servers

load balancer

clients

r/w requests

scaling web requests


writeread

database load on a simple

web application

6

85% 15%


readwrite

database load on a successful web

application7

20% 80%


8

database server

web servers

load balancer

clients

r/w requests✘

scaling up means buying

a bigger database

server


9

readwrite

the bigger database server will eventually

have the same problem

80%20%


read/writemaster

read/onlyslaves

web servers

R/W

R/O

load balancer

load balancer

clients

a web application

scheme with replication


read/writemaster

read/onlyslaves

readwriteread

database load with

replication11

85% 15% 100%


read/writemaster

read/onlyslaves

readwriteread

scaling database load

with replication

12

85% 15% 100%


Replication assessment

13

without replication with replication

database handling

performance

Point in Time recovery

failover

write scaling

backup

read scaling

easy harder

high lower (binary logs)

none easy

none possible

none minimal

with downtime without downtime

none easy


14

Table of contents

Why replication


Using replication



replication

Adding a slave

Replacing a slave




master

slaveIO thread

SQL thread

reads

reads

client

transaction

binary log

relay log replication concepts


16

Table of contents

Why replication


Using replication



replication

Adding a slave

Replacing a slave




17

1 SHUT DOWN THE DATABASE SERVER

Master


18

2 MAKE A BACKUP COPY

Master


19

3 ENABLE THE MASTER

Configuration file[mysqld]log-bin=mysql-binserver-id=1

Master


20

4 RESTART THE MASTER

Master


21

5 CREATE REPLICATION USER

SQL commandGRANT REPLICATION SLAVE ON *.* to 'slave_user@'10.10.100.%' IDENTIFIED BY 'slave_pass';

Master


22

6 INSTALL MySQL on the slave

Slave 1

Make sure that:• You're using the same version of MySQL • You have the same directory structure• The server is not started yet


23

7 COPY THE MASTER DATA to the slave

Slave 1


24

8 ENABLE THE SLAVE

Configuration file[mysqld]server-id=2relay-log=mysql-relayread-only# optional:log-bin=mysql-bin

Slave 1


25

9 START THE SLAVE SERVER

Slave 1


26

10 INITIALIZE THE SLAVE

SQL commandSET MASTER TOMASTER_HOST=master_IP,MASTER_PORT=3306,MASTER_USER=slave_user,MASTER_PASSWORD='slave_pwd';

Slave 1


27

11 START THE SLAVE SERVICE

SQL commandSTART SLAVE;

Slave 1


28

12 CHECK THE SLAVE

SQL commandSHOW SLAVE STATUS \G... Slave_IO_Running: YesSlave_SQL_Running: Yes...

Slave 1


Troubleshooting

• SHOW SLAVE STATUS says SLAVE_IO_RUNNING=No

• Make sure that the slave host can connect to the master

• Make sure that master and slave have different Server-id

• Check the error log of both master and slave


Testing the slave

• Create a table in the master.

• Make sure that the slave has replicated the table.

• Insert data in the master

• read that data in the slave


31

Table of contents

Why replication


Using replication



replication

Adding a slave

Replacing a slave




32

1 NO NEED TO STOP THE MASTER!

Master


33

2 STOP THE SLAVE

SQL commandSTOP SLAVE IO_THREAD;# wait until the SQL_THREAD# has done everythingSTOP SLAVE SQL_THREAD;# STOP THE SERVER

Slave 1


34

3 MAKE A COPY OF THE DATA DIRECTORY

Slave 1


35

4 RESTART THE SLAVE

Slave 1


36

5 INSTALL MySQL on the new slave

Slave 2

Make sure that:• You're using the same version of MySQL • You have the same directory structure• The server is not started yet


37

6 COPY THE old slave DATA on the slave

Slave 2


38

7 ENABLE THE NEW SLAVE

Configuration file[mysqld]server-id=3relay-log=mysql-relayread-only # optional:log-bin=mysql-bin

Slave 2

must be unique!


39

8 START THE NEW SLAVE

Slave 2


40

9 CHECK THE SLAVE

SQL commandSHOW SLAVE STATUS \G... Slave_IO_Running: YesSlave_SQL_Running: Yes...

Slave 2


Why it works

• No need to issue a CHANGE MASTER TO command.

• Because we cloned the old slave

• The new slave gets its parameters from the .info files in the data directory


42

Table of contents

Why replication


Using replication



replication

Adding a slave

Replacing a slave




From single server application

43

r/w requests


To replication-aware application

44

read/writemaster

read/onlyslaves

R/WR/O

load balancer


Single server application$link = mysql_connect(

$server_IP, 'mysql_user', 'mysql_password');$result = mysql_query('INSERT INTO table_name (x) VALUES (1)',$link);$result = mysql_query('SELECT * FROM table_name WHERE x=1',$link);


Making an application aware of replication

46

connectreadwrite

IPuserpassword

<<R/W database handling>>

db

connect

IPuserpassword

<<database handling>>db

read

IPuserpassword

<<read-only database handling>>

db

connectreadwrite

IPuserpassword

<<R/W database handling>>

db


Using replication: the WRONG way

47

R/W split by

statement

Write statement?

Connect to the master

Yes

Write to the master

Stop

No

Connect to the next available

slave

Read from slave


Why statement split is wrong

• Breaks transactions

• High risk of inconsistency

• Loses or corrupts data


Using replication:the RIGHT way

49

R/W split by function

Write function?

Connect to the master

Yes

Read and write from master

Stop

No

Connect to the next available

slave

Read from slave

more queries?

No

Yesmore

queries?

No

Yes


50

Table of contents

Why replication


Using replication



replication

Adding a slave

Replacing a slave





•MONITORING• … or die


52

Sample monitoring

master

slave

Get master binlog and position

get slave status

Running?

Yes

No

alert

Same or later binlog/position?

check table contents

NoYes


53

monitoringcontents

GET table CRC

GET table CRC

GET table CRC

master slave

same?

Yes

No

alert

GET table CRC

Stop


54

Table of contents

Why replication


Using replication



replication

Adding a slave

Replacing a slave




55

Replacing a slave

Slave crashes

are there more slaves?

STOP the master STOP one

slave

YesNo

add the first slave

add another slave

Stop


56

Table of contents

Why replication


Using replication



replication

Adding a slave

Replacing a slave




57


Master crashes

Let all slaves catch up with

execution

FIND the most up to date slave

STOP replication in

all slaves

make it the master

Stop

FIND which transactions are

missing from other slaves

connect all slaves to the new master

run missing transactions

to other slaves


58

Table of contents

Why replication


Using replication



replication

Adding a slave

Replacing a slave




59

read/writemaster

read/onlyslaves

R/WR/O

load balancer

load balancing


60

backup master

slaves

STOP SLAVE

remove slave from load balancer

START SLAVE

perform backup

attach slave to load

balancer

Let slave catch up


master

slaves

STOP SLAVE

remove slave from load balancer

START SLAVE

calculate summary

tables

attach slave to load

balancer

Let slave catch up

61

makesummary

tables


62

master

slave

innodbnon partitioned

slave

innodbnon partitionedinnodb

partitioned by range

slave

MyISAMpartitioned by range

Partitionsfor heavy statistics


63

master

slave


slave


ARCHIVEpartitioned by range

(date)

slave


(product)

slave


(location)

Simulating multiple

dimensions


Semi-synchronous replication

• Available in 5.5 and higher

• Makes sure that at least one slave has copied the data.

• Increases reliability


65

transaction with regular replication

clientmaster

slave

commit

binary log

execute

returns to client

replication


66

transaction with semi-

synchronous replication

clientmaster

slave

commit

binary log

execute

returns to client

sends transaction

to slave

gets acknowledgement

relay log


scaling your web app with mysql replication

Technology

slave slave

slave service slave

master leveraging replication

master data

slave host

slave serverslave

everythingstop slave

sql command set master