mysql replication: pros and cons

MySQL Replication: Pros and Cons Achieve Higher Performance, Uptime, Reliability and Simplicity

for Real-World Use Cases.

Darpan Dinker @darpandinker VP of Engineering Schooner Information Technology

p2 | © 2011 Schooner Information Technology. All rights reserved. © 2012 Schooner Information Technology, p2 © 2012 Schooner Information Technology, p2

Agenda

•  Quick tour of MySQL asynchronous, semi-synchronous and synchronous replication schemes

•  20 real-world use cases: which replication mechanisms to choose and why

•  SchoonerSQL Q&A


•  MySQL Master-Master •  DRBD •  Resilience •  Point-in-time recovery (PITR) •  Delayed Slave •  Very large databases •  Automated replication failover •  Mixed hardware •  Virtualized or Cloud env. •  Elasticity requirements

•  High write rate •  Read scaling •  Use of Flash memory •  High Availability •  Server sprawl, inefficient HW util. •  Full and incremental backup •  WAN, multi-DC deployments •  Online schema updates •  Online maintenance & upgrades •  Minimize administration

20 Real-‐World Use Cases


•  Concepts that effect MySQL replication –  Parallelism –  Flow control – Consistency & Serializability

•  Quick tour of replication schemes for MySQL –  Asynchronous –  Semi-synchronous –  Synchronous

Part One


Parallelism, or the Lack of

•  Commodity hardware: 4-10 core processors –  MySQL Slave uses <1 core to apply replication –  Can MySQL Master sustain write transactions

using >1 core ?

•  Hard-disks have >5ms latency (random) –  Single threaded Slave cannot make >200

random reads/sec in foreground –  Can a MySQL Slave use more disk throughput ?

•  Flash memory offers >50k IOPS –  Add Flash memory to improve performance of a

single thread –  At what price/performance ?

p6 | © 2011 Schooner Information Technology. All rights reserved. © 2012 Schooner Information Technology, p6

Answer to Parallelism: SchoonerSQL

2-‐10X throughput on SchoonerSQL Slaves

1,693,476

5,326,066 5,080,958

1,782,231

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

Percona 5.5.20 Schooner ASYNC

Schooner SYNC Stock 5.5.21

commit/1800s

Slave commits on Sysbench

3,235,410 3,239,277

11,174,862

9,456,770

0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

Percona 5.5.20 Stock 5.5.21 Schooner SYNC Schooner ASYNC

Commit/ 3900 sec

Slave commits on DBT2

2200

1598

2190

700

0

500

1000

1500

2000

2500

Schooner 5.1.3 Async Percona 5.5.20 Async

Commits / sec

TPCC-‐mysql Throughput @ 5000W

Master

Slave


Flow Control


No flow control

•  Slave lags the Master –  Stale reads on Slave –  Failover onto Slave is tricky

–  Action: make Slave faster (add Flash, more memory)

–  Action: shard database without hitting H/W limits

–  Action: use SchoonerSQL parallel async replication & reduce slave-lag

With flow control

•  Slave is in lock-step with Master – Reads are near or fully

consistent –  Failover is low or no risk –  Slave may throttle Master

–  Action: use SchoonerSQL parallel sync replication to eliminate data-loss and slave-lag

–  Action: use similar H/W within a cluster

Flow Control in ReplicaSon


Serializability and Consistency

•  Serializability: a transaction schedule is serializable if its outcome (e.g., the resulting database state) is equal to the outcome of its transactions executed serially, i.e., sequentially without overlapping in time.

•  MySQL Slave is single threaded, i.e. executes and commits in a serial order, hence serializable.

•  SchoonerSQL Slave (async/semi-sync*/sync) – Checks for conflicts –  Executes in parallel – Commits transactions in the same order as the Master –  Strong consistency, 100% compatibility, Binlog in identical sequence


Things to Watch Out for in Parallel ReplicaSon ImplementaSons TransacSon Ordering (1/2): Master

T5 T3

T1 T4

T2

Commit order

T3 T1 T2

T1 T2 T3 T4 T5

(In InnoDB and binary log)

(TransacSons on Master)


Things to Watch Out for in Parallel ReplicaSon ImplementaSons TransacSon Ordering (2/2): Slave or 2nd Master

T1 T2 T3 T4 T5

Commit order

T3 T1 T2

T5 T4 T3 T2 T1 (In InnoDB and binary log)

(order in relay log or sync replicaSon buffers)


MySQL ReplicaSon Technology

1.  Asynchronous – Oldest, most popular and widely deployed

2.  Semi-synchronous –  Introduced in v5.5. – Objective: avoid many data-loss situations

3.  Synchronous (“Virtual Synchrony”, not 2PC) – Not available from Oracle/ MySQL –  SchoonerSQL: >1 year in production –  XtraDB Cluster: announced last week


#1 MySQL Asynchronous ReplicaSon

•  Loosely coupled master/slave relationship

•  Master does not wait for Slave •  Slave determines how much to read

and from which point in the binary log •  Slave can be arbitrarily behind master in

reading and applying changes

•  Read on slave can give old data •  No checksums in binary or relay log

stored on disk, data corruption possible •  Upon a Master’s failure

•  Slave may not have latest committed data resulting in data loss

•  Fail-over to a slave is stalled until all transactions in relay log have been committed – not instantaneous

Master mysqld Slave mysqld

Relay log

InnoDB Tx log

DB MySQL Bin log

InnoDB Tx log

DB

Last tx=100 Last tx=100 Last tx=70 Last tx=50

Log events pulled by Slave

Tx.Commit(101) Repl.apply(51)

tx=101 tx=101 tx=51 tx=51

Read version based on tx=50

Read Stale data on Slave No flow control CorrupPon Master failure = mess Data loss Single-‐threaded Slave


#2 MySQL Semi-‐synchronous ReplicaSon

•  Semi-coupled master/slave relationship •  On commit, Master waits for an ACK

from Slave •  Slave logs the transaction event in relay

log and ACKs (may not apply yet) •  Slave can be arbitrarily behind master in

applying changes

•  Read on slave can give old data •  No checksums in binary or relay log

stored on disk, data corruption possible •  Upon a Master’s failure

•  Fail-over to a slave is stalled until all transactions in relay log have been committed – not instantaneous


Relay log

InnoDB Tx log

DB MySQL Bin log

InnoDB Tx log

DB

Last tx=100 Last tx=100 Last tx=100 Last tx=50

Log for tx=100 pulled by Slave

Tx.Commit(101) Repl.apply(51)

tx=101 tx=101 tx=51 tx=51


Slave ACK for tx=100

Read Stale data on Slave No flow control CorrupPon Master failure = mess Data loss Single threaded Slave


#3 SchoonerSQL Synchronous ReplicaSon

•  Tightly-coupled master/slave relationship

•  After commit, all Slaves guaranteed to receive and commit the change

•  Slave in lock-step with Master

•  Read on slave gives latest committed data

•  Checksums in replication paths •  Upon a Master’s failure

•  Fail-over to a slave is fully integrated and automatic

•  Application writes continue on new master instantaneously

•  No data loss


InnoDB Tx log

DB MySQL Bin log

InnoDB Tx log

DB

Last tx=100 Last tx=100 Last tx=100

Log for tx=100 pushed to Slave

Tx.Commit(101)

tx=101 tx=100


Slave ACK for tx=100 Repl.apply(100)

Read Stale data on Slave No flow control CorrupPon Master failure = mess Data loss Single threaded Slave


QualitaSve Comparison

MySQL 5.5 Asynchronous

SchoonerSQL Asynchronous

MySQL 5.5 Semi-‐

Synchronous

SchoonerSQL Semi-‐

Synchronous*

XtraDB Cluster Synchronous

SchoonerSQL Synchronous

Parallel replicaSon (for same schema) N Y N Y Y Y

Throughput Medium High Medium High Medium High

High network latency tolerant Y Y N N N N

MulS-‐level failure detecSon N Y N Y N Y

Global transacPon IDs N Y N Y N Y

Capacity with transient failures Y Y Y Y N Y

Commit consistency Y Y Y Y N Y

Auto recovery with high consistency Medium High Medium High Medium High

Auto replicaSon failover and repair N Y N Y Y Y

Large update/ insert/ LOAD INFILE Y Y Y Y N Y

Unexpected aborts & deadlocks N N N N Y N

MulS-‐Master Cluster N N N N Y N

* Future release


20 real-world use cases: which replication mechanisms to choose and why?

Part Two


#1 High Write Rate

•  Async/ semi-sync –  Issue: Slave lags the Master –  Issue: Master hits limits (code + H/W)

•  Typical solutions – Use more main-memory and/or Flash memory –  Shard database

•  SchoonerSQL –  SchoonerSQL is optimized for multi-cores, Flash memory and fast

network – Master scales vertically with H/W resource (CPU, memory, storage). –  Async and sync Slave have parallel threads for replication that match

the speed of the Master.


#2 Read Scaling

0

50000

100000

150000

200000

250000

300000

1 2 4 8 Nodes in Schooner Cluster

Sysbench Read/Write

Reads/s

Writes/s

•  Async/ semi-sync/ sync –  Issue: Scale read queries for a read

intensive application

•  Typical solution – MySQL replication allows unlimited

number of Slaves

•  SchoonerSQL –  Schooner sync supports consistent reads from up to 7 Slaves –  Schooner async allows same unlimited number of Slaves


#3 Use of Flash Memory

•  Async/ semi-sync/ sync –  Issue: MySQL/InnoDB is IO latency sensitive, adding dozens of hard disks

does not scale well –  Issue: Slave is unable to keep-up with Master –  Issue: InnoDB flushing & check-pointing has unstable performance

•  Typical solutions –  Use Flash to provide more IOPS to single threaded Slave –  Use Flash for faster random access to read and write database files

•  SchoonerSQL –  SchoonerSQL is designed for fast storage (such as Flash memory) to reduce

writes, increase flushing and checkpoint efficiencies, executing more read and write IOs in parallel.

–  Slave parallelism and vertical scalability helps avoid sharding and provides factors of better price/performance, order of magnitude when compared with disks


#4 High Availability (HA)

•  Async/ semi-sync –  Issue: MySQL provides weak out-of-the-box availability –  Issue: Replication and application connections are not failed over

•  Sync –  Issue: Requires failover of application connections

•  Typical solutions – Use a proxy – Use Virtual IPs (MMM)

•  SchoonerSQL for mission-critical applications –  Includes integrated VIP management and automatic failover – Global transaction IDs in binary log allow sync, semi-sync* and async

replication connections to be automatically repaired/ redirected


#4 High Availability (HA): SchoonerSQL

Base

Sync re

plicaS

on (LAN)

Auto client con

necSon

failover

Sync nod

e failures (up to

7)

Auto sync con

necSon

repair

Fault tolerance (sync -‐>

async -‐> sync)

Async W

AN con

necSon

repair

WAN level failover

Two Se

r failure detecSo

n

Self he

aling

Availabilit

y


#5 Server Sprawl, Inefficient Hardware USlizaSon

•  Async/ semi-sync/ sync –  Issue: Software bottlenecks force premature sharding

Issue: Servicing reads or taking backup on Slave conflicts with single threaded Slave

–  Issue: DRBD stand-by servers

•  Typical solutions – Use a Slave solely for backups –  Add Slave servers

•  SchoonerSQL –  Provides high consolidation via high

vertical scalability and 2-10X faster Slave replication

1,693,476

5,326,066 5,080,958

1,782,231

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

Percona 5.5.20 Schooner ASYNC

Schooner SYNC Stock 5.5.21

commit/1800s

Slave commits on Sysbench


#5 Server Sprawl, Inefficient Hardware USlizaSon SchoonerSQL VerScal Scalability

0

200

400

600

800

1000

1200

1400

1600

1 4 16 64

UPlizaP

on (C

PU th

read

s)

Total concurrent connecPons

CPU Scalability

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

1 4 16 64

IOPS


IO Scalability

Read IOPS

Write IOPS

0

5

10

15

20

25

30

1 4 16 64

Band

width (M

Bps)


Network Scalability

Network in

Network out

SchoonerSQL scales on mul9-‐cores & Flash

Benchmark: 5000 warehouse TPCC-‐MySQL


#6 Full and Incremental Backup

•  Async/ semi-sync –  Issue: Slave instance may lag due to backup

•  Sync –  Issue: Flow control may be triggered slowing down the commits

•  Typical solutions – Reserve a slave solely for backup (non-sync mode) –  Schedule or execute backup during periods with low write activity –  Throttle down backup speed (increasing the time taken to backup) –  Separate storage device and controller for backup target

•  SchoonerSQL –  Slave parallelism reduces or eliminates these issues –  Schooner backup includes adaptive throttling to reduce contention


#7 WAN and MulS-‐Data Center Deployments

•  Semi-sync/ Sync –  Issue: Increases in network latency slow down commits –  Issue: Failure detection over higher latency networks is slow

•  Typical solutions – Use semi-sync or async within DC (or with metro-area networks) – Use async for high latency networks (WAN)

•  SchoonerSQL – Use sync and/or async within DC (or with metro-area networks) – Use async for WAN (automatic failover supported) – High throughput maintained across WAN, as permitted by network

bandwidth & quality


MASTER

Read Master

READS

REPL Read Master

Read Master

REPL

READS

READS READS

Data Center #1

WRITES

WRITES

REPL

Synchronous Cluster

MASTER

Read Master

READS

REPL Read Master

Read Master

REPL

REPL

READS

READS

READS

Data Center #2

Synchronous Cluster

Parallel Async

#7 WAN and MulS-‐Data Center Deployments: SchoonerSQL


#7 WAN and MulS-‐Data Center Deployments: SchoonerSQL

SchoonerSQL asynchronous replication performance


#8 Online Schema Updates

•  Async/ semi-sync/ sync –  Issue: Certain schema changes take minutes-hours and lock table for

the duration (blocking write transactions that write to this table) –  Issue: Adding index has similar effects

•  Sync –  Issue: Requires extra machinery if async is not supported

•  Typical solutions – Use Flash memory for small-medium tables –  Leverage one of several solutions (open-ark, pt-online-schema..)

•  SchoonerSQL –  Supports asynchronous and synchronous for an instance. –  Fully compatible with existing mechanisms to update schema.


#9 Online Maintenance and Upgrades

•  Async/ semi-sync/ sync –  Issue: Application failover is required before taking a MySQL instance

offline •  Sync

–  Issue: To service load during maintenance, a temporary server may need to be used and require sufficient manual work

•  Typical solutions – Use scripts, manual steps

•  SchoonerSQL – GUI and CLI includes features to migrate an instance to another

server and moving application connections. SchoonerSQL supports automated full and incremental database provisioning and recovery.

–  Automation reduces errors.


#10 Minimize AdministraSon

•  Async/ semi-sync/ sync –  Issue: MySQL requires considerable configuration and administration

for replication, failure-detection, failover, backup/restore

•  Typical solutions –  Add MySQL experienced members on the dev/ops team

•  SchoonerSQL –  Auto provisioning of a Peer/ Slave –  Mark a sync instance as Master –  Auto-sync after an instance is restarted –  Schedule full & incremental backups –  Alerts for notification –  State and progress of startup and shutdown


#10 Minimize AdministraSon SchoonerSQL: GUI for Management

•  Integrated GUI simplifies administration –  Powerful 1-click actions for

•  On-line provisioning of servers and MySQL instances •  On-line database migration, upgrades •  Master role is set on an instance with a click •  Create / Start / Stop / Remove MySQL instances

•  Easy Server Management –  Assign VIPs to Masters, Read Masters and

Asynchronous Masters and Slaves –  Create Synchronous and Asynchronous groups or

clusters –  Change the configuration parameters through GUI


#10 Minimize AdministraSon SchoonerSQL: Monitoring & Backup Schedules

•  Monitoring and optimization: •  Extensive displays with resource utilization statistics

•  Physical (cores, storage, network) and logical (buffers, locks,…)

•  CPU, Disk, RAM usage •  IO Read and Write kb/s •  Bytes In & Out kb/s

•  Integrated full & incremental online backup –  Scheduled backup ( daily, monthly) –  Supports full database restore –  Adaptive backup that minimizes effect on database


#10 Minimize AdministraSon SchoonerSQL: Alerts

•  SchoonerSQL provides email based alerts that are very useful in monitoring the database environment. Alerts include name, date & time, severity, description and have configurable thresholds.

•  Sample alerts

–  Instance created/deleted –  Instance up/down –  Instance attached/detached – Group created/removed –  Async failover –  VIP configuration


#11 MySQL Master-‐Master Setup (AcSve/Passive)

•  Async/ semi-sync –  Issue: Require quick failover upon active Master’s failure

•  Async –  Issue: May loose transactions upon active Master’s failure

•  Typical solutions – Configure two MySQL instances as Master-Master (setup to replicate

to and from each other) – Use external failure-detection and load (re-) direction mechanisms

(e.g. MMM, Flipper, HAProxy)

•  SchoonerSQL –  SchoonerSQL sync cluster setup is similar to an active-passive

Master-Master setup. Automatic failure detection and application failover is integrated in the solution.


#11 MySQL Master-‐Master Setup (AcSve/Passive)

MASTER

Read Master

READS

REPL Read Master

Read Master

REPL

READS

READS READS

SchoonerSQL

WRITES

WRITES

REPL

Synchronous Cluster

MASTER

Read Master

READS

Read Master

Read Master

REPL

READS

READS READS

SchoonerSQL

WRITES

WRITES REPL

Synchronous Cluster

Instantaneous Failover


#12 MySQL with DRBD Setup

•  Async –  Issue: Possibility of data loss when active Master fails & immediate

failover is required –  Issue: Reconnecting Slaves to new Master is not straightforward

•  Typical solutions – Use semi-sync replication in v5.5 – Use DRBD to replicate at physical storage block device level

•  SchoonerSQL –  Instead of wasting resources on a stand-by and long failover time

(recovery and warm-up), SchoonerSQL sync cluster provides logical replication (avoiding corruption propagation in DRBD) with no data loss, instantaneous failover and automatic failover of replication connections (sync or async).


#13 Resilience

•  Sync –  Issue: Sensitive to N/W quality, splits cluster at any fault –  Issue: Read capacity compromised

•  Typical solutions –  Semi-sync falls back to async mode –  Async Slaves disconnect without affecting the Master, and reconnect

at a later time

•  SchoonerSQL –  Sync falls back to async* without affecting the Master or

compromising the read capacity

* As of SchoonerSQL 5.1.5


#14 Point In Time Recovery (PITR)

•  Sync –  Issue: Binary logs within a cluster may not contain transactions in the

same sequence. PITR becomes difficult if the same instance that created backup files is unavailable.

•  Typical solutions –  Backup database and binary log for all or majority of cluster

instances (increases disk space requirements and may cause slow down during backup)

•  SchoonerSQL –  SchoonerSQL Slave instances using any replication type commit in

the same order as Master, even when parallelizing writes. Global transaction IDs help stitch state of any cluster member. Backup from one instance of a cluster is sufficient.


#15 Delayed Slave

•  Sync –  Issue: Instances in the cluster are in lock-step with Master and do not

read binary log (as an async Slave does)

•  SchoonerSQL –  SchoonerSQL supports multiple replication types in the same

database cluster. An async Slave instance can be used for this purpose.


#16 Very Large Databases

•  Async/ semi-sync –  Issue: Working set may be larger than

main-memory -> slow Slave replication –  Issue: Large databases take longer

to backup

•  Typical solution –  Throttle backup and control database size –  Use snapshots for backup

•  SchoonerSQL –  Adaptive backup helps with file copy based backup –  Parallel Slave replication helps to hide high rate of reads –  Gains seen hard disks as well as Flash

2200

1598

2190

700

0

500

1000

1500

2000

2500

Schooner 5.1.3 Async

Percona 5.5.20 Async

Commits / sec

TPCC-‐mysql Sample @ 0.5 TB

Master

Slave


#17 Automated ReplicaSon Failover

•  Async/ semi-sync –  Issue: MySQL does not include failover of replication

•  Typical solutions – Use custom external tool-kits – Manual work when instances fail

•  SchoonerSQL –  Integrated support for failover of all replication types


#17 Automated ReplicaSon Failover (with Sync replicaSon)

MASTER

Read Master

READS

REPL Read Master

Read Master

REPL

READS

READS READS

SchoonerSQL

WRITES

WRITES

REPL

Synchronous Cluster

MASTER

Read Master

READS

Read Master

Read Master

REPL

READS

READS READS

SchoonerSQL

WRITES

WRITES REPL

Synchronous Cluster

Instantaneous Failover


#17 Automated ReplicaSon Failover (with Async replicaSon) – 1/2

MASTER

Read Master

READS

REPL Read Master

Read Master

REPL

READS

READS READS

Data Center #1

WRITES

WRITES

REPL

Synchronous Cluster

MASTER

Read Master

READS

REPL

Read Master

REPL

READS READS READS

Data Center #2

Synchronous Cluster

Async


#17 Automated ReplicaSon Failover (with Async replicaSon) – 2/2

Async

MASTER

Read Master

READS

Read Master

Read Master

REPL

READS

READS READS

Data Center #1

WRITES

WRITES REPL

Synchronous Cluster

MASTER

Read Master

READS

REPL

Read Master

REPL

READS READS READS

Data Center #2

Synchronous Cluster

ReplicaPon Failover


#18 Mixed Hardware

•  Async/ semi-sync –  Issue: Smaller hardware for Slave causes it to lag

•  Sync –  Issue: Cluster executes commits at the pace of the weakest H/W

•  Typical solutions – Use better (or identical) hardware for Slave servers – Use Flash memory in Slave servers (alleviate single threaded Slave)

•  SchoonerSQL –  SchoonerSQL sync commits at the pace of weakest H/W –  SchoonerSQL async with parallel replication has higher chances of

staying close with Master’s commit speed


#19 Virtualized or Cloud Environments

•  Async/ semi-sync/ sync –  Issue: IO latency and throughput is typically poor

•  Sync –  Issue: Frequent failure detections stemming from resource starvation

•  Typical solutions – Working-set or database is maintained in main-memory (avoid read

IOs) – Databases are heavily sharded (to reduce read and write IOs)

•  SchoonerSQL –  Support sync only on well provisioned servers, typically with elevated

failure detection timeouts. –  Async provides speedup (parallel threads hide longer latencies)


#20 ElasScity Requirements

•  Async/ semi-sync/ sync –  Issue: Dynamically add/remove Slaves based on load or trends

•  Sync –  Issue: Disables an instance while a new instances is provisioned

•  Typical solutions – Manual work, scripts, proxies

•  SchoonerSQL –  Sync instance is online while provisioning another instance –  A click in the GUI provisions a Slave that starts servicing read load. A

click removes an instance from the cluster – Dynamically add virtual IPs that load balance between instances


What is SchoonerSQL ?

Part Three

•  SchoonerSQL is a complete distribution of MySQL+InnoDB –  mysqld smells & feels the same –  100% client compatible, uses same

database files

•  Installer •  Cluster manager

–  GUI, CLI –  Administer and monitor

•  Backup & schedules •  System and MySQL metric

collection & display •  Utility to capture an incident

(optionally call-home)

•  High performance & high availability

•  Minimizes replication administration –  Automatic app failover –  Automatic repl connection repair –  Single click provisioning

•  Online upgrades •  Online maintenance •  Elastic (user-driven) cluster •  Alerts


SchoonerSQL Benefits Summary

•  4-‐20x more throughput/server vs. MySQL 5.5 • Highest performance synchronous and asynchronous replicaSon clusters

High Performance and Scalability

•  No service interrupSon for planned or unplanned database downSme • Instant automaSc fail-‐over •  On-‐line upgrade and migraSon •  90% less downSme vs. MySQL 5.5 •  WAN/DR auto-‐failover

High Availability

Ease of AdministraPon

•  No error-‐prone manual processes •  Monitoring and OpSmizaSon

•  No lost data •  Highest data consistency

High Data Integrity

•  Full MySQL + InnoDB: not a toolkit •  Free your staff to build your business, not a custom database • TCO 50% cheaper than MySQL 5.5

Compelling Economics

Out-‐of-‐the-‐box Product

100% MySQL Enterprise InnoDB CompaPble

Broad Industry Deployment •  eCommerce, Social Media, Telco, Financial

Services, EducaSon •  High volume web sites •  Geographically distributed websites

Thank You!

www.schoonersql.com

@schoonerinfo