a tour of mysql high availability

8/14/2019 A Tour of MySQL High Availability

1/87

A tour of MySQL

High AvailabilityPHP Quebec, 2009

Morgan Tocker, firstname at percona dot com

Consultant, Percona Inc.


2/87

Introduction

Were going to be talking about varioustechnologies and topologies we can use to improveavailability.

High Availability != High Performance Often we want both, but beware some solutions come

without the other!

Yes, at the end well cover backups!


3/87

Audience Poll!

Who needs 100% availability?


4/87

Trick Question.

Sorry, it doesnt exist.


5/87

Second question...

Lets just say in a magical worldit did exist...


6/87

Yep, another trick question.

You cant afford it.


7/87

Its Expensive.

HA is like the Noahs Ark of computing.

You need at least two of everything.

Anything less less than two and you have a single point

of failure (SPOF).


8/87

Its also a sliding scale

You?


9/87

How do you measure availability?

Availability is measured in nines;

Two, three, four, five and six nines.

Is it the time your application is running minus

unpredicted outages? Does scheduled maintenance count?

How about network downtime?


10/87

Comparing you options

Availability % Downtime per year Downtime per month*

90% 36.5 days 72 hours

95% 18.25 days 36 hours

98% 7.30 days 14.4 hours

99% 3.65 days 7.20 hours

99.9% 8.76 hours 43.2 min

99.99% 52.6 min 4.32 min99.999% 5.26 min 25.9s

99.9999% 31.5 s 2.59s

* For monthly calculations, a 30-day month is used.

Source: http://en.wikipedia.org/wiki/High_availability
http://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nines


11/87

How do you do this?

You likely have a boss that demands 100%availability. Part of Implementing HA is a social problem, heres how

to deal with this....


12/87

1. Identify all failure scenarios

Example Problem:RAID controller thinks its April fools day and startswriting 1s instead of 0s and 0s instead of 1s.

The Master Database server is down!


13/87

2. Estimate the recovery...

02:05 AM - Crash occurs.

02:12 AM - Someone notices a problem with the website - callsthe Sales Manager, no answer.

02:17 AM - Sales Manager answers phone, figures he cant fixthe problem.

02:20 AM - Sales Manager calls you, you log in to diagnosethe problem.

02:26 AM - You find out you cant log into the system remotely

and have to lodge a ticket with your hosting company. .....


14/87


15/87

Estimate (cont...)

Fast forward...

Youre lucky you can get back online within the day. Youprobably have to abandon that server and restore the

backup on a new one.

And these estimates are me at my most optimistic!


16/87

3. Pass the responsibility on

Now put this in a report in size 12 font, double linespacing and give it to your manager:

Failure Scenariosi. Database Server FailureCurrent ETA is 16 hours for full recovery from backup (explain here ...)

Potential Improvement Associated Costs New Recovery ETA

Use Servers under Warranty + $3000 8 hours

Rent Servers + $50/month 4 hours

Have Hot Spare + $300/month 10 minutes


17/87

Some additional notes

The story I gave is very common.

There are a few warning lights that come up straightaway, and theyre not expensive things to fix.

Pingdom.com is $10/month, can do quite advancedchecks, and will Send you SMS.


18/87

Having said allthat...

Lets get back to technical challenges.

What should you care about with regards toavailability?


19/87

Simple Answer:

Availability is what your customerthinks it is.

If you can still provide them their requiredfunctionality - then youre still running.

Even if youve just had a critical database server crash!


20/87

Example


21/87

Example (cont.)

..

/*Checkifwehaveadatabase linkbeforerenderingoptionalfunctionality.

*/

if(!degrade_gracefully()){ show_add_comment_box();}else

{ print"Commentsisundermaintenance,checkbacksoon!";

}

?>


22/87

A simple example, but..

There are probably many cases like this in yourapplication.


23/87

High Availability Tools

mysql-master-master


24/87

MySQL Replication

Asynchronous

Ill explain that in a second.

Statement-Based (until 5.1 at least)

Spreads read load. Does not spread writes.


25/87

How replication works...


26/87



27/87



28/87



29/87



30/87

DRBD

A Linux Kernel Module

Think of it as Network Level RAID1.

Not just for MySQL!

Can be used to mirror other files. Doesnt spread reads.

Doesnt spread writes.

Can be Synchronous.


31/87

How DRBD works


32/87

How DRBD works


33/87

How DRBD works


34/87

How DRBD works


35/87

How DRBD works


36/87


37/87

How DRBD works


38/87

How DRBD works


39/87

How DRBD works


40/87

Comparing Replication to DRBD

Feature Replication DRBD

Fast Resync on inconsistency No Yes

Online Resync Yes if InnoDB Yes

Syncronous No Yes

Network Traffic Low to Medium High

Corruption on primary spreads Unlikely Certain

Secondary can be time delayed from primary Yes No

Secondary can be utilized Yes (reads) Yes (backups)

OS Support All Linux


41/87

Comparing (cont.)

Feature Replication DRBD

Consistency Check Not Built in Yes

Failover Capability Fairly Easy Easy

Multiple Spares Easy No

Remote Spares Yes Unsuitable

myslqdump backup from spare Yes No*

Secondary hardware requirements Either faster/slower than master At least as fast as master

Allows replicating partial dataset Yes No

Replication Event Checksums No Yes


42/87

Heartbeat

A Linux cluster manager.

Can monitor just about anything

Including DRBD and MySQL Replication.

Common setup is to use shared IP addressbetween two nodes.

Can script whatever you like happens on node failure.


43/87

Heartbeat

192.168.1.3

192.168.1.1 192.168.1.2

Webservers

Database Servers

#1#2


44/87

Heartbeat

192.168.1.3

192.168.1.1 192.168.1.2

Webservers

Database Servers

#1#2


45/87


46/87

Heartbeat

192.168.1.3

192.168.1.1 192.168.1.2

Webservers

Database Servers


47/87

Heartbeat

192.168.1.3

192.168.1.1 192.168.1.2

Webservers

Database Servers

#1


48/87

MMM

Similar concept to Heartbeat with a floatingapplication facing IP address. But more MySQL Aware than heartbeat.

Under active development by Percona.


49/87

Memcached

Dead simple front-end caching.

Access data from cache, alleviate database loadserver.


50/87

Cacti

A graphing tool.

Monitoring health important:


51/87

Cacti (cont.)


52/87

When we combine our powers together, we havetopologies.....


53/87

Starting small...

Web Servers

Database Server Database Slave


54/87

A little larger...

Web Servers



55/87

May work

Web Servers



56/87

Active-Passive Master-Master

Web Servers



57/87

Larger again

Web Servers

Front EndCaching (Memcached) Database Slave

Database Server


58/87

Clusters arent all fun and games

Split Brains A situation where both nodes think that they are

the primary.

ImPrimary

ImPrimary


59/87

Another issue...

Fencing and STONITHs

Force a take down of a failed node.

I wontdie.

I kill you!


60/87

Backups

No HA talk would be complete without plugging theimportance of backups!


61/87

What makes a good backup?

What are you insulating against? Require protection from failures as well as accidental/

Malicious deletes.

Quick to Backup

Quick to Recovery

Flexible recovery granularity Can recover just one row, or an entire table.

Consistent


62/87

An inconsistent backup

user_friendship

user_1 user_2

1 2

2 1

Users

id name email

1 Fred [email protected]

2 Mary [email protected]

user_blog_posts

id user_id post_contents

1 1 I hate my parents.2 2 Ponnies! lol. :D
mailto:[email protected]:[email protected]:[email protected]:[email protected]


63/87


user_friendship

user_1 user_2

1 2

2 1

Users

id name email



user_blog_posts


1 1 I hate my parents.2 2 Ponnies! lol. :D


64/87


user_friendship

user_1 user_2

1 2

2 1

Users

id name email



user_blog_posts


1 1 I hate my parents.

2 2 Ponnies! lol. :D


65/87


user_friendship

user_1 user_2

1 2

2 1

Users

id name email



Users

id name email



3 Barry [email protected]

user_blog_posts



mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]


66/87

A i i b k


67/87


user_friendship

user_1 user_2

1 2

2 1

3 2

2 3

user_friendship

user_1 user_2

1 2

2 1

Users

id name email




user_blog_posts




A i i t t b k


68/87


user_friendship

user_1 user_2

1 2

2 1

3 2

2 3

Users

id name email




user_blog_posts




A i i t t b k


69/87


user_friendship

user_1 user_2

1 2

2 1

3 2

2 3

Users

id name email




user_blog_posts




A i i t t b k


70/87


user_friendship

user_1 user_2

1 2

2 1

3 2

2 3

Users

id name email




user_blog_posts




Wh t ( t )


71/87

Users

id name email



3 Jim [email protected]

Whats wrong (cont.)

user_friendship

user_1 user_2

1 2

2 1

3 2

2 3user_blog_posts




A i t t b k


72/87

A consistent backup

user_friendship

user_1 user_2

1 2

2 1

Users

id name email



user_blog_posts




A i t t b k


73/87

A consistent backup

user_friendship

user_1 user_2

1 2

2 1

Users

id name email



user_blog_posts




A i t t b k


74/87

A consistent backup

user_friendship

user_1 user_2

1 2

2 1

Users

id name email



user_blog_posts




A i t t b k


75/87

A consistent backup

user_friendship

user_1 user_2

1 2

2 1

Users

id name email



user_blog_posts




I DB ith MVCC


76/87

InnoDB with MVCC

Users

id name email





I DB ith MVCC
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]


77/87

InnoDB with MVCC

Users

id name email





I DB ith MVCC


78/87

InnoDB with MVCC

Users

id name email





InnoDB ith MVCC


79/87

InnoDB with MVCC

Users

id name email





InnoDB with MVCC


80/87

InnoDB with MVCC

Users

id name email





3 Barry

[email protected]

InnoDB with MVCC
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]


81/87

InnoDB with MVCC

Users

id name email





3 Barry

[email protected]

MySQL Backup Overview
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]


82/87

MySQL Backup Overview

mysqldump Text based backups

Easier to recover individual rows

Raw Backup Copy the data files

Filesystem snapshot

InnoDB Hot Backup (commercial)

Backup from a Slave

Comparisons


83/87

Comparisons

mysqldump Raw backups SnapshotsInnoDB HotBackup*

BlockingNo (if allInnoDB)

Yes PartiallyNo (if allInnoDB)

Backup Speed Medium Fast Fast Fast

Recovery

Speed

Slow Fast Medium Fast

RecoveryGranularity

Best Depends Not Good Good

Point in Time Recovery


84/87

Point in Time Recovery

Really slick, just enable binary logging. An example backup:

mysqldump --flush-logs --all-databases--master-data=2 > mybackup.sql

Recovery:

mysql < mybackup.sqlmysqlbinlog mylog.000005 | mysql

Need more info?


85/87

Need more info?

We have a blog on everything aboutMySQL Performance:

www.mysqlperformanceblog.com

Pictures used with CC Attribution
http://www.mysqlperformanceblog.com/http://www.mysqlperformanceblog.com/


86/87

Pictures used with CC Attribution

Hot Rod:http://www.flickr.com/photos/januszbc/1353696160/

Hyundai:http://www.flickr.com/photos/daveseven/1538984615/

The End
http://www.flickr.com/photos/daveseven/1538984615/http://www.flickr.com/photos/daveseven/1538984615/http://www.flickr.com/photos/daveseven/1538984615/http://www.flickr.com/photos/daveseven/1538984615/http://www.flickr.com/photos/januszbc/1353696160/http://www.flickr.com/photos/januszbc/1353696160/


87/87

The End

Questions?

a tour of mysql high availability

Documents