![Page 1: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/1.jpg)
©Continuent 2012.
Multi-Master, Multi-Site MySQL Databases Made Easy
with Continuent TungstenRobert Hodges, CEO
Giuseppe Maxia, Director of QA
![Page 2: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/2.jpg)
©Continuent 2012
Introducing Continuent
2
• The leading provider of clustering and replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance scaling and data management for MySQL
• Replication - Flexible, high-performance data movement
![Page 3: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/3.jpg)
©Continuent 2012
Quick Continuent Facts
• Largest Tungsten clustering installation processes about 500M transactions each day
• Tungsten Replicator was application of the year at the 2011 MySQL User Conference
• Continuent is well known in the MySQL community for its clustering and replication
• Over 90 customers currently deployed
3
![Page 4: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/4.jpg)
©Continuent 2012
Select Continuent Customers
4
![Page 5: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/5.jpg)
©Continuent 2012 5
Why Tungsten?
![Page 6: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/6.jpg)
©Continuent 2012
MySQL is a Great OLTP Database
6
But how do I...
• Handle DBMS failures?
• Perform maintenance and upgrades without taking applications o!ine?
• Load balance SQL across replicas?
• Create a disaster recovery site?
• Set up multi-master replication?
• Integrate with another DBMS?
![Page 7: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/7.jpg)
©Continuent 2012 7
Tungsten Master/Slave Clustering
![Page 8: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/8.jpg)
©Continuent 2012
Introducing Tungsten Clustering
8
Tungsten clusters combine o"-the-shelf open source DBMS servers into data services with:
• 24x7 data access • Scaling of load on replicas• Simple management commands
...without app changes or data migration
AmazonUS West
apache/php
GonzoPortal.com
Connector Connector
![Page 9: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/9.jpg)
©Continuent 2012
Doesn’t Replication Do This Already?
9
Application
Libmysqlclient.a
Application
Libmysqlclient.a
Replicator
Db2
Replicator
Db3Db1
Replicator
Slave SlaveMaster
![Page 10: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/10.jpg)
©Continuent 2012
Replication Limitations
Replication alone cannot enable constant access to data...
• What happens if the master fails?
• What happens if a slave is lagging too far behind to be usable?
• How do I maintain the DBMS without stopping apps?
10
![Page 11: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/11.jpg)
©Continuent 2012
From Replicating to Clustering
11
Application
Tungsten Connector
Application
Tungsten Connector
Replicator
Db2
Replicator
Db3Db1
Replicator
Slave SlaveMaster
Manager Manager Manager
Monitoring and Control
Monitoring and Control
Data Service: sjc
![Page 12: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/12.jpg)
©Continuent 2012
Basic Cluster Operations
12
• Switch -- Promote a slave node to master
• Shun -- Kick node out of cluster
• Welcome -- Return node to cluster
• Backup -- Run a backup
• Restore -- [Re-]provision a node
• Recover -- Return a broken node to slave
• Heartbeat -- Check cluster liveness
![Page 13: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/13.jpg)
©Continuent 2012
Automatic Failover
13
Application
Tungsten Connector
![Page 14: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/14.jpg)
©Continuent 2012
Automatic Failover
14
Application
Tungsten Connector
1. Detect non-responsivemaster
2. Halt master connections
![Page 15: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/15.jpg)
©Continuent 2012
Automatic Failover
15
Application
Tungsten Connector
3. Select most advanced slave and wait for pending transactions to apply
![Page 16: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/16.jpg)
©Continuent 2012
Automatic Failover
16
Application
Tungsten Connector
4. Promote to master, reconfigure slave(s), and re-enable connections
![Page 17: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/17.jpg)
©Continuent 2012
Automatic Failover
17
Application
Tungsten Connector
5. Recover failed master
![Page 18: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/18.jpg)
©Continuent 2012
Rolling SQL Schema Upgrade
18
Final UpgradeSwitchSlave Upgrade Slave Upgrade
• Shun slave 1, run upgrade, and return to cluster. Discard and recover on failure
• Upgrade old master. Cluster schema is now fully upgraded
• Repeat for additional slave(s)
• Switch master to promote an upgraded slave
![Page 19: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/19.jpg)
©Continuent 2012
Backup, Recovery, and Restore
19
• Backups
/cluster > datasource DB1 backup
• Recover
/cluster > datasource DB2 recover
• Restore
/cluster > datasource DB2 restore
• Out-of-box support for mysqldump/XtraBackup; simple integration for others
![Page 20: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/20.jpg)
©Continuent 2012 20
Managing Data Across Multiple Sites
![Page 21: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/21.jpg)
©Continuent 2012
Topics:
• Tungsten Replicator basics
• Disaster recovery sites
• Multi-master topology
• Fan-in
21
![Page 22: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/22.jpg)
©Continuent 2012
Tungsten Replicator Overview
22
Master
(Transactions + Metadata)
Slave
THL
DBMSLogs
Replicator
(Transactions + Metadata)
THLReplicator
Download transactions via network
Apply using JDBC
![Page 23: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/23.jpg)
©Continuent 2012
Tungsten Replication Service
23
Extract Filter Apply
StageExtract Filter Apply
StageExtract Filter Apply
Stage
Pipeline
MasterDBMS
TransactionHistory Log
In-MemoryQueue
SlaveDBMS
![Page 24: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/24.jpg)
©Continuent 2012
Parallel Replication of Sharded Data
24
Extract Filter Apply
StageExtract Filter Apply
StageStage
Pipeline
MasterDBMS
TransactionHistory Log
In-MemoryQueue
SlaveDBMS
Extract Filter ApplyExtract Filter ApplyExtract Filter Apply
![Page 25: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/25.jpg)
©Continuent 2012
Multiple Services Per Replicator
25
NYCReplicator
London
Service nyc
FrankfurtReplicator
Service fra
Replicator
Service nyc
Service fra
![Page 26: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/26.jpg)
©Continuent 2012
Multi-Master Replication
26
• Updates on 2+ sites (active-active mode)
• Enables geographic distribution of data
• No failover necessary if network fails or site becomes unavailable
• Not all applications can handle multi-master
• Applications must avoid con#icts
• Careful testing required
• Restoration of broken systems may not be easy
![Page 27: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/27.jpg)
©Continuent 2012
Simple Multi-Master Con$guration
27
NYC FrankfurtReplicator
fra (master)
nyc (slave)
Replicator
fra (slave)
nyc (master)
Database-to-Database
![Page 28: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/28.jpg)
©Continuent 2012
Clustered Multi-Master Con$guration
28
NYC Frankfurt
master
slave fra
fra
nyc
nyc
master
slave
![Page 29: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/29.jpg)
©Continuent 2012
Clustered Multi-Master Con$guration
29
NYC Frankfurt
slave
master fra
fra
nyc
nyc
master
slave
NYC Master Offline
![Page 30: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/30.jpg)
©Continuent 2012
Clustered Multi-Master Con$guration
30
NYC Frankfurt
master
slave fra
fra
nyc
nyc
master
slave
NYC Replicator Offline
![Page 31: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/31.jpg)
©Continuent 2012
Complex Multi-Master Topologies
31
StarReplication
AllMasters
![Page 32: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/32.jpg)
©Continuent 2012
Disaster Recovery Sites
32
• Updates go to primary site master
• DR site receives updates in real time
• Simple command promotion
• Tungsten connectors can route to primary and DR sites
• No application changes required
• Simplest way to protect against region/site failures
![Page 33: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/33.jpg)
©Continuent 2012
Tungsten Composite Data Services
33
App Server+ Connector
App Server+ Connector
Master Slave
San Jose New York
sj_nyc
![Page 34: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/34.jpg)
©Continuent 2012
Switching and failover
34
• Planned switch
/sj_nyc > switch to nyc
• Failover
/cluster > failover
• Failover between sites is not automated!
![Page 35: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/35.jpg)
©Continuent 2012
System-of-Record
35
• Application data are “homed” in di"erent locations
• Each MySQL schema has a single master
• Within a single DBMS there can be both masters and slave schemas
• Minimal application changes required
• Allows you to move chunks of data closer to users
![Page 36: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/36.jpg)
©Continuent 2012
System of Record Con$guration
36
NYC Frankfurt
Replicator
fra (master)
nyc (slave)
Replicator
fra (slave)
nyc (master)
acme
spitze
superb
acme
spitze
superb
Acme & Spitze mastered in NYC
Superb mastered in Frankfurt
![Page 37: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/37.jpg)
©Continuent 2012
Implementing System of Record
37
• Con$gure shard $lters on installation to set policies for unknown/bad shards
• De$ne shard homes in shard.map $le
shard_id master criticalacme nyc falsespitze nyc falsesuperb fra false
• Load shard de$nitions
trepctl shard -‐insert < shard.map
• Move shard homes by editing reloading the shard map
![Page 38: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/38.jpg)
©Continuent 2012
Fan-In Replication
38
• Transactions feed from multiple locations into a single slave
• Aggregate data for reporting
• Consolidate data from multiple servers into one server
• Transfer shards between non-connected clusters/locations
• Con#ict avoidance is up to applications
![Page 39: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/39.jpg)
©Continuent 2012
Fan-In Topology
39
Replicator
Replicator
nyc (master)
fra (master)
sfo (master)
nyc (slave)
fra (slave)
sfo (slave)
Replicator
New York
Frankfurt
Replicator
San Francisco
London
![Page 40: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/40.jpg)
©Continuent 2012
How Do I Obtain Tungsten?
40
• Register at Continuent to get downloads for Oracle replication: http://www.continuent.com/downloads/software
• Documentation is available on Continuent website
• MySQL replication is available from Tungsten Replicator project on code.google.comhttp://code.google.com/p/tungsten-replicator/
![Page 41: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/41.jpg)
©Continuent 2012
One more thing:
WE’RE HIRING!!!
41
![Page 42: Multi-Master, Multi-Site MySQL Databases Made Easy with](https://reader035.vdocument.in/reader035/viewer/2022062401/58649c3b1a28ab0e3093ebb5/html5/thumbnails/42.jpg)
©Continuent 2012.
Continuent Web Page:http://www.continuent.com
Tungsten Replicator 2.0:http://code.google.com/p/tungsten-replicator
Our Blogs:http://scale-out-blog.blogspot.comhttp://datacharmer.org/bloghttp://www.continuent.com/news/blogs
560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009e-mail: [email protected]