©Continuent 2013
Tungsten University: Geographically Distributed
Multi-Master MySQL ClustersRobert Hodges, CEO
Giuseppe Maxia, Director of QA
©Continuent 2013
Introducing Continuent
2
• The leading provider of clustering and replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance scaling and data management for MySQL
• Replication - Flexible, high-performance data movement
©Continuent 2013
Geographic Distribution of Data
3
©Continuent 2013
Why Multi-Master Across Sites?
4
• Move applications close to customers
• Online game pro"les
• Mapping app information
• Protect against site failures
• Hosted call centers
• Credit card processing
Multi-master is NOT a performance scaling solution
©Continuent 2013
Tungsten Cluster Architecturein 3 Minutes or Less
5
©Continuent 2013
Overview of a Tungsten Cluster
6
Application
Tungsten Connector
Application
Tungsten Connector
Replicator
Db2
Replicator
Db3Db1
Replicator
Slave SlaveMaster
Manager Manager Manager
Monitoring and Control
Monitoring and Control
Data Service: sjc
©Continuent 2013
Tungsten Replicator Architecturein 2 Minutes or Less
7
©Continuent 2013
Tungsten Replicator Overview
8
Master
(Transactions + Metadata)
Slave
THL
DBMSLogs
Replicator
(Transactions + Metadata)
THLReplicator
Download transactions via network
Apply using JDBC
©Continuent 2013
Simple Multi-Master Con"guration
9
London New YorkReplicator
ny (master)
lon (slave)
Replicator
ny (slave)
lon (master)
Multiple services per replicator
©Continuent 2013
Cross-Site Clustering in Action
10
©Continuent 2013
Clustered Multi-Master Con"guration
11
London New York
master
slave
ny
ny
lon
lon master
slave
te1
te2
te3
te4
Cross-site replicators read from slaves
Cross-site updates are unlogged
©Continuent 2013
Reconnect to Slave after Switch
12
slave
master
master
slave
(Master switch to te2)
London New York
ny
ny
lon
lonte1
te2
te3
te4
©Continuent 2013
Connect to Master when Slave Fails
13
master
master
slave
slave
London New York
ny
ny
lon
lonte1
te2
te3
te4
(Slave failure/maintenance)
©Continuent 2013
Multi-Master Best Practices
14
©Continuent 2013
Eventually Consistent Design
15
• Each site is independent application with local HA cluster
• Replication links clusters
• Transfer is asynchronous
• Think through business implications
• What does it mean if a site is down for several days?
• How do I get up-to-date picture of data?
©Continuent 2013
Avoid Con#icting Primary Keys
16
• Method #1: Auto-increment o$sets
[my.cnf]server-id=1auto-increment-offset = 1auto-increment-increment = 4
• Method #2:
create table test.keys ( server_id int primary key, counter int);insert into test.keys(server_id, counter) values(@@server_id, 0);
©Continuent 2013
Use Row Replication
17
• Row replication sends changes to each row:
[my.cnf]binlog_format=row
• Statement replication results in ambiguous changes depending on state of replication
update employees set salary = salary * 1.1;
Can affect different numbers of employees depending on state of
replication between sites
©Continuent 2013
Watch Out for...
18
• Triggers -- Tungsten cannot turn them o$; you have to alter the code
• Scheduled Events -- Be careful where they run!
• MyISAM -- Not crash safe and just generally bad for replication
• Very large transactions --
©Continuent 2013
Look Out for Con#icts
19
• UPDATE/DELETE on same rows across di$erent sites
• Generation of globally unbroken sequences (e.g., invoice numbers)
• Reports that need global state from all sites
©Continuent 2013
Use Filters to Detect/Avoid Con#icts
20
Name Purpose
ignore_server Drop transactions from server-id
rename Rename schemas/tables/columns
replicate Control schemas/table replication
shardfilter Control shard replication
You can also write your own filters!
©Continuent 2013
Test, Test, Test!!!
21
• Set up a QA environment
• Put load on every master on every site
• Test backup and restore on every DBMS node
• Test switch
• Test failover
• Compare data between sites
• Develop a reset procedure for fast test cycles
©Continuent 2013
Be Ready For Trouble
22
• Ensure you have multiple days of logs so you can be o%ine for a while
• Learn how to use data repair tools like pt-table-checksum
• Remove constraints to make repair easier
• Monitor replication!
• Get a support contract for when things go wrong
©Continuent 2013
Advanced Multi-Site Clustering
23
©Continuent 2013
Complex Multi-Master Topologies
24
StarReplication
AllMasters
©Continuent 2013
Snow#ake Topology
25
HQ
Region 1 Region 2
CM1 CM2 CM1 CM2
©Continuent 2013
Wrapping Up
26
©Continuent 2013
Tungsten University Sessions
27
• Replicate between MySQL and Oracle(May 2 and 7)
Send any feedback to: [email protected]
©Continuent 2012.
Continuent Web Page:http://www.continuent.com
Tungsten Replicator 2.0:http://code.google.com/p/tungsten-replicator
Our Blogs:http://scale-out-blog.blogspot.comhttp://datacharmer.org/bloghttp://www.continuent.com/news/blogs
560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009e-mail: [email protected]
