building simple and complex clusters with tungsten replicator

©Continuent 2012.

Giuseppe MaxiaQA DirectorContinuent, Inc

@datacharmer

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.

Building simple and complex

replication clusters with Tungsten

Replicator

1

1Tuesday, February 7, 12

From the user'sstandpoint


BINARY LOG

REPLICATION

MySQLDBMS

trans

actio

n

trans

actio

n

trans

actio

n

trans

actio

n

trans

actio

ntra

nsac

tion

trans

actio

ntra

nsac

tion

trans

actio

ntra

nsac

tion

trans

actio

n

MySQLDBMS

transactiontransaction






MySQL replicationis single threaded


MySQLDBMS

MySQLDBMS

MySQLDBMS

MySQLDBMS

master

slave slaveslave

MySQLDBMS

MySQLDBMS

MySQLDBMS

MySQLDBMS

master master master

slave

single source multi source (fan-in)

multiple sources?4Tuesday, February 7, 12

MySQLDBMS

MySQLDBMS

MySQLDBMS

MySQLDBMS

mastermaster

master master

from this to this

multiple masters?


MySQLDBMS

MySQLDBMS

MySQLDBMS

MySQLDBMS

mastermaster

master master

Avoiding conflicts?

INSERT RECORD

A

INSERT RECORD

A

MODIFY RECORD

B

MODIFY RECORD

B


MySQLDBMS

MySQLDBMS

MySQLDBMS

master

slaveslave

MySQLDBMS

MySQLDBMS

MySQLDBMS

master

slaveslave

MySQLDBMS

MySQLDBMS

MySQLDBMS

master

slaveslave

Seamless failover?


MySQLDBMS

MySQLDBMS

mysql master

postgresqlmysql

Replicating to something else?

oracle mongodb


All these examples tell us:

Nice dream, but MySQL can’t do it


Enter Tungsten Replicator


http://code.google.com/p/tungsten-replicator

100% GPL v2Open Source


What can it do?

• Easy failover

• Multiple masters

• Multiple sources to a single slave

• Conflict prevention

• Parallel replication

• Replicate to Oracle and PostgreSQL database


©Continuent 2012.

star-schema

master-slave Heterogeneous Direct slave

fan-in slave all-masters

MySQL

Oracle

Oracle

MySQL

RegularMySQL Master


Main components

• Transaction History Logs (THL)

• roughly corresponding to MySQL relay logs

• have a lot of metadata

• Service database

• contains metadata for latest transactions

• Metadata is committed together with data

• Makes slaves crash proof


Parallel replication


Parallel replication facts

✓Sharded by database

✓Good choice for slave lag problems

❖Bad choice for single database projects


Parallel Replication test

binary logs

MySQL slave

Tungsten slave

OFFLINE

STOPPED

replicator alpha

direct: alpha(slave)

Concurrent sysbenchon 30 databasesrunning for 1 hour

TOTAL DATA: 130 GBRAM per server: 20GB

Slaves will have 1 hour lag17Tuesday, February 7, 12

measuring results

binary logs

MySQL slave

Tungsten slave

ONLINE

START

replicator alpha

direct: alpha(slave)

Recording catch-up time


MySQL native replication

slave catch up in 04:29:30


Tungsten parallel replication

slave catch up in 00:55:40


Parallel replication made simpler

FROM HERE ....21Tuesday, February 7, 12


TO HERE22Tuesday, February 7, 12

parallel replicationdirect slave facts



✓No need to install Tungsten on the master




✓Tungsten runs only on the slave





✓Replication can revert to native slave with two commands (trepctl offline; start slave)






✓Native replication can continue on other slaves






✓Native replication can continue on other slaves

❖Failover (either native or Tungsten) becomes a manual task


Multiple masters


multiple masters

• Tungsten Replicator recipe: use more services


Bi-directional replication


Bi-directional replication with slaves


Multiple sources


multiple sources

• Tungsten Replicator recipe is still valid: use more services


Multiple source replication


Multiple masters replication: 3 nodes


Multiple masters replication: 4 nodes


star topologies


Conflict prevention


Conflict prevention facts



• Sharded by database




• Defined dynamically





• Applied either at the master or at the slave






• methods:






• methods:

• make replication fail






• methods:


• drop silently






• methods:


• drop silently

• drop with warning36Tuesday, February 7, 12

host1host3

master

host2

M

MS

SS

host4

S

Tungsten Conflict prevention points


host1host3

master

host2

M

MS

SS

host4

S

Prevention methods: Fail on master


host1host3

master

host2

M

MS

SS

host4

S

Prevention methods: Fail on masterINSERT A x,y

INSERT Ax,z


host1host3

master

host2

M

MS

SS

host4

S

Prevention methods: Fail on slave


host1host3

master

host2

M

MS

SS

host4

S

Prevention methods: Fail on slaveINSERT A x,y

INSERT Ax,z


host1

master

host2

M

M

host3

S

Prevention methods: Fail on slave(Multiple sources)


host1

master

host2

M

M

host3

S

Prevention methods: Fail on slaveINSERT A x,y

INSERT Ax,z

(Multiple sources)


host1host3

master

host2

M

MS

SS

host4

S

Prevention methods: DROP on master


host1host3

master

host2

M

MS

SS

host4

S

Prevention methods: DROP on masterINSERT B x,y


host1host3

master

host2

M

MS

SS

host4

S

Prevention methods: DROP on masterINSERT B x,y

INSERT Bx,z


host1host3

master

host2

M

MS

SS

host4

S

Prevention methods: DROP on master

INSERT Bx,z


host1

master

host2

M

M

host3

S

Prevention methods: DROP on slave(Multiple sources)


host1

master

host2

M

M

host3

S

Prevention methods: DROP on slaveINSERT A x,y

INSERT Ax,z

(Multiple sources)


host1

master

host2

M

M

host3

S

Prevention methods: DROP on slave

INSERT Bx,z

(Multiple sources)

INSERT B x,y


host1

master

host2

M

M

host3

S

Prevention methods: DROP on slave

INSERT Bx,z

(Multiple sources)


Tungsten in practiceInstallation


Installation

• Check the requirements

• Get the binaries

• Expand the tarball

• Run ./tools/tungsten-installer


REQUIREMENTS

• Java JRE or JDK (Sun/Oracle or Open-jdk)

• Ruby 1.8 (only during installation)

• ssh access to the same user in all nodes

• MySQL user with all privileges


Installation types

• master / slave

• slave - direct


master/slave

binlog

THL

THL

slave

slave

master

host1host2

host3

THL


slave direct

binlogTHL

slavemaster

relay log

host1host2

host3

THL

slave

relay log


sample installation

• 4 servers

• qa.r1.continuent.com r1




• MySQL 5.1

• tungsten-replicator-2.0.4


Installation (1)# starting at node 4, but any would do

for N in 1 2 3 4do ssh r$N mkdir tinstalldone

cd tinstalltar -xzf /path/to/tungsten-replicator-2.0.4.tar.gzcd tungsten-replicator-2.0.4


Installation (2)export TUNGSTEN_BASE=$HOME/tinstall

./tools/tungsten-installer \ --master-slave \ --master-host=r1 \ --datasource-user=tungsten \ --datasource-password=secret \ --service-name=dragon \ --home-directory=$TUNGSTEN_BASE \ --cluster-hosts=r1,r2,r3,r4 \ --start

1 2 3 4 5 6 7 8 9


Installation (2)export TUNGSTEN_BASE=$HOME/tinstall

./tools/tungsten-installer \ --master-slave \ # installation mode --master-host=r1 \ # who’s the master --datasource-user=tungsten \ # mysql username --datasource-password=secret \ # mysql password --service-name=dragon \ # name of the service --home-directory=$TUNGSTEN_BASE \ # where we install --cluster-hosts=r1,r2,r3,r4 \ # hosts in cluster --start # start replicator after installing


What does the installation do

host4

1: Validate all servers

✔✗✔

✔

host1

✔✗✔

✔

host2

✔✗✔

✔

host3

✔✗✔

✔

Report all errors54Tuesday, February 7, 12


host4

1 (again): Validate all servers

✔

✔

✔

host1

✔

✔

✔

host2

✔

✔

✔

host3

✔

✔

✔

✔✔ ✔ ✔



2: install Tungsten in all servers

host3

$HOME/ tinstall/ config/ releases/ relay/ logs/ tungsten/

host4host1

host2


examplessh r2 chmod 444 $HOME/tinstall./tools/tungsten-installer \ --master-slave --master-host=r1 \ --datasource-user=tungsten \ --datasource-password=secret \ --service-name=dragon \ --home-directory=$HOME/tinstall \ --thl-directory=$HOME/tinstall/logs \ --relay-directory=$HOME/tinstall/relay \ --cluster-hosts=r1,r2,r3,r4 --start

ERROR >> qa.r2.continuent.com >> /home/tungsten/tinstall is not writeable


examplessh r2 chmod 755 $HOME/tinstall./tools/tungsten-installer \ --master-slave --master-host=r1 \ --datasource-user=tungsten \ --datasource-password=secret \ --service-name=dragon \ --home-directory=$HOME/tinstall \ --thl-directory=$HOME/tinstall/logs \ --relay-directory=$HOME/tinstall/relay \ --cluster-hosts=r1,r2,r3,r4 --start

# no errors


Installation - direct./tools/tungsten-installer \ --direct \ --master-host=r1 \ --slave-host=r4 \ --master-user=tungsten \ --slave-user=tungsten \ --master-password=secret \ --slave-password=secret \ --service-name=Castor \ --channels=1 \ --home-directory=$TUNGSTEN_BASE \ --start-and-report


http://bit.ly/tr20_cookbook




© Continuent 2011© Continuent 2011

Tungsten Replicator vs. Enterprise

Feature Replicator EnterpriseSimple admin with trepctl X XMySQL Row/Statement 5.0/5.1/5.5 X XMulti-master replication X XParallel apply (2.0) X XDistributed management with cctrl XAutomated failover and recovery XZero-downtime upgrade XSQL routing and load-balancing XFloating IP support X


Conclusion and Q&A


©Continuent 2012. 63

Continuent Web Site:http://www.continuent.com

Tungsten Replicator 2.0:http://code.google.com/p/tungsten-replicator

Check out our blogs!http://scale-out-blog.blogspot.comhttp://datacharmer.blogspot.comhttp://flyingclusters.blogspot.com

560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel (866) 998-3642 Fax (408) 668-1009e-mail: [email protected]


http://www.continuent.com

http://www.continuent.com



http://scale-out-blog.blogspot.com

http://scale-out-blog.blogspot.com

http://datacharmer.blogspot.com

http://datacharmer.blogspot.com

http://flyingclusters.blogspot.com

http://flyingclusters.blogspot.com

mailto:[email protected]

mailto:[email protected]

building simple and complex clusters with tungsten replicator

Technology

slave replication

native slave

master tungsten

multiple masters replication

mysql native replicationslave

gbtungsten slave direct

mysql relay logs

parallel replicationtuesday