tungsten university: setup & operate tungsten replicator

34
©Continuent 2013 Tungsten University: Set Up and Operate Tungsten Replication Robert Hodges, CEO

Upload: continuent

Post on 24-Jun-2015

892 views

Category:

Technology


4 download

DESCRIPTION

Do you have the background necessary to take full advantage of Tungsten Replicator in your environments? Tungsten offers enterprise-quality replication features in an open source package hosted on Google Code. This virtual course will teach you how to set up innovative topologies that solve complex replication problems. We start with single MySQL servers running MySQL replication and show a simple path migration path to Tungsten. Course Topics - Checking host and MySQL prerequisites - Downloading code from http://code.google.com/p/tungsten-replicator/ - Installation using the tungsten-installer utility - Transaction filtering using standard filters as well as customized filters you write yourself - Enabling and managing parallel replication - Configuring multi-master and fan-in using multiple replication services - Backup and restore integration - Troubleshooting replication problems - Logging bugs and participating in the Tungsten Replicator community. Replication is a powerful technology that takes knowledge and planning to use effectively. We give you the background that makes replication easier to set up, and allows you to take full advantage of the Tungsten Replicator benefits. Learn how to configure and use it more effectively for your projects in the cloud as well as on-premise hardware.

TRANSCRIPT

Page 1: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Tungsten University: Set Up and Operate

Tungsten Replication Robert Hodges, CEO

Page 2: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Introducing Continuent

2

• The leading provider of clustering and replication for open source DBMS

• Our Product: Continuent Tungsten

• Clustering - Commercial-grade HA, performance scaling and data management for MySQL

• Replication - Flexible, high-performance data movement

Page 3: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Introducing Tungsten Replicator

3

Page 4: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Tungsten Replicator Overview

4

Master

(Transactions + Metadata)

Slave

THL

DBMSLogs

Replicator

(Transactions + Metadata)

THLReplicator

Download transactions via network

Apply using JDBC

Page 5: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Tungsten Replication Service

5

Extract Filter Apply

StageExtract Filter Apply

StageExtract Filter Apply

Stage

Pipeline

MasterDBMS

TransactionHistory Log

In-MemoryQueue

SlaveDBMS

Page 6: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Overview of Installation Process

1. Set up hosts

2. Prepare MySQL replicas

3. Download software

4. Install using tpm utility

6

Amazon Setup:https://docs.continuent.com/wiki/display/TEDOC/

Preparing+EC2+Servers

Page 7: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Old Way of Installing Tungsten...

• tungsten-installer - Perform initial installation

• con!gure-service - Set up individual replication services

• update - Change parameters on services after installation

7

The tpm utility replaces all of this!

Page 8: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

How tpm Works

8

db1

db2

db3

Staging copy of files

check prereqscopy code

configure topology

Page 9: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

What Is Tungsten Doing?

9

Page 10: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Commands to Check Replication

• trepctl services -- All services

• trepctl status -- Overall pipeline status

• trepctl status -name stages -- Stage summary

• trepctl status -name stores -- Store summary

• trepctl status -name tasks -- Task detail

• trepctl [-service name] heartbeat -- Is it Alive?

10

Page 11: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Changing Replicator Con!guration

11

Page 12: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Commands to Change Con!guration

• tpm update - Alter con"guration

• tpm update - Upgrade the replicator to a new version

12

Hint: Don’t change replicator configuration files directly...

Page 13: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Recon!guring a Replicator

• No restart on local host

trepctl offlinetools/tpm update ... --no-restarttrepctl configuretrepctl online

• With automatic restart

tools/tpm update ...

13

Page 14: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Enabling Parallel Replication

14

Page 15: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Parallel Replication of Sharded Data

15

Extract Filter Apply

StageExtract Filter Apply

StageStage

Pipeline

MasterDBMS

TransactionHistory Log

ParallelQueue

SlaveDBMS

Extract Filter ApplyExtract Filter ApplyExtract Filter Apply

Page 16: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Fixing Problems

16

Page 17: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

The Scene of the Crime

17

trepctl -host db2 statusProcessing status command...NAME VALUE---- -----...pendingError : Event application failed: seqno=1957469 fragno=0 message=java.sql.SQLException: Statement failed on slave but succeeded on masterpendingErrorCode : NONEpendingErrorEventId : mysql-bin.000008:0000000461736666;0pendingErrorSeqno : 1957469pendingExceptionMessage: java.sql.SQLException: Statement failed on slave but succeeded on master create table trouble(id int primary key) /* ___SERVICE___ = [university] */...Finished status command...

Page 18: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Correcting problems

1. Look at the transaction (thl list -seqno NNN)

2. Look at the replicator message log

3. Fix the problem and go online

OR

4. Skip over the problem and go online

OR

5. Reprovision the slave

18

Page 19: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Tungsten Logs

19

Page 20: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Simple Facts about Logs

• Replicator log = THL (Transaction History Log)

• One THL per replication service

• 2x the size of MySQL binlog

• Default retention of 7 days

• Take service o#ine to move or delete logs safely

• The thl tool shows you log contents

20

Page 21: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Resetting Replication

21

Page 22: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Resetting and Reinstalling

Reset

1. Stop replicator

2. Remove THL "les

3. Remove tungsten_<svc> databases

Reinstall

1. All of the above, plus delete /opt/continuent/*

2. Rerun installation

22

Page 23: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Backup and Restore

23

Page 24: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

What Backup Tools Does MySQL Have?

24

MySQL Server

Process

File System

DBMS Files

mysqldump(dataset < 10Gb)

xtrabackup(10Gb < dataset < 1Tbtar/rsync(dataset < 1Tb)

file system snapshot(1Tb << dataset)

Page 25: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Advanced Installation using Cookbooks

25

Page 26: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

How Do I Install Fan-In Replication?

26

db1Replicator

db3

Service db1

db2Replicator

Service db2

Replicator

Service db1

Service db2

Page 27: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

How Do I install Multi-Master?

27

db1 Replicator

Service db1

Service db2

Service db3db3

Service db1

Service db2

Service db3db2

Replicator

Service db1

Service db2

Service db3

Page 28: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Tpm Understands Topologies!

• Use --topology argument!

28

Fan-In StarAll-Masters

--topology=fan-in --topology=all-masters --topology=star

Page 29: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

For More Help: Tungsten Cookbook

29

• Installs complex topologies automatically

• Directions in cookbook/README

• Parameters in cookbook/USER_VALUES.sh

• Node names in COMMON_NODES.sh

• Installation script for each supported topology

http://datacharmer.blogspot.com/2013/07/tpm-multi-master-composer.html

Page 30: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Joining the Community

30

Page 31: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Tungsten Replicator is Open Source

31

• Project home:

http://code.google.com/p/tungsten-replicator/

• Log bugs, "nd builds, post in replicator discussion group

• Documentation:

https://docs.continuent.com/wiki/display/TEDOC/Tungsten+Documentation+Home

Page 32: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

Wrapping Up

32

Page 33: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2013

More Webinars and Talks!

33

• How Gittigidiyor (a subsidiary of eBay) Replicates Data in Real Time from MySQL to Oracle(6 November 2013 10am EDT/3pm GMT)

• Come see us at Percona Live London! (11-12 November 2013)

Send any feedback to: [email protected]

Page 34: Tungsten University: Setup & Operate Tungsten Replicator

©Continuent 2012.

Continuent Web Page:http://www.continuent.com

Tungsten Replicator 2.0:http://code.google.com/p/tungsten-replicator

Our Blogs:http://scale-out-blog.blogspot.comhttp://datacharmer.org/bloghttp://www.continuent.com/news/blogshttp://"yingclusters.blogspot.com/

560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009e-mail: [email protected]