tungsten use case: how gittigidiyor (a subsidiary of ebay) replicates data in real time from mysql...
DESCRIPTION
Learn how Gittigidiyor (a subsidiary of eBay) replicates data in real time from MySQL to Oracle with Continuent Tungsten. Gittigidiyor consolidates reporting data in a single MySQL server. Transactions arriving on this server must be replicated in real time to an Oracle instance. Levent Kurt, Site Operations Manager at Gittigidiyor, describes how Gittigidiyor solved their problems of handling large tables in MySQL by replicating 20 GB of binlog data/day from MySQL 5.5 Community Edition to Oracle Enterprise Edition 11G with Continuent Tungsten, in a hosted VMware ESX environment.TRANSCRIPT
![Page 1: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/1.jpg)
©Continuent 2013.
How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time
From MySQL To OracleLevent Kurt, Site Operations Manager, Gittigidiyor
Giuseppe Maxia, Director of Quality Assurance, Continuent Linas Virbalas, Senior Software Engineer, Continuent
![Page 2: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/2.jpg)
©Continuent 2013
Introducing Continuent
���2
• The leading provider of clustering and replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance scaling and data management for MySQL
• Replication - Flexible, high-performance data movement
![Page 3: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/3.jpg)
©Continuent 2013
A Review of Tungsten Replicator
���3
![Page 4: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/4.jpg)
©Continuent 2013
Tungsten Replicator Overview
���4
Master
(Transactions + Metadata)
Slave
THL
DBMS Logs
Replicator
(Transactions + Metadata)
THLReplicator
Download transactions via network
Apply using JDBC
![Page 5: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/5.jpg)
©Continuent 2013
Master Replication Service
���5
Extract Filter Apply
StageExtract Filter Apply
Stage
Pipeline
MySQL Master
Transaction History Log
In-Memory Queue
Slave ReplicatorsBinlog
tcp/ip
![Page 6: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/6.jpg)
©Continuent 2013
Slave Replication Service
���6
Extract Filter Apply
StageExtract Filter Apply
StageExtract Filter Apply
Stage
Pipeline
Transaction History Log
In-Memory Queue
Slave DBMS
Master Replicator
tcp/
ip
![Page 7: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/7.jpg)
©Continuent 2013
star
master-slave Heterogeneous
fan-in slave all-masters
MySQL
Oracle
Oracle
MySQL Oracle
Oracle
MySQL MySQL
���7
![Page 8: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/8.jpg)
©Continuent 2013
Heterogeneous Replication
![Page 9: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/9.jpg)
©Continuent 2013
Steps to Heterogeneous Replication
1. Prepare (translate) schema for the slave DBMS
2. Set up replication
3. Provision initial data
4. Continue real-time replication
ddlscan
tpm
Parallel ApplyMySQL Blackhole
Tungsten Replicator
![Page 10: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/10.jpg)
©Continuent 2013
Replicating from MySQL to Oracle
![Page 11: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/11.jpg)
©Continuent 2013
Use Case: Real-Time Sales/Order Fulfillment
Real-Time Replication
Web-based Sales Purchase Ordering
3M Order Items
~1M Xacts/ Day
![Page 12: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/12.jpg)
©Continuent 2013
1. Translating schema for the slave
![Page 13: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/13.jpg)
©Continuent 2013
Translating Schema
���13
• Goal - set up heterogeneous replication
Replication
![Page 14: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/14.jpg)
©Continuent 2013
Translating Schema
���14
• Beginning - how to convert tables?
empty
Tables
![Page 15: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/15.jpg)
©Continuent 2013
Translating Schema
���15
• Beginning - how to convert tables?
empty
Tables
•Data types? •Column lengths? •Naming conventions? •Reserved words?
![Page 16: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/16.jpg)
©Continuent 2013
Translating Schema
���16
• ddlscan automates these conversions
empty
Tables
ddlscan
![Page 17: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/17.jpg)
©Continuent 2013
ddlscan
���17
• Part of Tungsten Replicator, GPL v2
• Translates schema with replication in mind
• Provides errors and warnings
• Can rename schema/tables/columns
![Page 18: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/18.jpg)
©Continuent 2013
Usage (MySQL to Oracle Example)
���18
$ cd tungsten-replicator/bin !
$ ./ddlscan \ -db test \ -template ddl-mysql-oracle.vm \ -user tungsten \ -pass secret
![Page 19: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/19.jpg)
©Continuent 2013
Translating Schema
���19
• ddlscan looks into source schema
empty
Tables
![Page 20: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/20.jpg)
©Continuent 2013
Translating Schema
���20
• ddlscan translates and renders DDL commands
empty
Tables
oracle-ddl.sql
![Page 21: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/21.jpg)
©Continuent 2013
Result of ddlscan
���21
CREATE TABLE 3colors /* ERROR: table starts with a number, use rename option and a filter */
( id NUMBER(10, 0), color VARCHAR2(1) /* ENUM('R','G','B') */, enabled NUMBER(1) /* BIT(1) - no constraints for 0 or 1 */, acolumnthatdoesntfittooracleat NUMBER(10, 0)
/* WARN: truncated column name exceeding 30 characters (acolumnthatdoesntfittooracleatall) */
! /* ERROR: table must have a primary key! */ ); !CREATE TABLE talks ...
![Page 22: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/22.jpg)
©Continuent 2013
Translating Schema
���22
• You run resulting SQL file on Oracle
empty
Tables
oracle-ddl.sql
![Page 23: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/23.jpg)
©Continuent 2013
Translating Schema
���23
• Tables are ready!
Tables Translated tables (empty)
![Page 24: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/24.jpg)
©Continuent 2013
2. Set Up Replication
![Page 25: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/25.jpg)
©Continuent 2013
Replicator Installation
���25
./tools/tpm configure my_ora \ --enable-heterogenous-service=true \ --install-directory=/opt/mysql_to_ora/continuent \ --members=alpha,bravo \ --master=alpha !./tools/tpm configure my_ora --hosts=alpha \ --replication-user=tungsten \ --replication-password=secret !./tools/tpm configure my_ora --hosts=bravo \ --replication-user=tungsten_frommysql \ --replication-password=secret \ --datasource-type=oracle \ --datasource-oracle-service=ORCL \ --svc-applier-filters=dropstatementdata !./tools/tpm install my_ora !./tools/tpm start my_ora
!
![Page 26: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/26.jpg)
©Continuent 2013
Replication Under the Hood
���26
MySQL Binlog
binlog_format=row
Service frommysql
Tungsten Master Replicator
MySQLExtractor Special Filters •Transform
ENUM to string •Transform SET
to string
Service frommysql
Tungsten Slave Replicator
Special Filters •Drop DDL •Map names to upper case •Rename too long or reserved
objects •Optimize updates to remove
unchanged columns •Ignore extra tables
OracleApplierMySQLExtractor
![Page 27: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/27.jpg)
©Continuent 2013
3. Provisioning an Oracle slave with
auditing trail
![Page 28: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/28.jpg)
©Continuent 2013
The Requirements
• Export data from MySQL master to the Oracle slave before starting replication
• The Oracle slave should also take trace of the changes
• Convert the MySQL tables to stream of raw data in THL format
• Data is about 400 GB
���28
![Page 29: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/29.jpg)
©Continuent 2013
The Practical Steps
1. Export data from master
2. Keep track of replication position
3. Start the replicator from the sandbox
4. Load data into slave with parallel apply
5. Switch replicator to real master
6. Dismiss the sandbox
7. Replication from MySQL to Oracle live
���29
![Page 30: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/30.jpg)
©Continuent 2013
Provisioning (1)
���30
slave
Oracle
master
MySQL
table data
table structuretable data
table structuretable data
table structure
![Page 31: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/31.jpg)
©Continuent 2013
table structure
table data
table structure
Provisioning (2)
���31
slave
Oracle
master
MySQL
table data
table structuretable data
table structure table structuretable structureddlscan
![Page 32: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/32.jpg)
©Continuent 2013
table structure
Provisioning (3)
���32
slave
Oracle
master
MySQL
table structuretable structure
MySQL sandbox
CONVERT to BLACKHOLE
table data
table structuretable data
table structuretable data
table structure
![Page 33: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/33.jpg)
©Continuent 2013
Provisioning (4)
���33
slave
Oracle
master
MySQL
MySQL sandbox
table data
table structuretable data
table structuretable data
table structure
not replicating
backup
![Page 34: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/34.jpg)
©Continuent 2013
Provisioning (5)
���34
slave
OracleMySQL sandbox
backup
bin logs
tungsten replicator THL
tungsten replicator
(ROW)
![Page 35: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/35.jpg)
©Continuent 2013
How Parallel Apply Works
ExtractFilter
(Shard)Apply
Extract/Filter/Apply
Stage Stage StagePipeline
Master DBMS
Slave DBMS
Transaction History Log
Parallel Queue
Extract/Filter/ApplyExtract/Filter/Apply
Extract/Filter/Apply
ProvisioningMutli-tenant source
![Page 36: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/36.jpg)
©Continuent 2013
4. Continue Real-Time Replication
![Page 37: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/37.jpg)
©Continuent 2013
Provisioning (6)
���37
slave
Oracle
master
MySQL
MySQL sandbox
table data
table structuretable data
table structuretable data
table structure
tungsten replicator
bin logs
THLtungsten replicator
![Page 38: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/38.jpg)
©Continuent 2013
Gittigidiyor (subsidiary of eBay)
architecture and needs
![Page 39: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/39.jpg)
©Continuent 2013 1
• Gittigidiyor • A subsidiary of eBay • Founded in 2001, joined the eBay family in June 2011 • Most widely-used e-shopping site in Turkey • More than 9 million registered users • Over 900 thousand daily visitors and • 5 million listings at any given time. • 700 thousand sellers all around Turkey • Selling 1 item every 3 seconds
![Page 40: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/40.jpg)
©Continuent 2013
Gittigidiyor Datawarehouse The Problem !• 400 GB of data • 450 tables • over 150 million rows • 20GB binlog/day • MySQL inefficiency for large
data handling • Migration to Oracle • Need to keep auditing trail • Tried Golden Gate, but
• technical problems • too expensive
Replication
Master
ETL
CDC
DWH
���40
![Page 41: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/41.jpg)
©Continuent 2013
Tungsten Solution !• DDL generation • Replication to Oracle master • Handling data correctly • CDC generation • Fail-safe
Master
ETL
Replicator
ORA Master
ORA CDC ORA DWH
Replication
Tungsten Replication
Tungsten Filters
���41
![Page 42: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/42.jpg)
©Continuent 2013
The steps we took !1. Proof of Concept
a. replicated 5 tables b. created the DDL
Generator. 2. Set up the replication to
Oracle Master 3. Set up filters for Oracle
CDC generation. 4. Dumped all data in MySQL
Master and MySQL CDC to Oracle equivalents.
Master
ETL
Replicator
ORA Master
ORA CDC ORA DWH
Replication
Tungsten Replication
Tungsten Filters
���42
![Page 43: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/43.jpg)
©Continuent 2013
MySQL to Oracle Demo
![Page 44: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/44.jpg)
©Continuent 2013
Support for Tungsten Replicator
���44
• Community support is available on project mailing list
• Continuent offers implementation services and annual 24/7 support subscriptions
• For high availability or disaster recovery needs, check out Continuent Tungsten clustering
• Meanwhile, have fun and try it out!
![Page 45: Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle](https://reader033.vdocument.in/reader033/viewer/2022052823/55515e55b4c905a8768b4e48/html5/thumbnails/45.jpg)
©Continuent 2013 ���45
Continuent Web Page: http://www.continuent.com
!
Tungsten Replicator: http://tungsten-replicator.org
Our Blogs: http://flyingclusters.blogspot.com http://scale-out-blog.blogspot.com http://datacharmer.blogspot.com http://continuent-tungsten.blogspot.com
560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: [email protected]