a mysql odyssey - a blackhole crossover

A MySQL Odyssey - a Blackhole crossover

Keith Hollman

MySQL Principal Sales Consultant EMEA


The following is intended to outline our general product direction. It is

intended for information purposes only, and may not be incorporated

into any contract.

It is not a commitment to deliver any material, code, or functionality,

and should not be relied upon in making purchasing decisions.

The development, release, and timing of any features or functionality

described for Oracle’s products remain at the sole discretion of Oracle.

Safe Harbor Statement


Agenda MySQL Replication: the crossover.

From

Dev…

…thru’ the

BLACK

HOLE…

Knowledge

is key

Non-

implosion

non-

destructive

…into Pre /

Production

Windows

¿Why?

Oracle Linux

The

Future

Nebulae


¿Why?

Sharing how to automate a cross-platform data transfer with

MySQL, applying data filters at the same time.

Small, quick and easy to setup, minimizing performance & resource

overheads.

Maneuvering the High Availability door to other possibilities, such as

Security control, Auditing, horizontal & vertical scalability, read-write

segregations.

What’s this all about:


¿Why?

A Developer using MySQL on Win (Testing).

Need to export & import data / schema definition into DBA’s /

Sys.Admin’s Linux environment (Pre-Prod, UAT, Production).

Certain changes are required and it has to be easy, and unobtrusive.

And data sensitive columns?

Wouldn’t it be nice if I could have it all transferred directly, automatically

and not have to worry about platform differences?

Just “a” Scenario


¿Why? The Setup

“Dev” pc

Win.7 Pro SP1 Intel i5 2.67GHz 8Gb RAM 64bit

CREATE TABLE `crossover` …… ENGINE=InnoDB;

“BLACKHOLE” env

Oracle Linux 6.3 UEK 2.6.39 VM 1xCPU 512Mb RAM 32bit 5Gb disk

ALTER TABLE `crossover` ENGINE=BLACKHOLE;

“Prod” env

SLES 11 SP2 3.0.74 VM 1xCPU 1Gb RAM 10Gb disk

ALTER TABLE `crossover` ENGINE=InnoDB;

MySQL 5.6.11 EE Replication Setup

using GTID’s: Master (Win)

-> Slave / Master (OL) -> Slave (SLES)


¿Why?

BLACKHOLE storage engine:

“.. accepts data but throws it away and does not store it. Retrievals always return an empty

result”

– Row Based Replication best (vs Statement, autoinc & PK’s)

– UPDATE’s & DELETE’s triggers aren’t activated: will break replication.

– All operations on BLACKHOLE will be replicated to SLES slave.1

Others:

– Replicating with lower_case_table_names=1.

– replicate-ignore-db, replicate-do-table, replicate-wild-*.

– Double check FW (iptables –F for easiness).

MySQL BLACKHOLE & Other Awareness

1 Should we be tempted to temporarily ‘fix’

replication by changing SE and back.


The BLACKHOLE Crossover


From Dev… thru’ the BLACKHOLE…

Logical Dump of the master database “sakila” from Win to OL: [mysql@ol63uek01 ~]$ mysqldump -uroot -poracle -h 192.168.56.101 -P3307 -B sakila \ --master-data=2 | mysql -uroot

ol63 10:40 sakila> show databases;

Set Replication up…

… and make sure we only replicate “sakila”: [mysql@ol63uek01 ~]$ vi /etc/my.cnf : [mysqld] replicate-do-db =sakila

Check it’s working: ol63 11:00 sakila> show slave status\G

Win 11:00 sakila> call BlackholeInsert (1000); ..& look for the “*–relay.bin*” logs.

[ From Win to Oracle Linux ]


Win 11:01 sakila> select count(*) from crossover;

Stepping into the BLACKHOLE: ol63 11:02 sakila> select count(*) from crossover; ol63 11:02 sakila> show create table crossover; ol63 11:02 sakila> alter table crossover engine=blackhole;

Query OK, 1001 rows affected (0.16 sec) Records: 1001 Duplicates: 0 Warnings: 0

Confirming: ol63 11:02 sakila> show create table crossover; ol63 11:03 sakila> select count(*) from crossover;

From Dev… thru’ the BLACKHOLE… Continued…

+----------+ | count(*) | +----------+ | 1001 | +----------+ 1 row in set (0.00 sec)


| crossover | CREATE TABLE `crossover` ( ÌD` int(7) NOT NULL AUTO_INCREMENT, `Name` char(20) NOT NULL DEFAULT '', PRIMARY KEY (ÌD`) ) ENGINE=InnoDB AUTO_INCREMENT=1002 DEFAULT CHARSET=utf8 | 1 row in set (0.00 sec)

| crossover | CREATE TABLE `crossover` ( ÌD` int(7) NOT NULL AUTO_INCREMENT, `Name` char(20) NOT NULL DEFAULT '', PRIMARY KEY (ÌD`) ) ENGINE=BLACKHOLE DEFAULT CHARSET=utf8 | 1 row in set (0.00 sec)



… thru’ the BLACKHOLE… into Prod.

Logical dump of ol63uek BLACKHOLE table to SLES Production env: [mysql@suse01 ~]$ mysqldump -uroot -p -h 192.168.56.102 -B sakila --master-data=2 \ | mysql -uroot suse01 11:11 sakila> show databases;

Set Replication up… again (ol63 is our Master now) & check it works: suse01 11:12 sakila> show slave status\G

Coming out of the BLACKHOLE, into InnoDB@SLES: suse01 11:12 sakila> alter table crossover engine=InnoDB;

Pull data from Win into SLES (non-blackhole tables): [mysql@suse01 ~]$ mysqldump -uroot –p –h 192.168.56.101 -P3307 --no-create-info \ --no-create-db –B sakila | mysql -uroot


Non-Implosion and Non-Destructive.

http://ourwikiworld.blogspot.com.es/2011/04/amazing-physics-stunning-building.html


Non-implosion & non-destructive

Truncate table crossover;

Bulk INSERT’s.

DDL modifications.

Finer Replication capabilities.

Schema manoeuvrability.

Performance analysis.

Located on same Master server.

Isolated env’s & modular.

! UPDATE & DELETE triggers.

! Added Operational commitments.

! Data models, PK & Autoinc cols.

! IO monitoring / impact evaluation

removed.

! Infrastructure becomes inter-

dependent (replication).


Non-implosion & non-destructive cont. Win ol63uek SLES

OBJECT_NAME crossover Diff vs orig. crossover diff crossover diff

COUNT_STAR 10,002 -1 10,001 0 40,007 -20,004

SUM_TIMER_WAIT 231,905,415,096 -26,335,040 7,504,322,436 0 3,783,883,427,646 -9,161,505,807

MIN_TIMER_WAIT 11,728,944 0 22,464 0 24,384 0

AVG_TIMER_WAIT 23,185,664 -376 750,087 0 94,580,202 94,127,574

MAX_TIMER_WAIT 50,530,335,064 0 426,302,487 0 428,082,829,761 0

COUNT_READ 1 -1 0 0 30,006 -20,004

SUM_TIMER_READ 26,335,040 -26,335,040 0 0 15,641,005,167 -9,161,505,807

MIN_TIMER_READ 26,335,040 -26,335,040 0 0 24,384 0

AVG_TIMER_READ 26,335,040 -26,335,040 0 0 521,208 126,492

MAX_TIMER_READ 26,335,040 -26,335,040 0 0 374,708,547 0

COUNT_WRITE 10,001 0 10,001 0 10,001 0

SUM_TIMER_WRITE 231,879,080,056 0 7,504,322,436 0 3,768,242,422,479 0

MIN_TIMER_WRITE 11,728,944 0 22,464 0 24,384 0

AVG_TIMER_WRITE 23,185,288 0 750,087 0 376,786,521 0

MAX_TIMER_WRITE 50,530,335,064 0 426,302,487 0 428,082,829,761 0

COUNT_FETCH 1 -1 0 0 30,006 -20,004

SUM_TIMER_FETCH 26,335,040 -26,335,040 0 0 15,641,005,167 -9,161,505,807

MIN_TIMER_FETCH 26,335,040 -26,335,040 0 0 24,384 0

AVG_TIMER_FETCH 26,335,040 -26,335,040 0 0 521,208 126,492

MAX_TIMER_FETCH 26,335,040 -26,335,040 0 0 374,708,547 0

COUNT_INSERT 10,001 0 10,001 0 10,001 0

SUM_TIMER_INSERT 231,879,080,056 0 7,504,322,436 0 3,768,242,422,479 0

MIN_TIMER_INSERT 11,728,944 0 22,464 0 24,384 0

AVG_TIMER_INSERT 23,185,288 0 750,087 0 376,786,521 0

MAX_TIMER_INSERT 50,530,335,064 0 426,302,487 0 428,082,829,761 0

COUNT_UPDATE 0 0 0 0 0 0

SUM_TIMER_UPDATE 0 0 0 0 0 0

We could use

common O.S. activity

measuring tools.

Using MySQL’s own

Performance_Schema:

After a: truncate table crossover;

Obtained via a: select * from table_io_waits_summary_by_table where object_schema ='sakila' and object_name = 'crossover‘ \G


Knowledge & The Future

http://ourwikiworld.blogspot.com.es/2011/04/amazing-physics-stunning-building.html


mysql CLI & plenty of commands.

MySQL Workbench Utilities.

– mysqldbcopy, mysqlfailover, mysqlreplicate, mysqlrpladmin, mysqlrplcheck, etc.

MySQL Performance_Schema.

MySQL Enterprise Monitor.

– Replication Advisor

– Query Analyzer

Knowledge is key: Monitoring & Utilities.

MySQL Solutions


The Future Nebulae

Existing / inherited Applications.

– eg.Using CSV type tables, from Win to Linux or vice-versa.(Address books, User

provisioning, etc.)

Cloud / Virtualization platforms.

– HA Potential: Replication, 1x Blackhole for 10+ slaves) & MySQL Cluster.

– Column filtering / Replication filtering / Triggers from Blackhole.

Eliminate sensitive or un-required columns.

Create a number of slaves from BLACKHOLE each with it’s own dataset.

Big Data / BI / custom solutions.

– Repeat bulk loads for troubleshooting / performance comparisons.

– Analyze binlogs on BLACKHOLE / backup strategies, etc.

Who can benefit?


Reference material

The BLACKHOLE Storage Engine (ref.man.)

MySQL Replication Tutorial (TWP)

The CSV Storage Engine (ref.man.)

MySQL Workbench download. (documentation)

Download Utilities without Workbench: http://dev.mysql.com/downloads/tools/utilities/1.3.html

MySQL Enterprise Monitor:

– Trial 30 day evaluation http://edelivery.oracle.com

Performance Schema documentation. (table_io_waits_summary_by_table descr.)

Oracle Linux & Unbreakable Enterprise Kernel (Getting Started).

SUSE Linux Enterprise Server 11 SP2 for x86 included packages.

Advanced Linux Command Mastery

http://dev.mysql.com/doc/refman/5.6/en/blackhole-storage-engine.html

http://www.mysql.com/why-mysql/white-papers/mysql-replication-tutorial/

http://dev.mysql.com/doc/refman/5.6/en/csv-storage-engine.html

http://dev.mysql.com/doc/refman/5.6/en/csv-storage-engine.html

http://dev.mysql.com/downloads/tools/workbench/

http://dev.mysql.com/doc/workbench/en/index.html

http://dev.mysql.com/downloads/tools/utilities/1.3.html

http://dev.mysql.com/doc/mysql-monitor/2.3/en/index.html

http://edelivery.oracle.com/

http://dev.mysql.com/doc/refman/5.6/en/performance-schema.html

http://dev.mysql.com/doc/refman/5.6/en/table-waits-summary-tables.html

https://linux.oracle.com/

http://www.oracle.com/technetwork/articles/servers-storage-admin/uek-rel2-getting-started-1555632.html

https://www.suse.com/LinuxPackages/packageRouter.jsp?product=server&version=11&service_pack=sp2&architecture=i386&package_name=index_group

http://www.oracle.com/technetwork/articles/linux/part3-090589.html


[email protected]


Efxaristo OSC13, SUSE, Thessaloniki, everyone. (Thank you)


Graphic Section Divider

Keynotes

Conferences Sessions

Hands-on Labs

Birds-of-a-feather sessions

Tutorials

Receptions

Demo Pods

Exhibition Hall

San Francisco, September 21-23 Additional Day of Tutorials Oracle.com/mysqlconnect

Early Bird

Discount: Register

by July 19 and Save

US$500!

Email to [email protected] for

exhibition and sponsorship opportunities

mailto:[email protected]



a mysql odyssey - a blackhole crossover

Technology