gtid replication - wordpress.com of contents 5 3 gtid replication implementation 4 gtid fail-over,...

GTID ReplicationImplementation and Troubleshooting

Abdel-MawlaGharieb2015/04/15

Who am I ?

• Abdel-Mawla Gharieb (Moll)

• From Alexandria, Egypt.

• Graduated in 2008.

• I like sports.

• Playing piano.

• 7 years of experience.

• Sr. DB Engineer at

2

Rakuten, Inc.

• Founded in Feb 1997.

• Largest e-commerce company in Japan.

• 4th Largest e-commerce company in the world.

• More than 11 thousands employees worldwide.

• In 2008, started expanding outside Japan.

• No.1 Internet company in the world is our target.

3

Rakuten Group4

Table of Contents5

3 GTID Replication Implementation

4 GTID Fail-over, Practical Demo

5 Migration to GTID Replication

6 GTID Replication Troubleshooting

１ What is GTID?

2 GTID Benefits

Table of Contents6

１ What is GTID?

2 GTID Benefits





What is GTID?

• GTID: Global Transaction IDentifier.

• Unique among all servers inside the replication

• Consists of two parts {source_id:transaction_id}:

– Source_id: Server’s UUID

– Transaction_id: Sequence number.

7

b1ee1e40-de96-11e4-b3a7-f4c709d514e2:10

Table of Contents 8

１ What is GTID?

2 GTID Benefits





GTID Benefits

• Setting up MySQL Replication is so simple now!

• Consistency is guaranteed between master and slaves.

• Simple to determine inconsistency.

• Simple to do Point in Time Recovery (PiTR).

• Useful to get consistent backup in MTS.

• Fail-over process become much easier.

• Automatic fail-over scripts are easy to implement.

9

GTID Benefits10

Master A Master B

Slave

mastera-binlog.73847547645

masterb-binlog.69893457845

slave-binlog.482847645

CHANGE MASTER TOMASTER_HOST=‘Master_B’,MASTER_LOG_FILE= ??,MASTER_LOG_POS= ??;

Classic Replication

GTID Benefits11

Master A Master B

Slave

uuid_A:1973 uuid_A:1973

uuid_A:1973CHANGE MASTER TOMASTER_HOST=‘Master_B’,MASTER_AUTO_POSITION = 1;

GTID Replication

Table of Contents12

１ What is GTID?

2 GTID Benefits





GTID Replication Implementation

GTID important variables• gtid-mode: ON|OFF• enforce-gtid-consistency: Prevent executing the non-transactionally safe

statements…– CREATE TABLE … SELECT statements.– CREATE TEMPORARY TABLE statements inside transactions.– Transactions or statements which update both transactional and non-

transactional tables.

• gtid-purged: Transactions have been purged from the binary logs.• gtid-executed: Transactions already executed on the server.• gtid-next: GTID value for the next transaction.

13


• Fresh installation

• Adding slave to an existing GTID replication

14


Fresh installation

• Master’s side configuration.

• Slave’s side configuration.

15

Fresh installation

Master’s side configuration• Add the following variables to my.cnf:

– server-id = 1

– log-bin = mysql-bin

– gtid-mode = ON

– enforce-gtid-consistency

– log-slave-updates

16

Fresh installation

Master’s side configuration• Restart MySQL to apply the configuration changes:

• Create a MySQL user to be used by the slave:

17

SQL> GRANT REPLICATION SLAVE ON *.* TO ‘repl’@’slave_host’ IDENTIFIED BY ‘p@ssw0rd’;

shell> service mysql restart

Fresh installation

Slave’s side configuration• Add the following variables to my.cnf:

– server-id = 2

– log-bin = mysql-bin

– gtid-mode = ON

– enforce-gtid-consistency

– log-slave-updates

18

Fresh installation

Slave’s side configuration• Restart MySQL to apply the configuration changes:

• Execute the CHANGE MASTER TO command:

19

shell> service mysql restart

SQL> CHANGE MASTER TOMASTER_HOST=‘master_ip’,MASTER_PORT=3306,MASTER_USER=‘repl’,MASTER_PASSWORD=‘p@ssw0rd’,MASTER_AUTO_POSITION=1;

Fresh installation

Slave’s side configuration• Start the replication:

20

SQL> START SLAVE;

Fresh installation

Slave’s side configuration• Checking the replication status:

– SQL> SHOW SLAVE STATUS|GSlave_IO_State: Waiting for master to send eventMaster_Host: 127.0.0.1Master_User: replMaster_Port: 3306. . .Slave_IO_Running: YesSlave_SQL_Running: Yes. . .Retrieved_Gtid_Set: b1ee1e40-de96-11e4-b3a7-f4c709d514e2:1-5Executed_Gtid_Set: b1ee1e40-de96-11e4-b3a7-f4c709d514e2:1-5Auto_Position: 1

21


Adding slave to an existing

GTID replication

22

Adding slave to an existing GTID replication

• Backup the master server:

– shell> mysqldump -u root -p --all-databases --flush-privileges --single-transaction --flush-logs --triggers --routines --events --hex-blob >/path/to/backupdir/full_backup-$TIMESTAMP.sql

• Start MySQL service on the slave with GTID variables.

• Restore the master backup.

• Execute change master to with MASTER_AUTO_POSITION=1

• Start the slave “START SLAVE;”

23

Adding slave to an existing GTID replication

• How does the slave get the correct GTID value to start replication from?!!

mysqldump knows about GTID!!

24

---- GTID state at the beginning of the backup--SET @@GLOBAL.GTID_PURGED='b1ee1e40-de96-11e4-b3a7-f4c709d514e2:1-7';

Table of Contents25

１ What is GTID?

2 GTID Benefits





Table of Contents26

１ What is GTID?

2 GTID Benefits





Migration to GTID Replication (5.6)

• Make the master server read only: SET GLOBAL read_only=ON;

• Allow all slaves to catch up with the master.

• Shutdown MySQL service on ALL servers.

• ADD the GTID variables.

• Add read_only in the master’s and skip_slave_start in the slaves’.

• Start MySQL service on ALL servers.

• Issue CHANGE MASTER TO MASTER_AUTO_POSITION=1; on all slaves and start them.

• Make the master writable: SET GLOBAL read_only=OFF;

27


Shutting down all servers at the same time means the GTID migration requires downtime. Why shutdown then?

– gtid_mode is a read only variable.

– Different gtid_mode values among replication servers is not acceptable.

Any workarounds ?!• Facebook and Booking.com patches.

• Facebook patch is ported in Percona Server starting from 5.6.22-72.0 (http://www.percona.com/blog/2015/02/10/online-gtid-rollout-now-available-percona-server-5-6/)

28

http://www.percona.com/blog/2015/02/10/online-gtid-rollout-now-available-percona-server-5-6/


enforce_gtid_consistency• OFF: non-transactionally safe statements are OK.

• WARN: non-transactionally safe statements will raise a warning but also OK.

• ON: non-transactionally safe statements will raise errors and not be executed.

30


GTID_MODE• OFF: Master will not generate GTIDs, slave will not replicate GTIDs.

• OFF_PERMISSIVE: same like OFF but the slave can replicate both GTID and non GTID transactions.

• ON: Master generates GTIDs only, slave replicates GTIDs only.

• ON_PERMISSIVE: same like ON but the slave can replicate both GTID and non GTID transactions.

31


Online migration

• SET GLOBAL ENFORCE_GTID_CONSISTENCY = WARN; on all servers• Check after a while if some warnings were printed or not.• SET GLOBAL ENFORCE_GTID_CONSISTENCY = ON; on all servers• SET GLOBAL GTID_MODE = OFF_PERMISSIVE; on all servers• SET GLOBAL GTID_MODE = ON_PERMISSIVE; on all servers• Wait for the following actions:

32


Online migration– Ongoing transactions require gtid-free were finished.

– All gtid-free transactions were replicated on all slaves.

– Binary log files that contain gtid-free events were expired.

• SET GLOBAL GTID_MODE = ON; on all servers

• STOP SLAVE;CHANGE MASTER TO MASTER_AUTO_POSITION=1;START SLAVE; on all slaves

33

Table of Contents34

１ What is GTID?

2 GTID Benefits





GTID Replication Troubleshooting

• Skip a transaction.

• Re-build a broken slave.

35


Skip a transaction

36

Skip a transaction

• Sample errors:

37

Last_SQL_Error: Could not execute Write_rows event on table test.t1; Duplicate entry '4'forkey'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log mysql-bin.000304, end_log_pos285

Skip a transaction

• Sample errors:

38

Last_SQL_Error: Could not execute Delete_rows event on table test.t1; Can't find record in 't1', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000304, end_log_pos 688

Last_SQL_Error: Could not execute Update_rows event on table test.t1; Can't find record in 't1', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000304, end_log_pos 492

Skip a transaction

Problem solution• Check which transaction is causing the problem:

39

SQL> SHOW SLAVE STATUSG ..Retrieved_Gtid_Set: b1ee1e40-de96-11e4-b3a7-f4c709d514e2:1-5Executed_Gtid_Set : b1ee1e40-de96-11e4-b3a7-f4c709d514e2:1-4Auto_Position: 1

Skip a transaction

Problem solution• Inject an empty transaction:

40

SQL> SET GTID_NEXT='b1ee1e40-de96-11e4-b3a7-f4c709d514e2:5';SQL> BEGIN;COMMIT;SQL> SET GTID_NEXT='AUTOMATIC';SQL> START SLAVE;


Re-build a broken slave.

41

Re-build a broken slave

• Sample error:

42

Last_IO_Errno: 1236Last_IO_Error : Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.'


Problem solution• Backup the master’s database:

43

shell> mysqldump -u root -p --all-databases --flush-privileges –single-transaction --master-data=2 --flush-logs --triggers --routines --events –hex-blob >/path/to/backupdir/full_backup-$TIMESTAMP.sql

shell> head -n 50 /path/to/backupdir/full_backup-$TIMESTAMP.sql|grep PURGED SET @@GLOBAL.GTID_PURGED='b1ee1e40-de96-11e4-b3a7-f4c709d514e2:1-8';


Problem solution• Restore the backup on the broken slave:

44

shell> mysql -u root -p < /path/to/backupdir/full_backup-$TIMESTAMP.sqlERROR 1840 (HY000): @@GLOBAL.GTID_PURGED can only be set when @@GLOBAL.GTID_EXECUTED is empty


Problem solution• Empty the GTID_EXECUTED on the slave:

• Restore the backup again

45

SQL> RESET MASTER;

shell> mysql -u root -p < /path/to/backupdir/full_backup-$TIMESTAMP.sql


Problem solution• Make sure that GTID_EXEUCTED is correct:

46

SQL> SHOW GLOBAL VARIABLES LIKE 'gtid_executed';+---------------------+---------------------------------------------------------------+| Variable_name | Value |+---------------------+---------------------------------------------------------------+| gtid_executed | b1ee1e40-de96-11e4-b3a7-f4c709d514e2:1-8 |+---------------------+---------------------------------------------------------------+1 row in set (0.00 sec)


Problem solution• Start the slave:

47

SQL> START SLAVE;

Before Using GTID …

• Migration to GTID replication requires downtime (5.6).

• Non-transactionally safe statements will not be executed (5.6).

• MySQL performance in GTID replication is a little bit slower.

• MariaDB GTID is not compatible with MySQL server or Percona server.

• Errant transactions might cause troubles.

• New protocol, new problems and new solutions.

48

Conclusion …49





１ What is GTID?

2 GTID Benefits

Q & A

THANK [email protected]

We are hiring!!http://global.rakuten.com/corp/careers/

50

http://global.rakuten.com/corp/careers/

gtid replication - wordpress.com of contents 5 3 gtid replication implementation 4 gtid fail-over,...

Documents