become a mysql dba: performing live database upgrades - webinar slides

1

Your host & some logistics

I'm Jean-Jérôme from the Severalnines Team and I'm your host for today's webinar!

Feel free to ask any questions in the Questions section of this application or via the Chat box.

You can also contact me directly via the chat box or via email: [email protected] during or after the webinar.

Confidential

Copyright 2013 Severalnines AB

ClusterControl Overview

2

Confidential


ClusterControl In a nutshell

3

Manage Scale

Monitor Deploy

Confidential

Copyright Severalnines AB

ClusterControlAutomation & Management

! Provisioning ! Deploy a cluster in minutes ! On-premises or in the cloud (AWS)

! Monitoring ! Systems view ! 1sec resolution ! DB / OS stats & performance advisors ! Configurable dashboards ! Query Analyzer ! Real-time / historical

4

Confidential


ClusterControlAutomation & Management

! Management ! Multi cluster/data-center ! Automate repair/recovery ! Database upgrades ! Backups ! Configuration management ! Cloning ! One-click scaling

5

Confidential


Supported DatabasesSQL

! MySQL (Oracle/Percona/MariaDB)

! MySQL Galera Cluster

! MySQL Cluster (NDB)

! MariaDB Cluster

! MySQL Replication 5.6

! Standalone MySQL/MariaDB

! Postgres 9.4

NoSQL

! Percona TokuMX Cluster

! MongoDB Sharded Cluster

! MongoDB Replica Set

6

Confidential


Customers

7


Performing MySQL database upgrades in Replication and Galera setups

November 24, 2015

Krzysztof Książek

Severalnines

[email protected]

8

mailto:[email protected]


! We want to help all non-DBA people who look after MySQL infrastructures

! Share tips and good practices

! Watch the replays of the previous webinars on our Slideshare page

! http://www.slideshare.net/Severalnines/videos

9

“Become a MySQL DBA” series

http://www.slideshare.net/Severalnines/videos


! Why upgrade?

! Different types of upgrade

! How to prepare for an upgrade

! Minor version upgrade

! for MySQL and Galera

! Major version upgrade

! for MySQL and Galera

! ClusterControl and MySQL upgrades

10

Agenda


Why upgrade?

11


! Security incidents and downtimes related to bugs can induce high costs

! Patches and upgrades require constant attention - these have to be installed on a regular basis to protect systems from known stability problems or security holes.

! Many companies are failing to do this

! On the next slide you can find some interesting exceptions from MySQL 5.6.27 changelog

12

Why upgrade?


! InnoDB: Setting lower_case_table_names=0 on a case-insensitive file system could result in a hang condition when running an INSERT INTO ... SELECT ... FROM tbl_name operation with the wrong tbl_name letter case.

! Replication: Repeatedly checking for ERR_LOCK_WAIT_TIMEOUT (as done, for example by repeatedly executing SHOW SLAVE STATUS) during a prolonged write lock on a table led to an assert.

! An assertion could be raised due to incorrect error handling if a SELECT ... FOR UPDATE subquery resulted in deadlock and caused a rollback.

! DELETE could check privileges for the wrong database when table aliases were used.

13

Why upgrade?


! MySQL 5.6.27 (2015-09-30)

! MySQL 5.6.26 (2015-07-24)

! MySQL 5.6.25 (2015-05-29)

! MySQL 5.6.24 (2015-04-06)

! MySQL 5.6.23 (2015-02-02)

! MySQL 5.6.22 (2014-12-01)

! MySQL 5.6.21 (2014-09-23)

! MySQL 5.6.20 (2014-07-31)

14

Why upgrade?


Different types of upgrade

15


! Minor version upgrades

! MySQL 5.6.x -> 5.6.y

! Versions should be compatible with regards to features

! Backward compatibility should be maintained

! The biggest concern is performance - execution plans may change, MySQL host has to be restarted

16



! Major version upgrades

! 5.x -> 5.y or even x.y -> a.b (for example 4.1 -> 5.7)

! Everything may have been changed

! Changes may be incompatible

! Extensive testing is required

! Read through the documents covering upgrade process

17



How to prepare for an upgrade?

18


! We’ll focus on major version upgrade as it has much bigger potential to impact your workload than simple 5.6.x -> 5.6.y upgrade

! First step - plan your upgrade process. Will you use logical dumps or binary in-place upgrade? Are you going to use a slave?

! Test your plan - is it doable? Did everything work as planned? Sometimes differences between versions make some upgrade plans impossible to execute

19



! Once you have a test database on a new version, upgraded in a way you plan to upgrade production, it’s time to run further test

! You need to check if app works ok with a new version of the database

! Check for both logical issues and performance regressions

! More profiling you can do from within your app (or external tool like New Relic), more data to compare

20



! Performance regressions

! Some settings may need to be tuned differently

! Query execution plans may change from version to version

! Logical issues

! Implicit row order may differ

! SQL syntax may have changed

21



! Minor version upgrades should be compatible (after GA)

! They tend to contain bug fixes and some simple improvements

! Ideally, you’d perform the same suite of tests like you’d perform when upgrading 5.x -> 5.y

! As a minimum, check for performance regressions

! Plan for warming up the node - MySQL restart will be required

22



! pt-upgrade is one of tools you can use to automate regression testing

! It uses data from slow log and replay them on two MySQL nodes (i.e. running old and new version)

! Queries are checked (among others) for the execution time and their result set (missing rows, incorrect order)

! Helps DBA to pinpoint regressions and potential issues

23



Minor version upgrade

24


! This process involves performing binary upgrade

! apt-get, yum, binary package - depending on the distro

! Once you perform binary upgrade, keep in mind to run mysql_upgrade script

! It’s designed to check schemas for any possible incompatibilities and fix them if needed

25

Minor version upgrade - MySQL and Galera


! You want to go slave by slave, upgrading them serially

! Keep the majority of traffic off upgraded slave to allow it to warm up

! You can preload buffer pool data (available in Oracle’s MySQL 5.6 and Percona Server for MySQL 5.5) to minimize the impact

! Keep in mind that the rest of the environment will have to handle higher load for the duration of upgrade and warmup phase

! Once you upgraded all the slaves, failover needs to be performed and then, finally, old master has to be upgraded

26

Minor version upgrade - MySQL


! You want to perform upgrade node by node

! Nodes will have to be restarted so traffic needs to be steered out of them for the duration of a maintenance

! Warmup phase applies here too - ensure that upgraded node is warmed up correctly

! Again, preloading buffer pool may speed up the process significantly

27

Minor version upgrade - Galera


Major version upgrade

28


! Much more complex process

! Logical upgrade is a recommended way to perform upgrade (only in 5.6 -> 5.7 Oracle removed this recommendation)

! For large databases this is serious issue to solve as dumping and then loading data takes very long time

! Setting up new nodes and connecting them via the replication may be needed to stay online for the duration of the upgrade

29

Major version upgrade - MySQL


! In-place upgrade is possible (just run apt-get/yum) but you have to be sure it works for you - tests have to be very detailed

! We’ve seen databases haunted by issues long time after the upgrade

! This is definitely not the best option to choose

30



! Assuming you can’t take a downtime for the duration of the upgrade, step-by-step approach is required

! First - build a new slave or use one of currently used in the topology (if the rest of them can handle the workload)

! Stop the replication at a known position

! Dump the data using mysqldump, mydumper or any other mean (SELECT INTO OUTFILE)

31



! Clean up the host - remove MySQL data directory, remove MySQL packages

! Install MySQL in the new version, clean install

! Load the data

! Setup the host as a slave using binlog position from the dump

! Let the new slave catch up with the master (it can be even days behind, all depends on the dataset size and dump/load speed)

32



! Use binary backup (xtrabackup/snapshots) to upgrade other nodes

! Finally, failover to one of the slaves with the new version and upgrade the old master

33



! Galera cluster can be treated as a single MySQL instance

! Yet it is possible (given some precautions) to run with a mix of versions in the cluster

! For larger data sets it may be not doable to dump/reload data before gcache will be filled - after that SST will be triggered

! Replication solves this problem - you can build a slave and it will catch up even if initial lag is in days

34

Major version upgrade - Galera


! Beginning is similar to MySQL replication - build a new slave to Galera Cluster

! Dump the data on the slave (keep in mind binlog coordinates)

! Upgrade to the new MySQL/Galera version

! Bootstrap the node as a separate cluster

! Load the data35



! Set it up as a slave to the old Galera Cluster

! One by one, remove nodes from the old cluster and add them to the new one

! Upgrade MySQL/Galera to the new version, wipe out old data directory - nodes will be provisioned by SST

36



! All the time you need to ensure that the production cluster (initially it will be old version but later it will be the new one) has enough nodes to handle the traffic

! Finally, you will have to switch production traffic to the new Galera Cluster

37



! If you can dump and reload data while last applied writeset still exists in gcache, you can try in-place upgrade

! Take the node off the cluster, dump, upgrade, reload, bring it back to the cluster

! If IST executes, things are looking good

! Such node has to be read only - writeset replication may not work from newer to older MySQL version

! If you also upgrade Galera (i.e. 2.x -> 3.x) some settings may have to be added to my.cnf (socket.checksum for example)

38



! Plan your upgrade process

! Check if it works

! Test your application on new MySQL version

! Check for any regressions

! Execute your upgrade process

! Remember about warmup phase

39

Short summary


MySQL Upgrades and ClusterControl

40


! ClusterControl can be used to some extend to help you in preparations for upgrade

! It’s also important to keep in mind few things when upgrading clusters managed by ClusterControl

! Building a slave - ClusterControl has option to automate slave provisioning for both MySQL replication and Galera Cluster

41




42

! You can leverage it to provision additional slave which then be upgraded manually to the new version

! Remove the slave from the cluster once you setup new Galera there

! Add new Galera Cluster to the ClusterControl once it’s set up


! It is important to disable at least node recovery in the ClusterControl when performing manual upgrades

! This is to prevent ClusterControl automated recovery from interrupting upgrade process

! You can also use Nodes -> Node’s IP -> Shutdown Node to stop MySQL on given host and ensure that ClusterControl won’t start it again

43



! More blogs in “Become a MySQL DBA” series:

! http://severalnines.com/blog/become-mysql-dba-blog-series-optimizer-hints-faster-query-execution

! http://severalnines.com/blog/become-mysql-dba-blog-series-understanding-mysql-error-log

! http://severalnines.com/blog/become-mysql-dba-blog-series-galera-cluster-diagnostic-logs

! Contact: [email protected]

44

Thank You!

http://severalnines.com/blog/become-mysql-dba-blog-series-optimizer-hints-faster-query-execution

http://severalnines.com/blog/become-mysql-dba-blog-series-understanding-mysql-error-log

http://severalnines.com/blog/become-mysql-dba-blog-series-galera-cluster-diagnostic-logs

mailto:[email protected]

become a mysql dba: performing live database upgrades - webinar slides

Technology