Confidential

MySQL Replication BlueprintApril 5, 2015

Art van Scheppingen

Senior Support Engineer, Severalnines

art@severalnines.com

Confidential

Your host & some logistics

I'm Jean-Jérôme from the Severalnines Team and I'm your host for today's webinar!

Feel free to ask any questions in the Questions section of this application or via the Chat box.

You can also contact me directly via the chat box or via email: jj@severalnines.com during or after the webinar.

Confidential

Agenda

☐ Why a Blueprint for Replication?

☐ MySQL Replication Blueprint - Components & Topology

☐ Monitoring & Trending

☐ Management

☐ Load balancing

☐ Q&A

Confidential

Why the need for a MySQL Replication blueprint?

Confidential

Replication in the pre-MySQL 5.6 era

☐ Pre-MySQL 5.6 replication☐ Easy to set up☐ Easy to break☐ Difficult and sometimes impossible to repair

☐ Tools become necessary☐ Failover tools (MHA, PRM, Mysqlfailover)☐ Percona-Toolkit (pt-heartbeat, pt-table-sync, pt-slave-restart)

Confidential

Is this production ready?

☐ Enable master binlog☐ Have slave(s) read from master☐ Are we done?

Confidential

Issues to be addressed in production (1 / 2)

☐ Master as a SPOF - failover (automatic vs manual)☐ Data consistency☐ Read write split☐ Dynamic topology changes☐ Load balancing

Confidential

Issues to be addressed in production (2 / 2)

☐ Config changes☐ Version upgrades☐ Schema changes☐ Backup strategies☐ Scaling☐ Monitoring

Confidential

Making MySQL Replication Production Ready

☐ Holistic approach needed☐ Deployment automation☐ Failover tools☐ Virtual ip addresses☐ Load Balancers☐ Backups

☐ Topology logic implemented many times (in each component)

Confidential

Only one version of the truth

☐ Who is the current master?☐ In case of failover, who is the new master?☐ Which are the slaves?☐ Application requests always directed to the right server

(depending on reads vs writes)

Confidential

GTID - A Stronger Foundation for Replication

☐ Global Transaction Identifier (GTID) introduced in 5.6☐ Used to identify and associate each transaction to its origin☐ Makes the transaction unique in the topology

☐ Different GTID implementation in MySQL and MariaDB

Confidential

Management: Repair A Broken Topology

☐ Slave promotion without GTID☐ Find the most advanced slave☐ Elect this node as the new master☐ Find difference between new master and slaves☐ Find binlog position per slave☐ Re-attach slaves to new master

☐ Why do we need to find the binlog position?☐ Slaves have their own binary logs (slave updates)☐ Position in slave logs do not correspond to the master

Confidential

Management: Repair A Broken Topology

☐ Slave promotion with GTID☐ Find the most advanced slave☐ Elect this node as the new master☐ Re-attach slaves to new master with auto-position

☐ Auto-position☐ Master finds the binlog file and position for the slave☐ Slaves will automatically receive remaining transactions☐ Transactions are in the correct order

Confidential

Introducing the MySQL Replication Blueprint

Components and Topology

Confidential

Components and topology

Confidential

Components and Topology: Master/Slave

☐ Most simplistic MySQL replication topology

Confidential

Components and Topology: Master/Slave

Confidential

Components and Topology: Multi Master

☐ Similar to Master/Slave☐ All nodes are master and replica☐ Circular replication

☐ Active/Passive☐ One node is the master☐ Other node(s) are hot standby.

☐ Active/Active☐ All nodes are master

Confidential

Components and Topology: Multi Master

Confidential

Components and Topology: Multi Master

☐ Prevent infinite loops☐ log-slave-updates=1 + replicate-same-server-id=0☐ Even with GTID

☐ Prevent conflicts☐ Writing to the same data/schemas

☐ Replication bottleneck☐ Active/Active: two replication streams☐ After a master fails you only have one replication stream

Confidential

Components and Topology: Parallel Replication

☐ MySQL replication is single threaded☐ Replication blocked☐ Long running queries☐ DDL changes

☐ Parallel Replication☐ Multiple replication streams

Confidential

Components and Topology: Parallel Replication

Confidential

Components and Topology: Parallel Replication

☐ MySQL 5.6☐ Replication thread per schema

☐ MySQL 5.7☐ Intra schema☐ Group commit large write sets

Confidential

Components and Topology: Multi Source Replication

☐ Slave replicates from multiple masters☐ No conflicting schemas

☐ Replication channels☐ Little support with replication tools☐ Unmanaged slave

☐ Useful for☐ Data Warehousing☐ Delayed slaves☐ Data locality

Confidential

Components and Topology: Multi Source Replication

Confidential

Components and Topology: Monitoring

☐ Replication Manager / Failover tool☐ Monitors replication stream☐ Monitor slave lag☐ Flap protection☐ Global state of the topology

☐ Performance monitoring (Trending)☐ Replication stream☐ Slave lag

Confidential

Components and Topology: Management

☐ Replication Manager / Failover tool☐ Repair broken replication☐ Scaling☐ Backup Management☐ Updates☐ Schema Changes☐ Configuration Management

Confidential

Components and Topology: Load Balancers

☐ Various types of load balancers☐ Layer 4 (TCP/IP)☐ Layer 7 (Application)

☐ Needs to monitor the topology

Confidential

Monitoring & Trending

Confidential

Confidential

Monitoring vs Trending

☐ Monitoring☐ Keeps an eye on your systems☐ Will alert if a threshold is met

☐ Trending☐ Insight into the system internals☐ Trends can warn you before anything has happened yet☐ Keep historical state/data

Confidential

Monitoring: Availability

☐ Do more than SELECT 1;☐ Measure true status of nodes and cluster☐ Test read/write☐ Open essential schemas☐ Check the full topology

Confidential

Trending: Why do we need trends?

☐ Trending☐ Plot trends of key (performance) metrics☐ Find problems before they arise☐ Pre-emptive problem management

☐ Trending tools☐ Granularity of sampling☐ More datapoints = better

Confidential

Trending: What metrics to store?

☐ Resource monitoring☐ CPU☐ Memory☐ IO capacity☐ Diskspace

☐ Replication monitoring☐ Slave lag☐ Transactions per second

Confidential

Confidential

Trending: Correlate

☐ Correlate your data☐ Include other metrics into your graphs

☐ Deployments☐ Failovers☐ Application metrics

Confidential

Confidential

Trending: Solutions

☐ Storage☐ Grafana☐ Prometheus☐ InfluxDB

☐ Trending tools☐ Grafana☐ Cacti

☐ Hosted☐ VividCortex

☐ Integrated Framework☐ ClusterControl

Confidential

Alerting

☐ Get notified when it goes wrong☐ Integrate alerting with monitoring tools

☐ Ready made integrations☐ APIs

☐ Alerting services☐ Pagerduty☐ OpsGenie☐ VictorOps

Confidential

Management

Confidential

Management: Scaling Out

☐ Add new slaves☐ Prime new slave with a copy of the data☐ Add to existing topology

☐ Monitoring☐ Failover tools☐ Load balancers

Confidential

Confidential

Confidential

Management: Scaling Out

☐ Performance issues:☐ Too many replication threads on master

☐ Add intermediate slaves☐ Adds more complexity☐ Not all failover tools support intermediate slaves

Confidential

Management: Delayed Slave

☐ Why slow down the replication?☐ Quick access to historical state☐ Accidentally wiped data☐ Hacked databases

☐ MySQL 5.7☐ CHANGE MASTER TO … MASTER DELAY=3600;

☐ Other versions☐ pt-slave-delay

Confidential

Management: Repair A Broken Topology

☐ MySQL replication is fragile☐ Master failure☐ Network problems☐ Data inconsistency (data drift)

Confidential

Management: Repair A Broken Topology

☐ Failover automation☐ Promote slaves automatically☐ Controlled failover☐ Move virtual ip addresses☐ Perform other pre- and post-failover tasks

☐ Failover tools☐ MySQL Master HA (MHA)☐ Percona Replication Manager (PRM)☐ Orchestrator☐ Mysqlfailover

Confidential

Management: Backups

☐ Logical backups☐ Dump of all tables and rows☐ Mysqldump

☐ Physical backups☐ File copy of the MySQL data directory☐ Xtrabackup☐ MySQL Hotcopy☐ Filesystem snapshots (LVM, ZFS)

Confidential

Management: Backups

☐ Why would you make a backup?☐ Disaster recovery☐ Testing/staging/development☐ Priming new slaves

☐ Replication is not a backup☐ DDL, Truncate and delete statements are replicated☐ Delayed slave will not help if your DC is down

Confidential

Management: Backups

☐ Full or incremental backups?☐ Xtrabackup allows incremental backups☐ Only copy altered data☐ Smaller backups☐ Full backup necessary as starting point☐ Slower recovery

Confidential

Management: Backups

☐ Scheduling☐ Off-peak hours scheduling☐ Don’t overlap schedules

☐ Backup suites☐ MySQL Enterprise backup☐ Bacula☐ Zmanda Recovery Manager (ZRM)☐ Holland Backup

Confidential

Management: Backups

☐ Backup validation☐ Unpack backups☐ Start MySQL from recovered backup☐ Rebuild slave from backup☐ Rebuild topology☐ Scheduled verification

☐ Offsite backups☐ Copy to another DC☐ Copy to cloud (Amazon Galcier / S3)

Confidential

Management: Updating or Upgrading

☐ Minor version updates☐ New features☐ Bugfixes☐ Security fixes☐ Easy to update, rolling restart

☐ Major version upgrades☐ Need preparation☐ Cluster downtime

Confidential

Management: Schema Changes

☐ DDL Changes propagated through replication☐ DDL statements are blocking the replication stream☐ Can cause slave lag

☐ Apply DDL Changes locally☐ Node by node change☐ Master failover necessary☐ DDL change has to be compatible

☐ Apply DDL Changes through Percona Online Schema Change

Confidential

Management: Schema Changes

☐ Percona Online Schema Change☐ Creates the new table structure☐ Creates triggers on the old table☐ Backfills data in the background☐ Swaps the two tables once done

☐ Not 100% compatible with all DDL changes☐ Always test your changes first!

Confidential

Management: Configuration Changes

☐ Manage your configurations☐ Deployment systems (Puppet, Chef, etc.)☐ Keep a copy inside Git/SVN

☐ Sync your configuration☐ Settings may differ per host☐ Runtime changes not propagated to configuration files☐ Visual differences (MonYog, ClusterControl)

Confidential

Load Balancing

Confidential

Load Balancing

☐ Ensure any component can/may fail☐ Master node is a single point of failure☐ Master failover must be handled application side as well

☐ Configuration managers (Zookeeper, Consul)☐ Virtual ip addresses (not always available / allowed)☐ Load Balancers

Confidential

Confidential

Confidential

Load Balancers: Benefits

☐ Transparent☐ Sits between application and database☐ Application does not have to be topology aware

☐ Split database connections (reads and writes)☐ Runtime read/write splitting

☐ Instant failover☐ Failover happens for all hosts at the same time

Confidential

Load Balancing: Proxies

☐ Which proxies are available today?☐ Layer 3 (network)

☐ HAProxy☐ LVS

☐ Layer 7 (application)☐ MySQL Proxy☐ MaxScale☐ ProxySQL

Confidential

Load Balancing: Proxies

☐ MaxScale☐ Understands SQL☐ Understands MySQL topologies☐ Runtime read/write splitting☐ Query rewriting☐ Can be resource intensive☐ Topology discovery may be different

Confidential

Load Balancing: Proxies

☐ ProxySQL☐ Understands SQL☐ Understands MySQL topologies☐ Runtime read/write splitting☐ Query rewriting☐ Query caching☐ Relatively new (GA since November 2015)

Confidential

Load Balancing: Proxies

☐ HAProxy☐ Fast☐ Proven technology☐ MySQL scripts readily available

Confidential

Load Balancing: Query Caching

☐ MySQL Query Cache flawed☐ Shared memory space☐ Locks on every read/write operation ( seconds)☐ Evictions take longer☐ High concurrency

☐ ProxySQL☐ No locks☐ Regular expression match☐ Caching for milliseconds☐ Performs similar as Redis and Memcache

Confidential

Load Balancing: Query Rewriting

☐ Why would you rewrite queries?☐ Bad performing queries☐ Broken queries (bad code drop)☐ Partitioning/sharding☐ Negate SQL injections / hacks

☐ Which proxies?☐ ProxySQL☐ MaxScale

Confidential

Our implementation

Confidential

ClusterControl: An Integrated Framework

Confidential

ClusterControl: Demo

Confidential

Q & A

Confidential

Additional Resources

☐ Ebook: MySQL Replication for High Availability☐ www.severalnines.com/whitepapers

☐ Become a MySQL DBA blog series☐ severalnines.com/blog-categories/db-ops

☐ Blog: MySQL Replication failover - MaxScale vs MHA☐ severalnines.com/blog/mysql-replication-failover-maxscale-vs-mha-part-1

☐ Load Balancing Tutorial☐ severalnines.com/tutorials/mysql-load-balancing-haproxy-tutorial

☐ We will soon have a whitepaper on the MySQL Replication Blueprint