oss4b: installing & managing mysql like a real devops

OSS4B: Installing & managing MySQL like a real devops

Automation in production

Frédéric Descamps20 Sep 2013

Who am I ?

• Frédéric Descamps “lefred”• @lefred• http://about.me/lefred• Percona Consultant since 2011• Managing MySQL since 3.23 (as far as

I remember)• devops believer

2

Agenda

• devops ?• Installation automation• Test automation• Schema management automation

3

devops ?

devopsis a movement willing to help

aligning IT with company needs

(starting by aligning devs and ops)

@endemics - Gildas Le Nadan - “The Devops movement”, Mixit Lyon 2011

We can say that devops is bringing together people

and practices from development and

operations

devops ?

We can say that devops is bringing together people

and practices from development and

operations

devops ? I am a happy dev !I am a happy op !

devops ?

devops is about CAMS

devops ?

CULTUREAUTOMATION

MEASUREMENTSHARING

@botchagalupe @damonedwards http://www.opscode.com/blog/2010/07/16/what-devops-means-to-me/

devops ?

What techniques can I use ?

MonitoringInfrastructure as Code

Continuous Improvement

devops & DBAs ?

What about DBAs ?

devops & DBAs !

Join the team, follow the

movement !

devdbaops

You are already between devs

and ops

devdbaops

You are already between devs

and ops

I am a happy DBA !

CAM

CULTUREAUTOMATION

MEASUREMENTSHARING

CAMS 1st principle: CULTURE

Communication !

Usually DEVS communicate with managers (all kinds of!)

CAMS 1st principle: CULTURE

Communication !

OPS communicate with support, ISP's and... their servers

CAMS 1st principle: CULTURE

Communication !

DBAs just don't communicate

CAMS 1st principle: CULTURE

The interaction between developers, sysadmins, dbas and all the actors

should begin from the start of any projects !

and do itin team !

CAMS 1st principle: CULTURE

Discuss everything database related with developers and sysadmins:

disks and raid design

schema design & partitions

additional columns & indexes

tables repair, ...

CAMS 1st principle: CULTURE

This is just a culture change

CAM

CULTUREAUTOMATIONMEASUREMENT

SHARING

CAMS 2nd principle: AUTOMATION

Can we automate DBAs job ?

Should we ??

Even if we have only one database ?

CAMS 2nd principle: AUTOMATION

CAMS 2nd principle: AUTOMATION

YES !YES !

CAMS 2nd principle: AUTOMATION

A lot of things to take care

binaries installation

configs

cron jobs

monitoring

grants

databases

replication

CAMS 2nd principle: AUTOMATION

Documentation is needed and must be up to date

CAMS 2nd principle: AUTOMATION

Everything needs to be reproducible

quickly without missing anything !

CAMS 2nd principle: AUTOMATION

The “easy” part to automate is the part

close to the dark side of the force: sysadmins

CAMS 2nd principle: AUTOMATION

The “easy” part to automate is the part

close to the dark side of the force: sysadmins

CAMS 2nd principle: AUTOMATION

Installation, configuration, cron jobs, monitoring and even grants can be automated using a configuration management system

Consider your infrastructure as code

CAMS 2nd principle: AUTOMATION

Example

node percona1 {        include percona::repository        include percona::toolkit

        Class['percona::repository'] ­> Class['percona::server']

        class {                'percona::server':                        mysql_version   => "5.6",                        perconaserverid => "1",                        enable          => "true",                        ensure          => "running"        }}

CAMS 2nd principle: AUTOMATION

But how do you manage schemas ?

Is it automated ?

And in production ?

V 1 V 2

Database Schema Migration

There are some existing tools:– Liquibase (probably the best right now)

– Flyway

– c5-db-migration

– dbdeploy

– MyBatis

– Autopatch

– Sqitch

Database Schema Migration

Most of them are ...

Integrated with a framework or an ORM

Database Schema Migration

Most of them are ...

for JAVA applications

Database Schema Migration

Most of them use ...

XML definitions

What is easier ? This:

<?xml version="1.0" encoding="UTF­8"?> <databaseChangeLogxmlns="http://www.liquibase.org/xml/ns/dbchangelog"xmlns:xsi="http://www.w3.org/2001/XMLSchema­instance"xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangeloghttp://www.liquibase.org/xml/ns/dbchangelog/dbchangelog­2.0.xsd"> <changeSet id="1" author="lefred"><createTable tableName="department"><column name="id" type="int"><constraints primaryKey="true" nullable="false"/></column><column name="name" type="varchar(50)"><constraints nullable="false"/></column><column name="active" type="boolean" defaultValueBoolean="true"/></createTable></changeSet> </databaseChangeLog>

Or this ?

CREATE TABLE department (id INT NOT NULL PRIMARY KEY,name VARCHAR(50) NOT NULL

);

Database Schema Migration

also they are...

blocking

schema changes

Database Schema Migration

and finally...

not made to be used

via a configuration management system

Is there an alternative ?

Yes ;-)

online-migration

Online-migration: what is it ?

standalone Python script

command line only

Uses Oracle mysql-utilities

Uses pt-online-schema-change

Uses normal SQL statements

Can be easily integrated with puppet as a type and a provider are available

Online-migration: features

schema versioning and status

$ ./online­migration.py status worldMigration of schema world :+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­+| VERSION | APPLIED             | STATUS           | COMMENT                |+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­+|    0000 | 2013­02­22 09:17:16 |               ok |           Initial file ||    0001 |                none |          pending |                   none ||    0002 |                none |          pending |                   none ||    0003 |                none |          pending | add nice column to cit |+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­+

Online-migration: features

schema migration up$ ./online­migration.py up world to 3INFO : You want to migrate up to version 0003INFO : Ok this version is pendingINFO : Preparing migration to version 0001INFO : Applied changes match the requested schemaINFO : Preparing migration to version 0002INFO : Applied changes match the requested schemaINFO : Preparing migration to version 0003INFO : Applied changes match the requested schema

$ ./online­migration.py status worldMigration of schema world :+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­+| VERSION | APPLIED             | STATUS           |                COMMENT |+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­+|    0000 | 2013­02­22 09:17:16 |               ok |           Initial file ||    0001 | 2013­02­22 09:17:46 |               ok |                   none ||    0002 | 2013­02­22 09:18:24 |               ok |                   none ||    0003 | 2013­02­22 09:18:36 |               ok | add nice column to cit |+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­+

Online-migration: features

schema checksum

$ ./online­migration.py status worldMigration of schema world :+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­+| VERSION | APPLIED             | STATUS           |                COMMENT |+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­+|    0000 | 2013­02­22 09:17:16 |               ok |           Initial file ||    0001 | 2013­02­22 09:17:46 | checksum problem |                   none |+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­+

$ ./online­migration.py checksum worldworld's current schema checksum = 3239e7245d98d27681020e36d0085676

Online-migration: features

schema migration down (rollback)$ ./online­migration.py down world 2INFO : You want to migrate down 2 version(s)INFO : rollback from 0003 to 0002INFO : rollback from 0002 to 0001

$ ./online­migration.py status worldMigration of schema world :+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­+| VERSION | APPLIED             | STATUS           |                COMMENT |+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­+|    0000 | 2013­02­22 09:17:16 |               ok |           Initial file ||    0001 | 2013­02­22 09:17:46 |               ok |                   none ||    0002 | 2013­02­22 09:18:24 |         rollback |                   none ||    0003 | 2013­02­22 09:18:36 |         rollback | add nice column to cit ||    0002 |                none |          pending |                   none ||    0003 |                none |          pending | add nice column to cit |+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­+

Online-migration: features

schema dif$ ./online­migration.py diff worldWARNING : Schema of world doesn't have expected checksum(4478d85870969436400bac023f2b2b7c)

TABLE `City`­ `CountryCode` varchar(10) DEFAULT NULL,+ `CountryCode` char(3) NOT NULL DEFAULT '',

Online-migration with puppet

Example

node percona1 { include percona::repository include percona::toolkit include online­migration

        Class['percona::repository'] ­> Class['percona::server']

        mysql_schema { "world":                     ensure  => present,         require => [ Package["online­migration"], 

   Service['mysql'] ],                     version => 1,                      cwd     => "/root/om",        }        ...}

CAM

CULTUREAUTOMATION

MEASUREMENTSHARING

CAMS 3rd & 4th principles: MEASUREMENT & SHARING

This can be summarized in two other words :

TEST&

VISIBILITY

TEST

TEST

What should you test as a

devops DBA ?

TEST as a devops DBA

Test if you can deploy everything and you didn't forget anything

Test every changes (even small configuration change)

Test also the performance ! Is it faster with this change ?

Automate your tests !

TEST as a devops DBA

To test performance you can use several tools from Percona like:– pt-query-digest

– pt-upgrade

– Percona-playback

Visibility

Share the following metrics with everybody in

your company

Visibility: tests results

Share the results of the tests:

jenkins dashboard

Visibility: monitoring

Monitor your database with an alerting system

(nagios)

Visibility: monitoring

But don't alert too much !

Visibility: monitoring

Your alerting system should answer those questions:

Is the database server running ?

Does it allow connections

Is replication lagging on production slaves ?

Visibility: trending

The most interesting facts for a DBA are the trending results !

Use cacti with Percona Monitoring Plugins, they provide very good metrics

http://www.percona.com/doc/percona­monitoring­plugins/cacti/mysql­templates.html

Visibility: trending

Correlate changes in the trending results with your

configuration changes...

everything should be tracked

Time for a demo ?

Installation of a development and a production environment (using vagrant)

You can view this part of demo at http://tinyurl.com/kafboua

demo

Let's create a schema for our application in development

And push it to production

demo

Now let's modify the schema

And deploy it to production

demo

Now let's modify the schema

And deploy it to production

V 0 empty database “world”

demo

v1 (release 1 of the schema)

SQL can be downloaded at http://tinyurl.com/n3upcyq

Demo available at http://tinyurl.com/k7lm2yf

demo

v2

SQL can be downloaded at http://tinyurl.com/ld7pwf7

alter table City    add PolutionLevel int;alter table City    add index pol_idx(PolutionLevel);

demo

v3

SQL can be downloaded at http://tinyurl.com/knqx4ql

alter table City    add nice tinyint default 1;

Warning !

A current bug in pt-online-schema-change prevents the program to work as expected when a table has constraints

https://bugs.launchpad.net/percona-toolkit/+bug/1207186

demo

Now let's upgrade this in production

Preparation

– copy all meta files into our puppet module

– specify the version in our manifest

Deployment

– run puppet

Demo at http://tinyurl.com/mq6sw33

demo

 mysql_schema { "world":                        ensure  => present,                        require => [ 

Package["online­migration"],  Service['mysql'] 

 ],                        version => 1,                        cwd     => "/root/om",        }

...info: Mysql_schema[world](provider=mysql_schema): performing the check, last_version is 0info: Mysql_schema[world](provider=mysql_schema): we need to migrate to another version (1)info: Mysql_schema[world](provider=mysql_schema): performing the migration...

Resources

Percona – http://www.percona.com

Percona playback - http://www.percona.com/doc/percona-playback/

online-migration - https://github.com/lefred/online-migration

pt-online-schema-change - http://www.percona.com/doc/percona-toolkit/pt-online-schema-change.html

mysql-utilities - http://dev.mysql.com/downloads/tools/utilities/1.3.html

Vagrant schema versioning environment - https://github.com/lefred/schemas-versioning

shelr.tv – http://shelr.tv/users/4f85fb1b9660807969000005

Percona provides

24 x 7 Support Services Quick and Easy Access to Consultants Same Day Emergency Data Recovery Remote DBA Services QUOTE OSS4B for Special Consulting Offer – Free Mini Health Audit with Gold Support and Free Performance Audit with Platinum Support Valid September-October 2013

sales@percona.com or 00442081330309

Thank you !

Questions?