oss4b: installing & managing mysql like a real devops

OSS4B: Installing & managing MySQL like a real devops

Automation in production

Frédéric Descamps20 Sep 2013

Who am I ?

• Frédéric Descamps “lefred”• @lefred• http://about.me/lefred• Percona Consultant since 2011• Managing MySQL since 3.23 (as far as

I remember)• devops believer

2

http://about.me/lefred

Agenda

• devops ?• Installation automation• Test automation• Schema management automation

3

devops ?

devopsis a movement willing to help

aligning IT with company needs

(starting by aligning devs and ops)

@endemics - Gildas Le Nadan - “The Devops movement”, Mixit Lyon 2011

We can say that devops is bringing together people

and practices from development and

operations

devops ?

We can say that devops is bringing together people

and practices from development and

operations

devops ? I am a happy dev !I am a happy op !

devops ?

devops is about CAMS

devops ?

CULTUREAUTOMATION

MEASUREMENTSHARING

@botchagalupe @damonedwards http://www.opscode.com/blog/2010/07/16/what-devops-means-to-me/

devops ?

What techniques can I use ?

MonitoringInfrastructure as Code

Continuous Improvement

devops & DBAs ?

What about DBAs ?

devops & DBAs !

Join the team, follow the

movement !

devdbaops

You are already between devs

and ops

devdbaops

You are already between devs

and ops

I am a happy DBA !

CAM

CULTUREAUTOMATION

MEASUREMENTSHARING

CAMS 1st principle: CULTURE

Communication !

Usually DEVS communicate with managers (all kinds of!)


Communication !

OPS communicate with support, ISP's and... their servers


Communication !

DBAs just don't communicate


The interaction between developers, sysadmins, dbas and all the actors

should begin from the start of any projects !

and do itin team !


Discuss everything database related with developers and sysadmins:

disks and raid design

schema design & partitions

additional columns & indexes

tables repair, ...


This is just a culture change

CAM

CULTUREAUTOMATIONMEASUREMENT

SHARING

CAMS 2nd principle: AUTOMATION

Can we automate DBAs job ?

Should we ??

Even if we have only one database ?


YES !YES !


A lot of things to take care

binaries installation

configs

cron jobs

monitoring

grants

databases

replication


Documentation is needed and must be up to date


Everything needs to be reproducible

quickly without missing anything !


The “easy” part to automate is the part

close to the dark side of the force: sysadmins


Installation, configuration, cron jobs, monitoring and even grants can be automated using a configuration management system

Consider your infrastructure as code


Example

node percona1 { include percona::repository include percona::toolkit

Class['percona::repository'] > Class['percona::server']

class { 'percona::server': mysql_version => "5.6", perconaserverid => "1", enable => "true", ensure => "running" }}


But how do you manage schemas ?

Is it automated ?

And in production ?

V 1 V 2

Database Schema Migration

There are some existing tools:– Liquibase (probably the best right now)

– Flyway

– c5-db-migration

– dbdeploy

– MyBatis

– Autopatch

– Sqitch


Most of them are ...

Integrated with a framework or an ORM


Most of them are ...

for JAVA applications


Most of them use ...

XML definitions

What is easier ? This:

<?xml version="1.0" encoding="UTF8"?> <databaseChangeLogxmlns="http://www.liquibase.org/xml/ns/dbchangelog"xmlns:xsi="http://www.w3.org/2001/XMLSchemainstance"xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangeloghttp://www.liquibase.org/xml/ns/dbchangelog/dbchangelog2.0.xsd"> <changeSet id="1" author="lefred"><createTable tableName="department"><column name="id" type="int"><constraints primaryKey="true" nullable="false"/></column><column name="name" type="varchar(50)"><constraints nullable="false"/></column><column name="active" type="boolean" defaultValueBoolean="true"/></createTable></changeSet> </databaseChangeLog>

Or this ?

CREATE TABLE department (id INT NOT NULL PRIMARY KEY,name VARCHAR(50) NOT NULL

);


also they are...

blocking

schema changes


and finally...

not made to be used

via a configuration management system

Is there an alternative ?

Yes ;-)

online-migration

Online-migration: what is it ?

standalone Python script

command line only

Uses Oracle mysql-utilities

Uses pt-online-schema-change

Uses normal SQL statements

Can be easily integrated with puppet as a type and a provider are available

Online-migration: features

schema versioning and status

$ ./onlinemigration.py status worldMigration of schema world :+++++| VERSION | APPLIED | STATUS | COMMENT |+++++| 0000 | 20130222 09:17:16 | ok | Initial file || 0001 | none | pending | none || 0002 | none | pending | none || 0003 | none | pending | add nice column to cit |+++++


schema migration up$ ./onlinemigration.py up world to 3INFO : You want to migrate up to version 0003INFO : Ok this version is pendingINFO : Preparing migration to version 0001INFO : Applied changes match the requested schemaINFO : Preparing migration to version 0002INFO : Applied changes match the requested schemaINFO : Preparing migration to version 0003INFO : Applied changes match the requested schema

$ ./onlinemigration.py status worldMigration of schema world :+++++| VERSION | APPLIED | STATUS | COMMENT |+++++| 0000 | 20130222 09:17:16 | ok | Initial file || 0001 | 20130222 09:17:46 | ok | none || 0002 | 20130222 09:18:24 | ok | none || 0003 | 20130222 09:18:36 | ok | add nice column to cit |+++++


schema checksum

$ ./onlinemigration.py status worldMigration of schema world :+++++| VERSION | APPLIED | STATUS | COMMENT |+++++| 0000 | 20130222 09:17:16 | ok | Initial file || 0001 | 20130222 09:17:46 | checksum problem | none |+++++

$ ./onlinemigration.py checksum worldworld's current schema checksum = 3239e7245d98d27681020e36d0085676


schema migration down (rollback)$ ./onlinemigration.py down world 2INFO : You want to migrate down 2 version(s)INFO : rollback from 0003 to 0002INFO : rollback from 0002 to 0001

$ ./onlinemigration.py status worldMigration of schema world :+++++| VERSION | APPLIED | STATUS | COMMENT |+++++| 0000 | 20130222 09:17:16 | ok | Initial file || 0001 | 20130222 09:17:46 | ok | none || 0002 | 20130222 09:18:24 | rollback | none || 0003 | 20130222 09:18:36 | rollback | add nice column to cit || 0002 | none | pending | none || 0003 | none | pending | add nice column to cit |+++++


schema dif$ ./onlinemigration.py diff worldWARNING : Schema of world doesn't have expected checksum(4478d85870969436400bac023f2b2b7c)

TABLE `City` `CountryCode` varchar(10) DEFAULT NULL,+ `CountryCode` char(3) NOT NULL DEFAULT '',

Online-migration with puppet

Example

node percona1 { include percona::repository include percona::toolkit include onlinemigration

Class['percona::repository'] > Class['percona::server']

mysql_schema { "world": ensure => present, require => [ Package["onlinemigration"],

Service['mysql'] ], version => 1, cwd => "/root/om", } ...}

CAM

CULTUREAUTOMATION

MEASUREMENTSHARING

CAMS 3rd & 4th principles: MEASUREMENT & SHARING

This can be summarized in two other words :

TEST&

VISIBILITY

TEST

What should you test as a

devops DBA ?

TEST as a devops DBA

Test if you can deploy everything and you didn't forget anything

Test every changes (even small configuration change)

Test also the performance ! Is it faster with this change ?

Automate your tests !

TEST as a devops DBA

To test performance you can use several tools from Percona like:– pt-query-digest

– pt-upgrade

– Percona-playback

Visibility

Share the following metrics with everybody in

your company

Visibility: tests results

Share the results of the tests:

jenkins dashboard

Visibility: monitoring

Monitor your database with an alerting system

(nagios)


But don't alert too much !


Your alerting system should answer those questions:

Is the database server running ?

Does it allow connections

Is replication lagging on production slaves ?

Visibility: trending

The most interesting facts for a DBA are the trending results !

Use cacti with Percona Monitoring Plugins, they provide very good metrics

http://www.percona.com/doc/perconamonitoringplugins/cacti/mysqltemplates.html

Visibility: trending

Correlate changes in the trending results with your

configuration changes...

everything should be tracked

Time for a demo ?

Installation of a development and a production environment (using vagrant)

You can view this part of demo at http://tinyurl.com/kafboua

http://tinyurl.com/kafboua

demo

Let's create a schema for our application in development

And push it to production

demo

Now let's modify the schema

And deploy it to production

demo

Now let's modify the schema

And deploy it to production

V 0 empty database “world”

demo

v1 (release 1 of the schema)

SQL can be downloaded at http://tinyurl.com/n3upcyq

Demo available at http://tinyurl.com/k7lm2yf

http://tinyurl.com/n3upcyq

http://tinyurl.com/k7lm2yf

demo

v2

SQL can be downloaded at http://tinyurl.com/ld7pwf7

alter table City add PolutionLevel int;alter table City add index pol_idx(PolutionLevel);

http://tinyurl.com/ld7pwf7

demo

v3

SQL can be downloaded at http://tinyurl.com/knqx4ql

alter table City add nice tinyint default 1;

http://tinyurl.com/knqx4ql

Warning !

A current bug in pt-online-schema-change prevents the program to work as expected when a table has constraints

https://bugs.launchpad.net/percona-toolkit/+bug/1207186



demo

Now let's upgrade this in production

Preparation

– copy all meta files into our puppet module

– specify the version in our manifest

Deployment

– run puppet

Demo at http://tinyurl.com/mq6sw33

http://tinyurl.com/mq6sw33

demo

mysql_schema { "world": ensure => present, require => [

Package["onlinemigration"], Service['mysql']

], version => 1, cwd => "/root/om", }

...info: Mysql_schema[world](provider=mysql_schema): performing the check, last_version is 0info: Mysql_schema[world](provider=mysql_schema): we need to migrate to another version (1)info: Mysql_schema[world](provider=mysql_schema): performing the migration...

Resources

Percona – http://www.percona.com

Percona playback - http://www.percona.com/doc/percona-playback/

online-migration - https://github.com/lefred/online-migration

pt-online-schema-change - http://www.percona.com/doc/percona-toolkit/pt-online-schema-change.html

mysql-utilities - http://dev.mysql.com/downloads/tools/utilities/1.3.html

Vagrant schema versioning environment - https://github.com/lefred/schemas-versioning

shelr.tv – http://shelr.tv/users/4f85fb1b9660807969000005

http://www.percona.com/

http://www.percona.com/doc/percona-playback/

https://github.com/lefred/online-migration

http://www.percona.com/doc/percona-toolkit/pt-online-schema-change.html

http://dev.mysql.com/downloads/tools/utilities/1.3.html

https://github.com/lefred/schemas-versioning

http://shelr.tv/users/4f85fb1b9660807969000005

Percona provides

24 x 7 Support Services Quick and Easy Access to Consultants Same Day Emergency Data Recovery Remote DBA Services QUOTE OSS4B for Special Consulting Offer – Free Mini Health Audit with Gold Support and Free Performance Audit with Platinum Support Valid September-October 2013

[email protected] or 00442081330309

mailto:[email protected]

Thank you !

Questions?

oss4b: installing & managing mysql like a real devops

Technology