database version control and deployment - model or migration scripts

@_AlexYates_

#SQLRelay

State-Based or Migrations-Based Database Development?

A Review of the Pros and Cons…Alex Yates

@_AlexYates_

#SQLRelay

DLM Consultant

workingwithdevs.com

alex.yates@dlmconsultants.com

Alex Yates

@_AlexYates_

@_AlexYates_

#SQLRelay

@_AlexYates_ | #SQLRelay

@_AlexYates_

#SQLRelay

AgileScrumLeanIterativeContinuous Delivery

Developers

@_AlexYates_

#SQLRelay

MonitoringDeployment

IntegrityPerformance

DBAs

@_AlexYates_

#SQLRelay

@_AlexYates_

#SQLRelay

Woah! Deployment fail?

@_AlexYates_

#SQLRelay

#worksOnMyMachine

Woah! Deployment fail?

@_AlexYates_

#SQLRelay

A cursor?! You just CAN’T do that?!

@_AlexYates_

#SQLRelay

#worksOnMyMachine

A cursor?! You just CAN’T do that?!

@_AlexYates_

#SQLRelay

Hey, you just dropped my hot-fix!

@_AlexYates_

#SQLRelay

#worksOnMyMachine

Hey, you just dropped my hot-fix!

@_AlexYates_

#SQLRelay

Farm Credit Services of America (FCSA)

• 100 person IT team, 14 sub-teams• Database version control inconsistent

• Deployment process manual• Delivery was slow and

unreliable

@_AlexYates_

#SQLRelay

Farm Credit Services of America (FCSA)

• Deployments easier to review

• Delivery faster and more reliable

• Standardised source control and delivery process

• Automated manual deployment tasks

@_AlexYates_

#SQLRelay

The automated deployment pipeline

@_AlexYates_

#SQLRelay

Databases are hard

• Schema changes vs existing data• Reference data vs production data• Teamwork and testing• Database drift (change outside

process, e.g. production hot-fixes)

@_AlexYates_

#SQLRelay

There’s more than one way to skin a cat

@_AlexYates_

#SQLRelay

There’s more than one way to skin automatea cat

@_AlexYates_

#SQLRelay

V1 V2

@_AlexYates_

#SQLRelay

V1 V2

Migrations-based solutions

@_AlexYates_

#SQLRelay

V1 V2

State-based solutions

@_AlexYates_

#SQLRelay

@_AlexYates_

#SQLRelay

“There's nothing more reliable than keeping track of exactly the scripts you intend to run, and running them, without trying to compare state and guess.”

@_AlexYates_

#SQLRelay

“There's nothing more reliable than keeping track of exactly the scripts you intend to run, and running them, without trying to compare state and guess.”

Paul Stovell, Octopus Deploy

http://docs.octopusdeploy.com/display/OD/SQL+Server+databases

@_AlexYates_

#SQLRelay

“As soon as you have multiple changes on a single aspect of an object, ordering and the ability to detect which change needs to be made gets very complicated.”

@_AlexYates_

#SQLRelay

“As soon as you have multiple changes on a single aspect of an object, ordering and the ability to detect which change needs to be made gets very complicated.”

Gert Drapers, built DataDude

https://blogs.msdn.microsoft.com/gertd/2009/06/05/declarative-database-development/

@_AlexYates_

#SQLRelay

Migrations vs state

http://workingwithdevs.com/delivering-databases-migrations-vs-state/

@_AlexYates_

#SQLRelay

Migrations vs state

http://workingwithdevs.com/delivering-databases-migrations-vs-state/

@_AlexYates_

#SQLRelay

Migrations vs state

http://workingwithdevs.com/delivering-databases-migrations-vs-state/

@_AlexYates_

#SQLRelay

Migrations vs state

http://workingwithdevs.com/delivering-databases-migrations-vs-state/

@_AlexYates_

#SQLRelay

Migrations vs state

http://workingwithdevs.com/delivering-databases-migrations-vs-state/

@_AlexYates_

#SQLRelay

Migrations vs state

http://workingwithdevs.com/delivering-databases-migrations-vs-state/

@_AlexYates_

#SQLRelay

Migrations vs state

http://workingwithdevs.com/delivering-databases-migrations-vs-state/

@_AlexYates_

#SQLRelay

11

2 Create view: kittenTrainersNrLdn

Selects only: kittenTrainers.FullName

Edit table: kittenTrainersAdd column: AcceptsTigerCubs BIT

Both work!Same changes, versioned and deployed two ways

@_AlexYates_

#SQLRelay

Demo

@_AlexYates_

#SQLRelay

11

2 Create view: kittenTrainersNrLdn

Selects only: kittenTrainers.FullName

Edit table: kittenTrainersAdd column: AcceptsTigerCubs BIT

Both work!Same changes, versioned and deployed two ways

@_AlexYates_

#SQLRelay

11Edit table: kittenTrainersSplit column trainerFullName

into trainerFirstName and trainerLastName

But both suck.A tale of two work items…

2

Edit view: kittenTrainersNrLdnAdd column: AcceptsTigerCubs

@_AlexYates_

#SQLRelay

@_AlexYates_

#SQLRelay

0124_edit_view.sqlALTER VIEW kittenTrainersNrLdnAS SELECT fullName, acceptsTigerCubs FROM kittenTrainers

@_AlexYates_

#SQLRelay

0124_edit_view.sql 0125_edit_table.sqlALTER VIEW kittenTrainersNrLdnAS SELECT fullName, acceptsTigerCubs FROM kittenTrainers

ALTER TABLE kittenTrainers

*FUN WITH STRINGS* END

ALTER VIEW kittenTrainersNrLdnAS SELECT firstName, lastName FROM kittenTrainers

@_AlexYates_

#SQLRelay

ALTER TABLE kittenTrainers

*FUN WITH STRINGS* END

ALTER VIEW kittenTrainersNrLdnAS SELECT firstName, lastName FROM kittenTrainers

0124_edit_view.sql 0125_edit_table.sql

!

ALTER VIEW kittenTrainersNrLdnAS SELECT fullName, acceptsTigerCubs FROM kittenTrainers

@_AlexYates_

#SQLRelay

Conflicts easily missedChanges overwritten, hard to spot

Order mattersLast script wins

The “winding path” problem

How do you fix a bug that has been deployed to some environments but not others?

The problem with migrations

@_AlexYates_

#SQLRelay

V123kittenTrainersNrLdn.sqlCREATE VIEW kittenTrai…AS SELECT fullName FROM kittenTrainers

@_AlexYates_

#SQLRelay

V124kittenTrainersNrLdn.sqlCREATE VIEW kittenTrai…AS SELECT fullName, acceptsTigerCubs FROM kittenTrainers

V123kittenTrainersNrLdn.sqlCREATE VIEW kittenTrai…AS SELECT fullName FROM kittenTrainers

@_AlexYates_

#SQLRelay

V125kittenTrainersNrLdn.sqlCREATE VIEW kittenTrai…AS SELECT firstName, lastName FROM kittenTrainers

V124kittenTrainersNrLdn.sqlCREATE VIEW kittenTrai…AS SELECT fullName, acceptsTigerCubs FROM kittenTrainers

V123kittenTrainersNrLdn.sqlCREATE VIEW kittenTrai…AS SELECT fullName FROM kittenTrainers

@_AlexYates_

#SQLRelay

V125kittenTrainersNrLdn.sqlCREATE VIEW kittenTrai…AS SELECT firstName, lastName FROM kittenTrainers

V124kittenTrainersNrLdn.sqlCREATE VIEW kittenTrai…AS SELECT fullName, acceptsTigerCubs FROM kittenTrainers

V123kittenTrainersNrLdn.sqlCREATE VIEW kittenTrai…AS SELECT fullName FROM kittenTrainers

!

@_AlexYates_

#SQLRelay

@_AlexYates_

#SQLRelay

REVISION 123 REVISION 124kittenTrainers.sqlCREATE TABLE kittenTrainers(fullName,

acceptsTigerCubs)kittenTrainersNrLdn.sqlCREATE VIEW kittenTrainersN…AS SELECT fullName

FROM kittenTrainers

kittenTrainers.sqlCREATE TABLE kittenTrainers(firstName, lastName, acceptsTigetCubs)

kittenTrainersNrLdn.sqlCREATE VIEW kittenTrainersN…AS SELECT firstName, lastName, acceptsTigerCubs FROM kittenTrainers

@_AlexYates_

#SQLRelay

Diff script:

DROP COLUMN fullName

ALTER TABLE kittenTrainers ADD FirstName NVARCHAR(50), LastName NVARCHAR(50)

ALTER VIEW kittenTrainersNrLdn AS  …

@_AlexYates_

#SQLRelay

Diff script:

DROP COLUMN fullName

ALTER TABLE kittenTrainers ADD FirstName NVARCHAR(50), LastName NVARCHAR(50)

ALTER VIEW kittenTrainersNrLdn AS  …

!

@_AlexYates_

#SQLRelay

Script:

DROP COLUMN fullName

ALTER TABLE kittenTrai… ADD FirstName, LastName

ALTER VIEW kittenTrain… AS  …

Need to understand your tool

It should be obvious to you that your tool won’t work

What is the Plan B / override?

Because one day you’ll need it

Test for data loss Automatically (naturally)

The problem with state

@_AlexYates_

#SQLRelay

So what is better?

State MigrationsVS

@_AlexYates_

#SQLRelay

State Easier (less control) Better for

sprocs/functions Better for

large/distributed teams Better for frequent

changes Better for dependency

nightmares Drift: rolled back

Migrations More control

(harder/needs discipline) Better for data

migrations Better for small teams Better for infrequent

changes Better for simple data

stores Drift: ignored

VS

@_AlexYates_

#SQLRelay

State Easier (less control) Better for

sprocs/functions Better for

large/distributed teams Better for frequent

changes Better for dependency

nightmares Drift: rolled back Better for development

Migrations More control

(harder/needs discipline) Better for data

migrations Better for small teams Better for infrequent

changes Better for simple data

stores Drift: ignored Better for automation

VS

@_AlexYates_

#SQLRelay

Pick appropriate tooling

State

DevArt SchemaCompare

DB Maestro

Migrations

DbUp

Flyway

Hybrid

@_AlexYates_

#SQLRelay

Let’s talk about:

https://msdn.microsoft.com/en-us/library/mt204009.aspx

@_AlexYates_

#SQLRelay

How does SSDT work?

Most scripts do not require manual scripting

Manual scripting causes problems

@_AlexYates_

#SQLRelay

How does SQL Source Control work?

Most scripts do not require manual scripting

Manual scripting causes problems

Perhaps just these ones?So only write these ones

@_AlexYates_

#SQLRelay

How does SSDT work?https://msdn.microsoft.com/en-US/library/hh272704(v=vs.103).aspx

https://msdn.microsoft.com/en-us/library/jj889461(v=vs.103).aspx

<< RefactorLog

Pre/Post-Deploy scripts >>

@_AlexYates_

#SQLRelay

Let’s talk about:

http://www.red-gate.com/sql-source-control/

@_AlexYates_

#SQLRelay

How does SQL Source Control work?

(This diagram again)We still only need a few upgrade scripts

@_AlexYates_

#SQLRelay

How does SQL Source Control work?

@_AlexYates_

#SQLRelay

How does SQL Source Control work?

https://documentation.red-gate.com/display/SOC5/How+migration+scripts+work c

@_AlexYates_

#SQLRelay

Let’s talk about:

http://www.red-gate.com/readyroll/

@_AlexYates_

#SQLRelay

State Easier (less control) Better for

sprocs/functions Better for

large/distributed teams Better for frequent

changes Better for dependency

nightmares Drift: rolled back

Migrations More control

(harder/needs discipline) Better for data

migrations Better for small teams Better for infrequent

changes Better for simple data

stores Drift: ignored

Better for sprocs/functions Better data migrations

How does ReadyRoll work?

@_AlexYates_

#SQLRelay

How does ReadyRoll work?Programmable objects

(Views, stored procedures etc)V125myproc.sqlEXEC sp_rename 'table_foo', 'table_bar‘

Migrations(Tables, reference data etc)

@_AlexYates_

#SQLRelay

How does ReadyRoll work?Programmable objects

(Views, stored procedures etc)V125myproc.sqlEXEC sp_rename 'table_foo', 'table_bar‘

Migrations(Tables, reference data etc)

@_AlexYates_

#SQLRelay

How does ReadyRoll work?Programmable objects

(Views, stored procedures etc)V125myproc.sqlEXEC sp_rename 'table_foo', 'table_bar‘

Migrations(Tables, reference data etc)

https://documentation.red-gate.com/display/RR1/Programmable+Objects

@_AlexYates_

#SQLRelay

But haven’t we been doing hybrid for years?

@_AlexYates_

#SQLRelay

ProdTestDev

@_AlexYates_

#SQLRelay

ProdTestDev

@_AlexYates_

#SQLRelay

ProdTestDev S0

@_AlexYates_

#SQLRelay

ProdTestDev S0

S0

@_AlexYates_

#SQLRelay

ProdTestDev S0

S0

State Migrations

@_AlexYates_

#SQLRelay

ProdTest

DevS0

“Late migrations”

S1

ProdTest

Dev S0

“Early migrations”

@_AlexYates_

#SQLRelay

Early vs Late

• Test deploy script early

• Easier to automate

• Developers own deployments (typically)

• Optimised deployment scripts (no winding path)

• Easier to intervene

• DBAs own deployments (typically)

@_AlexYates_

#SQLRelay

Drift

@_AlexYates_

#SQLRelay

@_AlexYates_

#SQLRelay

“When making changes directly on production, you are making a decision that the delay due to poor cycle time is more expensive than the risk of making a mistake.”

@_AlexYates_

#SQLRelay

“When making changes directly on production, you are making a decision that the delay due to poor cycle time is more expensive than the risk of making a mistake.”

Alex Yates, Built this slide

@_AlexYates_

#SQLRelay

The bad stuff:

• Accidental roll-backs (state)

• Failed deployments (migrations)

• Environment inconsistency (migrations)

@_AlexYates_

#SQLRelay

DriftStrategies to help

Improve cycle time Strict security policies Monitor drift (play with DDL

triggers) Redgate DLM Dashboard

(free)https://www.simple-talk.com/sql/database-administration/database-deployment-the-bits-database-version-drift/

@_AlexYates_

#SQLRelay

DriftStrategies to help

Improve cycle time Strict security policies Monitor drift (play with DDL

triggers) Redgate DLM Dashboard

(free)http://www.red-gate.com/products/dlm/dlm-dashboard/

@_AlexYates_

#SQLRelay

Demo

@_AlexYates_

#SQLRelay

Farm Credit Services of America (FCSA)

• Standardised DLM processes• FCSA delivering much more efficiently

• The FCSA model being rolled out at FCMA (sister org)

• The team all have excellent CV’s

@_AlexYates_

#SQLRelay

@_AlexYates_

#SQLRelay

Siloed sparrows suck at DLM

@_AlexYates_

#SQLRelay

Heated hippos are closed minded

@_AlexYates_

#SQLRelay

State Easier (less control) Better for

sprocs/functions Better for

large/distributed teams Better for frequent

changes Better for dependency

nightmares Drift: rolled back Better for development

Migrations More control

(harder/needs discipline) Better for data

migrations Better for small teams Better for infrequent

changes Better for simple data

stores Drift: ignored Better for automation

VS

Clever people consider options

@_AlexYates_

#SQLRelay

State

DevArt SchemaCompare

DB Maestro

Migrations

DbUp

Flyway

Hybrid

Hybrid tools are awesomeTools are pretty good

@_AlexYates_

#SQLRelay

ProdTest

DevS0

“Late migrations”

S1

ProdTest

Dev S0

“Early migrations”

It’s a balance, not a choice

@_AlexYates_

#SQLRelay

Hunt his smug face - make pain on it

@_AlexYates_

#SQLRelay

This stuff makes a big difference

http://www.codeaperture.io/2016/09/13/how-redgate-helped-define-our-process/

@_AlexYates_

#SQLRelay

Questions?

@_AlexYates_

#SQLRelay

DLM Consultant

workingwithdevs.com

alex.yates@dlmconsultants.com

Alex Yates

@_AlexYates_

@_AlexYates_

#SQLRelay

PREMIER SPONSOR

GOLD SPONSORS

SILVER SPONSORS

BRONZE SPONSORS

SUPPORTERS

@_AlexYates_

#SQLRelay

Please give us your feedback:

sqlrelay.co.uk/feedbackThank you

@_AlexYates_

#SQLRelay

Image sourcesAuthor Source Information

Chiltepinster Wikimedia Commons Mocking Bird Argument.jpg – Wikimedia Commons. This file is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license. Source on Wikimedia Commons: “Own work”

Bit Boy Flickr The elephant in the room – Flickr. This file is licensed under the Creative Commons Attribution 2.0 Generic license.

Nils Rinaldi Flickr Hippo fight 2/3 – Flickr. This file is licensed under the Creative Commons Attribution 2.0 Generic license.

My own collection Taken by/property of Alex Yates Kitten, “There’s more than one way to skin a cat!”

Memegenerator.net Memegenerator.net I don’t always edit database. Content designed to be shared and delivered with credit to memegenerator.net.

Ctrl.Alt.Design ctrla.lt Social Media share icons