migration or state based?
TRANSCRIPT
![Page 1: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/1.jpg)
Migration or state based?How to put your database under source control
![Page 2: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/2.jpg)
#JOYFULBI2
Agenda
• Motivation
• State or migration – main gotchas
• Demo
• Summary
![Page 3: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/3.jpg)
#JOYFULBI3
About me
• SQL Server Support Engineer @Joyful Craftsmen
• Former SQL Server Lead DBA from ČS, a.s.
• Czech PASS Leader & SQL Saturday co-organizer
@malekpav [email protected] https://joyfulcraftsmen.com/
Czech PASS SQL Saturday Prague #889
![Page 4: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/4.jpg)
![Page 5: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/5.jpg)
DLM!What problems we are trying to solve?
![Page 6: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/6.jpg)
#JOYFULBI6
Bug density
"Over the past seven years, the Coverity Scan service has
analyzed nearly 850 million lines of code from more than 300
open source projects including Linux, PHP and Apache. ... The
analysis found an average defect density of .69“
Source: https://www.helpnetsecurity.com/2013/05/07/analyzing-450-million-lines-of-software-code/
![Page 7: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/7.jpg)
#JOYFULBI7
The cost of defects
Implementation
UnitTesting
IntegrationTesting
SystemTesting
In-Service
100xC
ost
to
Fix
Def
ect
When Detected
![Page 8: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/8.jpg)
#JOYFULBI8
Source Control
Release Management
Monitoring & Operations
Testing
QA
Staging
Prod
Continuous Integration
Build
Test
SSDT
DBUP
![Page 9: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/9.jpg)
#JOYFULBI9
Versioning of databases
• Database is state based – it can’t be easily changed
• Two basic approaches• Migration Based• State Based
Source Control
![Page 10: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/10.jpg)
#JOYFULBI10
State or migrations?
State
• Your source of truth is how the database should be
Migrations
• Your source of truth is how the database should change
![Page 11: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/11.jpg)
#JOYFULBI11
Neverending fights
![Page 12: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/12.jpg)
#JOYFULBI12
State based more explained
• Source control contains• All* the objects in the database
• Deployment• A magic wand which compares the state in source control
and in a target database and creates a change script, which includes all the changes, regardless what the previous state was
![Page 13: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/13.jpg)
#JOYFULBI13
State Based Automation
A
B B
Source Control / Artifact UAT Production
CREATE TABLE A
CREATE TABLE A ALTER TABLE B
Automatically generated script (based on schema comparison) Run post deployment script
Run post deployment scriptPossible migrations or metadata refresh
Post
Dep
loym
ent
Scri
pts
A A
SSDTsqlpackage.exe
+dacpac
Red Gate Source Control*
DLM RedGate PS module
+ nuget package
![Page 14: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/14.jpg)
#JOYFULBI14
Migration based explained
• Source control contains• All the scripts we want to run against our database
• Deployment• Simple run all the scripts against the target which haven’t
run yet
![Page 15: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/15.jpg)
#JOYFULBI15
Migration based automation
Src: https://github.com/sqlcollaborative/dbops
![Page 16: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/16.jpg)
#JOYFULBI16
Major problems with each approach by example1. Include MiddleName into dbo.GetAccountInfo
procedure output
2. Rename the table dbo.Usr to dbo.User
![Page 17: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/17.jpg)
#JOYFULBI17
Migration based
Two developers start to work on their takssimultaneously…
ALTER PROC dbo.GetAccountInfo ASBEGIN
SELECT Name, MiddleName FROM dbo.UsrEND
EXEC sp_rename 'Usr', 'User’
ALTER PROC dbo.GetAccountInfo ASBEGIN
SELECT Name FROM dbo.UserENDRev 112 Rev 113
![Page 18: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/18.jpg)
#JOYFULBI18
One developer “refactor” the database
State based
State A State B
?
![Page 19: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/19.jpg)
#JOYFULBI19
One developer “refactor” the database
DROP TABLE dbo.Usr
CREATE TABLE dbo.User (id int, Name varchar(50), MiddleNamevarchar(50))
ALTER PROC dbo.GetAccountInfo ASBEGINSELECT Name, MiddleName FROM dbo.UserEND
State based
![Page 20: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/20.jpg)
#JOYFULBI20
Trouble with both approaches
• Migrations• Changes are overwritten and conflicts are missed• Last revision always wins• Harder to review or manage pull requests• How we get the change?
• State based• Need to understand and trust your tool• Must have a clear rollback plan• Must have tests for the data loss• Usually longer deployment times
![Page 21: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/21.jpg)
#JOYFULBI21
One more thing - drifts explanation
• Changed performed out of normal pipeline
• Samples:• New index• Nullability of columns• Computed columns• Partitions
What wasn’t tested in production can’t
go to test!
I don’t often change the database. But if
so, I do it in production!
![Page 22: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/22.jpg)
#JOYFULBI22
Some of the other key factors
• How easy we work with tool• Refactoring• Loss of changes• Multiplatform• Customization• Price• IDE• Rollback scenarios• Suitable for selected development model
![Page 23: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/23.jpg)
#JOYFULBI23
Some tools for DLM
State Based Migration Based
SSDTDBUP
![Page 24: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/24.jpg)
#JOYFULBI24
DemoLet’s see the tools in the action!
![Page 25: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/25.jpg)
#JOYFULBI25
Fine tuned state based approach
![Page 26: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/26.jpg)
#JOYFULBI26
Fine tuned migration based approach
Handle all programmable objects as state based
Modify the state a track the change
Include migrations for all data movement related
changes
![Page 27: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/27.jpg)
#JOYFULBI27
Hybrid approach 1/2Via SQL Change Automation
Baseline state z produkce
Feature / Sprint / Verze
Změnové data releated skriptya případné (meta)datové úpravy
State všech programmable objects
Co
mm
it
Release
1. Pre-Deployment skripty
2. Postupný běh migračních skriptů
3. Programmableobjects statbasedeployment
4. Post-Deploymentskripty
![Page 28: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/28.jpg)
#JOYFULBI28
Run migration scripts from 1.01-3
Hybrid approach 2/2Via SQL Change Automation
21 3
Source Control UAT Produkce
Run migration scripts from 1.01-15
1211 14 15
2221
1.0
a1
.01
1.0
2
History table--------1.01-3
History table--------1.01-15
2221
1.0
2
1211
1.01
2221
1.0
2
1413
Pro
gram
mab
leo
bje
cts
Compare programmable objects Compare programmable objects
![Page 29: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/29.jpg)
#JOYFULBI29
One more thing!
![Page 30: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/30.jpg)
#JOYFULBI30
Dedicated vs shared development model
Dedicated
• Everyone has his own copy of the database
• When work is done, commit goes to a shared repo
Shared
• Multiple developers on one database
• When work is done, selected changes goes to a repo
![Page 31: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/31.jpg)
#JOYFULBI31
Dedicated vs shared development model
Dedicated +-• Fast change and commit• Tests “consistency”• Branches can be easily
used• Allows experiments• My house, my castle
Shared +-• Demanding for proper
communication• Slow for changes
detection• Cannot run test until
without risking a break
![Page 32: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/32.jpg)
#JOYFULBI32
Source Control
Release Management
Monitoring & Operations
Testing
QA
Staging
Prod
Continuous Integration
Build
Test
![Page 33: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/33.jpg)
#JOYFULBI33
Summary• State
• Easier• Way better for programmable
objects• Works well in large/distributed
teams• Great for frequent changes• Can handle large dependencies• DRIFT: rolled back
• Migrations• More control (harder to
manage)• Better for changes which
change data schema (migrations)
• More suitable for small teams • Preferred if nr of changes is
relatively small• No that good for many
dependent objects• DRIFT: remains
![Page 34: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/34.jpg)
#JOYFULBI34
What to choose
• When you have a fortune• Choose SQL Change Automation or SQL Source Control
• When you are more app developer• Use DbUp or FlyWay
• When you develop a warehouse or DB centric app• Use SSDT bud think about migration based approach
(DbUp)
![Page 35: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/35.jpg)
#JOYFULBI35
Pavel
Need support? Let us know!
DATA-DRIVEN DECESIONS
SMARTLY.
EVERYBODY MAKING
Contact: [email protected]
![Page 36: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/36.jpg)
Thank you!Questions?
![Page 37: Migration or state based?](https://reader033.vdocument.in/reader033/viewer/2022061523/629f3657069ccd51ac222527/html5/thumbnails/37.jpg)