the challenges and pitfalls of database deployment automation

Yaniv Yehuda

The Challenges and Pitfalls of Database Deployment Automation

2

• You will be on mute for the duration of the event

• We are now talking so please type a message in the Questions box in the Control Panel if you can’t hear us (please check your speakers and GoToWebinar audio settings first)

• There will be a Q+A session at the end, you can start submitting you questions on the Q&A bar on your gotowebinar dashboard.

• A recording of the full webinar will be put up online

Before We Begin

3

Presenter

Yaniv Yehuda• CTO, Co-Founder at DBmaestro

About DBmaestro

• Founded in 2008, product launched in 2010• Founded by Yariv Tabac and Yaniv Yehuda• Headquartered in Israel, Global Operations

5

• Doing better with less• Reacting quickly to market needs• Getting ahead of competition

• Just can’t wait 6 months for that next release…• Agile Development• Process Automation• DevOps

Agile world…

Dealing with Risk

Smaller and more focused changes are easier to manage (Agile…) Automation of repeating tasks lowers risk of (human) error Development and Operations should work in synergy (DevOps)

7

Continuous Integration

Continuous Delivery

Continuous Deployment

Automation is the Name of the Game…

8

• Principles and practices• Focus on streamlining development

• Developers integrate code into shared repository• Each check-in is verified• Automated builds• Automated tests• High visibility

• Easier & quicker to prevent and find problems, less back tracks => short integrations

What is Continuous Integration?

9

• Next step after continuous integration• Becoming lean, and even more Agile

• Make sure each change is releasable • Develop-> build-> test-> move to staging-> acceptance test

• Build a process to release with a push of a button• Deploy to production-> test production

• Actual deployment to production is manually actuated

=> Ensure risk mitigation and high efficiency

And Continuous Delivery?

10

• Rapid changes• Reacting quickly to market needs• Getting ahead of competition

• Fewer changes backed out• Better collaboration• Fewer defects

• Ultimately better service • Happy customers • Profitability

Why Continuous Delivery?

11

• Team and process• Version everything• Automate your tests• Fix it, properly (no out of process changes!)

• Automate your deployments• Create the deployment pipeline

Focus points

12

But…

13

• The database holds your essential information• Any changes can impact the entire system• Need to be synchronized with other changes• Often overlooked

Database is a Key Component

14

• There is more to a database than SQL scripts• Schema structure• Code• Content and meta-content• Internal dependencies• …

• Ensure that changes are not made without authorization

• Ensure no out-of-process changes

Reaching Inside the Database

15

• Old adage but true• The database is often neglected and therefore can

become the weakest link• Essential from a compliance and business point of

view• Should be the strongest link

The Weakest Link In a Chain

16

• Silos exist…• Don’t always communicate effectively• Need to share knowledge• Need to follow same procedures & best

practices

Developers and DBAs

17

Real-life tales

18

Real-life tales…

“it was difficult to track who made a change to a database object and what change they made.” (working-around file based version control)

“it took hours to get releases working. some changes were not documented and left out. we actually preferred crashes in integration. It is much worse when something works, but works wrong. in production…” (manual and error prone releases)

19

Real-life tales…

“We recently had a disaster - the script in the version control was not updated and when executed in production, ran the wrong revision. That cost tens of thousands of $”(an out-of-process update to QA that was not properly tracked)

“We had multiple releases to production every day. That is one release a week with multiple follow up fixes, and yet more fixes”(code overrides, partial versions, wrong versions – all pushed to production)

20

Real-life tales…

“we had an incident where a trigger was not correctly implemented during a code release. a trigger from a previous build was used instead which was only detected on Tuesday morning on the first business day after our code release. this was a customer-facing application and made our team look, and feel, bad about the release.

we realized that we needed to bring more discipline and rigor to our database changes.”(manual process are hard to repeat over and over without errors)

21

Challenges…

Un-coordinated Process

Silos in development

Deployment delays …

Out-of-Process Changes

Errors in production

Lack of confidence in automation

Code overrides

Different methodologies

Lack of historyMissing changes

wrong versions

22

• Root Causes:• Manual script based version control process• Deployment tools unaware of version control• No change process red-flags…

23

Two isolated Processes

Version Control Process (file based)

Development Process

Check-Out Script

Modify Script

Get updated Script from DB

Check-In Script

Compile Scriptin DB

Debug Scriptin DB

?

??

?

A

A’

24

Scripts

Pit falls

25

Scripts & Version Control

Challenges… • Code-overrides• Working on the wrong revisions • Scripts do not always find their way to the version control solution• Out of process updates go unnoticed• Hard to locate outdated update scripts

Playing safe? what we really need: • The actual code of the object• The upgrade script• A roll-back script

26

Testing Scripts

Single object based scripts• Hard to test in their entirely (holistically)• Hard to test due to colliding dependencies• Need to run in a specific order…

Large multi object based script• Represents the entire update - can deal with dependencies• Much harder to deal with project scope changes• Hard to mange – a big list of commands

27

X1.11.1.11.11.21.31.41.51.61.7

Scripts… Build Once Deploy Many

Int QA Stage Prod

Database Deploy Script

Environment

Re-Base (due to defects)

DevDev

DevModel

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.11.11.41.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

Out of Process Change

X

X

X

X

X

? 1.1.1

X

28

Scripts are static…

Scripts, unless super sophisticated:• Unaware of changes made in the target environment• Time passed from their coding to the time they are run• Potentially overriding production hot-fixes or work done in parallel

by another team

Content changes are very hard to manage• Metadata & lookup content does not practically fit into the VC• In most cases they are simply not managed

29

Dealing with challenges…

Coordinated Process Traceability

Start in the Beginning

No Out-of-Process Changes

Impact Analysis

Automation

Task Based Development

Well Defined Processes

30

Version Control - One Enforced Process

31

Dealing with challenges…

Integrated Version Control process• Leverage proven version control best practices

• Forcing check in & out for changes• Labels• etc..

• No code-overrides• Always working with the correct revision• All changes are documented• Always know who did what, when, why and from where• No out-of-process changes

• Supporting structure, code and content

No time spent on manual coding of the change scripts

32

Bonus Points

Task based development…• Correlate each database change with a change request

• Task ID• Work Item• Trouble Ticket• CR• etc…

…enables task based deployments• Partial deployments (a feature, a collection of bugs, etc…)• Scope changes easily synced between code and database

33

Change Policy Enforcement

34

Change Policy Enforcement

For deployment automation we need:

Automatic, repeatable & safe

Using tools make sense …

36

1.11.21.31.41.51.61.7

Build & Deploy On Demand

*

Int QA Stage Prod

Database Deploy Script

Environment* Execute the same script being executed at the Stage environment

Re-Base (due to defects)

DevDev

DevModel

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.4

1.4 1.7

1.1.1 1.7

1.1 1.1 1.11.41.7

File Based Version Control

Out of Process Change

1.1.11.7 1.1.11.7

Real World Deployment Automation

Test cases using compare & sync tools:

An index exists in source (QA) but not in target (Production)What should we do? Add the index or not?

38

Compare & Sync tools

Safe to automate?

Sure…

Deployment Automation

An index exists in Target (Production) but not in source (QA)What should we do? Drop the index or not?

40

Compare & Sync tools

Safe to automate?

No. Requires manual inspection…

41

Safe?

Simple, right?NO! we are going to BREAK production without even

knowing…

42

Why break production???

A compare & sync tool:• Is unaware of any changes that occurred before the time it ran• Has no knowledge of changes that took place at the target environment

• Does not leverage version control for more information• Unable to deal with conflicts & merges between different teams

• Requires manual inspection • Requires detailed knowledge regarding each change as part of the

process

So… no automation… as automating problems into production is a major risk!!!

43

We need to leverage version control into deployment decisions…

Safety Net For Deployment Automation

Source vs. Target

Action

= No Action

≠ ?

Source vs. Baseline

Target vs. Baseline

Action

= = No Action

≠ = Deploy Changes

= ≠ Protect Target

≠ ≠ Merge Changes

You do not have all of the information

With Baselines and 3 way analysis the unknown is now known

Simple Compare & Sync Baseline aware Analysis

45

Protecting Target Environment

Development BaselinePrevious Label /

Production Golden Copy

Production

No index in baseline =>

we should protect the NEW index on production!!!

(Protect Target)

46

Protecting Target Environment

Development BaselinePrevious Label /

Production Golden Copy

Production

BUT… If we had the index in the baseline =>

we should take it down from production…

(Deploy Change)

Deployment Automation

And Merge???

48

Conflict Resolving – Database Code

49

Conflict Resolving – Meta Data/Content

50

Impact Analysis

51

Safety Net For Deployment Automation

Database Safe Deployment Automation:• Leverages version control (baselines & previous revisions)

• Has a flexible scope (deploy multi schema to single task or work item)

• Can be run as a batch process (repeatable & consistent)

• Integrates to ALM (labels, CRs, Continuous Integration & Delivery)

• Deals with conflicts & merges to match code agility

Can raise red flags to stop the line…

if requires human intervention

Summary - What is DBmaestro TeamWork?

• Database Enforced Change Management solution+ Database version control+ Enforce best practices+ Plugs into the ALM (change request, tickets & work items)

+ Database merge & change impact analysis+ Know who can do what, where, when & why

• DevOps Solution for databases + Baseline aware deployment automation, rollback & recovery+ Reduce database deployment issues+ Plugs into release management & Continuous Delivery

Summary - How?

• Database version control • Enforced Check Out/In • Labels• Rollback/Undo • Audit trail reports

• Database impact analysis • Utilizes version control repository information • 3-way analysis

• Database deployment automation• API for automation• Baselines • Conflict resolution • Customized business logic

• Continuous EVERYTHING!

54

Q & A