continuous delivery distilled

Con$nuous Delivery Dis$lled

Ma# Callanan @mcallana

Why Con/nuous Delivery?

•  “The most important problem that we face as so?ware professionals is this: –  If somebody thinks of a good idea, how do we deliver it to users as quickly as possible?” – CD Book

•  “Our highest priority is: –  to sa/sfy the customer through early and con$nuous delivery of valuable so?ware.” – Agile Manifesto

Why Con/nuous Delivery?

•  Cycle /me should be measured in hours – not months –  How long does a single change take to release?

•  Quality should be built-‐in to the process –  Not an a?erthought of manual inspec/on

•  Feedback should be close as possible to point of failure –  Late feedback is expensive

What is Con/nuous Delivery?

•  A set of prac/ces and principles –  aimed at, building, tes/ng, and releasing so?ware faster and more frequently

Principles of So>ware Delivery

Create a Repeatable, Reliable Process for Releasing So?ware

Automate Almost Everything

Version Control Everything

If It Hurts, Do It More O?en – “Bring the Pain Forward”

Build Quality In

“Done” Means Released

Everybody Is Responsible for the Delivery Process

Con/nuous Improvement


•  A way of reducing cycle /me –  How long does a single change take to release? –  I.e. how long it takes a simple code change to get to produc/on?


•  A way of joining development, deployment, tes/ng & release ac/vi/es –  coordina/ng them to make the process as efficient and reliable as possible

–  reducing feedback delays by automa/ng manual processes

Develop Deploy Test Release

Central Concept: Deployment Pipeline

•  Make process visible •  Enable automated tes/ng –  Ensure quality is built into process

•  Enable automated deploys to any environment •  Improve feedback cycles

Commit stage

Acceptance stage

Exploratory tes$ng UAT Capacity

Tes$ng Produc$on

What is not Con/nuous Delivery: An/-‐pa#erns

•  Deploying so?ware manually •  Deploying to prod-‐like env only a?er dev is complete •  Manual configura/on management of produc/on

Deployment Pipeline

Agile / Lean / DevOps / Culture

Configura$on Management Tes$ng Deployment

Automa$on Disciplined

Development

Risk Management

Con$nuous Integra$on

Building Blocks

Requirements • Analyst & Customer

Design • Architect

Development • Developer

QA • Tester

Release • Opera/ons

Months / Years!

Waterfall

Agile -‐> Con/nuous Delivery

Agile Culture

Analyst/Customer

Testers Devs

Acceptance Criteria

Automa/on

DevOps Culture

Analyst/Customer

Testers

Ops

Devs

Acceptance Criteria

Automa/on

Silos

•  Ul/mately, we succeed or fail as a team –  not as individuals

•  Symptoms of Silos –  Dev throws work over wall to Test throws over wall to Ops

–  End up spending as much /me blaming each other as fixing defects that arise from siloed approach

h#p://www.gesilos.com.au/images/pellet/3%20x%2026%20Tonne%20Pellet%20Silos.jpg

Repairing Silos

•  How to break down silos –  Get everybody involved together from start of project –  Ensure they can communicate on a frequent, regular basis –  Increase visibility and self-‐serviceablilty

Lean

•  Example Value Stream Map

Require-‐ ments

Develop-‐ment Tes$ng Staging

Capacity Tes$ng

Release

Value-‐ Added Time

Elapsed Non-‐Value Added

Time

Business Customer

Batch Size vs Risk

h#p://www.slideshare.net/jallspaw/ops-‐metametrics-‐the-‐currency-‐you-‐pay-‐for-‐change

•  Reducing Batch Size Reduces Risk

Batch Size Science

High Release Cost = High Batch Size

Batch Size

Cost

Release Cost

Total Cost

Low Release Cost = Low Batch Size

Batch Size

Cost

Release Cost

Ac/ng on Feedback •  The delivery team must receive feedback

and then act on it •  Involve everybody in feedback process

–  Work in cross-‐func$onal teams or meet o?en –  Retrospec$ve: discuss how to improve

delivery process next itera/on

•  Broadcast the informa/on –  Big, visible dashboards ensures feedback gets into someone’s head

•  Feedback must be acted upon –  Requires discipline and planning –  Stop and decide on a course of ac$on –  Only once this is done should the team carry on with their work

Deployment Pipeline




Development

Risk Management

Building Blocks


Con/nuous Integra/on Developer • Commit to Version Control

Build Server • Compile

Tes/ng • Unit Tests • Component Tests

Repor/ng • Test Results • Coverage • Sta/c Analysis

Disciplined Development


Automa$on

Deployment Pipeline

Deployment Pipeline



Development

Risk Management


Building Blocks


Deployment Pipeline

Commit stage

Acceptance stage

Exploratory tes$ng UAT Capacity

Tes$ng Produc$on

Increasing confidence in build’s produc/on readiness

Environments become more produc/on-‐like

Faster feedback

•  Tension: – Environmental Confidence vs Feedback Speed

Ar/fact Repository (e.g. Yum)

Version Control (e.g. Git)

Deployment Pipeline Source Code

Commit stage

Compile Sta/c analysis Unit Tests Coverage

Build binary ar/facts

Acceptance stage

Configure Env Deploy

Smoke Tests Acceptance Tests

ar6facts ar6facts

UAT

Deploy Compliance Tests

Smoke Tests

Capacity Stage


Smoke Tests Load Tests

Produc$on


Smoke Tests

ar6facts

CM code/data, Orchestra6on, Puppet, Chef, Fabric, Ansible,

etc

Testers Self-‐service deploys

Ops Push-‐bu4on releases

Env & App Config

Java, Ruby, Groovy, Scala, config, Maven,

Gradle, etc

Deployment Pipeline •  Every commit is considered a release candidate •  Not every release candidate is released •  Candidates must first pass: –  Compile, Unit Test, Sta/c Analysis, Create/Upload RPMs –  Deploy RPM, Start server –  Smoke Tests –  Acceptance Tests –  Performance Tests –  UAT / Manual Exploratory –  Business decision to go-‐live –  Approval from gatekeepers

Deployment Pipeline

h#p://con/nuousdelivery.com/2010/02/con/nuous-‐delivery/

Deployment Pipeline -‐ Scripts •  Deployment pipelines are powered by scripts •  Script Ideals –  Obey the Unix Philosophy – do one thing well –  Environment-‐agnos/c –  Sensible exit code

•  Script Architecture –  Build Scripts

•  Convert build output into deployment input –  Deploy Scripts

•  Environment-‐independent instruc/ons to deploy so?ware remotely

•  Can run on central command-‐and-‐control server –  Test Scripts

•  Repeatable instruc/ons to invoke automated tes/ng


Automa$on

Deployment Pipeline

Deployment Pipeline



Development

Risk Management


Building Blocks



Disciplines Incremental development

Feature switches

Mainline development •  Can’t even do CI with mul/ple branches

Small releases

Keep code releasable

Keep automated tests green •  TDD: Red before Green


Configura$on Management

Deployment Automa$on


Deployment Pipeline

Deployment Pipeline



Development

Risk Management


Building Blocks

Tes$ng


Tes/ng

h#p://lisacrispin.com/2011/11/08/using-‐the-‐agile-‐tes/ng-‐quadrants/

Tes/ng

You can not save money if you are more worried about money, than you are about quality.

W. Edwards Deming:

Costs ↓ Focus on Quality ↑

Costs ↑ + Quality ↓ Focus on Costs ↓

Tes/ng Triangle

Unit

Manual

Automated

Other

Exploratory

System

Acceptance

Unit

Tes/ng •  “Eliminate the need for mass inspec/ons, as the way of life to

achieve quality, by building quality into the product in the first place.”

•  “Quality doesn’t come from inspec/on, but from improvement of the process. Improve the process so that defects aren’t produced in the first place. This eliminates the need for inspec/on on a mass basis.”

•  “Rou/ne inspec/on is the same as planning for defects, acknowledging that the process isn’t correct, or that the specifica/ons made no sense in the first place.”

•  “Inspec/on is too late as well as ineffec$ve and costly.”

h#p://www.signsculpt.com.au/wp-‐content/uploads/2013/03/focus-‐on-‐quality.jpg h#p://leanandkanban.wordpress.com/2011/07/15/demings-‐14-‐points/

Prod-‐Like Environments •  Every diff with Prod is a risk •  Need to take calculated risks –  Trade-‐offs for efficiency – Need to understand the cost

•  E.g. – DBs –  Configura/on Management –  Load Balancers – Deployment Process

•  Spot the Difference = waste

h#p://www.nairaland.com/a#achments/1500785_spot_jpge3cbd8220a9ec91ea49adz6c79aeb2

Configura$on Management Tes$ng Disciplined

Development

Deployment Pipeline

Deployment Pipeline



Development

Risk Management


Building Blocks

Deployment Automa$on


Deployment Automa/on

•  Releases should be low-‐ceremony, stress-‐free events – We’ve already prac/ced deployment 100s of /mes using the exact same process in prod-‐like environments

–  Don’t have to remember complex manual steps or rely on wri#en instruc/ons

–  Only tes/ng required is to verify the environment •  (not func/onality – already tested with every commit) •  Tes/ng is automated

–  Releases take minutes (not hours)

An/pa#ern: Deploying So?ware Manually

•  An/pa#ern Signs: –  Extensive, detailed release documenta$on –  Reliance on manual tes$ng to confirm app is correct –  Explaining why deployment is going wrong on release day –  Frequent correc$ons to release process during release –  Environments that differ in their configura$on –  Releases that take more than a few minutes to perform –  Releases that are unpredictable in their outcome

Deployment Automa/on •  Deployments should tend towards being fully automated

–  Human: 1) pick version & env, 2) press “deploy” bu#on •  Automated deployment scripts = up-‐to-‐date doco

–  Encourages collabora$on – everything is explicit in script •  Don’t depend on the deployment expert

–  Boring, repe$$ve manual tasks requiring significant exper$se == most certain way of ensuring human error

•  Automated deployment process is cheap and easy to test –  Deployment smoke tests

•  Automated processes are fully auditable •  Automated process must be used by everybody

–  Should be the only way in which the so?ware is ever deployed

Con/nuous Deployment? •  Idea: Take pipeline and make final “deploy to prod” step automa/c •  Tests

–  Automated tests have to be fantas/c -‐ automated UT/CT/FAT/NFAT covering en/re app –  Have to write all tests first, so only when story is complete will check-‐ins pass ATs

•  Intui/ve objec/on: too risky –  But, more frequent releases ! lower risk in any par/cular release –  Amount of change between releases goes down –  Releasing every change ! risk is limited to just risk inherent in one change –  Con/nuous deployment is a great way to reduce the risk of release –  Forces you to do the right thing

•  Pre-‐requisites: –  You can’t do it without automa/ng your en/re build, deploy, test, & release process –  You can’t do it without a comprehensive, reliable set of automated tests –  You can’t do it without wri/ng system tests that run against a prod-‐like env

Con/nuous Deployment? •  If you can’t actually release every set of changes that passes all tests, aim

to create process that lets you do so •  Pipelines support this process by:

–  Crea/ng repeatable, reliable, automated system for ge{ng changes into prod ASAP –  Crea/ng highest quality so?ware using highest quality process -‐ massively reducing risks

•  Con/nuous deployment takes this approach to its logical conclusion •  It should be taken seriously, because it represents a paradigm shi> in the

way so?ware is delivered •  Even if you have good reasons for not releasing every change you make—

and there are less such reasons than you might think—you should behave as if you were going to do so

Database Migra/on

DB v10

App v200 compa/ble with DB v10 & v11

App v210 compa/ble with

DB v11

App v220 compa/ble with DB v11 & v12

Time

App v230 compa/ble with

DB v12 …

DB v11

DB v12

… …

•  Decouple database changes from applica/on deployment

Tes$ng Deployment Automa$on


Deployment Pipeline

Deployment Pipeline



Development

Risk Management


Building Blocks

Configura$on Management


Configura/on Management

•  Provisioning & management should be automated

•  Keep everything you need to create/maintain infra under version control –  E.g. Kickstart, Puppet, DNS zone files, DHCP, SMTP, firewall, scripts

–  Inputs to deployment pipeline (same as source code)

Deployment Toolchain

CI -‐> Repo -‐> Orchestra/on -‐> CM •  Examples – TeamCity -‐> Yum -‐> Fabric -‐> Puppet/Hiera – Bamboo -‐> Yum -‐> Rundeck -‐> Chef –  Jenkins -‐> Nexus -‐> Ansible

DevOps Toolchain

h#p://www.slideshare.net/AnthonyShortland/dto-‐chefconf2012

Example DevOps Toolchain

h#p://www.slideshare.net/AnthonyShortland/dto-‐chefconf2012

Deployment Pipeline



Development


Building Blocks

Risk Management


Risk Management

•  Repeatability = confidence

h#p://www.slideshare.net/jallspaw/ops-‐metametrics-‐the-‐currency-‐you-‐pay-‐for-‐change

Risk Management •  What about PCI?

–  Lock down who is able to access “privileged” envs –  Change mgmt process for making changes to privileged envs –  Approvals from mgmt before deployments can be performed –  Protect against poten/al malicious interven/ons –  Audit all deployments

•  Deployment pipeline makes it possible to enforce these strategies while enabling efficient delivery process –  Automa$on over Documenta/on –  Enforce Traceability

Risk Management •  Process for managing approvals

–  Automated CR mgmt system –  Adding access control to deployment pipeline is a trivial exercise

•  How does CAB decide whether change should be executed? –  If risks outweigh benefits, change should not be made

•  Principles: –  Keep metrics on the system & make visible:

•  How long, how many wai$ng, propor$on denied? •  MTBF/MTTR •  Cycle $me

–  Regular retrospec$ves & improve system based on feedback

CD Maturity Model

Figure 15.1 Maturity model

CD Maturity Model

h#ps://flic.kr/p/afDTK9

Below this line, we’re slipping backwards

Above this line, we’re climbing the ladder of maturity

In One Slide

Quality Cycle Time

Beyond the Book

•  How to manage interconnected pipelines? –  Independent Releases: release one service at a /me

– Ensure all releases are backwards compa/ble with current consumers

– Discipline Required •  Short cycle /me & small batch size •  Feature Switching •  System tes/ng/monitoring

–  E.g. synthe/c transac/ons

Beyond the Book

•  Consumer Driven Contracts – Allows independent development and release of dependent services

h#p://mar/nfowler.com/ar/cles/consumerDrivenContracts.html

C

B B

A

A C

Code

Tests

B dev/tester

Maintains

Calls

As ‘C’ changes, its tests for consumers ‘A’ and ‘B’ are executed

continuous delivery distilled

Software

deployment

connuous delivery

version control

rst place

release candidate

single change

release process

deployment