agile australia conference 2011 - devops live accounts- continuous delivery_st

LiveAccounts: Continuous Delivery in the Real World

Nish Mahanty

Lawrence Song

“Rapidly deliver value to our customers” CTO

Good news journey Quadrupled our feature throughput Shortened our delivery times by 87% Improved team satisfaction Metrics Team empowerment Tools

Just the facts Ma’am

Legacy code

Manual Build

Unpredictability

Distributed team

Real bug?

Late night deployments

Develop Smoke

Functional

Smoke Regression

Smoke

Delivery Value Stream

3 3 1

Prod Staging QA 1 3 0.3

Compile Assemble Package Restore DB Apply DB Changes Deploy Package

Assemble Package Restore DB Apply DB Changes Deploy Package

Assemble Package Apply DB Changes Deploy Package

Dev

Dude, where’s my new feature?

(and what we don’t)

What we believe in

Code quality

Current Before

Automated functional tests

0

21

56

79

110

132 146

160 173

S O N D J F M A M

# of Scenarios

60 65 65 65

80 83

93 94 94

S O N D J F M A M

% Coverage

Unit test coverage – new code

Code branches

5

2 1

Before Now Next

Deployment Duration

S O N D J F M

130 98

Cycle Time (Features and Defects)

S O N D J F M

30

9.8

Delivery

Duration

87% faster release

11

1

How we did it

One team

Develop Smoke

Functional

Smoke Regression

Smoke 3 3 1

Prod Staging QA 1 3 0.3

Dev

Build a common vision

Common KPIs

Build Trust

Improve Comms

Empower the team to change things

Approach

• Close collaboration

• Incremental

• Data driven

• Automation

• Consensus base

Develop Smoke

Functional

Smoke Regression

Smoke

Development Environment

3 3 1

Prod Staging QA 1 3 0.3

Dev

3 days to set up environment Shared database Shared environment Manual build No continuous integration build

30 min to set up Database change management Development sandbox Automated builds

Evolution, not revolution

Maven Buildr

Hudson Go

Perforce Git

Manual

No CI

Code Quality

• Agreed on coding standards as a team

• Unit Test Code Coverage: Minimum Value 80%

• Cyclomatic Complexity <= 7

• Checkstyle

• JsLint

• Fail a build

Develop Smoke

Functional

Smoke Regression

Smoke

Deployment

3 3 1

Prod Staging QA 1 3 0.3

Dev

3 days to QA environment 20 page checklist 1 day getting prod data to env Similar timeframes for stg/prod Frustrated QAs

30 min to each environment Auto configuration management 1 click deployment 1 click restore prod db

Develop Smoke

Functional

Smoke Regression

Smoke

Testing

3 3 1

Prod Staging QA 1 3 0.3

Dev

4 days (4 QAs) Tedious regression Slow feedback cycle

1 day (2 QAs) Automated functional tests

Behaviour Driven Dev’t by empowered testers

Automated continuous delivery pipeline

Dev Team

Version Control

Build, Test & Assemble

Automated regression test

QA Env & Smoke test

Staging Smoke Test

check in

trigger

Prod & Smoke Test

trigger

feedback

feedback

one click

one click

one click

feedback

feedback

feedback

What did we learn?

• Automation, Automation, Automation

• Continuous feedback/improvement

• Takes longer to automate for Windows

• Put infrastructure configuration into version control

• Run automated smoke tests for each deployment

Change in value stream times

1 hr 7 hrs 1 hr

Develop Smoke

Functional

Smoke Regression

Smoke 3 3 1

Prod Staging QA

1 3 2 hrs

Dev

30 min 30 min 30 min

Delivery

Duration

87% faster release

11

1

How you can do it

How do I sell this to my stakeholders?

Emphasise business benefits

– increased delivery predictability

– shorter time-to-market

– less rework

– lower risk

– lower application total cost of operation

Show them the metrics!

Where do I start?

Team: What’s the major pain point?

Automate manual processes

Measure stuff!

Use experts to accelerate adoption and reduce learning curve

We’re hiring...

@nishmahanty

nish.mahanty@myob.com

@lawrencesong

lawrence.song@myob.com