releasing to production every week

Releasing to Production Every

Week

1 year

46 releases

1 rollback

summary

lessons learned

poll:

how long is your release cycle?

Company

real-time energy monitoring

building energy efficiency

save $

save kW

save

aggregate data

hosted solution

SaaS

Java

learn

Team

• 9 developers

• 9 developers• 1 product manager

• 9 developers• 1 product manager• 1 graphic designer

• 9 developers• 1 product manager• 1 graphic designer• 1 tester*

• 9 developers• 1 product manager• 1 graphic designer• 1 tester*

* recently added

what’s missing?

operations?

support?

QA?

project manager?

DBA?

generalizing specialists

fungibility

rotating responsibility

daily support duty

few hand-offs

empowered

Process

Goal:• to surface risk as early

as possible to keep problems out of production

maximize feedback

shrink time

Continuous Integration

Continuous Integration

• 3-4 Commits/dev/day

Automated Build

Automated Build

• 3 minute build

Automated Build

• 3 minute build• 30-40 builds/day

Automated Build

• 3 minute build• 30-40 builds/day

Automated Test Deploy

Automated Test Deploy

• 4-5 times per day

Automated Test Deploy

• 4-5 times per day• Scheduled nightly at

11PM

Daily Standup

Daily Standup

• Weekly company standup

Daily Review

Daily Review

• Quick peer feature review

Weekly Demo

Weekly Demo

• 15 minute company-wide user-driven demo

Mon Tues Wed Thurs Fri

9:30AMPlanningMeeting

10:00AMUser-driven

Demo

ReleaseTesting

5pmRelease

weekly cycle

“agile”

Scrum?

XP?

what’s missing?

user stories?

user stories?

user stories?

story boards?

story boards?

story boards?

estimation?

estimation?

estimation?fits or it doesn’t

pairing?

pairing?

pairing?as needed

pairing?as needed

code reviews

TDD?

TDD?

TDD?30% coverage

TDD?30% coverage

+ Selenium

TDD?30% coverage

+ Selenium

retrospectives?

retrospectives?

retrospectives?Release Review

retrospectives?Release Review

5 Whys

5 Whys

lightweight RCA

just-in-time

within 24 hours

1 Jira per question

Practices

Daily support rotation

5 Whys

Continuous Monitoring

Continuous Monitoring

App Server Socket Logger LogMaster

WARN | ERROR

Continuous Monitoring

Continuous Monitoring

• Proactive response to issues

Continuous Monitoring

• Proactive response to issues

• Clean logs

Continuous Monitoring

• Proactive response to issues

• Clean logs

• Message throttling

Continuous Monitoring

• Proactive response to issues

• Clean logs

• Message throttling

• Gmail

Continuous Monitoring

• Proactive response to issues

• Clean logs

• Message throttling

• Gmail

• Managing the signal-to-noise ratio

Continuous Monitoring

Continuous Monitoring

Continuous Monitoring

Continuous Monitoring

App Server

gmetad

web app

RRD

gmondUDP multicast

Continuous Monitoring

Continuous Monitoring

Continuous Monitoring

Continuous Monitoring

Test mirrors Prod

Trust Test

Zero-downtime Deployment

Zero-downtime Deploys

Load Balancer

App Server App Server App Server

Deploy

Zero-downtime Deploys

Zero-downtime Deploys

• Fully automated deployment

Zero-downtime Deploys

• Fully automated deployment

• One button deploy from TeamCity

Zero-downtime Deploys

• Fully automated deployment

• One button deploy from TeamCity

• Gracefully bring down servers

Zero-downtime Deploys

What about the DB?

Zero-downtime Deploys

Bering

Zero-downtime Deploys

001_create_login_table002_add_login_id_index003_create_user_table004_create_group_table

001_add_inv_constraint002_drop_alias_column

Expand Contract

Zero-downtime Deploys

Zero-downtime Deploys

• Database migration decoupled from the release

Zero-downtime Deploys

• Database migration decoupled from the release

• Simplified rollback

Zero-downtime Deploys

• Database migration decoupled from the release

• Simplified rollback

• Some additional complexity in writing migrations

Incremental Rollout

Incremental Rollout

Incremental Rollout

• New features are released to user subset (by role)

Incremental Rollout

• New features are released to user subset (by role)

• “Release is a marketing term”

Incremental Rollout

• New features are released to user subset (by role)

• “Release is a marketing term”

• Production levers

Incremental Rollout

• New features are released to user subset (by role)

• “Release is a marketing term”

• Production levers

• Selective degredation

Production DB Restore

Production DB Restore

Production DB Restore

• Nightly backups

Production DB Restore

• Nightly backups

• Automated Test DB refresh every Monday morning

Production DB Restore

• Nightly backups

• Automated Test DB refresh every Monday morning

• Local DB refresh on demand

Production DB Restore

• Nightly backups

• Automated Test DB refresh every Monday morning

• Local DB refresh on demand

• Cleansed

WANGMI

WANGMI

akathe discipline to defer

Single Feature Release

• Daily Support Rotation

• 5 Whys

• Continuous Monitoring

• Test mirrors Prod

• Zero-downtime deployment

• Incremental rollout

• Production DB restore

• WANGMI - the discipline to defer

• Single Feature Release

Tools

Questions?

owen@pulseenergy.com

Nov 3, 4, 5• Martin Fowler

• Mary Poppendieck

• Eric Evans

• Michael Feathers

• Michael Nygard

Questions?

owen@pulseenergy.com

• Eric Ries: Continuous Deployment

• Daily rotating support duty

• Log: 100MB/server/day

• 30 KLOC

• ThoughtWorks

• Working with clients to shorten their release cycle

• “Agile”

• CruiseControl.NET