releasing to production every week
DESCRIPTION
Presented at DevTeach Vancouver 2009TRANSCRIPT
Releasing to Production Every
Week
1 year
46 releases
1 rollback
summary
lessons learned
poll:
how long is your release cycle?
Company
real-time energy monitoring
building energy efficiency
save $
save kW
save
aggregate data
hosted solution
SaaS
Java
learn
Team
• 9 developers
• 9 developers• 1 product manager
• 9 developers• 1 product manager• 1 graphic designer
• 9 developers• 1 product manager• 1 graphic designer• 1 tester*
• 9 developers• 1 product manager• 1 graphic designer• 1 tester*
* recently added
what’s missing?
operations?
support?
QA?
project manager?
DBA?
generalizing specialists
fungibility
rotating responsibility
daily support duty
few hand-offs
empowered
Process
Goal:• to surface risk as early
as possible to keep problems out of production
maximize feedback
shrink time
Continuous Integration
Continuous Integration
• 3-4 Commits/dev/day
Automated Build
Automated Build
• 3 minute build
Automated Build
• 3 minute build• 30-40 builds/day
Automated Build
• 3 minute build• 30-40 builds/day
Automated Test Deploy
Automated Test Deploy
• 4-5 times per day
Automated Test Deploy
• 4-5 times per day• Scheduled nightly at
11PM
Daily Standup
Daily Standup
• Weekly company standup
Daily Review
Daily Review
• Quick peer feature review
Weekly Demo
Weekly Demo
• 15 minute company-wide user-driven demo
Mon Tues Wed Thurs Fri
9:30AMPlanningMeeting
10:00AMUser-driven
Demo
ReleaseTesting
5pmRelease
weekly cycle
“agile”
Scrum?
XP?
what’s missing?
user stories?
user stories?
user stories?
story boards?
story boards?
story boards?
estimation?
estimation?
estimation?fits or it doesn’t
pairing?
pairing?
pairing?as needed
pairing?as needed
code reviews
TDD?
TDD?
TDD?30% coverage
TDD?30% coverage
+ Selenium
TDD?30% coverage
+ Selenium
retrospectives?
retrospectives?
retrospectives?Release Review
retrospectives?Release Review
5 Whys
5 Whys
lightweight RCA
just-in-time
within 24 hours
1 Jira per question
Practices
Daily support rotation
5 Whys
Continuous Monitoring
Continuous Monitoring
App Server Socket Logger LogMaster
WARN | ERROR
Continuous Monitoring
Continuous Monitoring
• Proactive response to issues
Continuous Monitoring
• Proactive response to issues
• Clean logs
Continuous Monitoring
• Proactive response to issues
• Clean logs
• Message throttling
Continuous Monitoring
• Proactive response to issues
• Clean logs
• Message throttling
• Gmail
Continuous Monitoring
• Proactive response to issues
• Clean logs
• Message throttling
• Gmail
• Managing the signal-to-noise ratio
Continuous Monitoring
Continuous Monitoring
Continuous Monitoring
Continuous Monitoring
App Server
gmetad
web app
RRD
gmondUDP multicast
Continuous Monitoring
Continuous Monitoring
Continuous Monitoring
Continuous Monitoring
Test mirrors Prod
Trust Test
Zero-downtime Deployment
Zero-downtime Deploys
Load Balancer
App Server App Server App Server
Deploy
Zero-downtime Deploys
Zero-downtime Deploys
• Fully automated deployment
Zero-downtime Deploys
• Fully automated deployment
• One button deploy from TeamCity
Zero-downtime Deploys
• Fully automated deployment
• One button deploy from TeamCity
• Gracefully bring down servers
Zero-downtime Deploys
What about the DB?
Zero-downtime Deploys
Bering
Zero-downtime Deploys
001_create_login_table002_add_login_id_index003_create_user_table004_create_group_table
001_add_inv_constraint002_drop_alias_column
Expand Contract
Zero-downtime Deploys
Zero-downtime Deploys
• Database migration decoupled from the release
Zero-downtime Deploys
• Database migration decoupled from the release
• Simplified rollback
Zero-downtime Deploys
• Database migration decoupled from the release
• Simplified rollback
• Some additional complexity in writing migrations
Incremental Rollout
Incremental Rollout
Incremental Rollout
• New features are released to user subset (by role)
Incremental Rollout
• New features are released to user subset (by role)
• “Release is a marketing term”
Incremental Rollout
• New features are released to user subset (by role)
• “Release is a marketing term”
• Production levers
Incremental Rollout
• New features are released to user subset (by role)
• “Release is a marketing term”
• Production levers
• Selective degredation
Production DB Restore
Production DB Restore
Production DB Restore
• Nightly backups
Production DB Restore
• Nightly backups
• Automated Test DB refresh every Monday morning
Production DB Restore
• Nightly backups
• Automated Test DB refresh every Monday morning
• Local DB refresh on demand
Production DB Restore
• Nightly backups
• Automated Test DB refresh every Monday morning
• Local DB refresh on demand
• Cleansed
WANGMI
WANGMI
akathe discipline to defer
Single Feature Release
• Daily Support Rotation
• 5 Whys
• Continuous Monitoring
• Test mirrors Prod
• Zero-downtime deployment
• Incremental rollout
• Production DB restore
• WANGMI - the discipline to defer
• Single Feature Release
Tools
Nov 3, 4, 5• Martin Fowler
• Mary Poppendieck
• Eric Evans
• Michael Feathers
• Michael Nygard
• Eric Ries: Continuous Deployment
• Daily rotating support duty
• Log: 100MB/server/day
• 30 KLOC
• ThoughtWorks
• Working with clients to shorten their release cycle
• “Agile”
• CruiseControl.NET