releasing to production every week india
DESCRIPTION
TRANSCRIPT
Releasing to Production Every Week
Owen RogersTwitter: @exortech
http://exortech.com/blog
1 year
49 releases
49 releases(excluding patch releases)
~ 1 release / week
summary
lessons learned
process
practices
tools
Poll:
how long is your release cycle?
Me
Vancouver
> 15 teams
Agile methods
Company
start up
real-time energy monitoring
building energy efficiency
save $
save kW
save
aggregate data
hosted solution
SaaS
Java
Team
• 9 developers
• 9 developers• 1 product manager
• 9 developers• 1 product manager• 1 graphic designer
• 9 developers• 1 product manager• 1 graphic designer• 1 tester
what’s missing?
operations?
support?
QA?
project manager?
DBA?
generalizing specialists
fungibility
rotating responsibility
daily support duty
few hand-offs
empowered
Process
Goal:• to surface risk as early
as possible to keep problems out of production
maximize feedback
shrink time
Continuous Integration
Continuous Integration
• 3-4 Commits/dev/day
Automated Build
Automated Build
• 3 minute build
Automated Build
• 3 minute build• 30-40 builds/day
Automated Build
• 3 minute build• 30-40 builds/day
Automated Test Deploy
Automated Test Deploy
• 4-5 times per day
Automated Test Deploy
• 4-5 times per day• Scheduled nightly at
11PM
Daily Standup
Daily Standup
• Weekly company standup
Daily Review
Daily Review
• Quick peer feature review
Weekly Demo
Weekly Demo
• 15 minute company-wide user-driven demo
Mon Tues Wed Thurs Fri
9:30AMPlanningMeeting
10AMUser-drivenDemo Release
Testing
5pmRelease
weekly cycle
11pmCut-off
Sustainable Pace
Sustainable Pace
Sustainable Pace
• regular release window
Sustainable Pace
• regular release window• quality of life and code
Sustainable Pace
• regular release window• quality of life and code• part of culture
Sustainable Pace
• regular release window• quality of life and code• part of culture• company heartbeat
“agile”
Scrum?
XP?
what’s missing?
user stories?
user stories?
user stories?
story boards?
story boards?
story boards?
estimation?
estimation?
estimation?fits or it doesn’t
“muda”
pairing?
pairing?
pairing?as needed
pairing?as needed
code reviews
pairing?as needed
code reviewsopen workspace
TDD?
TDD?
TDD?52% coverage
TDD?52% coverage+ Selenium
TDD?52% coverage+ Selenium
retrospectives?
retrospectives?
retrospectives?Release Review
retrospectives?Release Review
5 Whys
lightweight RCA
5 Whys
just-in-time
within 24 hours
1 Jira per question
Practices
Daily support rotation
5 Whys
Continuous Monitoring
Continuous Monitoring
App Server Socket Logger LogMaster
WARN | ERROR
(async)
Continuous Monitoring
Continuous Monitoring
• Proactive response to issues
Continuous Monitoring
• Proactive response to issues
• Clean logs
Continuous Monitoring
• Proactive response to issues
• Clean logs
• Message throttling
Continuous Monitoring
• Proactive response to issues
• Clean logs
• Message throttling
• Gmail
Continuous Monitoring
• Proactive response to issues
• Clean logs
• Message throttling
• Gmail
• Managing the signal-to-noise ratio
Continuous Monitoring
Continuous Monitoring
Continuous Monitoring
Continuous Monitoring
App Server
gmetad
web app
RRD
gmondUDP multicast
Continuous Monitoring
Continuous Monitoring
Continuous Monitoring
Continuous Monitoring
Zero-downtime Deployment
Zero-downtime Deploys
Load Balancer
App Server App Server App Server
Deploy
Zero-downtime Deploys
Zero-downtime Deploys
• Fully automated deployment
Zero-downtime Deploys
• Fully automated deployment
• One button deploy from TeamCity
Zero-downtime Deploys
• Fully automated deployment
• One button deploy from TeamCity
• Gracefully bring down servers
Zero-downtime Deploys
What about the DB?
Zero-downtime Deploys
Bering
Zero-downtime Deploys
012_rename_login_id_to_user_name
ALTER TABLE USER RENAME COLUMN login_id user_name
Zero-downtime Deploys
012_rename_login_id_to_user_name
012_add_user_name_column 02_remove_login_id_column
expansion contraction
Zero-downtime Deploys
Zero-downtime Deploys
• expansion
Zero-downtime Deploys
• expansion
• preserves backwards compatibility
Zero-downtime Deploys
• expansion
• preserves backwards compatibility
• contract
Zero-downtime Deploys
• expansion
• preserves backwards compatibility
• contract
• clean up
Zero-downtime Deploys
Zero-downtime Deploys
• Typically safe expansion operations:
Zero-downtime Deploys
• Typically safe expansion operations:
• add table, add column, insert data
Zero-downtime Deploys
• Typically safe expansion operations:
• add table, add column, insert data
• Typically unsafe expansion operations:
Zero-downtime Deploys
• Typically safe expansion operations:
• add table, add column, insert data
• Typically unsafe expansion operations:
• rename or remove columns or tables
Zero-downtime Deploys
• Typically safe expansion operations:
• add table, add column, insert data
• Typically unsafe expansion operations:
• rename or remove columns or tables
• add constraint
Zero-downtime Deploys
/expand/.../143001_create_login_table002_add_login_id_index003_create_user_table004_create_group_table
/.../contract/.../143001_add_inv_constraint002_drop_alias_column
/...
Zero-downtime Deploys
Zero-downtime Deploys
• Database migration decoupled from the release
Zero-downtime Deploys
• Database migration decoupled from the release
• Simplified rollback
Zero-downtime Deploys
• Database migration decoupled from the release
• Simplified rollback
• Some additional complexity in writing migrations
Release Options
Traditional Release
Traditional Release
Traditional Release
• high risk
Traditional Release
• high risk
• not scalable
Traditional Release
• high risk
• not scalable
• prolonged release: “Don’t release until it’s all ready”
Release Options
Release Options
Release Options
• release to user subset (private beta)
Release Options
• release to user subset (private beta)
• split test (experimentation)
Release Options
• release to user subset (private beta)
• split test (experimentation)
• incremental rollout
Release Options
• release to user subset (private beta)
• split test (experimentation)
• incremental rollout
• selective enabling/downgrading
Release Options
• release to user subset (private beta)
• split test (experimentation)
• incremental rollout
• selective enabling/downgrading
• disable feature (feature darkmode)
Release Options
• release to user subset (private beta)
• split test (experimentation)
• incremental rollout
• selective enabling/downgrading
• disable feature (feature darkmode)
• defer commit
Release Options
• release to user subset (private beta)
• split test (experimentation)
• incremental rollout
• selective enabling/downgrading
• disable feature (feature darkmode)
• defer commit
• defer release
Release Options
Release Options
• Features running in production for weeks before fully available
Release Options
• Features running in production for weeks before fully available
• lots of data
Release Options
• Features running in production for weeks before fully available
• lots of data
• “Release is a marketing term”
Release Options
• Features running in production for weeks before fully available
• lots of data
• “Release is a marketing term”
• Production levers
Release Options
• Features running in production for weeks before fully available
• lots of data
• “Release is a marketing term”
• Production levers
• WANGMI: the discipline to defer
Use Production Data
Use Production Data
Use Production Data
• Upload Nightly backups
Use Production Data
• Upload Nightly backups
• Automated Test DB refresh after every release
Use Production Data
• Upload Nightly backups
• Automated Test DB refresh after every release
• Local DB refresh on demand
Use Production Data
• Upload Nightly backups
• Automated Test DB refresh after every release
• Local DB refresh on demand
• Cleansed
Use Production Data
• Upload Nightly backups
• Automated Test DB refresh after every release
• Local DB refresh on demand
• Cleansed
• Test data is a liability
Single Feature Release
• Daily Support Rotation
• 5 Whys
• Continuous Monitoring
• Zero-downtime deployment
• Release Options
• Use Production Data
• Single Feature Release
Tools