releasing the monolith on a daily basis - codemash
TRANSCRIPT
VINCENT KOK • DEVELOPMENT MANAGER • ATLASSIAN • @VINCENTKOK
Releasing the Monolith On a Daily Basis
@vincentkok
http://geekandpoke.typepad.com/geekandpoke/2011/08/hudson-status-monitor.html
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Cool stories
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Got issues?
https://flic.kr/p/fSopv4
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Journey
https://flic.kr/p/8WQutt
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Monolith
https://flic.kr/p/nA4S49
C O N F L U E N C E P I P E L I N E
M I C R O S E RV I C E E N V Y
4 P R O B L E M S
Agenda
C U LT U R E
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Confluence Pipeline
https://flic.kr/p/jus4Bt
Nr of builds last week
15000
Nr of jobs last week
81000
Nr of tests
31992
Nr of build agents
1000
Flakes
NaN
Size of Pipeline team
3
A typical pipeline
ReleaseInner Outer
Publish Deploy
The Confluence pipeline
ReleaseInner Outer
Publish Deploy
The Confluence pipeline
ReleaseInner Outer
Publish Deploy
Staging deploy
The Confluence pipeline
ReleaseInner Outer
Publish Deploy
Staging deploy
Manual release branch
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Culture
https://flic.kr/p/DsetEn
GreenDex
GreenDex
The Oz Principle
Experience
The Oz Principle
The way you’ve been shaped
BeliefsExperience
The Oz Principle
The way you’ve been shaped
Beliefs are based on your experience
BeliefsExperience Actions
The Oz Principle
The way you’ve been shaped
Beliefs are based on your experience
You take actions based on your beliefs
ResultsBeliefsExperience Actions
The Oz Principle
The way you’ve been shaped
Beliefs are based on your experience
You take actions based on your beliefs
The outcome of our actions
Clear goal Early results Momentum
Theory in real life
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Microservice Envy
https://flic.kr/p/b7ak8e
Smaller Clear ownership Independent Lifecycle
Microservice traits
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Monolith
https://flic.kr/p/nA4S49
< 1 hour
< 1 day
Release cadence Test automation
Ownership Complexity
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
#1: Complexity
https://flic.kr/p/cFM3cd
The real version
ReleaseInner Outer
Publish Deploy
Staging deploy
Manual release branch
Manual tasks Hard to understand
Inconsistent feedback
Impact
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Fix the model
https://flic.kr/p/8ykpkW
Fork / Join
Release
Publish Deploy
Inner Outer
Fork
Join
Fork/Join details
Inner loop Build times Build stability
Iterate
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
#2: Ownership
https://flic.kr/p/bfRuoX
Getting the model wrong
Fire fighting Broken windows No improvements
Impact
Dev Speed Feedback loop MTTR
Impact
Ownership
Development Teams Site Reliability Engineering
Ownership
Development Teams
Sustainable environment
Site Reliability Engineering
Pipeline team - BuildOps
Where does your team sit?
Busywork
Improving team life
Improving team life
Enforce, don’t fix
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Sustainable System
https://flic.kr/p/awVLLs
Early result
Move the needle
Measure everything
VisibilityCreate awareness in the wider team
Gather data to allow data driven decisions and prioritisation
Focus on quick wins to move the needle
Define rules Shared OwnershipEnforce by automation
Iterate
Shared ownership
Shared ownership
First responder
First responder
First responder
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
#3: Testing
https://flic.kr/p/cBbxuQ
Unit
Service
UI
Unit
Service
UI
Slowness Duplication Flakes
Impact
The impact of flakes
Broken windows
Culture
Build timesDue to restarts build times get out of control really quickly
Builds will go red people will start to ignore this; Culture of green out of the window
It starts with one, many will follow soon
Retries as a pain killerAutomatic retries to relieve some pain. Remember this is a workaround
Early result
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Fix the root cause
https://flic.kr/p/J44Nm
Iterate
Test strategy Flake policy
Fix or RemoveIssueDetect Ignore
Flake policy
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Deleting tests, seriously?
https://flic.kr/p/8BYXV4
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
#4: Releases
https://flic.kr/p/91MFkh
Release cycle
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Congestion
https://flic.kr/p/bRgBZP
Dogfooding becomes hard
Big bangEnds at master Cancellations
Impact
For once a free lunchFork and Join solved most of our problems already
Early result
Iterate
Reduce build times Fix dogfooding
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Dogfooding
https://flic.kr/p/dxpLTm
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Feedback
https://flic.kr/p/dxaAdk
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Soak time
https://flic.kr/p/fmfG2q
Decouple code and features
Merge often One process
Feature flags
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
How far in are we?
https://flic.kr/p/8WQutt
Release cadence Test automation
Ownership Complexity
Clear goal Early results Momentum
Changing beliefs
• Big cool statistic
• 2,569
• Add-Ons in Marketplace
Sustainable System
https://flic.kr/p/awVLLs
http://geekandpoke.typepad.com/.a/6a00d8341d3df553ef015392a77686970b-pi
Thank you!
VINCENT KOK • DEVELOPMENT MANAGER • ATLASSIAN • @VINCENTKOK