state of mobile continuous delivery at spotify
TRANSCRIPT
State of mobile CD at SpotifyKristian Lindwall@klindwall
@ixhd@klindwall
30
5 years ago
40050
300
Now
30
100+ teams in 5 sites
Cross functional and full stack
Self-organizing
“Feel like a mini-startup”
End to end responsibilitySystems runs, works, scales
Section name 10
Section name 11
IOS Android
Feature squads(1-2 devs per platform in each team)
Client infrastructure Squads(tooling, support, Releases)
Backend Infrastructure squads
Daily commits
Quick history of mobile CD
Section name 13
• Desktop first• Quality process completely detached from dev
process, quality unknown• Little or no automated testing• A lot of work on feature branches, painful
merges• Irregular releases, occured because of company
requirements or external events
State of mobile CD
Section name 14
• Mobile equally important to desktop• Feature set determined release date• Low predictability leading to scope creep and
even more risk (“this has to go out with the release”)
State of mobile CD
Section name 15
• Introduced release manager role• Introduced three-week schedule.• Started taking test automation seriously• Had a hard time getting engineering and feature
development squads to “conform” to time-bound releases
• Missed releases. Unpredictably still because of low quality
State of mobile CD
Section name 16
• Specialised infra teams• Put tests together with source code for apps and
library• Started running and enforcing automated tests
with code changes• Nightly dogfooding
State of mobile CD
Section name 17
• Buy-in and mindset in the tech and product organizations caught up with our intention
• Bi-weekly releases• Almost all releases hit the market within 14 days of branching
date (almost predictable!)• Client branches and dependencies are cut automatically - no
judgement call
State of mobile CD
Section name 18
• Current challenge: Make releases predictable to within a few days of branch cut
• Moving towards shorter cycles (1 week probably)
State of mobile CD
Development of the Spotify clients today
CI System(Teamcity)
• 1 project per client• 80-120 commits
per day• ~1000 commits
per release
• 30-50 active developers per client and day
• Avg. time for pull request verification: <10m
• ~70% pull requests merged in <4hrs
• ~10000 unit tests/client (as well as a number of UI-based tests and integration tests)
Testing strategy
Snapshot:Releasing mobile
at Spotify
Current release cycle
Release branch only accept bugfixes
Current release cycleDistributionAndroid: Private Google Play channeliOS: Crashlytics
Current release cycleDistributionAndroid: Google Play Store, Beta channeliOS: App Store, Testflight programFeedback in beta forum
Current release cycle
DistributionAndroid: Google Play Store, gradual rolloutiOS: Beta has sufficient usage and quality -> release to 100% on App Store
Current painpoints
Developer pain points
Flakiness
What is “flaky tests”?• Unstable tests – timing issues, logical errors, hard to write good tests• Infrastructure – network, new Xcode versions, build framework,
unreliable test data• Backend – latency, changes, regional
Why is it bad?• Broken window syndrome• No trust in test results
Flaky Tests
Elastic
CI System
Logs
Test
resu
ltsBuil
d res
ults
Smart Alerting KibanaOdeneye
Dealing with flakiness
Dealing with flakiness
Time
Build times
Releasability
Releases Android Feb 2015-Mar 2016
Lessons learnt Getting people dedicated to target mobile CD
was a game changer
Weed out bottlenecks one at a time. Slowness, flakiness etc will cause people to do the worng thing.
Continuous delivery is mostly about changing behavior, not technology
Thank you!
Kristian Lindwall@klindwall