hacker news meetup april 2014
DESCRIPTION
How Songkick's technology team does continuous delivery and uses experiments to develop our products.TRANSCRIPT
Text
What do you know?How Songkick uses continuous delivery and experimentation to learn from our users
About me: Dan Crow@crowquine
CTO, Songkick
!
Visiting Professor, University of Leeds co-Founder Blurb Apple, Google, Unicru, Verb
Songkick• The live music startup
• One of original Silicon Roundabout startups
• Second largest live music service globally
• Growth:
Back in 2012• Simplified product
• Simplified code
• Moved to SOA
• Moved to continuous integration
Results• Releases per month went way up
!
!
!
!
!
• Team happiness too
Text
Continuous Delivery
Aim• Fast feedback from users
• Get code live as fast as possible
• Minimize developer context switching
• Lots of small changes, instead of few large releases
Our principles• Create a continuous flow of work
• Fast feedback trumps exhaustive testing
• Waiting for builds kills focus
• Developers own product quality
• Business metrics tell you what is happening
What we do• Develop on main
• Jenkins CI server
• Automated tests on staging, 10 minute maximum
• Push live on green
• Features accumulate behind feature flippers
• Manual testing live in production
Feature flippers• Group changes together
• Control who can see changes: • No-one • VPN only • percentage of traffic • everyone
• Admin UI controls
Lessons learned• Speed is essential
• Small changes create speed
• Small changes easier to debug
• Immediate feedback leads to: • better product • happier team
It’s about culture, people• CD is cultural change, not a technical one
• Fast and small is (initially) scary
• Roll forwards becomes possible
• Minimize process once everyone is comfortable
• Developers own quality and testing
Text
Experiments
Everything is an experiment• Your startup
• Your products
• Each feature
• Each change
Embrace the scientific method
How we experiment• Answer a question
• Figure out the lowest cost way to answer the question • Example: sharing experiment
• Run as many experiments as possible
• Always throw away the code and design
Throw away the experiments• 80% of experiments fail, so do lots of experiments
• If the experiment fails • remove the changes asap
• If the experiment succeeds • remove the changes asap • rebuild the code
Mechanics• Document the question and the expected results
• A/B testing using feature flippers
• Variety of packages to analyse results: • Google Analytics • skab • MixPanel
It’s about culture, people• Always find the MVE
• Be comfortable with rough quality
• Get used to experiments failing
• Experiment often and fast
• Trust your users, not your instincts
• Trust your tests
Results• Run multiple experiments at once
• Experiments typically last 2-5 days
• Most web features now subject to tests
• Sometimes short-circuit formal A/B testing
Limitations• Your UI changes often
• Can get stuck in local minima
• Everyone needs to understand statistical significance
• Need lots of traffic to get fast results
• Mobile development cycles are too slow
Obligatory hiring appeal
Songkick are hiring: [email protected]
@crowquine
!
Silicon Milkroundabout #7, May 10-11