dan crow - becoming a data driven company leanconf 2013

Songkick: becoming a data-driven company

Dan Crow

whoisDan CrowCTO, Songkick@crowquine

Ten years in Silicon Valley● Apple● Verb● Unicru● Co-founder: Blurb

Five years at Google● Google Squared● Ads and Android

Songkick

We do live music● Started in 2007● One of first 10 startups in London● Tell us who your favorite bands are, we’ll

alert you when they play in your town● Second largest live music service in the

world

Core product

BoomtownIt’s a rat trap, and you’ve been caught

Where we wereMid 2010:● 2M monthly uniques● Big ball of Ruby mud● One big dev team● One release every 2 weeks● Velocity trending to zero● Unhappy dev team

But: talent + agile = hope

Simplify the productAfter four years our product had accrued cruft● Complexity in user interface● Complexity in features

Time to radically trim what we delivered● What do users actually use?● What do we need to support core use

cases?

Simplify the technologySimpler product means:● Delete a lot of code● Service-oriented architecture● Page-component web app model● Reduced build times● Continuous integration● Web apps separated from backends● Create new apps fast● Support many more apps

Technology impacts teamComponentized tech means:● Small, independent teams● Clear goals● Clearer roles, e.g. Tech Lead● Distributed, devolved decision making● Need for process across teams

Results● Build time from 4 hours -> 25 minutes● Developer velocity improved 4x● Site performance improved 2x● User satisfaction increased

○ faster site○ less cluttered, easier UI

The impactre

leas

es p

er w

eek

Continuous deploymentI made my way to the top of the hill...

How it works for us● Jenkins build server● Ten minute build time● Automatic push to production● Features can accumulate behind flippers● Acceptance testing in production● Manual testing in production with real data● Business metric monitoring● Roll forward

CD workflow

Mostly a cultural change● Quality becomes a dev issue● Whole company must buy in● Business must be willing to break things● Development is a flow, not a cycle● Some people won’t adapt to it● Took us 9 months to do right

Experiments

How to experimentEverything in a startup is an experiment: your company, each product, each feature● Experiments answer a question

○ Don’t present a fully formed feature● What’s the lowest cost way to the answer?

○ Most experiments fail, so move fast○ e.g. Do our users want to share? Sharing buttons

that do nothing

How we do itMultivariate testing environment:● Feature flippers allow us to direct traffic● Split traffic across versions A/B/C…● Use a variety of stats packages to analyze

○ Google Analytics○ Songkick “skab” package

● Code and design are always thrown away● Successful experiments = feature rebuild

Again, a cultural change● Always find the MVE● Be comfortable with lower quality● Get used to experiments failing● Experiment often● Trust your users, not your intuition

Downsides● Frequently changing user experience● Can get stuck in local minima● Hard to maintain consistency● Stats can be challenging to interpret● You need traffic to make it work

Benefits● Only users can tell you what works● Avoids question-asking biases● Ties feature improvements to business goals● Results constantly surprise us

Wrap upData is good

Where we are now● 9M monthly unique users● iPhone and Android app (2M users)● Tourbox - portal for artists● Detour - crowdfunding for concerts● Integrations with Spotify, Foursquare● Comprehensive ticketing product● Smaller dev team, moving faster

In summary● Getting the technology right is critical● Experiments are the heart of lean● It’s always about the culture

○ Hardest thing to change○ Takes time and effort○ You need everyone to work that way○ Has the biggest impact