building a data driven business
Post on 17-Feb-2017
377 Views
Preview:
TRANSCRIPT
Building a data driven business
Konstantin Savenkov, CEO, Intento
Konstantin Savenkov, PhDChief Research Officer, Zvooq COO, Bookmate and Dream Industries now Founder & CEO
Turned a number of companies data-driven
• B(2B)2C content services
• edutainment
• ad tech• commercial space management
• consulting: online2offline, UGC, biotech
Bookmate, Zvooq
Theory&Practice, Exchanges
Unisound
DI Telegraph
@ksavenkov
Without data, people take
decisions based on biases and
beliefs@ksavenkov
…data replaces beliefs and provides a competitive
advantage@ksavenkov
If you’re competitor-focused, you have to
wait until there’s a competitor doing something.
Being customer-focused allows you to be more
pioneering.Jeff Bezos Amazon
@ksavenkov
…data helps to be customer-centric
@ksavenkov
Information is the oil of the 21st century, and analytics is the combustion engine.
Peter Sondegaard Gartner
@ksavenkov
…check the gas quality and do the maintenance
@ksavenkov
Half of the money spent on advertising
is wasted; the trouble is I don’t know which half.
John Wanamaker or William Lever@ksavenkov
…channel attribution and ROI will tell you
@ksavenkov
The average conversion rate in the United States whether you are selling
elephants or iPods, will be 2%
(с) Авинаш Кошик@ksavenkov
…data get you above the average
@ksavenkov
MasteryBusiness Metamorphosis
Data AwareData Monetisation
Data-drivenBusiness Optimisation
AnalyticsBusiness Insights
Data-Driven Maturity Index
Collect DataBusiness
Monitoring
@ksavenkov
@ksavenkov
To be data driven, it’s not enough to
collect and mine data. You need the culture.
@ksavenkov
Don’t optimise for data! Optimise for learning!
@ksavenkov
All initiatives must improve KPI
Formulate hypotheses
Run experiments@ksavenkov
Data validate hypotheses
behind your own decisions.
Data does not make decisions for you!
@ksavenkov
DATA
CulturePeople
ProcessTechnology
Analytics Business model
PlanningIterations
KPI
Hypotheses
Experiments
Changes
@ksavenkov
ISetting Goals
and KPI
@ksavenkov
Looks simple
MRR MarginMAU
but there are issues
@ksavenkov
IT’S UNCLEARwho’s responsible?
how to imrove?
how does that relate with resources spent?
(besides CEO)
(besides “work better”)
(when most of the business processes are automatic)
@ksavenkov
KPI Trees
MAU
New Loyal Returned
Traffic Conversion Retention Reactivation
@ksavenkov
KPI TreesMRR
MRR stable net new MRR
list MRR
MRR upgrades
MRR downgrades
New clients
MRR expansion
new MRR
Reactivation
direct sales
@ksavenkov
KPI Trees
Margin
LTVCAC
COGSARPU
Lifetime
Commissions
@ksavenkov
ADVANCED MODE
Ratio Analysis
Cohort Analysis
@ksavenkov
HELPS TOIdentify bottlenecks
Define roles and responsibilities
Generate ideas
Measure impact of initiatives
@ksavenkov
IIMeasure Impact
of Initiatives
@ksavenkov
For planning, groups tasks in Epics that matter
Track results of launched Epics
A culture of data success learning
@ksavenkov
Case studyMarketing project were tracked until launched
Study of a half-year long cross-promo campaign discovered a significant loss via COGS.
We’e identified key mechanics that led to the loss.
We’e adjusted the mechanics, re-negotiated with partners on the ongoing campaigns.
PROFIT@ksavenkov
IIIHypotheses and
experiments
@ksavenkov
It doesn’t work this way:
marketing campaign
@ksavenkov
impact!
product feature launched
billing failure
pupils back from holidays
AppStore featuring
people back from vacations
@ksavenkov
It doesn’t work this way:
marketing campaign
impact!
product feature launched
billing failure
pupils back from holidays
AppStore featuring
people back from vacations
@ksavenkov
It doesn’t work this way:
marketing campaign
impact!
DIFFERENT CHANNELS CONVERT WITH A
DIFFERENCE IN ORDER OF MAGNITUDE
ALSO:A lot of Epics that shouldn’t have been launched at all
A culture of praising a random success and explaining failures by external causes
Impossibility to learn by mistakes
@ksavenkov
formulate measurable hypotheses
carefully plan experiments
define a condition of success/failure prior to implementation
data collection and attribution, split-testing, controlled variables,
statistical significance
demonstrate explicit risks use models built on past data
prioritisation aid
and how the failure affects the roadmap
ability to prove multiple hypotheses simultaneously
@ksavenkov
Case StudiesEstimating mechanics of marketing projects before they launched
Product improvement that results in 2.5x conversion, 2x lifetime
Split testing of targeting and creative materials for ad campaigns, resulting in conversion 2-3 times higher than organic
Immediate increase of “conversion” from all initiatives to successful ones
This approach is behind all conclusions in these slides
lots of failed experiments success despite of all external factors
NO OTHER WAY@ksavenkov
Case Study: Recommender system for Conversion
• Hypotheses to prove:1. There’re enough users who will use RS output2. Their conversion will be above average
• A/B testing is the only way:– different channels convert with up to 20x difference– current traffic mix is unpredictable and hard to control
in case of app installs
• Do pilots:– Run with limited resources, then extrapolate and decide
if run full-scale
Case Study: Recommender system for Conversion
Let’s look at the economics
• In case of using a third-party RS on a CPO basis, in this case the CPO is limited by $0.14 (actually, much less)
• In case of a flat fee of $1000**/month, this is feasible starting from 7143 new subscribers/month, or $35K of marketing budget.
* CAC and marketing budget are model data** some arbitrary number
IVLeading indicators
@ksavenkov
Make decisions on monthly KPI or
finished projects is like going backwards
@ksavenkov
Look ahead or at least watch your step, not
backwards
Make the data work for you
@ksavenkov
Daily indicators
Incremental indicators
Leading indicators
Predictive models
spot problems and anomalies just in time
example: baremetrics.io
a perfect input for inbound marketing
accurate goals and perfect financial planning
@ksavenkov
Case StudySome loyal subscribers churned away for 2-3 months, to come back later. It was historically attributed to holidays and other external factors.
The daily indicators have shown that all churned users subscribed on weekdays. What a riddle!
It turned out that on weekdays the code base is frequently deployed to the production server, flushing the message queue of subscription renewals.
The fix increased a lifetime of paying users by 20%
@ksavenkov
Case StudyPredicting a lifetime for users that registered right now (10% accuracy)
Accurate unit economics for contracts with B2B2C partners,
content providers, pricing@ksavenkov
Case Study
A probabilistic model for segmenting users
Input data for chained communication
(inbound marketing)
@ksavenkov
Case StudyAccurate goals
loyal
new
churned
guaranteed growth?
inevitable stagnation!*
*unless KPI are increased@ksavenkov
Case StudyAccurate financial planning
Operational model
Marketing plan-fact
Financialplan-fact
Marketing budget
CAC, conversion
organic forecast
deal terms
unit economics deals forecast revenue forecasts
@ksavenkov
VBusiness process
automatisation
@ksavenkov
Hire another employee
or train another model?
@ksavenkov
Affects all business processes that scale linearly with a headcount:
customer supporteditorial office
content managementmarketing
@ksavenkov
Case StudiesUser base grows from 1M to 2M, doubling the headcount in customer support?
Implemented auto-reply using our knowledge base and smart templates for support engineers
A number of markets increased, adding more editors?Created an algorithm to provide a short-lists
based on a user behavior
An amount of UGC explodes, more content-managers?
Improving reduplication and computer aid based on the collected data
@ksavenkov
VIImproving the
Company Itself
@ksavenkov
Operational analytics - improve the process of KPI improvement
(productivity)
@ksavenkov
ITERATION
Case StudyImproving the Agile process
the expectations:
@ksavenkov
ROADMAPS DESIGN DEVELOPMENT QA SHIPPED
PLANNING
ITERATION
Case StudyImproving the Agile process
the reality:
@ksavenkov
INFLATING EFFORT
ROADMAPS DESIGN DEVELOPMENT QA SHIPPED
BACKLOG
NEW STUFF
TECH. DEBT
PLANNING
BUGS
UX
SOFTWARE OPS
URGENT STUFF
UNCLEAR DESIGN
UNCLEAR TECH
ITERATING
Data as an Asset
@ksavenkov
Case Stuies• Compare B2B2C deals through unit economics
• Estimate traffic quality for partner ad networks
• Data partnerships
• Targeted user communication
• Personalisation and recommender systems (the next slide)
• Bonus track: Investigating a large number of purchase returns for an internet retailer
@ksavenkov
Using Recommender Systemsand personalisation
CAC
LTV
ContentCosts
Marketing Expenses
New Customers
ARPU
Lifetime
Consumed Content Mix
Conversion
Retention
Reactivation
ExposedContent Mix
÷
×
* the recommendation fairy
*
Innovative business experiments, as there’s no
recipes
To pioneer, you need to iterate quicker than competitors
@ksavenkov
Data, models built over the data and experimental results is the
main asset created and exploited
by the innovative business
@ksavenkov
“THE UNFAIR ADVANTAGE”
top related