croll lean analytics workshop (3h) - lean ux nyc april 2014

130
Lean Analytics Lean UX NYC April, 2014 @acroll

Upload: lean-analytics

Post on 11-Aug-2014

4.278 views

Category:

Design


5 download

DESCRIPTION

How to use data to build a better business faster. Based on the book Lean Analytics, this presentation looks at startup metrics and offers a framework for deliberate growth and iterative improvement of a new business. It also includes examples from larger organizations trying to change from within.

TRANSCRIPT

Page 1: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Lean Analytics

Lean UX NYCApril, 2014

@acroll

Page 2: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Some housekeeping.

Page 3: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Don’t sell what you can make. Make what you can sell.

Kevin Costner is a lousy entrepreneur.

Page 4: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

The core of Lean is iteration.

Page 5: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Most startups don’t know what they’ll be when they grow up.

Hotmailwas a database company

Flickrwas going to be an MMO

Twitterwas a podcasting company

Autodeskmade desktop automation

Paypalfirst built for Palmpilots

Freshbookswas invoicing for a web design firm

Wikipediawas to be written by experts only

Mitelwas a lawnmower company

Page 6: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Unfortunately,we’re all liars.

Page 7: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Everyone’s idea is the best right?

People love this part!

(but that’s not always a good thing)

This is where things fall apart.

No data, no learning.

Page 8: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Analytics can help.

Page 9: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Analytics is the measurement of movement towards your business

goals.

Page 10: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

In a startup, the purpose of analytics is to iterate to product/market fit

before the money runs out.

Page 11: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

I have two kids.At least one of them is a girl.

Page 12: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

What are the chancesthe other is a boy?

Page 13: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

BB BG

GB GG

Page 14: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

2 of 3 (66%) are boys.

GB GG BG

Page 15: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Some fundamentals.

Page 16: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

A good metric is:

Understandable

If you’re busy explaining the data, you won’t be busy acting on it.

Comparative

Comparison is context.

A ratio or rate

The only way to measure change and roll up the tension between two metrics (MPH)

Behaviorchanging

If you’re busy explaining the data, you won’t be busy acting on it.

Page 17: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Thesimplestrule

badmetric.

If a metric won’t change how you behave, it’s a

h"p://www.flickr.com/photos/circasassy/7858155676/

Page 18: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Metrics help you know yourself.

Acquisition

Hybrid

Loyalty

70%of retailers

20%of retailers

10%of retailers

You are just like

Customers that buy >1x in 90d

Once

2-2.5per year

>2.5per year

Your customers will buy from you

Then you are in this mode

1-15%

15-30%

>30%

Low acquisition cost, high checkout

Increasing return rates, market share

Loyalty, selection, inventory size

Focus on

(Thanks to Kevin Hillstrom for this.)

Page 19: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Qualitative

Unstructured, anecdotal, revealing, hard to aggregate, often too positive & reassuring.

Warm and fuzzy.

Quantitative

Numbers and stats. Hard facts, less insight, easier to analyze; often sour and disappointing.

Cold and hard.

Page 20: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Exploratory

Speculative. Tries to find unexpected or interesting insights. Source of unfair advantages.

Cool.

Reporting

Predictable. Keeps you abreast of the normal, day-to-day operations. Can be managed by exception.

Necessary.

Page 21: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Rumsfeld on Analytics

(Or rather, Avinash Kaushik channeling Rumsfeld)

Things we

know

don’tknow

we know Are facts which may be wrong and should be checked against data.

we don’tknow

Are questions we can answer by reporting, which we should baseline & automate.

we knowAre intuition which we should quantify and teach to improve effectiveness, efficiency.

we don’tknow

Are exploration which is where unfair advantage and interesting epiphanies live.

Page 22: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

MayAprMarFeb

Slicing and dicing data

Jan

0

5,000

Activ

e use

rs

Cohort:Comparison of similar groups along a timeline.(this is the April cohort)

A/B test:Changing one thing (i.e. color) and measuring the result (i.e. revenue.)

MultivariateanalysisChanging several things at once to see which correlates with a result.

☀☁☀☁

Segment:Cross-sectional

comparison of all people divided by

some attribute (age, gender, etc.)

Page 23: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Which of these two companiesis doing better?

Page 24: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

  January February March April May

Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50Is this company growing or stagnating?

Cohort 1 2 3 4 5

January

February

March

April

May

$5 $3 $2 $1 $0.5

$6 $4 $2 $1

$7 $6 $5

  $8 $7

      $9

How about this one?

Page 25: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Cohort 1 2 3 4 5

January

February

March

April

May

Averages

$5 $3 $2 $1 $0.5

$6 $4 $2 $1  

$7 $6 $5    

$8 $7      

$9        

$7 $5 $3 $1 $0.5

Look at the same data in cohorts

Page 26: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Lagging

Historical. Shows you how you’re doing; reports the news. Example: sales.

Explaining the past.

Leading

Forward-looking. Number today that predicts tomorrow; reports the news. Example: pipeline.

Predicting the future.

Page 27: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya)

If someone comes back to Zynga a day after signing up for a game, they’ll probably become an engaged, paying user (Nabeel Hyatt)

A Dropbox user who puts at least one file in one folder on one device (ChenLi Wang)

Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman)

A LinkedIn user getting to X connections in Y days (Elliot Schmukler)

Some examples

(From the 2012 Growth Hacking conference. http://growthhackersconference.com/)

Page 28: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Which means it’s time to talk about correlation.

Page 29: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

1

10

100

1000

10000

Ice cream consumption DrowningsJan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec

Page 30: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Correlated

Two variables that are related (but may be dependent on something else.)

Ice cream & drowning.

Causal

An independent variable that directly impacts a dependent one.

Summertime & drowning.

Page 31: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

A leading, causal metricis a superpower.

h"p://www.flickr.com/photos/bloke_with_camera/401812833/sizes/o/in/photostream/

Page 32: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Growth hacking, demystified.

Find correlation

Test causality

Optimize the causal factor

Pick a metric to change

Page 33: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Is social action a leading indicator of donation?

http

://bl

og.ju

stgi

ving.

com

/nine

-reas

ons-

why

-soc

ial-a

nd-m

obile

-are

-the-

futu

re-o

f-fun

drais

ing/

Page 34: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Is mobile use?ht

tp://

blog

.just

givin

g.co

m/n

ine-re

ason

s-w

hy-s

ocial

-and

-mob

ile-a

re-th

e-fu

ture

-of-f

undr

aising

/

Page 35: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Why is Nigerian spam so badly written?

Page 36: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Aunshul Rege of Rutgers University, USA in 2009

Experienced scammers expect a “strike rate” of 1 or 2 replies per 1,000 messages emailed; they expect to land 2 or 3 “Mugu” (fools) each week.One scammer boasted “When you get a reply it’s 70% sure you’ll get the money”“By sending an email that repels all but the most gullible,” says [Microsoft Researcher Corman] Herley, “the scammer gets the most promising marks to self-select, and tilts the true to false positive ratio in his favor.”

1000 emails

1-2 responses

1 fool and their money, parted.

Bad language (0.1% conversion)

Gullible (70% conversion)

1000 emails

100 responses

1 fool and their money, parted.

Good language (10% conversion)

Not-gullible (.07% conversion)

This would be horribly inefficient since

humans are involved.

Page 37: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Turns out the word “Nigeria” is the best way to identify promising prospects.

Page 38: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Nigerian spammersreally understand their target market.

They see past vanity metrics.

Page 39: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

The Lean Analytics framework.

Page 40: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Eric’s three engines of growth

Virality

Make people invite friends.

How many they tell, how fast they

tell them.

Price

Spend money to get customers.

Customers are worth more than

they cost.

Stickiness

Keep people coming back.

Approach

Get customers faster than you

lose them.

Math that matters

Page 41: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Dave’s Pirate MetricsAARRR

AcquisitionHow do your users become aware of you?

SEO, SEM, widgets, email, PR, campaigns, blogs ...

ActivationDo drive-by visitors subscribe, use, etc?

Features, design, tone, compensation, affirmation ...

RetentionDoes a one-time user become engaged?

Notifications, alerts, reminders, emails, updates...

RevenueDo you make money from user activity?

Transactions, clicks, subscriptions, DLC, analytics...

ReferralDo users promote your product?

Email, widgets, campaigns, likes, RTs, affiliates...

Page 42: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Stage

EMPATHY I’ve found a real, poorly-met need that a reachable market faces.

STICKINESS I’ve figured out how to solve the problem in a way they will keep using and pay for.

VIRALITY I’ve found ways to get them to tell their friends, either intrinsically or through incentives.

REVENUE The users and features fuel growth organically and artificially.

SCALE I’ve found a sustainable, scalable business with the right margins in a healthy ecosystem.

GateTh

e fiv

e st

ages

Page 43: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Empathy stage:Localmind hacks Twitter

Needed to find out if a core assumption—strangers answering questions—was valid.Ran Twitter experiment instead of writing codeAsked senders of geolocated Tweets from Times Square random questions; counted response rateConclusion: high enough to proceed

Page 44: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Stickiness stage:qidiq streamlines invites

Survey owner adds recipient to groupSurvey owner asks question

Recipient reads survey questionRecipient responds to questionRecipient sees survey results

(Later, if needed…)Recipient visits site; no password!Recipient does password recovery

One-time link sent to emailRecipient creates password

Recipient can edit profile, etc.

Survey owner adds recipient to group

Survey owner asks question

Recipient gets invite

Recipient reads survey question

Recipient responds to question

Recipient installs mobile app

Recipient creates account, profile

Recipient sees survey results

Recipient can edit profile, etc.

10-2

5% R

ESPO

NSE R

ATE

70-9

0% R

ESPO

NSE R

ATE

Page 45: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Six business model archetypes(Yours is probably a blend of these.)

Page 46: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

E-commerce SaaS (freemium?) Mobile app (gaming) Two sided marketplace Media User generated content

Page 47: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

(Which means eye charts like these.)

Customer Acquisition Cost

paid direct search wom inherent virality

VISITOR

Freemium/trial offer

Enrollment

User

Disengaged User

Cancel

Freemium churn

Engaged User

Free user disengagement

Reactivate

Cancel

Trial abandonment rate

Invite Others

Paying Customer

Reactivationrate

Paid conversion

FORMER USERS

User Lifetime Value

Reactivate

FORMER CUSTOMERS

Customer Lifetime Value

Viral coefficientViral rate

Resolution

Support data

Account Cancelled Billing Info Exp.

Paid Churn Rate

Tiering

Capacity Limit

Upselling rate Upselling

Disengaged DissatisfiedTrial Over

Page 48: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Model + Stage = One Metric That Matters.

One MetricThat Matters.

The business you’re in

E-Com SaaS Mobile 2-Sided Media UCGEmpathy

Stickiness

Virality

Revenue

ScaleThe

stag

e yo

u’re

at

Page 49: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Really? Just one?

Page 50: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Yes, one.

Page 51: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

In a startup, focus is hard to achieve.

Page 52: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Having only one metricaddresses this problem.

Page 54: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Moz cuts down on metricsSaaS-based SEO toolkit in the scale stage. Focused on net adds.

Was a marketing campaign successful? Were customer complaints lowered? Was a product upgrade valuable?

Net adds up:

Can we acquire more valuable customers? What product features can increase engagement? Can we improve customer support?

Net adds flat:

Are the new customers not the right segment? Did a marketing campaign fail? Did a product upgrade fail somehow? Is customer support falling apart?

Net adds down:

Page 55: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Metrics are like squeeze toys.

http://www.flickr.com/photos/connortarter/4791605202/

Page 56: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Empathy

Stickiness

Virality

Revenue

Scale

E-commerce SaaS MediaMobile

appUser-gencontent

2-sidedmarket

Interviews; qualitative results; quantitative scoring; surveys

Loyalty, conversion

CAC, shares, reactivation

Transaction, CLV

Affiliates, white-label

Engagement, churn

Inherent virality, CAC

Upselling, CAC, CLV

API, magic #, mktplace

Content, spam

Invites, sharing

Ads, donations

Analytics, user data

Inventory, listings

SEM, sharing

Transactions, commission

Other verticals

(Money from transactions)

Downloads, churn, virality

WoM, app ratings, CAC

CLV, ARPDAU

Spinoffs, publishers

(Money from active users)

Traffic, visits, returns

Content virality, SEM

CPE, affiliate %, eyeballs

Syndication, licenses

(Money from ad clicks)

Page 57: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Better: bit.ly/BigLeanTable

Page 58: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

What other metricsdo you want to know about?

Page 59: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Drawing some lines in the sand.

Page 60: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

A company loses a quarter of its customers every year.

Is this good or bad?

Page 61: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Not knowing what normal ismakes you do stupid things.

Page 62: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Baseline:5-7% growth a week

“A good growth rate during YC is 5-7% a week,” he says. “If you can hit 10% a week you're doing exceptionally well. If you can only manage 1%, it's a sign you haven't yet figured out what you're doing.” At revenue stage, measure growth in revenue. Before that, measure growth in active users.

Paul Graham, Y Combinator

• Are there enough people who really care enough to sustain a 5% growth rate?

• Don’t strive for a 5% growth at the expense of really understanding your customers and building a meaningful solution

• Once you’re a pre-revenue startup at or near product/market fit, you should have 5% growth of active users each week

• Once you’re generating revenues, they should grow at 5% a week

Page 63: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Baseline:10% visitor engagement/day

Fred Wilson’s social ratios

30% of users/month use web or mobile app

10% of users/day use web or mobile app

1% of users/day use it concurrently

Page 64: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Baseline:2-5% monthly churn• The best SaaS get 1.5% - 3% a month. They have multiple Ph.D’s

on the job.• Get below a 5% monthly churn rate before you know you’ve got a

business that’s ready to grow (Mark MacLeod) and around 2% before you really step on the gas (David Skok)

• Last-ditch appeals and reactivation can have a big impact. Facebook’s “don’t leave” reduces attrition by 7%.

Page 65: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Who is worth more?

Today

A Lifetime:$200

Roberto Medri, Etsy

B Lifetime:$200

Visits

Page 66: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Baseline:Calculating customer lifetime

25%monthly churn

100/25=4The average

customer lasts 4 months

5%monthly churn

100/5=20The average

customer lasts 20 months

2%monthly churn

100/2=50The average

customer lasts 50 months

Page 67: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Baseline:CAC under 1/3 of CLV• CLV is wrong. CAC Is probably wrong, too.• Time kills all plans: It’ll take a long time to find

out whether your churn and revenue projections are right

• Cashflow: You’re basically “loaning” the customer money between acquisition and CLV.

• It keeps you honest: Limiting yourself to a CAC of only a third of your CLV will forces you to verify costs sooner.

Lifetime of 20 mo.$30/mo. per

customer$600 CLV

$200 CACNow segment those users!

1/3 spend

Page 68: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

The Lean Analytics cycle

Page 69: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Draw a new linePivot orgive up

Try again

Success!

Did we move the needle?

Measure the results

Make changes in production

Design a test

Hypothesis

With data:find a

commonality

Without data: make a good

guess

Find a potential improvement

Draw a linePick a KPI

Page 70: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Do AirBnB hosts get more business if their property is professionally photographed?

Page 71: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Gut instinct (hypothesis)Professional photography helps AirBnB’s business

Candidate solution (MVP)20 field photographers posing as employees

Measure the resultsCompare photographed listings to a control group

Make a decision Launch photography as a new feature for all hosts

Page 72: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

5,000 shoots per month by February 2012

Page 73: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Hang on a second.

Page 74: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Gut instinct (hypothesis)Professional photography helps AirBnB’s business

SRSLY?

Page 75: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Draw a new linePivot orgive up

Try again

Success!

Did we move the needle?

Measure the results

Make changes in production

Design a test

Hypothesis

With data:find a

commonality

Without data: make a good

guess

Find a potential improvement

Draw a linePick a KPI

Page 76: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

“Gee, those houses that do well look really

nice.”

Maybe it’s the camera.

“Computer: What do all the

highly rented houses have in

common?”

Camera model.

With data:find a commonality

Without data: make a good guess

Page 77: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Landing page design A/B testing

Cohort analysis General analytics

URL shortening

Funnel analytics

Influencer Marketing

Publisher analytics

SaaS analytics

Gaming analytics

User interaction Customer satisfaction KPI dashboardsUser segmentation

User analytics Spying on users

Page 78: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Some non-tech examples.

Page 79: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

I lied. Everyone is a tech company.

Page 80: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

http://www.flickr.com/photos/puuikibeach/4789015423 http://www.flickr.com/photos/elcapitanbsc/3936927326

Cost of experiments: down. Cost of attention: way up.

Page 81: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Let’s pick on restaurants for a while.

Page 82: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

A line in the sand

Labor costs

Gross revenue

30%

24%

20%

Too costly?

Just right

Understaffed?

=

Page 83: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

A leading indicator

http://www.flickr.com/photos/avlxyz/4889656453http://www.flickr.com/photos/mysticcountry/3567440970

50 reservationsat 5PM

250 coversthat night

(Varies by restaurant. McDonalds ≠ Fat Duck.)

Page 84: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

http://www.flickr.com/photos/southbeachcars/6892880699

Restaurant MVP

Page 85: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Is tip amount a leading indicator of long-term revenue?

Page 86: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Why does every table get the same menu?

Page 87: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Is purple ink better?http://tippingresearch.com/uploads/managing_tips.pdf

Page 88: Croll   lean analytics workshop (3h) - lean ux nyc april 2014
Page 89: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Growth hacking

(is a word you should hate but will hear a lot about.)

Page 90: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Growth hacking, demystified.

Find correlation

Test causality

Optimize the causal factor

Pick a metric to change

Page 91: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Guerrillamarketing

Data-drivenlearning

Subversiveness

GROWTHHACKING

Page 92: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya)

If someone comes back to Zynga a day after signing up for a game, they’ll probably become an engaged, paying user (Nabeel Hyatt)

A Dropbox user who puts at least one file in one folder on one device (ChenLi Wang)

Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman)

A LinkedIn user getting to X connections in Y days (Elliot Schmukler)

(from the 2012 Growth Hacking conference) (These are also great segments to analyze.)

The leading indicator

Page 93: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

• Growth hacking is simply what marketing should have been doing, but it fell in love with Don Draper and opinions along the way

• Optimize a factor you think is correlated with growth

The growth hack

Page 94: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

AirBnB and Craigslist

Page 95: Croll   lean analytics workshop (3h) - lean ux nyc april 2014
Page 96: Croll   lean analytics workshop (3h) - lean ux nyc april 2014
Page 97: Croll   lean analytics workshop (3h) - lean ux nyc april 2014
Page 98: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

What if you’re ina big organization?

Page 99: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

If a startup is an organization designed to search for a sustainable, repeatable business model, then an established company is an organization designed

to perpetuate one.

Page 100: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Intrapreneurs often have to use proxiesStage Startup metrics Intrapreneur metrics

EmpathyCustomers interviewed (needs &

solutions), assumptions quantified, TAM, monetization possibility

Non-customers interviewed; assumptions quantified, constraints identified, TAM, disruption potential

Stickiness Churn, engagement Support tickets, integration time, call center data, delays

Virality Viral coefficient, viral cycle time Net Promoter Score, referrals, case study willingness

Revenue Attention, engagementBillable activity; signed LOIs; pilot

programs; after-development profitability

Scale Automation Contribution, training costs, licensing

Page 101: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

The Lean Analytics lifecycle of an Intrapreneur

Empathy Find problems; don’t test demand. Skip the business case, do analytics

Entitled, aggrieved customers

Stickiness Know your real minimum based on expectations, regulations

Hidden “must haves”, feature creep

Virality Build inherent virality in from the start; attention is the new currency

Luddites who don’t understand sharing

Revenue Consider the ecosystem, channels, and established agreements

Channel conflict, resistance, contracts

Scale Hand the baton to others gracefully Hating what happens to your baby

Beforehand Get buy-in Political fallout

Page 102: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Some things that work.

Page 103: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Frame it like a studyProduct creation is almost accidental.Unlike a VC or startup, when the initiative fails the organization still learns.

http://www.flickr.com/photos/creative_tools/8544475139

Page 104: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Transformative isolation:Skunkworks

Page 105: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Use outliers and missed searches to hunt for good ideas & adjacencies

(Multi-billion-dollar hygiene product company)

1/8 men have an incontinence issue. 1/3 women do.When search results show a significant number of men searching, this suggests the adjacent (male) market is underserved.

Page 106: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Use data to create a taste for data

Sitting on Billions of rows of transactional dataDavid Boyle ran 1M online surveysOnce the value was obvious to management, got license to dig.

Page 107: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Focus on the desired behavior, not just the information.

http://www.psychologytoday.com/blog/yes/200808/changing-minds-and-changing-towels

26% increase in towel re-use with an appeal to social norms; 33% increase when tied to

the specific room.

Energy Conservation “Nudges” and Environmentalist Ideology: Evidence from a Randomized Residential Electricity

Field Experiment - Costa & Kahn 2011

The effectiveness of energy conservation “nudges” depends on an individual’s political ideology ... Conservatives who learn that their

consumption is less than their neighbors’ “boomerang” whereas liberals reduce their consumption.

Page 108: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Understand hidden constraints

That pencil story is a myth. Graphite is conductive and explosive. The Minimum Viable Product is Viable for a reason.

Page 109: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Know what has tobe built in-house

SAP integrationEmployment law

Page 110: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Run it as a consulting business first.

(Just don’t get addicted to it. Your goal is to learn and overcome integration challenges and find the 20% of features that 80% of the market

will pay for.)

Page 111: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

When in doubt, collect dataFrom tackling the FTA rate to visualizing the criminal justice supply chain.

Page 112: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Everything’s an excuse to experiment

Page 113: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Find other ways to collect data; everything is an experiment.

Page 114: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Don’t just collect data, chase it.

Page 115: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Some tools and traps

Page 116: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Traction graphs

Your business model

The stage you’re at

Your one metric

... change often if you’re doing it right.

So how do you track that over time?

Page 117: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Traction graphs

Jan Feb Mar Apr May Jun

Signupsper day

Conversionrate

Churnrate

Viralcoefficient

This axis changes for each metric

Page 118: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Traction graphs

Jan Feb Mar Apr May Jun

Signupsper day

Conversionrate

Churnrate

Viralcoefficient

0%

Page 119: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Use vanity to get to meaningful metrics

Your goal is to produce outcomes

If the outcomes require action, and vanity motivates actors, use it

But show how the vanity metric is a leading indicator of the real one

x

Web traffic

Revenue

Activation

CartSize

Conversion rate

Page 120: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

The three threesThreeassumptions

What big bets are you making?•“People will answer questions”•“Organizers are frustrated with how to run conferences”•“We'll make money from parents”•“Amazon is reliable enough for our users.”

Three actionsto take

What are you doing to make these assumptions happen (or identify they’re wrong and change course?)•Product enhancements•Marketing strategies

Three experimentsto run

•Feature tests•Continuous deployment•A/B testing•Customer survey

Page 121: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

The three threes

Threeassumptions

Three actionsto take

Three experimentsto run

Monthly

Weekly

Daily

Board, investors, founders

Executive team

Employees

Strategy

Tactics

Execution

Page 122: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

The three threesThreeassumptions

Three actionsto take

Three experimentsto run

Get more people

Increase answer %

Test betterquestions

Change the UI

Test timings

Questions from peers

Many people will answer questions

Page 123: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

The problem-solution canvasCURRENT STATUS

• List key metrics you’re tracking, where they’re at, and compare with last few weeks• How are things trending?

LAST WEEK’S LESSONS LEARNED AND ACCOMPLISHMENTS)

• What did you learn last week?• What was accomplished?• On track: YES / NO?

The Goal is to Learn

Page 124: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

The problem-solution canvasHYPOTHESIZED SOLUTIONS

• List possible solutions that you’ll start working on next week. Rank them.• Why do you believe each solution will help you solve or complete solve the problem?

METRICS / PROOF + GOALSProblem #1 (put name here)

• Metrics you’ll use to measure whether or not the solutions are doing what you hoped (solving the problem)• List proof (qualitative) you’ll use as well• Define goals for the metric

HYPOTHESIZED SOLUTIONS

• List possible solutions that you’ll start working on next week. Rank them.• Why do you believe each solution will help you solve or complete solve the problem?

METRICS / PROOF + GOALS

• Metrics you’ll use to measure whether or not the solutions are doing what you hoped (solving the problem)• List proof (qualitative) you’ll use as well• Define goals for the metric

Problem #2 (put name here)

Page 125: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

“The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them.”

Lloyd S. Nelson

Page 126: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Pic by Twodolla on Flickr. http://www.flickr.com/photos/twodolla/3168857844

Page 127: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

ARCHIMEDES HAD TAKEN

BATHS BEFORE.

Page 128: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Once, a leader convinced othersin the absence of data.

Page 129: Croll   lean analytics workshop (3h) - lean ux nyc april 2014

Now, a leader knowswhat questions to ask.