airbnb offline experiments

35
A/B Testing In The Real World How to run experiments in an “oine” setting Elena Grewal 2014-04-09 Big Data Innovation Summit

Upload: elena-grewal

Post on 04-Dec-2014

302 views

Category:

Data & Analytics


0 download

DESCRIPTION

Elena Grewal presented these slides on a/b testing in the real world (offline experiments not online) at the Big Data Innovation Summit on April 9, 2014.

TRANSCRIPT

Page 1: Airbnb offline experiments

A/B Testing In The Real WorldHow to run experiments in an “offline” setting

Elena Grewal 2014-04-09

Big Data Innovation Summit

Page 2: Airbnb offline experiments

The Plan

2

1

2

“Offline” experiments: what and why?

Some experiment pitfalls and advice

3 Conclusions

Page 3: Airbnb offline experiments

But first, you might ask: What is Airbnb?

Page 4: Airbnb offline experiments

Airbnb is an online marketplace for accommodations

4

Part of the “sharing economy”

Page 5: Airbnb offline experiments

Search in San Francisco

5

Page 6: Airbnb offline experiments

Come Stay In My Home!

6

Page 7: Airbnb offline experiments

That looks like a website. What do we mean by “offline”?

Page 8: Airbnb offline experiments

Guest Journey

Page 9: Airbnb offline experiments

Host Journey

Page 10: Airbnb offline experiments

Offline Operations Departments

+ Customer Support

+ Local Operations

+ Professional Photography

+ Many others…

10

Page 11: Airbnb offline experiments

Customer Support

Page 12: Airbnb offline experiments

Local Ops Teams

Page 13: Airbnb offline experiments

Photography

!

+ 3,000 Photographers worldwide

+ Over 100k listings photographed

+ Almost 2 million professional photos13

Page 14: Airbnb offline experiments

Stepping back

14

+ Many companies have offline operations + Can optimize these using experiments

!

!

!

Online Experiments:

We run these all the time too.

If you are curious about on our online experimentation see Jan Overgoor’s tech talk

http://nerds.airbnb.com/tech-talks/

Page 15: Airbnb offline experiments

Why Do We Need Experiments?

Page 16: Airbnb offline experiments

Before and after won’t work

16

• Often very little data before professional photos are added • Seasonality and other confounding factors bias results

Page 17: Airbnb offline experiments

Selection bias often impacts analysis

17

• Listings that opt to get professional photography are not the same as listings that do not get photography

Page 18: Airbnb offline experiments

Without an experiment, we don’t know the causal effect

18

This is the same reason we need online experiments

Date01−01 01−15 02−01 02−15 03−01 03−15

Product Launch

Product Rollback

Launch initiative: e.g. Offered Free Professional Photography

Page 19: Airbnb offline experiments

Traditional A/B Testing Online

Great sources: http://mcfunley.com/design-for-continuous-experimentation http://www.evanmiller.org/how-not-to-run-an-ab-test.html

Control Treatment

19

Page 20: Airbnb offline experiments

-5%

-4%

-3%

-2%

-1%

0%

1%

2%

3%

4%

5%

0 4 8 12 16 20 24 28 32 36

Del

ta

Treatment Effect for Price Filter Experiment

Initial Results Look Good

20

Δ > 0 : “positive”

0.00

0.10

0.20

0.30

0.40

0 4 8 12 16 20 24 28 32 36

p-va

lue

Days since start of experiment

P-Value

p < 0.05 : “significant”

Page 21: Airbnb offline experiments

-5%

-4%

-3%

-2%

-1%

0%

1%

2%

3%

4%

5%

0 4 8 12 16 20 24 28 32 36

Del

ta

Treatment Effect for Price Filter Experiment

Actually, NeutralStatistical significance by itself does not tell the whole story

p = 0.4 : “noise”

Δ = 0 : “neutral”

21

0.00

0.10

0.20

0.30

0.40

0 4 8 12 16 20 24 28 32 36

p-va

lue

Days since start of experiment

P-Value

p < 0.05 : “significant”

Page 22: Airbnb offline experiments

Offline Experiment Examples

Page 23: Airbnb offline experiments

Professional PhotographyLet’s run an experiment!

23

More bookings?

Page 24: Airbnb offline experiments

Beware of CannibalizationThe unit of randomization depends on the effect we want to estimate

24

!

!

Page 25: Airbnb offline experiments

Local Operations: Market Level Experiment

25

!

+ Smaller “long tail” markets < 100 reviewed listings

Randomize Markets 93 Treatment / 92 Control

Assess impact of operational strategy on market growth

+ Statistically measure the lift due to local ops teams + Measuring active listings, hosts, reviewed listings, and

bookings

Page 26: Airbnb offline experiments

Market Distribution U.S. & Europe

26

Page 27: Airbnb offline experiments

Finding: Local Ops Efforts Have Positive Impact on Growth

27

Active Listings

Control17% Growth

Local Ops Kickoff

Treatment 31% Growth

Page 28: Airbnb offline experiments

Case Study: Campos do Jordão, BR

28

+ Market grew 9x + Over 90% of the new listings are from new users + Low CPA + Primary approach is phone sales + Other approaches were less successful

+ 862%

+ 7%

Use qualitative research to understand what happened

Active Listing Growth

Treatment Control

Page 29: Airbnb offline experiments

Host EducationImproving listings through outreach

29

+ Initially not launched as an experiment and found positive impact + Launched as an experiment and found neutral impact + Don’t need market level approach here! !

Page 30: Airbnb offline experiments

Some takeawaysUse context to improve operations

30

+ Can investigate heterogeneity in treatment effects with higher N + Word of caution: can’t just compare those who were reached

by a call or email to the control (selection bias strikes again)

Page 31: Airbnb offline experiments

Compare entire treatment to entire control

31

!Treatment

!!

!Control

!!

Called

vs.

Page 32: Airbnb offline experiments

Additional Offline vs. Online Considerations

32

+ Opt-in biases + You know you are in an experiment (Hawthorne/John Henry effects) + Monetary incentives impact external validity, trade-off take-up rate

+ Takes time to adjust to a change + Sample size may be limited by ops capacity + Stakeholders may be less data-savvy + Real people delivering the experiment! + Ethical considerations !

Always partner with customer support.

!

Page 33: Airbnb offline experiments

Takeaways

+ Controlled experiments are the way to go if you want to make causal inference + Use them to optimize operations! !

but: + Level of randomization - what impact do you want to measure? + Cannibalization + Compare the right groups - no selection bias + Break down results to get the most from the analysis + Be practical/ethical - you are dealing with real people here

33

Page 34: Airbnb offline experiments

!

!

Questions? !

!

@elenatej [email protected] !

we’re hiring: www.airbnb.com/jobs

Page 35: Airbnb offline experiments