how to conclude online experiments in python

volodymyrk

How to conclude online experiments

in PythonVolodymyr (Vlad) Kazantsev

Head of Data Science at Product Madness

volodymyrk

volodymyrk

Goal of the tutorial

Uncover the “magic” behind statistics used for A/B testing and other online experiments

volodymyrk

● Head of Data Science (Social Gaming)

● Product Manager at King

● MBA at London Business School

● Visual Effect developer (Avatar, Batman, ...)

● MSc in Probability (Kiev Uni, Ukraine)

A quick bioNow

2004

volodymyrk

Different kinds of tests

● Classic A/B tests

● Long running activities with control groups

● Longitudinal tests

volodymyrk

Why bother?

● To test your hypothesis and learn● To avoid blindly following HiPPOs● To audit performance of product and

marketing teams

volodymyrk

Why Stats?

● To separate data from the noise● To quantify uncertainty

volodymyrk

Fruit Crush Epic

The Story of almost real mobile game, in the almost real gaming company.. and one Data Scientist

volodymyrk

Day-13 seconds panic-attack

volodymyrk

Day 1 - loading time panic-attack!Fruit Crush Epic

volodymyrkTaxonomy of Classical stat testing

Which Test?

1 Sample

2 Samples

>2 Samples

Mean

Proportion

Variance

σ known

σ unknown

z-test one sample

t-test one sample

z-test for proportion

Chi-squared test

Mean

Proportion

Variance

ANOVA

z-test for (μ1-μ2)

t-test for (μ1-μ2)

z-test or t-test for dependent samples

z-test, 2 proportions

independent

dependent samples

σ1,σ2 known

σ1,σ2 unknown

F-test

volodymyrkTaxonomy of Classical stat testing

Which Test?

1 Sample

2 Samples

>2 Samples

Mean

Proportion

Variance

σ known

σ unknown

z-test one sample

t-test one sample

z-test for proportion

Chi-squared test

Mean

Proportion

Variance

ANOVA

z-test for (μ1-μ2)

t-test for (μ1-μ2)

z-test or t-test for dependent samples

z-test, 2 proportions

independent

dependent samples

σ1,σ2 known

σ1,σ2 unknown

F-test

volodymyrk

One sample t-testNull Hypothesis:- avg. loading time <=3 seconds for last hour's observation

Alternative Hypothesis:- population mean is >3 seconds for last hour's observation

Test:- single sample, one-sided t-test.

volodymyrk

One sample t-test

t_value = t-test(samples, expected mean)

p-value: 0.086 probability of obtaining the result as extreme as observed, assuming Null-hypothesis is true

t-distribution lookup(t_value, sample_size)

volodymyrk

If you want to code it yourself

volodymyrk

Stats in Python

numpy

scipy.stats

statsmodels.stats

theano

pymc3

Classical Bayesian

* High-level view. Lot’s of stuff missing here. pymc3 uses statsmodels for GLM

volodymyrk

One sample t-test and z-test

volodymyrk

Confidence Interval

volodymyrk

Confidence Interval for the Mean

volodymyrk

Standard Error of the Mean in Python

volodymyrk

Next Day

volodymyrk

Day-2OMG, my Retention is low!

volodymyrk

Is my day-1 retention low?

Day-1 results:

installs 448

returned next day 123

Day-1 retention 27.46%

Retention target 30%

Fruit Crush Epic

volodymyrkTaxonomy of Classical stat testing

Which Test?

1 Sample

2 Samples

>2 Samples

Mean

Proportion

Variance

σ known

σ unknown

z-test one sample

t-test one sample

z-test for proportion

Chi-squared test

Mean

Proportion

Variance

ANOVA

z-test for (μ1-μ2)

t-test for (μ1-μ2)

z-test or t-test for dependent samples

z-test, 2 proportions

independent

dependent samples

σ1,σ2 known

σ1,σ2 unknown

F-test

volodymyrk

One sample z-test for proportionNull Hypothesis:- avg. retention >=30%

Alternative Hypothesis:- avg. retention <30%

Test:- single sample, one-sided z-test for proportion

volodymyrk

In Python...

volodymyrk

So what is my confidence interval?

volodymyrk

Day-5Connect with Facebook or Die!

The First A/B test

volodymyrk

A/B test 1 - connect to Facebook

volodymyrk

A/B test design

Group A

Group B Start Level 1

Start Level 1

Finish Level 1

50%

50%

Have seen prompt 2501

Connected 1104

Connect rate 44.1%

Have seen prompt 2141

Connected 1076

Connect rate 50.2%

Fruit Crush Epic

volodymyrk

Is it statistically significant?Fruit Crush Epic

volodymyrkTaxonomy of Classical stat testing

Which Test?

1 Sample

2 Samples

>2 Samples

Mean

Proportion

Variance

σ known

σ unknown

z-test one sample

t-test one sample

z-test for proportion

Chi-squared test

Mean

Proportion

Variance

ANOVA

z-test for (μ1-μ2)

t-test for (μ1-μ2)

z-test or t-test for dependent samples

z-test, 2 proportions

independent

dependent samples

σ1,σ2 known

σ1,σ2 unknown

F-test

volodymyrk

Two samples z-test for proportionNull Hypothesis:- avg. connection rate is the same. P1 = P2

Alternative Hypothesis:- P1 ≠ P2

Test:- two samples z-test for proportion. Two sided

volodymyrk

Two samples z-test for proportion in Python

volodymyrk

Confidence interval for difference in proportion

volodymyrk

In Python

volodymyrk

What should we measure, exactly?

1000

1000

150

400

450

30

390

430

160

840

40

400

400

connected: 47%retained: 82%

connected: 50%retained: 80%Start

Level 1

Start Level 1

Start Level 2

Start Level 2

volodymyrk

What about Bayesian Stats?

volodymyrk

Bayesian Credible Interval vs. CI

volodymyrk

Day-30Do you want to buy last chance?

A/B testing Revenue

volodymyrk

How much an extra life is worth?

LOSER!!!

Purchase another chance

for only..

$0.99

LOSER!!!

Purchase another chance

for only..

$1.99

Fruit Crush Epic

volodymyrk

How we are going to test it?Consider● There are multiple items to buy in game (lives, boosters, blenders, etc)● We expect more people to make a $0.99 purchase, so we hope to make

more money overall, even at lower priceA/B test Design● We will show A/B test to new users only● Will run for 2 months● We will measure overall revenue per user in the first 30 days● Null-hypothesis: we make more money from $0.99 group

Measurements● Difference in Average Revenue Per User (ARPU) in 30 days● Difference in Conversion Rate (%% of users who make at least 1 purchase)

volodymyrk

Results

count 450 390mean 151.9 214.225% 20.8 26.550% 55.3 69.475% 147.3 231.3max 3960 3647.8

Fruit Crush Epic

* random generator used in the example is available in ipython notebooks** distribution is made more extreme than what is normally observed in casual game, like our imaginary match-3 title

volodymyrk

Results

30,000 users in each group450 payers 390 payers

p-value = 0.037Significant

p-value = ???Is it Significant?

volodymyrkTaxonomy of Classical stat testing

Which Test?

1 Sample

2 Samples

>2 Samples

Mean

Proportion

Variance

σ known

σ unknown

z-test one sample

t-test one sample

z-test for proportion

Chi-squared test

Mean

Proportion

Variance

ANOVA

z-test for (μ1-μ2)

t-test for (μ1-μ2)

z-test or t-test for dependent samples

z-test, 2 proportions

independent

dependent samples

σ1,σ2 known

σ1,σ2 unknown

F-test

volodymyrk

Welch's t-test (σ1≠σ2)

Can we actually use t-test?

volodymyrk

Poor’s man non-parametric test: split 5

p < 3%

volodymyrkIf you don’t know enough stats - simulate!

This is very close to p-value from t-test

volodymyrk

Can we improve sensitivity?27 players, who have spent > $1000 in both group.10 in $0.99 group and 17 in $1.99 groupMax spent = $3960

volodymyrkAnd we re-run our analysis

Again, we can use t-test

volodymyrk

Final Thoughts

volodymyrk

Can we analyse distributions?

You can quantify difference between two curvesArea under the curve is Average Revenue per User

Fruit Crush Epic

* random generator used in the example is available in ipython notebooks** distribution is made more extreme than what is normally observed in casual game, like our imaginary match-3 title

volodymyrk

Is 30 day revenue a good metric?LTV projection A LTV projection B

Fruit Crush Epic

volodymyrk

Summary:

● There are only few stats tests that any Data Scientist must know

● t-tests are robust to be useful even with skewed data sets

● Bayesian and MCMC is cool, but don’t use MCMC for trivial cases

● It is hard to detect the difference in heavily-skewed cases

IPython Notebooks for this tutorial are available at: http://nbviewer.ipython.org/github/VolodymyrK/stats-testing-in-python