Spice up your conversions

#DDTT

Speaker

#DDTT

Hubert Wassner Chief Data Scientist

@hwassner

Christopher Broque Senior Business Developer christopher@abtasty.com

#DDTT

Paris Londres Cologne Sydney Madrid New-York

120+ Passionate people

450+ references

21+ awards

9+ billions Tested visitors

23ème

107ème

AB Tasty : CRO specialist !

The Data Science behind testing, optimization and personalization

#DDTT

A/B Test is easy !

5 % clicks

+16% clicks

-2 % clicks

#DDTT

But in production...

+1% clicks

#DDTT

What's wrong ?

Statistical tests are mandatory !

#DDTT

A/B test is not that simple...

First step

Variation B has 10 % (measured) gain & 95 % confidence index... but 1 % gain in production is likely!

The frequentist Approach

Provides a Pvalue = « The probability to see such data without any (real)

difference between A &B. »

Confidence index = (1-Pvalue)*100

Problem : It only qualifies the existence of a difference, not it's size! Example :

#DDTT

Smarter step

Bayesian Approach

give confidence interval on the gain Example : gain (A->B) is in [1%:15 %]

1. No bad surprise in production

&

2. Better choice when facing implementation cost. Example: testing a product recommender system

#DDTT

Real customer case

Context

Two recommender systems for a cosmetic & perfume retailer

A

B

C

#DDTT

What about media ?

E-commerce

N visits for a “unique” conversion => CTR [0:1]

Media

N visits for n “multiple” conversions (where n>N) => CTR [0:∞]

#DDTT

E-commerce ≠ media

E-commerce

1 visitor x 10 conversions =

10 visitors x 1 conversion

Media

1 visitor x 10 conversions <

10 visitors x 1 conversion

#DDTT

And then ?

Basic formulas doesn't work any more...

One need to handle more complex formulas.

#DDTT

Multiple conversion Analysis

A - Original 12 000 visitors 2 000 conversions CTR : 16,6%

Gain : [-3% : 28%] => 7,5% [0.7% : 15%] => 7,3% Chances of winning : 87% 98%

B A + 1 visitor, with 150 conversions CTR : 17,9%

C A + 15 visitors, with 10 conversions each CTR : 17,9%

12 000 visitors 2 000 conversions CTR : 16,6%

A + 1 visitor, with 150 conversions CTR : 17,9%

A + 15 visitors, with 10 conversions each CTR : 17,9%

Multiple conversion Analysis

A - Original

[-3% : 28%] => 7,5% [0.7% : 15%] => 7,3%

87% 98%

B C

Gain :

Chances of winning : #DDTT

We need to go faster !!

Exploitation time

Paper news < 8 days Private sale 2-4 days Classified ad < 8 days

No time for an A/B test...

Need to do a “smart” test

#DDTT

“Multi-armed bandit”

Imagine you have :

• N different slot machines • P coins

Objective :

• Win a maximum with minimum coins

#DDTT

Application to web data

Imagine you have :

• N variations • P visitors

Objective

• Making the more transaction you can

#DDTT

Application to web data

Intérêts : • Limiting regrets (lower T.C.O.) • Optimizing short living objects, like flash sales, newspaper article, ...

• Testing simultaneously more variations • Classic A/B test ~= testing A/B/C/D/E with « bandits »

#DDTT

Conclusion

A/B test can be more complex that you might think

It has evolved with specificity of web businesses.

And that's only the beginning! (Artificial Intelligence is on the road...)

Questions ?

@hwassner

Hubert Wassner Chief Data Scientist

#DDTT