M A R K E T I N GA N A L Y T I C S

M A R I O A V I N A S C O

D I R E C T O R D A T A S C I E N C E A N D A N A L Y T I C S

F I N T E C H S T A R T U P

F O R M E R U B E R T E C H N O L O G I E S

F O R M E R F A C E B O O K

. . . . . .

T O P I C S

Brief Intro to AI and ML

Models used in Marketing

Use Cases

Q&A

T H E M O S T P O P U L A R U S E O F A I I S I M A G E R E C O G N I T I O N

M O D E L S A T U B E R : " C H U R N "

0%

100%

True Positive

80% Threshold

METRIC

Accuracy how often is the classifier correct?

(TP+TN)/total

Precision When it predicts yes, how often is it correct?

TP/predicted yes

Recall When it's actually yes, how often does it predict yes?

TP/actual yes

O T H E R B R E A K T H R O U G H S I N A I

- Embedding: vector representations of user behavior, words, sounds

- Automated feature engineering: Transformation of the input data using math

- Reinforced learning: multiple neural nets providing feedback to each other

- Reverse bootstrap: data set contain many labels, with noise. This technique

partitions the data, uses 1st partition to score second partition and remove

ambiguous labels

C A S E S. . . . . . +

Communication Frequency

Channel Efficiency

Q&A

C A S E S. . . . . . +

Communication Frequency

Channel Efficiency

Q&A

H O W M A N Y E M A I L S S H O U L D W E S E N D T O O U R R I D E R S / D R I V E R S ?

What is the optimal level so we avoid un-subscription and

yet communicate well?

For a long time, we had a ‘One size’ fits all

We capped the number of marketing emails to our Riders, Drivers and Eaters at a fix number per week

Clearly, some segments expected less communications and unsubscribed accordingly, while others appreciated higher numbers.

Its not an easy problem to identify these segments

U S E S E V E R A L M L M O D E L S A S S E G M E N T A T I O N T O O L S

. . . . . .

By crossing predictive scores we

identify cohorts of high and low

risk for comms.

W E F O R M U L A T E D A N E W C O M M U N I C A T I O N S T R A T E G Y

. . . . . .

We recommended a reduction for the red

cohort and increase for the green one.We estimated that approximately 500K un-subscriptions were

avoided by implementing this model

M O D E L P E R F O R M A N C E – C R O S S S E L L

. . . . . .

The model identifies a segment

with +15X the conversion rate

Avg Conversion

High Conversion 15X

C A S E S

Communication Frequency

Channel Efficiency

Q&A

H O W D O E S M A R K E T I N G I N V E S T M E N T M O V E M Y K P I ?

Where and how much should I invest?

Optimization is the process of allocating resources (usually money) under constraints in order to maximize (or minimize) an outcome

For us, it usually means marketing spend to maximize driver’s trips or acquisition

THIS IS TYPICALLY A PROBLEM OF LOW DATA VOLUMES AND LOTS OF PARAMETERS:

Coefficients

Trend terms

Ad Stock

W H A T D O E S A M E D I A M I X M O D E L D O ?

. . . . . .

It answer: How does

marketing investment move

my KPI?

FORECAST/BENCHMARK

ATTRIBUTION PERFORMANCE

BUDGET OPTIMIZATION

C I T Y M O D E L S A R E C R E A T E D U S I N G T I M E S E R I E S F I T T I N G

SPEND VARIABLES RESPONSE

T I M E S E R I E S F I T T I N G W I T H P Y D L M

THE MODEL IS COMPRISED OF:

Trend

Seasonality

Regressors

P R O B L E M F O R M U L A T I O N

Excel prototype

M A R K E T I N G O P T I M I Z A T I O N R E S U L T S

Optimization is now automated and it reduces 2 weeks of planning and iterations to minutes

In Q3 – 2018 we were able to reduce marketing spend by ~15% in the USA and achieve our targets in terms of First Trips

Ad Stock

Coefficients

Trend terms

We developed a rapid iteration and prototyping process, and we can onboard new models in weeks or less as opposed to months

T H A N K Y O U !

M A R I O A V I N A S C O

M A R I O _ V I N A S C O @ A L U M N I . S T A N F O R D . E D U