daniel sheinberg | head of product management, adobe target...

© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. #AdobeSummit @DanielSheinberg @pradeepj2010

Maths models matter in last millisecond targeting Daniel Sheinberg | Head of Product Management, Adobe Target

Pradeep Javangula | Head of Engineering, Adobe Target

1

© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. #AdobeSummit @DanielSheinberg @pradeepj2010 2

Daniel Sheinberg Head of Product Management, Adobe Target [email protected]


Pradeep Javangula Head of Engineering, Adobe Target [email protected] twitter: pradeepj2010


Be a hero with Automated Personalisation

© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. #AdobeSummit @DanielSheinberg @pradeepj2010 © 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 5

Data Science and the New Marketer

7

Machines can help you reach your goal.



Today’s Narrative



The New Marketer Today’s Narrative



The New Marketer Today’s Narrative Data Science


Key Takeaways

Put the power of machine learning in

your hands

1 Marketers need to

focus where they drive differentiated value

2 The opportunity is

there today

3


What is Machine Learning?

13

• Machine learning is a type of • Artificial intelligence • Ability to learn without being explicitly

programmed. • Development of programs that can

teach themselves to grow and change when exposed to new data

• No underlying mathematical construct • Model based on statistics of prediction • Self-correct at run-time • Ad-hoc explore & exploit


The Hero

The Hero How do you optimize the hero spot?


Geography Segmentation

Basel

London

Athens

Paris

Barcelona


Affinity Segmentation

16

Surf

Cycle

Yoga

Run

Hike


How Many Segments Can You Target?

17

• Referrer Variables • Site Behaviour Variables • Offline Variables • Temporal Variables • Environmental Variables


What is the solution?

Is there a better solution? Is this the best way to display content to your users? How will you hit your KPI’s? Is this the best use of YOUR time?


®

Copyright 2012 Adobe Systems Incorporated. All rights reserved. Adobe confidential.

Do you think it’s potentially valuable to target different content to different audience segments? Are you targeting content on your site today? Are you targeting to multiple segments simultaneously? Are you updating targeting rules every 6 hours?

© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 20


Agenda

1 | Why use Machine Learning



4 | What’s ahead

5 | Q & A


Adobe Automated Personalisation

22

• Self-learning algorithm

• Visitors analysed uniquely at individual level

• Real time decision making

• Unlimited variables are used


Behind the Curtains of Adobe Target Automated Personalisation

Combiner

General Score

User Based Score

Multi-Armed Bandit


Generalised Score

• Is the conversion rate of an offer • General score does not take into account the user profile

24

2% of users click on banner 4% of users click on banner


What feeds the model?

environment variables + IP address

+ Country of origin

+ Time zone

+ Device type

+ Operating system

+ Browser type

+ Screen resolution

offline variables + CRM

+ Call center

+ Ticket sales

+ 3rd-party data

temporal variables + Time of day

+ Day of week

+ Recency

+ Frequency

referrer variables + Referring domain

+ Campaign ID

+ Affiliate

+ PPC

+ Natural search

+ Direct/bookmark

+ Social graph/login

site behaviour variables + Customer / Prospect

+ New/return visitor

+ Previous visit patterns

+ Previous Product interests – top level

+ Previous Product interests – low level

+ Searches

+ Previous online purchases

+ Previous Campaign exposure

+ Previous Campaign responses

User Based Score


Conversion Rate by Geography

26

0.00

0.01

0.01

0.02

0.02

0.03

0.03

0.04

0.04

0.05

0.05

London Paris Athens Basel

Cycling

Snowboarding


t-Test

• Identifies if the difference between the performance of two groups is statistically significant

• Evaluates signal to noise ratio

29

Most Significant


User Based Scoring

• Decision trees • Find the best split • Update data to account for the split • Repeat until stopping criteria

• The output is a score for each offer

30


Decision Tree

31

Decrease conversion rate -0.075 Country≠UK

Age<30 Increase conversion rate +0.05

Age=>30 Decrease conversion rate -0.04

1000 150 .15

Users Conversions Conversion Rate

Split is based on geography

600 120 .20

400 30 .075



Increase conversion rate +0.05 Country=UK


Model and Score

32

8 2 Conversion rate for this user profile

•Lives in Basel, Switzerland •24 year old male •Uses Safari •Shops work hours •Browsed winter sports category


How the two scores are used

33

The user based score and the general score are combined.

Combiner

User Based Score

General Score


Combiner

The general and user based scores are combined to generate a single number.

34

5 2 1 8

4 2

Mountain Bike Banner

Snowboarding Banner

User Based Score User Based Score

General Score

20 16 User A best predicted score

User B best predicted score


Multi-Armed Bandit

35

Multi-Armed Bandit


16 8

Multi Armed Bandit Algorithm

36 36

The Snowboard offer is twice as likely to be displayed

•Lives in Basel, Switzerland •24 year old male •Uses Safari •Shops work hours •Browsed winter sports category


Behind the Curtains of Adobe Target Automated Personalisation

Combiner

General Score

User Based Score

Multi-Armed Bandit

http://www.polleverywhere.com/app

http://www.polleverywhere.com/app/help

https://www.polleverywhere.com/multiple_choice_polls/djonqP1UMZitmtH?preview=true


Agenda




4 | What’s ahead

5 | Q & A


Let the machine do the work

40

Watch the money pile up


What is my role as a marketer?

41

Get the most out of the machine


How to use Machine Learning to your advantage

Identify where to use Automated Personalisation • High Volume pages • Pivotal navigation junction

Gather differentiated content • Use what you have to start

Ongoing optimisation • Develop new content alternatives • Leverage your audience data

42


Beyond the Machine

43

Optimize with Automated

Personalisation

Review Results

Generate Insights • Content Ideas • Audience

Opportunities

Set Up • Location • Content Options • Audiences

• Expand Automated Personalisation

• New A/B testing

Leverage Insights


How to get Started

44

• Identify the opportunity

• Bring in consulting

• Validate the concept

• Leverage expertise

• Get buy-in within your organisation

• Describe the potential to key stakeholders

• Share success stories from other companies

• Start small and build up



https://www.polleverywhere.com/multiple_choice_polls/ENuGAhCd6lc2UUl?preview=true



https://www.polleverywhere.com/multiple_choice_polls/2id19ukJIMWwjaS?preview=true


Agenda




4 | What’s ahead

5 | Q & A


Automated Personalisation Now and Tomorrow

49


New Algorithm: Random Forest

50


Key Takeaways

Put the power of machine learning in

your hands

1 Marketers need to

focus where they drive differentiated value

2 The opportunity is

there today

3


1 | Download the presentation to get started www.adobe.com/go/summit2014portal

Call to Action

2 | Be a Hero!


Questions and Answers


Daniel Sheinberg [email protected] Twitter: @DanielSheinberg LinkedIn: Daniel Sheinberg

Pradeep Javangula [email protected] LinkedIn: Pradeep Javangula


Every Session Complete 8 surveys to enter the prize draw

Go HERE, do THIS, and you could WIN!

Download the App

Survey available immediately after

the session

Give us feedback It takes less than a minute

Eight to win

Win a MeCam Record every minute

Win an iPad Air


May 14-15, 2014 | ICC ExCel, London

Learn more at summit.adobe.com/emea #AdobeSummit

daniel sheinberg | head of product management, adobe target...

Documents