jon puleston

PREDICTION SCIENCE

BACKGROUND

• Prediction protocol in survey are a great means of motivating respondents to think that can deliver back great data

• As a result we have started to use them a lot in many of the surveys we run

• Estimated we have conducted over 30 research on research experiments using various prediction protocols

• But lots of question marks exist over what the data means and what is the real accuracy of these predictions

• We have assembled today a lot of all this data and we are hoping you can help us start to make better sense of it.

TRADING GAMES

Fun

Inte

rest

ing

Thinkin

g time

0

40

80

120

Standard Trading game

-200%

-100%

0%

100%

200%

300%

400% Trading game

Standard rating

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101106111116121126131136141146151156161166171176181-100%

-80%

-60%

-40%

-20%

0%

20%

40%

60%

80%

100%

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101106111116121126131136141146151156161166171176181-100%

-80%

-60%

-40%

-20%

0%

20%

40%

60%

80%

100%

Standard rating certainty point

Trading game certainty point

If 20% of respondents are making up answers because they don’t care what they say, you need an extra 40% more sample to smooth out the systematic errors this introduces

THE TYPES OF EXPERIMENTS WE HAVE RUN• Betting on the future of brands• Predicting why people buy things• Predicting the price of things• Predicting the election prospects of political parties• Predicting football match results• Predicting the outcomes of TV game shows• Predicting the success of adverts• Predicting future sales of products• Predicting the future more generally

WHAT DO WE KNOW ABOUT PREDICTION?

PREDICTION CAN BE A LOT MORE ACURATE THAN TRADITIONAL POLLING

WHO WILL YOU VOTE FOR? 1,000 respondents

PREDICT WHO WILL WIN?20 people gambling

IOWA ELECTION MARKETIN 451 out of 596 OCCASIONS

ü

ELECTION FORECASTING

PREDICTION KNOWNS

The CXO Advisory group gathered 6,582 predictions from 68 different investing gurus made between 1998 and 2012, and tracked the results of those predictions how accurate were they?

48% correct

WILL YOU CLEAN UP?YES = 50%

PREDICT HOW MANY WILL CLEAN UP?YES = 15%

HOW MANY CLEANED UP?

WE ARE NOT GOOD AT PREDICTING OUR OWN BEHAVIOUR

13%

COGNITIVE BIASES CAN CORRUPT SO MANY ANSWERS TO TRADITIONAL SURVEY QUESTIONS

HEADS OR TAILS?

THIS IS AN OPPORTUNITY TO SWITCH FROM TREATING RESPONDENTS AS BIT OF DATA TO TREATING THEM AS A PROBLEM SOLVING ENGINE

BY REFRAMING QUESTIONS INTO PROBLEMS….

IF I TOSS THIS COIN 100 TIMES HOW MANY OCCASSIONS DO YOU PREDICT IT WILL BE HEADS?

EXAMPLE OF HOW THIS THINKING CAN BE APPLIED IN A SURVEY

HOW MANY IPADS ARE THERE IN CHINA?

• Do you own an Ipad? Yes = 30%• Predict how many of your friends, family and colleagues

own an ipad Average estimate = <5%

WHAT WE WANT TO KNOW!

OUR QUESTIONS

• How good are we are predicting things?• What are people good at predicting and what are they not

good at predicting?• Why?• What can influence our predictions?• What impact does emotion have on our predictive decision?• How do our predictions about other people’s behaviour

correlate with our self perception • What types of people are good at making predictions?• What makes a good predictor?• Is there such a thing as a good and bad predictor – can we

measure this?• Are there any gender differences in our predictive powers?

Key trend observed

“SOME PEOPLE ARE BETTER AT PREDICTING THAN OTHERS”

• Is that true or not?• What type of people?• How much big a difference can be observed?• Why?• What further research/analysis would you

recommend?

Philip Tetlock

“EXPERTS ARE NOT AS GOOD AT PREDICTING AS NON-EXPERTS”

“THE MORE FAMOUS YOU ARE THE WORSE YOU ARE AT MAKING PREDICTIONS”

Hedgehogs & Foxes

How confident are you in your prediction?How much are you going to gamble?Are you in instinctive or analytical thinker?

Can we see any evidence of this in our data?

Key trend observed

“WHY DO MEN PREDICT THE PRICE OF THINGS DIFFERENTLY TO WOMEN”

• Is that true or not?• What are the difference between men and

women?• Why?• What further research/analysis would you

recommend?

Key trend observed

“EMOTIONS GET IN THE WAY OF MAKING OBJECTIVE PREDICTIONS”

• Is that true or not?• How much big a difference can be observed and

on what type of questions?• Why?• What further research/analysis would you

recommend?

England 3 Montenegro 0

Germany 1 Northern Ireland 0

Key trend observed

“THERE ARE DIFFERENCES BETWEEEN PERSONAL AND PROJECTED PREDICTIONS”

• What are the key types of difference• Why?• How can we use this as a means of generating

insights?

Key trend observed

“THE NUDGE EFFECT: IS STRONGER ON SOME QUESTIONS THAN OTHERS”

• Is that true or not?• What questions are influenced more by nudges?• What is the relative impact of different types of

nudges?• Can Bayesian priors explain these difference • What further research/analysis would you

recommend?

Key trend observed

“WE ARE BETTER AT PREDICTING SOME THINGS MORE THAN OTHERS”

• Is that true or not?• What are we good and bad at predicting? • Why?• What further research/analysis would you

recommend?

Appendix

Analysis of pilot Gamified shopping survey experiment

VIRTUAL SHOPPING EXERCISE: WE ASKED RESPONDENTS TO IMAGINE THEIR LAST SHOPPING TRIP AND IMAGINE WHAT THEY WOULD HAVE PURCHASED

If for example they said they rushed round we encouraged them to do the task quickly

The offer was randomly varied

WE THEN ASKED THEM TO DO THE SAME SHOPPING TASK PROJECTIVELY IMAGINING WHAT A FRIEND WOULD BUY

WE PLANTED A RANGE OF DIFFERENT OFFERS TO EXPLORE THEIR RELATIVE PURCHASE APPEAL

& AT THE END WE SHOWED THEM ALL THE OFFERS AND ASKED THEM TO PICK OUT THE ONE THEY WOULD CHOOSE

PROMOTIONAL OFFER EXPERIMENT

• KEY QUESTIONS:

1. How did the offer selections differ between the personal and predictive tasks?

2. Could any of these techniques reproduce predicted behavioural economic premises?

3. What influence did the different shopping mindsets have on respondents virtual shopping behaviour?

HOW DID THE TECHNIQUES BROADLY COMPARE?

Virtual sales Projective sales

Preference -

2

4

6 2 for £2 now £1.09 33% off

Virtual sales Projective sales

Preference -

2

4

6 BOGOF Now £2 50% off

Virtual sales Projective sales

Preference -

1

2 now £2.25 BOGOF 50% off

Virtual sales Projective sales

Preference -

1

2

3 3 for 2 Buy 2 get 1 Free 33% off

Virtual sales

Projective sales

Preference -

1

2

3

4 Half price £2 voucher Free cond'er

Virtual sales

Projective sales

Preference -

1

2

3 Buy 2 get 1 Free Now £2.59 33% off

=95% confidence range

Virtual sales Projective sales

Preference -

2

4

6

8 Buy 1 get 1 Free Now £4 Free Lenor 50% off

Virtual sales Projective sales

Preference -

1

2

3 Now £1.24 BOGOF Free dip

Virtual sales

Projective sales

Preference -

4

8

12 BOGOF £3 Voucher 50% off

=95% confidence range

HOW DID THE TECHNIQUES BROADLY COMPARE?

OBSERVATIONS…

• In 8 out of 9 cases 1st choice preference of best offer matched the offer that generated the most personal virtual sales◦ Strength of choice preference has little or no correlation with

strength of virtual sales uplifts or projective sales uplifts• Personal v predictive choices were the same in 7 our of 9

cases◦ However in 5 out of 9 occasions there were differences in the

ranking of choice

Comparisons between personal choice and predictive choice…

Vine to

mato

s

Blueberri

es

Stawberri

es

Nescafe

Loreal E

lvive

Colgate

Arial

Pringle

s

3.34 3.34

1.14 1.18

2.43

1.17

4.53

2.03 2.29 2.46

1.22 1.80

0.46 1.15

5.79

1.40

Sales uplift of offer

Personal Projective

Why?

• We care less about others than ourselves• Personal brand preference distorts

POST EXERCISE:WHAT CONCLUSION CAN WE MAKE

WHY DOES PRESTIGIOUS/SMART GET SELECTED MORE BY PERSONAL & COOL/ STYLISH/ FRIENDLY LESS?

• Theories:

◦ These are metrics that advertising makes us believe are the language of consumers includes words like cool and stylish?

◦ could the word prestigious be more appealing to click on ?

◦ Words selected by individuals is less emotional?◦ As I have no experience of the brand I don't select

user friendly but I might guess that it is hence differences?

Luxury indulgent purchases the offer acts as an excuse to buy something they want and so has more personal impact…

Vine to

mato

s

Blueberri

es

Stawberri

es

Nescafe

Loreal E

lvive

Colgate

Arial

Pringle

s

3.34 3.34

1.14 1.18

2.43

1.17

4.53

2.03 2.29 2.46

1.22 1.80

0.46 1.15

5.79

1.40

Sales uplift

Personal Projective

=95% confidence range

Where there is strong personal brand preference offers have less impact that expected from projective sales

Vine to

mato

s

Blueberri

es

Stawberri

es

Nescafe

Loreal E

lvive

Colgate

Arial

Pringle

s

3.34 3.34

1.14 1.18

2.43

1.17

4.53

2.03 2.29 2.46

1.22 1.80

0.46 1.15

5.79

1.40

Sales uplift

Personal Projective

=95% confidence range

Observation about projective sales

• Projective sales appear to give a more objective viewpoint of an offer than personal sales as it is more detached from personal brand preference issues

• Comparing projective and personal sales reveals the impact that personal brand preference has on offer selection and so is potentially an interesting differentiating technique

• This technique of comparison is a good way of measuring levels of subconscious brand desire and brand loyalty.