jon puleston
TRANSCRIPT
PREDICTION SCIENCE
BACKGROUND
• Prediction protocol in survey are a great means of motivating respondents to think that can deliver back great data
• As a result we have started to use them a lot in many of the surveys we run
• Estimated we have conducted over 30 research on research experiments using various prediction protocols
• But lots of question marks exist over what the data means and what is the real accuracy of these predictions
• We have assembled today a lot of all this data and we are hoping you can help us start to make better sense of it.
TRADING GAMES
Fun
Inte
rest
ing
Thinkin
g time
0
40
80
120
Standard Trading game
-200%
-100%
0%
100%
200%
300%
400% Trading game
Standard rating
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101106111116121126131136141146151156161166171176181-100%
-80%
-60%
-40%
-20%
0%
20%
40%
60%
80%
100%
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101106111116121126131136141146151156161166171176181-100%
-80%
-60%
-40%
-20%
0%
20%
40%
60%
80%
100%
Standard rating certainty point
Trading game certainty point
If 20% of respondents are making up answers because they don’t care what they say, you need an extra 40% more sample to smooth out the systematic errors this introduces
THE TYPES OF EXPERIMENTS WE HAVE RUN• Betting on the future of brands• Predicting why people buy things• Predicting the price of things• Predicting the election prospects of political parties• Predicting football match results• Predicting the outcomes of TV game shows• Predicting the success of adverts• Predicting future sales of products• Predicting the future more generally
WHAT DO WE KNOW ABOUT PREDICTION?
PREDICTION CAN BE A LOT MORE ACURATE THAN TRADITIONAL POLLING
WHO WILL YOU VOTE FOR? 1,000 respondents
PREDICT WHO WILL WIN?20 people gambling
IOWA ELECTION MARKETIN 451 out of 596 OCCASIONS
ü
ELECTION FORECASTING
PREDICTION KNOWNS
The CXO Advisory group gathered 6,582 predictions from 68 different investing gurus made between 1998 and 2012, and tracked the results of those predictions how accurate were they?
48% correct
WILL YOU CLEAN UP?YES = 50%
PREDICT HOW MANY WILL CLEAN UP?YES = 15%
HOW MANY CLEANED UP?
WE ARE NOT GOOD AT PREDICTING OUR OWN BEHAVIOUR
13%
COGNITIVE BIASES CAN CORRUPT SO MANY ANSWERS TO TRADITIONAL SURVEY QUESTIONS
HEADS OR TAILS?
THIS IS AN OPPORTUNITY TO SWITCH FROM TREATING RESPONDENTS AS BIT OF DATA TO TREATING THEM AS A PROBLEM SOLVING ENGINE
BY REFRAMING QUESTIONS INTO PROBLEMS….
IF I TOSS THIS COIN 100 TIMES HOW MANY OCCASSIONS DO YOU PREDICT IT WILL BE HEADS?
EXAMPLE OF HOW THIS THINKING CAN BE APPLIED IN A SURVEY
HOW MANY IPADS ARE THERE IN CHINA?
• Do you own an Ipad? Yes = 30%• Predict how many of your friends, family and colleagues
own an ipad Average estimate = <5%
WHAT WE WANT TO KNOW!
OUR QUESTIONS
• How good are we are predicting things?• What are people good at predicting and what are they not
good at predicting?• Why?• What can influence our predictions?• What impact does emotion have on our predictive decision?• How do our predictions about other people’s behaviour
correlate with our self perception • What types of people are good at making predictions?• What makes a good predictor?• Is there such a thing as a good and bad predictor – can we
measure this?• Are there any gender differences in our predictive powers?
Key trend observed
“SOME PEOPLE ARE BETTER AT PREDICTING THAN OTHERS”
• Is that true or not?• What type of people?• How much big a difference can be observed?• Why?• What further research/analysis would you
recommend?
Philip Tetlock
“EXPERTS ARE NOT AS GOOD AT PREDICTING AS NON-EXPERTS”
“THE MORE FAMOUS YOU ARE THE WORSE YOU ARE AT MAKING PREDICTIONS”
Hedgehogs & Foxes
How confident are you in your prediction?How much are you going to gamble?Are you in instinctive or analytical thinker?
Can we see any evidence of this in our data?
Key trend observed
“WHY DO MEN PREDICT THE PRICE OF THINGS DIFFERENTLY TO WOMEN”
• Is that true or not?• What are the difference between men and
women?• Why?• What further research/analysis would you
recommend?
Key trend observed
“EMOTIONS GET IN THE WAY OF MAKING OBJECTIVE PREDICTIONS”
• Is that true or not?• How much big a difference can be observed and
on what type of questions?• Why?• What further research/analysis would you
recommend?
England 3 Montenegro 0
Germany 1 Northern Ireland 0
Key trend observed
“THERE ARE DIFFERENCES BETWEEEN PERSONAL AND PROJECTED PREDICTIONS”
• What are the key types of difference• Why?• How can we use this as a means of generating
insights?
Key trend observed
“THE NUDGE EFFECT: IS STRONGER ON SOME QUESTIONS THAN OTHERS”
• Is that true or not?• What questions are influenced more by nudges?• What is the relative impact of different types of
nudges?• Can Bayesian priors explain these difference • What further research/analysis would you
recommend?
Key trend observed
“WE ARE BETTER AT PREDICTING SOME THINGS MORE THAN OTHERS”
• Is that true or not?• What are we good and bad at predicting? • Why?• What further research/analysis would you
recommend?
Appendix
Analysis of pilot Gamified shopping survey experiment
VIRTUAL SHOPPING EXERCISE: WE ASKED RESPONDENTS TO IMAGINE THEIR LAST SHOPPING TRIP AND IMAGINE WHAT THEY WOULD HAVE PURCHASED
If for example they said they rushed round we encouraged them to do the task quickly
The offer was randomly varied
WE THEN ASKED THEM TO DO THE SAME SHOPPING TASK PROJECTIVELY IMAGINING WHAT A FRIEND WOULD BUY
WE PLANTED A RANGE OF DIFFERENT OFFERS TO EXPLORE THEIR RELATIVE PURCHASE APPEAL
& AT THE END WE SHOWED THEM ALL THE OFFERS AND ASKED THEM TO PICK OUT THE ONE THEY WOULD CHOOSE
PROMOTIONAL OFFER EXPERIMENT
• KEY QUESTIONS:
1. How did the offer selections differ between the personal and predictive tasks?
2. Could any of these techniques reproduce predicted behavioural economic premises?
3. What influence did the different shopping mindsets have on respondents virtual shopping behaviour?
HOW DID THE TECHNIQUES BROADLY COMPARE?
Virtual sales Projective sales
Preference -
2
4
6 2 for £2 now £1.09 33% off
Virtual sales Projective sales
Preference -
2
4
6 BOGOF Now £2 50% off
Virtual sales Projective sales
Preference -
1
2 now £2.25 BOGOF 50% off
Virtual sales Projective sales
Preference -
1
2
3 3 for 2 Buy 2 get 1 Free 33% off
Virtual sales
Projective sales
Preference -
1
2
3
4 Half price £2 voucher Free cond'er
Virtual sales
Projective sales
Preference -
1
2
3 Buy 2 get 1 Free Now £2.59 33% off
=95% confidence range
Virtual sales Projective sales
Preference -
2
4
6
8 Buy 1 get 1 Free Now £4 Free Lenor 50% off
Virtual sales Projective sales
Preference -
1
2
3 Now £1.24 BOGOF Free dip
Virtual sales
Projective sales
Preference -
4
8
12 BOGOF £3 Voucher 50% off
=95% confidence range
HOW DID THE TECHNIQUES BROADLY COMPARE?
OBSERVATIONS…
• In 8 out of 9 cases 1st choice preference of best offer matched the offer that generated the most personal virtual sales◦ Strength of choice preference has little or no correlation with
strength of virtual sales uplifts or projective sales uplifts• Personal v predictive choices were the same in 7 our of 9
cases◦ However in 5 out of 9 occasions there were differences in the
ranking of choice
Comparisons between personal choice and predictive choice…
Vine to
mato
s
Blueberri
es
Stawberri
es
Nescafe
Loreal E
lvive
Colgate
Arial
Pringle
s
3.34 3.34
1.14 1.18
2.43
1.17
4.53
2.03 2.29 2.46
1.22 1.80
0.46 1.15
5.79
1.40
Sales uplift of offer
Personal Projective
Why?
• We care less about others than ourselves• Personal brand preference distorts
POST EXERCISE:WHAT CONCLUSION CAN WE MAKE
WHY DOES PRESTIGIOUS/SMART GET SELECTED MORE BY PERSONAL & COOL/ STYLISH/ FRIENDLY LESS?
• Theories:
◦ These are metrics that advertising makes us believe are the language of consumers includes words like cool and stylish?
◦ could the word prestigious be more appealing to click on ?
◦ Words selected by individuals is less emotional?◦ As I have no experience of the brand I don't select
user friendly but I might guess that it is hence differences?
Luxury indulgent purchases the offer acts as an excuse to buy something they want and so has more personal impact…
Vine to
mato
s
Blueberri
es
Stawberri
es
Nescafe
Loreal E
lvive
Colgate
Arial
Pringle
s
3.34 3.34
1.14 1.18
2.43
1.17
4.53
2.03 2.29 2.46
1.22 1.80
0.46 1.15
5.79
1.40
Sales uplift
Personal Projective
=95% confidence range
Where there is strong personal brand preference offers have less impact that expected from projective sales
Vine to
mato
s
Blueberri
es
Stawberri
es
Nescafe
Loreal E
lvive
Colgate
Arial
Pringle
s
3.34 3.34
1.14 1.18
2.43
1.17
4.53
2.03 2.29 2.46
1.22 1.80
0.46 1.15
5.79
1.40
Sales uplift
Personal Projective
=95% confidence range
Observation about projective sales
• Projective sales appear to give a more objective viewpoint of an offer than personal sales as it is more detached from personal brand preference issues
• Comparing projective and personal sales reveals the impact that personal brand preference has on offer selection and so is potentially an interesting differentiating technique
• This technique of comparison is a good way of measuring levels of subconscious brand desire and brand loyalty.