primary research - marketing intelligenceintelligence.marketingonline.nl/sites/default/... ·...
TRANSCRIPT
![Page 1: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/1.jpg)
THE FUTURE PREDICTING
PRIMARY RESEARCH
1
![Page 2: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/2.jpg)
SO WHO IS THE BEST PREDICTOR IN THE AUDIENCE?
PRIZE: A REAL CRYSTAL BALL
PART 1: MAKES SOME PREDICTIONS
![Page 3: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/3.jpg)
IN SEPT 2013 WE ASKED A GROUP OF 400 UK PANELIST TO PREDICT THE SELLING PRICE OF THE IPAD MINI 3 WEEKS PRIOR TO ITS LAUNCH. HOW CLOSE DID THEY GET?
WHAT IS YOUR PREDICTION? WITHIN 10%, WITHIN 5%, WITHIN 3%, WITHIN 1%
*Note we told them the existing selling price of the old model
![Page 4: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/4.jpg)
PREDICT THE PRICE OF 100ML OF CHANEL PERFUME?
In €
![Page 5: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/5.jpg)
I AM GOING TO TOSS A COIN WILL IT BE HEADS OR TAILS
ADDITIONAL QUESTION…
PREDICT HOW MANY SAID HEADS?
![Page 6: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/6.jpg)
WHAT PROPORTION OF WINE DRINKERS IN THE UK
PREFER RED WINE? BASED ON A POLL OF 400 WINE DRINKERS IN UK WHO WERE ASKED IF THEY PREFER RED OR WHITE
WINE
WHAT % DRINK RED?
![Page 7: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/7.jpg)
PREDICT HOW MANY RESEARCHERS CHECK THEIR EMAIL BEFORE BREAKFAST? BASED ON POLL OF ATTENDEES AT ESOMAR CONGRESS 2014
![Page 8: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/8.jpg)
DO YOU CHECK YOUR EMAILS BEFORE BREAKFAST?
![Page 9: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/9.jpg)
England v Montenegro +3 +2 +1 0 -1 -2 -3
Germany v Rep. Ireland +3 +2 +1 0 -1 -2 -3
PREDICT WHAT MARGIN OF VICTORY OUR UK PANELISTS PREDICTED FOR THESE 2 FOOTBALL MATCHES
![Page 10: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/10.jpg)
WILL THE MARKET RESEARCH INDUSTRY BE BIGGER OR SMALLER IN 10 YEARS TIME?
![Page 11: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/11.jpg)
The CXO Advisory group gathered 6,582 buy or sell predictions from 68 different investing gurus made between 1998 and 2012, and tracked the results of those predictions. How accurate were they?
WHAT % WERE CORRECT?
![Page 12: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/12.jpg)
SWAP YOUR QUIZ SHEET WITH THE PERSON NEXT TO YOU READY FOR MARKING
![Page 13: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/13.jpg)
BACKGROUND Gamification à More prediction protocols in surveys à Fostered an interest in the science of prediction à Led to a series of dedicated prediction experiments à Exploration of the world of prediction market trading à Prediki
• 30+ Primary research experiments
• 500+ Predictions analysed
• 60+ Prediction markets v traditional research comparisons
![Page 14: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/14.jpg)
THE TYPES OF EXPERIMENTS WE HAVE RUN
• Betting on the future of brands • Predicting why people buy things • Predicting the behaviour of other people • Predicting the price of things • Predicting the election prospects of political parties • Predicting football match results • Predicting the outcomes of TV game shows • Predicting the success of adverts • Predicting future sales of products • Predicting the future more generally
![Page 15: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/15.jpg)
SO WHAT HAVE WE LEANT ABOUT PREDICTION?
![Page 16: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/16.jpg)
20%
42% 43%
32% 30%
38%
11%
21% 21% 22% 22%
29%
0% 5%
10% 15% 20% 25% 30% 35% 40% 45% 50%
Consumer purchasing estimates
Observed behaviour of
others
Observed opinion
Forecast Price prediction
Guesswork
Correct prediction Random chance
WHAT ARE WE GOOD AT PREDICTING AS INDIVUALS?
![Page 17: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/17.jpg)
SOME OF US ARE BETTER AT PREDICTING
0%
20%
40%
60%
80%
100%
120%
140%
160%
180%
score 1 score 2 score 3 score 4 score 5 score 6 score 7
Index of Prediction performance over 7 waves of experiments
top 100
bottom 100
![Page 18: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/18.jpg)
!"#$%&'%() *+,-%'. = /)0("1,'%() ×200("' ×345#&'%6%'.×(1−8%00%&+-'.) × :,)$(1)#;;
Note: Not prediction accuracy is not directly dependent on sample size
One person with access to the right information and able to objectively process it can be just as good as much larger groups. e.g. Nate Silver: Correctly predicted the outcome of all 52 states in the 2012 UK election
![Page 19: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/19.jpg)
16 IS A CROWD Jed Christianson, University of Birmingham calculates
LESS ABOUT SAMPLE SIZE MORE ABOUT SAMPLE DIVERSITY & INTELLIGENCE
Jed Christianson, estimates that the quality of consolidated group predictions does not improve much beyond 16 participants, if they are all well informed.
![Page 20: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/20.jpg)
..AND HOW YOU AGGREGATE CROWD WISDOM
MEAN, MODE, MEDIAN V TRADING & DOUBLE AUCTION TRADING
![Page 21: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/21.jpg)
1906 Plymouth County fair
Actual weight = 1198 lb Median average guess = 1207lb Error = <1%
WHAT WE KNOW
THE WISDOM OF CROWDS
Using larger samples and median aggregation techniques can help consolidate wisdom in situations where information and knowledge is dissipated.
![Page 22: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/22.jpg)
CROWD WISDOM IS BASED ON FILTERING THE SIGNAL FROM THE NOISE
Each person’s prediction is made up of 2 components: information & error.
If each individual’s judgement is independent & unbiased then the error will largely cancel itself out and the aggregation process then distils off the
inherent knowledge.
![Page 23: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/23.jpg)
Actual selling price = £319
Median average guess = £316 Error = 1%
SCORE: 1 POINT
2013 GMI online sample
WHAT WE KNOW
THE WISDOM OF CROWDS
![Page 24: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/24.jpg)
WITHOUT COLLECTIVE KNOWLEDGE CROWDS CAN BE PLAIN IGNORANT
WARNING
![Page 25: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/25.jpg)
2014 GMI online sample
Actual weight = 550kg
AN UNWISE CROWD
Median average guess = 350kg Error = 36%
Where there is no wisdom to consolidate, crowds just generate noise.
![Page 26: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/26.jpg)
CROWD WISDOM CAN BE A BIT BEHIND THE TIMES
-44%
-34% -33% -30%
-28% -26% -25% -25%
-17% -14%
2% 5%
9% 9%
21% 23%
-50%
-40%
-30%
-20%
-10%
0%
10%
20%
30%
Examples of price predicition errors
We measured an average 9% price lag on aggregated price predictions – our memory of price is based on historical purchase activity which is often lower that current pricing.
94% 87%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
Women Men
Price prediciton accuracy
Men are less price savvy than women
![Page 27: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/27.jpg)
Crowd €80 Actual €120
YOUR PREDICTIONS?
SCORE: +/-$10 2 POINTS +/-$20 1 POINT
![Page 28: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/28.jpg)
Aggregating crowd wisdom: “If each individual’s judgement is independent & unbiased then the error will largely cancel itself out”
THE PROCESS OF MAKING PREDICTIONS IS LITTERED WITH COGNITIVE BIASES
![Page 29: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/29.jpg)
68% predict heads DUE TO ORDER BIAS
HEADS OR TAILS?...YOUR PREDICTIONS
SCORE: +/-5% 2 POINT +/-10% 1 POINT
![Page 30: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/30.jpg)
WITH NO INFORMATION TO GUIDE US TINY NUDGES CAN HAVE BIG EFFECTS ON OUR PREDICTIONS
![Page 31: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/31.jpg)
HOW THIS EFFECT CORRUPTS PREDICTIONS….
54%
46% 46%
54%
0%
10%
20%
30%
40%
50%
60%
White Red
What percentage of people do you think prefer white wine?
What percentage of people do you think prefer red wine?
V
A 20% shift in prediction based simply on naming the colour of wine in the question!
![Page 32: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/32.jpg)
IN THE ABSENCE OF KNOWLEDGE WE PREDICT THE MAJORITY OF OTHER PEOPLE WILL DO & THINK THE SAME AS US
![Page 33: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/33.jpg)
- 10 20 30 40 50 60 70 80 90
100
Believers
Believe this How many people believe this
- 10 20 30 40 50 60 70 80 90
100
Non believers
Don’t believe How many people don't believe this
those holding minority opinions assume more people agree with them than those holding the majority opinion: is this the definition of delusion?
WHAT WILL THE WORLD BE LIKE IN 2050?
55% 68%
![Page 34: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/34.jpg)
CHECKING EMAIL YOUR PREDICTIONS
SCORE: +/-5% 2 POINT +/-10% 1 POINT
60%
20%
I check my emails before breakfast
I don't check my emails before breakfast
Prediction of how many other people at Lightspeed GMI check their emails
before breakfast*
*Source: office poll!
80% of Researcher said they check their email before breakfast
![Page 35: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/35.jpg)
OUR EMOTIONS REALLY CAN DOMINATE & BADLY DISTORT OUR PREDICTIONS
![Page 36: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/36.jpg)
England v Montenegro +3 +2 +1 0 -1 -2 -3
Germany v Rep. Ireland +3 +2 +1 0 -1 -2 -3
PREDICT WHAT SCORES OUR UK PANELISTS PREDICTED!
SCORE: CORRECT = 1 POINT PER QUESTION
![Page 37: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/37.jpg)
0%
10%
20%
30%
40%
50%
60%
new by 3 new by 2 new by 1 draw liv by 1 liv by 2 liv by 2
Newcastle Liverpool
0%
10%
20%
30%
40%
50%
chel by 3 chel by 2 chel by 1 draw car by 1 car by 2 car by 3
Chelsea Cardiff
0% 5%
10% 15% 20% 25% 30% 35% 40% 45% 50%
man by 3 man by 2 man by 1 draw south by 1
south by 2
south by 3
Southhampton Man U
0% 5%
10% 15% 20% 25% 30% 35% 40% 45% 50%
man by 3 man by 2 man by 1 draw south by 1
south by 2
south by 3
Southhampton Man U
FOOTBALL SCORE PREDICTIONS
These are the comparative football score predictions of competing fans.
![Page 38: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/38.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
Conservative Government
Labour Pary Government
Liberal Democrats Government
Conservative & Liberal Coalition
Labour and Liberal Coalition
Conservative, Liberal, & UKIP
Coalition
Conservatives Labour LiberalDemocrats UKIP
PREDICT WHO WILL FORM THE NEXT UK GOVERNMENT: BY PARTY AFFILIATION
The issue of emotional bias is particularly significant in the political arena where voting intention and predictions on who is likely to form a future government almost completely correlate.
![Page 39: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/39.jpg)
DIFFERENCES BETWEEN PREDICTING WHAT OTHERS WILL DO
V
WHAT I WILL DO
![Page 40: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/40.jpg)
SOCIAL COGNITIVE BIASES RENDER PREDICTIONS ABOUT OUR OWN BEHAVIOUR PARTICULARLY DIFFICULT
Will you tidy up after the meeting? Yes = 50%
TIDIED UP =13%
Predict how many will tidy up? = 15%
…we are much better at predicting the behaviour of others than our own behaviour. We like to think well of ourselves!
![Page 41: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/41.jpg)
WE ARE OFTEN TOO TIED UP IN THE DETAIL TO SEE THE BIGGER PICTURE
WILL THE MARRIAGE LAST? Yes/No Parents are much better than the married couples at predicting this
Source: Queens University Canada
![Page 42: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/42.jpg)
0,9 0,91 0,92 0,93 0,94 0,95 0,96 0,97 0,98 0,99
1
Experts Dillettantes (non
experts)
Chimps (random guesses)
Future Predictions accuracy
CA
LIBR
ATI
ON
Highly recommended reading
PHILIP TETLOCK STUDYING 15,000 GEO-POLITICAL PREDICTIONS
![Page 43: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/43.jpg)
0
0,01
0,02
0,03
0,04
0,05
0,06
Experts Dillettantes (non experts)
Chimps (random guesses)
Descrimination
Philip Tetlock
EXPERT POLITICAL JUDGEMENT
What he describes as “Dillettantes”, smart people with no specific expertise in the topic, prove to be more effective at making geo-political predictions than political pundits/experts.
![Page 44: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/44.jpg)
HOW MANY INVESTMENT GURUS 6,500 STOCK MARKET PREDICTIONS WERE CORRECT?
SCORE BELOW 50% = 1 POINT
48% 2% less accurate than a coin toss!
![Page 45: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/45.jpg)
WE ARE UNABLE TO SEPARATE THE SIGNAL FROM THE NOISE
Will the Market Research industry be bigger or smaller in 10 years time? Yes/No?
I don’t know but we in the market research industry are not probably the best people to ask.
![Page 46: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/46.jpg)
HOW TO WORK THE CROWD! EXPLORING THE BEST TECHNIQUES TO USE TO EXTRACT RELIABLE PREDICTIONS
SWAP BACK YOUR QUIZ SHEETS
10 VOLUNEERS
![Page 47: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/47.jpg)
WHICH OF IS THE MOST POPULAR AD?
BET 1, 2, 3, 4, 5 POINTS
![Page 48: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/48.jpg)
WHICH OF THESE CANDLES SELLS THE MOST
BET 1, 2, 3, 4, 5 POINTS
![Page 49: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/49.jpg)
WHICH OF THESE MUGS SELLS THE MOST?
BET 1, 2, 3, 4, 5 POINTS
Clue: These are mostly purchased as gifts for other people Clue 2: these are sold to English people!
Camping Cooking Gardening DIY (Home improvement)
![Page 50: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/50.jpg)
-200%
-150%
-100%
-50%
0%
50%
100%
150%
200%
250%
300%
350%
Evaluation of 20 different ads Monadic rating Performance prediction: Net Investment
0.89 correlation 5x differentiation
ADVICE: MAKE IT REWARDING
All the research out there show us that prediction made under competitive conditions where you can have the opportunity to win or lose, results in participants investing more thought and effort and this in tern improves the quality of predictions.
![Page 51: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/51.jpg)
WEIGHT BASED ON CONFIDENCE
33% 37% 36%
44%
0% 5%
10% 15% 20% 25% 30% 35% 40% 45% 50%
Total Guess Have a Hunch Fairly Sure Very Confident
Prediction accuracy
In our analysis of 500 predictions, where in each case we asked respondents how confident they were, we found the predictions from the most confident groups to be slightly more accurate.
![Page 52: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/52.jpg)
MONEY BET IS A PROXY TO CONFIDENCE
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
-3 -2 -1 +1 +2 +3
Bet amount: correlation with outcome
We also found an interesting relationship between money bet and prediction accuracy. A non-linear relationship. Small bets were less accurate than medium bets but larger bets were not more accurate than medium bets. The most accurate bets overall were in fact those that bet small amount against.
![Page 53: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/53.jpg)
EXPLOIT PREDICTION MARKET TRADING
• Prediction markets trading is a process where participants are given money to buy and sell shares in the outcome of an event with a chance to win real money.
• The most famous example being the Iowa Electronic Market a prediction market set up to predict the outcome of elections around the world: ◦ In 451 out of 596 head to head comparisons, prediction
market trading by groups on average of less than 20 people has more accurately predicted the result that the most accurate local opinion polls.
![Page 54: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/54.jpg)
• We conducted 32 head to head experiments to compare the prediction from prediction markets trading to predictions made in traditional surveys
• A survey based approach with random cells* of 15 participants who were asked to predict which products would sell more vs 15 people prediction markets trading – asked to buy or sell variable amounts to create a confidence weighted market
TESTING OUT USING PREDICTION MARKETS FOR MORE TRADTIONAL MARKET RESEARCH PURPOSES
![Page 55: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/55.jpg)
37%
55%
65%
0%
10%
20%
30%
40%
50%
60%
70%
Random guess Micro survey (sample of 15)
15 people prediction market trading
HEAD TO HEAD COMPARISON
Source: GMI/Prediki based on 32 direct head to head comparisons
Comparing the performance of small head to head samples, prediction markets trading groups did more accurately make correct predictions. The trading process more effectively aggregating group wisdom
![Page 56: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/56.jpg)
SOME ISSUES THOUGH
![Page 57: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/57.jpg)
OPINIONS IN PREDICTION MARKET TRADING CAN QUICKLY BE SET IN STONE IF NO NEW INFORMATION IS ADDED
ADDING MORE PEOPLE AFTER A CERTAIN POINT DOES NOT CHANGE THE RESULT
When we used larger samples of respondents in prediction markets, adding more respondents did not improve the quality of predictions – predictions rather rapidly got fixed in stone. The betting intentions of the earlier participants was having a significant impact on the later participants.
![Page 58: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/58.jpg)
37%
55%
65% 69%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Random guess Micro survey sample of 15
15 people prediction
markets trading
Standard survey: sample of 200
HEAD TO HEAD COMPARISON
The aggregated prediction of larger samples of people made standard surveys proved to be more accurate. These predictions were all made in isolation from the influence of other people’s opinion.
![Page 59: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/59.jpg)
INFORMATION SHARING IS KEY PREDICTION MARKETS FEED OF INFORMATION
• These initial experiments we did not allow the participants in the prediction markets trading process to share opinions all they could see was what others had bet.
• We then conducted a second series of experiments where we injected some intelligence by encouraging participants to discuss their decisions and allowed them all to see what other people thought.
![Page 60: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/60.jpg)
MARKETS WOULD REACT WHEN WE ADDED INFORMATION
Self generating clues
The participants began to self issue clues that would help the group collectively make more thoughtful predictions. The example above show how the market reacted to thoughtful comments made about which product they felt would sell more.
![Page 61: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/61.jpg)
THINK OF A QUESTION AS A CONUNDRUM: INFORMATION
PROVIDES CLUES TO HELP PEOPLE SOLVE THE PROBLEM
![Page 62: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/62.jpg)
DIALECTICAL BOOT STRAPPING ENCOURAGING CROWDS TO SELF-GENERATE THE INSIGHTS
NEEDED TO SOLVE PREDICTION CONUNDRUMS
Example = Board room decisions
Useful reference: Herzog and Hertwig (2009)
![Page 63: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/63.jpg)
37%
55%
65% 69%
81%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Random guess Micro survey (sample of 15)
15 people Micro prediction market
trading
Standard survey (sample of 200)
Micro prediction market with shared information & free
comments
THE VALUE OF ADDING INFORMATION TO PREDICTIVE MARKET TRADING SYSTEM
In this second series of experiments the prediction accuracy of these small group of individuals sharing information in prediction markets moved ahead of the predictions from larger samples of respondents in standard surveys
![Page 64: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/64.jpg)
THE PROBLEM OF HERDING EFFECTS
Information cascade
However there predictions were still not perfect, stray comments from individuals could occasionally steer the prediction markets off course.
![Page 65: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/65.jpg)
STUDYING THE IMPACT OF NUDGE EFFECTS: THE INFLUENCE ONE PERSON’S OPINION HAS ON ANOTHER
2%
6%
11%
15%
20%
Personal preferences Self evident predictions (e.g. ad
evaluation)
Factual (requiring knowledge)
Inverted personal preference (e.g. pedicting relative levels of dislike)
Complex estimates
Nudge influence by prediction task
Source: GMI research 2014
We conducted some unique experiments to measure the influence that one persons opinion could have on another and found the less certain people are and the harder the prediction is, the more we rely on other people’s opinions
![Page 66: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/66.jpg)
OUR SOLUTION = DIVIDING UP THE HERD
A simple solution to this is to run multiple prediction markets in isolation from each other and then aggregating their predictions.
![Page 67: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/67.jpg)
37%
55% 65% 69%
81% 93%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
Random guess Micro survey (sample of 15)
15 people Micro
prediction market trading
Standard survey
(sample of 200)
Micro prediction
market with shared
information & free comments
Aggrigation of 3 divided
herds
HEAD TO HEAD COMPARISON
By doing this we were able to improve the prediction accuracy from prediction markets even further.
![Page 68: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/68.jpg)
Please note these experiments must be treated only as anecdotal evidence to illustrate these issues. This is only one small series of experiments. A lot more work is need to properly validate these findings.
![Page 69: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/69.jpg)
EFFECTIVE USE OF PREDICTION MARKETS
• Incentivise - ideally with real money! • Allow active & dynamic trading • 16 is a crowd • Share as much information as possible • A moderator is important to stimulate debate and share
information • Divide the herd: run multiple micro markets
![Page 70: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/70.jpg)
SAMPLING 100’S SMALLER SMARTER GROUPS BEING ASKED SMARTER QUESTIONS & SHARING THOUGHTS & OPINIONS
![Page 71: PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal](https://reader034.vdocument.in/reader034/viewer/2022050111/5f48c3b9efae89316d61f783/html5/thumbnails/71.jpg)
SO WHO IS THE BEST PREDICTOR IN THE AUDIENCE?
PRIZE: A REAL CRYSTAL BALL