jon puleston - survey research: the science of ‘prediction’

70
PREDICTING THE FUTURE PRIMARY RESEARCH EXPLORING THE SCIENCE OF PREDICTION

Upload: baqmar

Post on 14-Jul-2015

249 views

Category:

Marketing


1 download

TRANSCRIPT

PREDICTING THE FUTUREPRIMARY RESEARCH EXPLORING THE SCIENCE OF PREDICTION

SO WHO IS THE BEST PREDICTOR IN THE

AUDIENCE?

PRIZE: A REAL CRYSTAL BALL

BOTTLE OF CHAMPAGNE

PART 1: MAKES SOME PREDICTIONS

PART 2:LIVE EXPERIMENT

WE ASKED A GROUP OF 400 UK PANELIST TO PREDICT THE

SELLING PRICE OF THE NEW* IPAD MINI 2 WEEKS PRIOR TO ITS

LAUNCH HOW CLOSE DID THEY GET?

WITHIN 10%, WITHIN 5%, WITHIN 3%, WITHIN 1%

*Note we told them the existing selling price of the old model

PREDICT THE PRICE OF 100ML OF CHANEL

PERFUME?

Price in €

HEADS OR TAILS

PREDICT HOW MANY

SAID HEADS?

WHAT PROPORTION OF WINE DRINKERS IN THE UK

PREFER RED WINE?BASED ON A POLL OF 400 WINE DRINKERS IN UK WHO WERE ASKED IF THEY PREFER RED OR WHITE

WINE

PREDICT IF IT WILL RAIN NEXT MONDAY

PREDICT HOW MANY RESEARCHERS CHECK

THEIR EMAIL BEFORE BREAKFAST?BASED ON POLL OF ATTENDEES AT ESOMAR CONGRESS

DO YOU CHECK YOUR EMAILS BEFORE

BREAKFAST?

England v Montenegro+3 +2 +1 0 -1 -2 -3

Germany v Rep. Ireland+3 +2 +1 0 -1 -2 -3

PREDICT WHAT MARGIN OF VICTORY OUR UK

PANELISTS PREDICTED FOR THESE 2 FOOTBALL

MATCHES

WILL THE MARKET RESEARCH INDUSTRY BE

BIGGER OR SMALLER IN 10 YEARS TIME?

The CXO Advisory group

gathered 6,582 buy or sell

predictions from 68 different

investing gurus made between

1998 and 2012, and tracked the

results of those predictions. How

accurate were they?

WHAT % WERE CORRECT?

SWAP YOUR QUIZ SHEET WITH THE PERSON

NEXT TO YOU READY FOR MARKING

BACKGROUND

Gamification More prediction protocols in surveys

Fostered an interest in the science of prediction Led to a

series of dedicated prediction experiments Exploration of

the world of prediction market trading Prediki

30+ Primary research experiments

500+ Predictions analysed60+ Prediction markets v traditional research comparisons

PREDIKI

THE TYPES OF EXPERIMENTS WE HAVE RUN

• Betting on the future of brands

• Predicting why people buy things

• Predicting the behaviour of other people

• Predicting the price of things

• Predicting the election prospects of political parties

• Predicting football match results

• Predicting the outcomes of TV game shows

• Predicting the success of adverts

• Predicting future sales of products

• Predicting the future more generally

SO WHAT HAVE WE

LEANT ABOUT PREDICTION?

20%

42% 43%

32% 30%

38%

11%

21% 21% 22% 22%

29%

0%5%

10%15%20%25%30%35%40%45%50%

Consumerpurchasingestimates

Observedbehaviour of

others

Observedopinion

Forecast Priceprediction

Guesswork

Correct prediction Random chance

WHAT ARE WE GOOD AT PREDICTING AS INDIVUALS?

SOME OF US ARE BETTER AT PREDICTING

0%

20%

40%

60%

80%

100%

120%

140%

160%

180%

score 1 score 2 score 3 score 4 score 5 score 6 score 7

Index of Prediction performance over 7 waves of experiments

top 100

bottom 100

𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛 𝑄𝑢𝑎𝑙𝑖𝑡𝑦=

𝐼𝑛𝑓𝑜𝑟𝑚𝑎𝑡𝑖𝑜𝑛 × 𝐸𝑓𝑓𝑜𝑟𝑡 × 𝑂𝑏𝑗𝑒𝑐𝑡𝑖𝑣𝑖𝑡𝑦 × 1 − 𝐷𝑖𝑓𝑓𝑖𝑐𝑢𝑙𝑡𝑦 × 𝑅𝑎𝑛𝑑𝑜𝑚𝑛𝑒𝑠𝑠

Note: Not directly dependent on sample size

Nate Silver: Correctly predicted the outcome of all 52 states in the 2012 UK election

16 IS A CROWDJed Christianson, University of Birmingham calculates

LESS ABOUT SAMPLE SIZE MORE ABOUT

SAMPLE DIVERSITY & INTELLIGENCE

..AND HOW YOU AGGREGATE

CROWD WISDOM

MEAN, MODE, MEDIAN V TRADING & DOUBLE

AUCTION TRADING

1906 Plymouth County fair

Actual weight = 1198 lb

Median average guess = 1207lb

Error = <1%

WHAT WE KNOW

THE WISDOM OF CROWDS

CROWD WISDOM IS BASED ON FILTERING

THE SIGNAL FROM THE NOISEEach person’s prediction is made up of 2 components: information & error.

If each individual’s judgement is independent & unbiased then the error

will largely cancel itself out and the aggregation process then distils off the

inherent knowledge.

Actual selling price = £319

Median average guess = £316

Error = 1% SCORE: 1 POINT

2013 GMI online sample

WHAT WE KNOW

THE WISDOM OF CROWDS

WITHOUT COLLECTIVE KNOWLEDGE

CROWDS CAN BE PLAIN IGNORANT

2014 GMI online sample

Actual weight = 550kg

AN UNWISE CROWD

Median average guess = 350kg

Error = 36%

CROWD WISDOM CAN BE A BIT BEHIND THE TIMES

-44%

-34% -33%-30% -28% -26% -25% -25%

-17%-14%

2%5%

9% 9%

21% 23%

-50%

-40%

-30%

-20%

-10%

0%

10%

20%

30%

Examples of price predicition errors

Average 9% price lag

94%87%

0%10%20%30%40%50%60%70%80%90%

100%

Women Men

Price prediciton accuracy

Men less price savvy

Crowd €80

Actual €120

YOUR PREDICTIONS

SCORE: +/-kr100 2 POINTS

+/-kr200 1 POINT

“If each individual’s judgement is independent &

unbiased then the error will largely cancel itself out”

THE PROCESS OF MAKING PREDICTIONS

IS LITTERED WITH COGNITIVE BIASES

68% heads DUE TO ORDER BIAS

YOUR PREDICTIONS

SCORE: +/-5% 2 POINT

+/-10% 1 POINT

WITH NO INFORMATION

TO GUIDE US TINY NUDGES

CAN HAVE BIG EFFECTS

ON OUR PREDICTIONS

HOW THIS EFFECT CORRUPTS PREDICTIONS….

54%

46%46%

54%

0%

10%

20%

30%

40%

50%

60%

White Red

What percentage of people do you think prefer white wine?

What percentage of people do you think prefer red wine?

V

20% SHIFT IN PREDICTION

PREDICT IF IT WILL RAIN NEXT THURSDAY

Predict the chances of it raining 5 days in advance

IF RAINING TODAY +20°%

Score If you predicted correctly = 1 point

STUDYING THE IMPACT OF NUDGE EFFECTS: THE INFLUENCE ONE PERSON’S OPINION HAS ON ANOTHER

2%

6%

11%

15%

20%

Personalpreferences

Self evidentpredictions (e.g.ad evaluation)

Factual (requiringknowledge)

Inverted personalpreference (e.g.pedicting relativelevels of dislike)

Complexestimates

Nudge influence by prediction task

Source: GMI research 2014

THE LESS CERTAIN PEOPLE ARE AND THE HARDER THE PREDICTION,

THE MORE WE RELY ON OTHER PEOPLE’S OPINIONS

IN THE ABSENCE OF

KNOWLEDGE WE

PREDICT THE MAJORITY

OF OTHER PEOPLE WILL

DO & THINK THE SAME

AS US

- 10 20 30 40 50 60 70 80 90

100

Believers

Believe this How many people believe this

- 10 20 30 40 50 60 70 80 90

100

Non believers

Don’t believe How many people don't believe this

those holding minority opinions assume more people agree with them than those holding the majority opinion: is this the definition of

delusion?`

WHAT WILL THE WORLD BE LIKE IN 2050?

55% 68%

YOUR PREDICTIONS

SCORE: +/-5% 2 POINT

+/-10% 1 POINT

50%

20%

I check my emails

before breakfast

I don't check my

emails before

breakfast

Prediction of how many other

people check emails before

breakfast*

*Source: office poll! 80% OF MARKET RESEARCHERS

OUR EMOTIONS REALLY CAN

DOMINATE & BADLY DISTORT OUR

PREDICTIONS

England v Montenegro

+3 +2 +1 0 -1 -2 -3

Germany v Rep. Ireland

+3 +2 +1 0 -1 -2 -3

PREDICT WHAT SCORES OUR UK PANELISTS PREDICTED!

SCORE: CORRECT = 1 POINT PER QUESTION

0%

10%

20%

30%

40%

50%

60%

new by 3 new by 2 new by 1 draw liv by 1 liv by 2 liv by 2

Newcastle Liverpool

0%

10%

20%

30%

40%

50%

chel by 3 chel by 2 chel by 1 draw car by 1 car by 2 car by 3

Chelsea Cardiff

0%

10%

20%

30%

40%

50%

man by 3 man by 2 man by 1 draw south by1

south by2

south by3

Southhampton Man U

0%

10%

20%

30%

40%

50%

man by 3man by 2man by 1 draw south by1

south by2

south by3

Southhampton Man U

FOOTBALL SCORE PREDICTIONS

0%

10%

20%

30%

40%

50%

60%

70%

80%

Conservative

Government

Labour Pary

Government

Liberal Democrats

Government

Conservative &

Liberal Coalition

Labour and Liberal

Coalition

Conservative,

Liberal, & UKIP

Coalition

Conservatives Labour LiberalDemocrats UKIP

PREDICT WHO WILL FORM THE NEXT UK GOVERNMENT: BY PARTY AFFILIATION

DIFFERENCES BETWEEN

PREDICTING WHAT OTHERS WILL DO V

WHAT I WILL DO

SOCIAL COGNITIVE BIASES RENDER

PREDICTIONS ABOUT OUR OWN

BEHAVIOUR PARTICULARLY DIFFICULT

Will you tidy up after the meeting?

Yes = 50%

TIDIED UP =13%

Predict how many will tidy up?

= 15%

THEREFORE WE WOULD ADVOCATE A

STEREOSCOPIC APPROACH

PERSONAL PREDICTIVE

WE ARE OFTEN TOO TIED UP IN THE

DETAIL TO SEE THE BIGGER PICTURE

WILL THE MARRIAGE LAST?

Yes/No

Parents much better than the married

couples at predicting this

Source: Queens University Canada

UNABLE TO SEPARATE THE SIGNAL

FROM THE NOISE

Will the Market Research industry be bigger

or smaller in 10 years time?Yes/No?

0.9

0.92

0.94

0.96

0.98

1

Experts Dillettantes

(non experts)

Chimps

(random

guesses)

Future Predictions accuracy

CA

LIB

RA

TIO

N

Highly recommended reading

PHILIP TETLOCK STUDYING 15,000

GEO-POLITICAL PREDICTIONS

0

0.01

0.02

0.03

0.04

0.05

0.06

Experts Dillettantes

(non

experts)

Chimps

(random

guesses)

Descrimination

Philip Tetlock

EXPERT POLITICAL JUDGEMENT

HOW MANY INVESTMENT GURUS STOCK

MARKET PREDICTIONS WERE CORRECT?

SCORE BELOW 50% = 1 POINT

48%2% less accurate than a coin toss!

HOW TO WORK THE CROWD!EXPLORING THE BEST TECHNIQUES TO USE TO EXTRACT RELIABLE PREDICTIONS

-200%

-150%

-100%

-50%

0%

50%

100%

150%

200%

250%

300%

350%

Evaluation of 20 different ads

Monadic rating

0.89 correlation 5x differentiation

MAKE IT REWARDING

WEIGHT BASED ON CONFIDENCE

33%37% 36%

44%

0%

10%

20%

30%

40%

50%

Total Guess Have a Hunch Fairly Sure Very Confident

Prediction accuracy

MONEY BET IS A PROXY

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

-3 -2 -1 +1 +2 +3

Bet amount: correlation with outcome

USE PREDICTION MARKETS

IOWA ELECTRONIC MARKET 480/590 OUT PREDICTED THE BEST POLL

SAMPLES OF UNDER 20

32 HEAD TO HEAD EXPERIMENTS

A survey based approach with random cells* of 15

participants who were asked to predict which products would

sell more

vs.

15 people prediction markets trading – asked to buy or sell

variable amounts to create a confidence weighted market

* Using Montecarlo simulation technique we aggregated the predictions of 10,000 randomly

selected group of 15 participants from a larger sample to make this

37%

55%

65%

0%

10%

20%

30%

40%

50%

60%

70%

Random guess Micro survey(sample of 15)

15 people predictionmarket trading

HEAD TO HEAD COMPARISON

Source: GMI/Prediki based on 32 direct head to head comparisons

SOME ISSUES THOUGH

OPINIONS IN PREDICTION MARKET TRADING CAN QUICKLY

BE SET IN STONE IF NO NEW INFORMATION ISADDED

ADDING MORE PEOPLE

AFTER A CERTAIN POINT

DOES NOT CHANGE THE

RESULT

37%

55%

65%69%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Random guess Micro surveysample of 15

15 peopleprediction

markets trading

Standard survey:sample of 200

HEAD TO HEAD COMPARISON

INFORMATION SHARING IS KEYPREDICTION MARKETS FEED OF INFORMATION

MARKETS WOULD REACT WHEN WE ADDED INFORMATION

Self generating

clues

THINK OF A QUESTION AS A

CONUNDRUM: INFORMATION

PROVIDES CLUES TO HELP PEOPLE

SOLVE THE PROBLEM

DIALECTICAL BOOT STRAPPINGENCOURAGING CROWDS TO SELF-GENERATE THE INSIGHTS

NEEDED TO SOLVE PREDICTION CONUNDRUMS

Example = Board room decisions

Useful reference: Herzog and Hertwig (2009)

37%

55%

65%69%

81%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Random guess Micro survey(sample of 15)

15 people Microprediction market

trading

Standard survey(sample of 200)

Micro predictionmarket with sharedinformation & free

comments

THE VALUE OF ADDING INFORMATION TO PREDICTIVE MARKET

TRADING SYSTEM

EFFECTIVE USE OF PREDICTION MARKETS

• Incentivise - ideally with real money!

• Allow active & dynamic trading

• 16 is a crowd

• Share as much information as possible

• A moderator is important to stimulate debate and share

information

• Divide the herd: run multiple micro markets

SAMPLING 100’S SMALLER SMARTER GROUPS

BEING ASKED SMARTER QUESTIONS &

SHARING THOUGHTS & OPINIONS

SO WHO IS THE BEST PREDICTOR IN THE

AUDIENCE?

PRIZE: A REAL CRYSTAL BALL