forecasting elections from voters' perceptions of candidates' ability to handle issues
DESCRIPTION
Bucharest Dialogues on Expert Knowledge, Prediction, Forecasting: A Social Sciences Perspective November 21, 2010TRANSCRIPT
![Page 1: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/1.jpg)
The PollyVote
Combining forecasts for
U.S. Presidential Elections
Andreas Graefe, Karlsruhe Institute of Technology
J. Scott Armstrong, Wharton School, University of Pennsylvania
Randall Jones, Jr., University of Central Oklahoma
Alfred Cuzán, University of West Florida
The full paper to this talk can be downloaded at: tinyurl.com/combiningelections.
Bucharest Dialogues on Expert Knowledge, Prediction, Forecasting: A Social Sciences PerspectiveNovember 21, 2010
![Page 2: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/2.jpg)
Background on the PollyVote project
The PollyVote project was begun in 2003 to demonstrate the value of forecasting principles by applying them to election forecasting.
The initial focus was on combining forecasts.
![Page 3: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/3.jpg)
Performance of the PollyVoteThe PollyVote combined forecasts to obtain highly accurate
forecasts of U.S. Presidential Election outcomes:– Prospectively for 2004 and 2008 (MAE: 0.4 percentage points)
– Retrospectively for 1992 to 2000
Across these five elections, the PollyVote was on average more accurate than each of its components: - Polls- Prediction markets- Experts- Statistical models
Polly achieved this without knowing anything about politics.
![Page 4: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/4.jpg)
Power of combining
Question: What is the ratio of students per teacher in primary schools in Romania?
Judge Estimate Error1 18 .52 19 1.5
Typical error of individual estimate 1Combined estimate 18.5 1
Error reduction through combining 0%
Judge Estimate Error1 18 .52 16 1.5
Typical error of individual estimate 1Combined estimate 17 0.5
Error reduction through combining 50%
![Page 5: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/5.jpg)
Procedure and conditions for combining forecasts
Procedure: Mechanically combine forecasts equal weights
(unless you have strong evidence for differential weights)
Conditions:1. Several forecasts available2. Uncertainty about which forecasts is most accurate
(although combing is often beneficial even when the best method is known beforehand)
Conditions for when combining is most beneficial:3. Different forecasting methods are available4. Forecasts rely upon different data
![Page 6: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/6.jpg)
Benefits of combining
1. Improves accuracy2. Avoids large errors3. Provides an additional assessment of
uncertainty4. Can be used for nearly all forecasting
problems.5. Simple to describe and apply.
![Page 7: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/7.jpg)
Costs of combining
1. Requires expertise with various methods2. Higher expenses with more methods
7
![Page 8: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/8.jpg)
8
Prior research
Meta-analysis of 30 studies on combining: 12% error reduction vs. error of typical component.
Recommendation: Combine forecasts from different methods that use different information
[Armstrong, 2001]
However, few studies have focused on the ex ante conditions of when combining is most beneficial.
![Page 9: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/9.jpg)
9
Polly’sComponents
Polly‘s components
PollsIEM
predictionmarket
ExpertsQuantitative
models
![Page 10: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/10.jpg)
10
Polly’sComponents
PollsProblem:
• Polls often unreliable, especially early in campaign
• Large differences in results of individual polls conducted around the same time
Polls
Within component Combining
![Page 11: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/11.jpg)
11
Polly’sComponents
IEMprediction
market
Within component Combining
• Polly’s prediction market: Iowa Electronic Markets (IEM)
• 7-day rolling average of daily market prices
• Adjust for overreactions of market such as information cascades
IEM prediction market
![Page 12: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/12.jpg)
12
Polly’scomponents Experts
Within component Combining
• Survey of experts
• Assumptions: Experts possess • Information from polls• Knowledge about the effect of
debates, campaigns, etc.
Experts
![Page 13: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/13.jpg)
13
Polly’scomponents
Quantitativemodels
Within combining Combining
Models focus on 2 to 7 variables, most often Incumbent‘s popularity State of economy
Individual accuracy of models varies across elections
Quantitative models
![Page 14: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/14.jpg)
14
Mean error reduction(93 days prior to Election Day,1992 to 2008)
Polly’scomponents
Gains from combining within components
Polls IEM Experts Models
Within components Combining Combining Combining Combining
14% 9% 21%18%
![Page 15: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/15.jpg)
Polly’scomponents
Combining across components
Polls IEM Experts Models
Within components Combining Combining Combining Combining
Across componentsCombining(unweighted
average)
PollyVote-Prediction
![Page 16: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/16.jpg)
Mean error reduction(93 days prior to Election Day,1992 to 2008)
Polly’scomponents
Gains from combining across components
Polls(combined)
IEM (combined)
Experts (combined)
Models (combined)
PollyVote-Prediction
50% 1% 32%43%
![Page 17: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/17.jpg)
Mean error reduction(93 days prior to Election Day,1992 to 2008)
Polly’scomponents
Gains from combining within & across components
TypicalPoll
OriginalIEM
TypicalExperts
TypicalModels
PollyVote-Prediction
58% 10% 58%52%
![Page 18: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/18.jpg)
18
If combining forecasts is so useful,
why is it seldom used?
![Page 19: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/19.jpg)
19
1. Managers do not believe combining helps
In four experiments with MBAs at INSEAD, most subjects did not realize that the error of the average forecast would be less than the error of the typical forecast.
Most subjects thought that averaging forecasts would yield average performance.
[Larrick & Soll, 2006]
![Page 20: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/20.jpg)
20
2. Some forecasters mistakenly believe they are combining properly
People often use unaided judgment to assign differential weights to individual forecasts.
Informal combining is likely to be harmful as people can select a forecast that suits their biases.
![Page 21: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/21.jpg)
21
3. Managers, forecasters, and researchers are persuaded by complexity
Simple models often predict complex problems better than more complex ones.
[Hogarth, in press]
These findings are difficult to believe. There is a strong belief that complex models are necessary to solve complex problems.
![Page 22: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/22.jpg)
4. Forecasters build reputation with extreme forecasts
Forecasters do not want to get lost in the crowd.
More extreme forecasts usually gain more attention and the media is more likely to report them.
[Batchelor, 2007]
![Page 23: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/23.jpg)
5. People mistakenly believe they can identify the most accurate forecast
In a series of experiments, when given two estimates as advice, most people chose one instead of averaging them – and thereby reduced accuracy.
[Soll & Larrick, 2009]
![Page 24: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/24.jpg)
24
Why doesn’t the PollyVote capture mass media attention?
The PollyVote varies little and, basically, is never wrong. Thus, no entertainment value.
Instead of accuracy, voters want excitement – and hope for their candidate.
![Page 25: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/25.jpg)
25
Accuracy problem is solved for major elections
PollyVote deviation averaged 0.4% for the 2004 and 2008 U.S. presidential elections and substantial improvements are scheduled for 2012.
Polly is available to researchers and practitioners for elections in the U.S., as well as in other countries.
![Page 26: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/26.jpg)
Applications of combining
All organizations can benefit from combining.
![Page 27: Forecasting elections from voters' perceptions of candidates' ability to handle issues](https://reader036.vdocument.in/reader036/viewer/2022070315/554f4967b4c905524c8b485f/html5/thumbnails/27.jpg)
ReferencesArmstrong, J. S. (2001). Combining forecasts. In: J. S. Armstrong (Ed.),
Principles of Forecasting: A Handbook for Researchers and Practitioners, Norwell: Kluwer, pp.417-439.
Batchelor, R. (2007). Bias in macroeconomic forecasts, International Journal of Forecasting, 23, 189-203.
Hogarth, R. (in press). When simple is hard to accept. In P. M. Todd, G. Gigerenzer, & The ABC Research Group (Eds.), Ecological rationality: Intelligence in the world. Oxford: Oxford University Press.
Larrick, R. P. & Soll, J. B. (2006). Intuitions about combining opinions: Misappreciation of the averaging principle. Management Science, 52, 111-127.
Soll, J. B. & Larrick, R. P. (2009). Strategies for revising judgment: How (and how well) people use others’ opinions, Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 780-805.