Bucharest Dialogues on Expert Knowledge, Prediction, Forecasting: A Social Sciences Perspective November 21, 2010

1. The PollyVote
Combining forecasts for
U.S. Presidential Elections
Andreas Graefe, Karlsruhe Institute of Technology
J. Scott Armstrong, Wharton School, University of Pennsylvania
Randall Jones, Jr., University of Central Oklahoma
Alfred Cuzán, University of West Florida
The full paper to this talk can be downloaded at: tinyurl.com/combiningelections.
Bucharest Dialogues on
Expert Knowledge, Prediction, Forecasting: A Social Sciences Perspective
November 21, 2010

2. Background on the PollyVote project
The PollyVote project was begun in 2003 to
demonstrate the value of forecasting principles
by applying them to election forecasting.
The initial focus was on combining forecasts.

3. Performance of the PollyVote
The PollyVote combined forecasts to obtain highly accurate
forecasts of U.S. Presidential Election outcomes:
– Prospectively for 2004 and 2008 (MAE: 0.4 percentage points)
– Retrospectively for 1992 to 2000
Across these five elections, the PollyVote was on average
more accurate than each of its components:
- Polls
- Prediction markets
- Experts
- Statistical models
Polly achieved this without knowing anything about politics.

4. Power of combining
Question: What is the ratio of students per teacher in
primary schools in Romania?
Judge Estimate Error
1 18 .5
2 19 1.5
Typical error of individual estimate 1
Combined estimate 18.5 1
Error reduction through combining 0%
Judge Estimate Error
1 18 .5
2 16 1.5
Typical error of individual estimate 1
Combined estimate 17 0.5
Error reduction through combining 50%

5. Procedure and conditions for combining forecasts
Procedure:
Mechanically combine forecasts equal weights
(unless you have strong evidence for differential weights)
Conditions:
1. Several forecasts available
2. Uncertainty about which forecasts is most accurate
(although combing is often beneficial even when the best
method is known beforehand)
Conditions for when combining is most beneficial:
1. Different forecasting methods are available
2. Forecasts rely upon different data

6. Benefits of combining
1. Improves accuracy
2. Avoids large errors
3. Provides an additional assessment of
uncertainty
4. Can be used for nearly all forecasting
problems.
5. Simple to describe and apply.

7. Costs of combining
1. Requires expertise with various methods
2. Higher expenses with more methods
7

8. Prior research
Meta-analysis of 30 studies on combining: 12% error
reduction vs. error of typical component.
Recommendation: Combine forecasts from
different methods that use different
information
[Armstrong, 2001]
However, few studies have focused on the ex ante
conditions of when combining is most beneficial.
8

9. 9
Polly’s
Components
Polly‘s components
Polls
IEM
prediction
market
Experts
Quantitative
models

10. 10
Polly’s
Components
Polls
Problem:
• Polls often unreliable, especially
early in campaign
• Large differences in results of
individual polls conducted around
the same time
Polls
Within component Combining

11. 11
Polly’s
Components
IEM
prediction
market
Within component Combining
• Polly’s prediction market: Iowa Electronic
Markets (IEM)
• 7-day rolling average of daily market
prices
• Adjust for overreactions of market
such as information cascades
IEM prediction market

12. 12
Polly’s
components Experts
Within component Combining
• Survey of experts
• Assumptions: Experts possess
• Information from polls
• Knowledge about the effect of
debates, campaigns, etc.
Experts

13. 13
Polly’s
components
Quantitative
models
Within combining Combining
 Models focus on 2 to 7 variables,
most often
 Incumbent‘s popularity
 State of economy
 Individual accuracy of models
varies across elections
Quantitative models

14. 14
Mean error reduction
(93 days prior to
Election Day,
1992 to 2008)
Polly’s
components
Gains from combining within components
Polls IEM Experts Models
Within components Combining Combining Combining Combining
14% 9% 21%18%

15. Polly’s
components
Combining across components
Polls IEM Experts Models
Within components Combining Combining Combining Combining
Across components
Combining
(unweighted
average)
PollyVote-Prediction

16. Mean error reduction
(93 days prior to
Election Day,
1992 to 2008)
Polly’s
components
Gains from combining across components
Polls
(combined)
IEM
(combined)
Experts
(combined)
Models
(combined)
PollyVote-Prediction
50% 1% 32%43%

17. Mean error reduction
(93 days prior to
Election Day,
1992 to 2008)
Polly’s
components
Gains from combining within & across components
Typical
Poll
Original
IEM
Typical
Experts
Typical
Models
PollyVote-Prediction
58% 10% 58%52%

18. If combining forecasts is so useful,
why is it seldom used?
18

19. 1. Managers do not believe combining helps
In four experiments with MBAs at INSEAD, most
subjects did not realize that the error of the average
forecast would be less than the error of the typical
forecast.
Most subjects thought that averaging forecasts would
yield average performance.
[Larrick & Soll, 2006]
19

20. 2. Some forecasters mistakenly believe
they are combining properly
People often use unaided judgment to assign
differential weights to individual forecasts.
Informal combining is likely to be harmful as people
can select a forecast that suits their biases.
20

21. 3. Managers, forecasters, and researchers are
persuaded by complexity
Simple models often predict complex problems better
than more complex ones.
[Hogarth, in press]
These findings are difficult to believe. There is a strong
belief that complex models are necessary to solve
complex problems.
21

22. 4. Forecasters build reputation with extreme
forecasts
Forecasters do not want to get lost in the crowd.
More extreme forecasts usually gain more
attention and the media is more likely to report
them.
[Batchelor, 2007]

23. 5. People mistakenly believe they can
identify the most accurate forecast
In a series of experiments, when given two
estimates as advice, most people chose one
instead of averaging them – and thereby reduced
accuracy.
[Soll & Larrick, 2009]

24. Why doesn’t the PollyVote
capture mass media attention?
The PollyVote varies little and, basically, is never
wrong. Thus, no entertainment value.
Instead of accuracy, voters want excitement – and
hope for their candidate.
24

25. Accuracy problem is solved for
major elections
PollyVote deviation averaged 0.4% for the 2004
and 2008 U.S. presidential elections and
substantial improvements are scheduled for
2012.
Polly is available to researchers and practitioners
for elections in the U.S., as well as in other
countries.
25

26. Applications of combining
All organizations can benefit from combining.

27. References
Armstrong, J. S. (2001). Combining forecasts. In: J. S. Armstrong (Ed.),
Principles of Forecasting: A Handbook for Researchers and
Practitioners, Norwell: Kluwer, pp.417-439.
Batchelor, R. (2007). Bias in macroeconomic forecasts, International
Journal of Forecasting, 23, 189-203.
Hogarth, R. (in press). When simple is hard to accept. In P. M. Todd, G.
Gigerenzer, & The ABC Research Group (Eds.), Ecological rationality:
Intelligence in the world. Oxford: Oxford University Press.
Larrick, R. P. & Soll, J. B. (2006). Intuitions about combining opinions:
Misappreciation of the averaging principle. Management Science,
52, 111-127.
Soll, J. B. & Larrick, R. P. (2009). Strategies for revising judgment: How
(and how well) people use others’ opinions, Journal of Experimental
Psychology: Learning, Memory, and Cognition, 35, 780-805.