Principles of Forecasting

Research on forecasting has been conducted since the 1930s. Empirical studies that compare methods and sources of evidence have been conducted to determine which forecasts are the most accurate. Researchers began to summarize, gather, collate and systematize these findings in the 1970s and 1980s.¹ In the mid-1990s, the Forecasting Principles Project was established with the objective of summarizing all useful knowledge about forecasting. The knowledge was codified as evidence-based principles. Evidence-based principles tell which methods to use under what conditions, based on the evidence to date. The strongest form of evidence comes from empirical studies that have compared the predictive ability of different forecasting methods.

The Forecasting Principles Project led to the Principles of Forecasting handbook: the work of 40 internationally-known experts on forecasting methods and 123 reviewers who were also leading experts on forecasting methods.² The forecasting principles are available on a site maintained and updated by the International Institute of Forecasters.³ For example, these forecasting principles were applied to the problem of predicting the outcome of the 2004 U.S. presidential election, and yielded predictions of a win for President Bush early and consistently, and predicted the vote more accurately than any other forecaster.

Following are examples of the principles that apply to long-term forecasts for complex situations where the causal factors are subject to uncertainty (as with climate).

Principle: Unaided judgmental forecasts by experts have no value. “Unaided” means evidence-based forecasting principles were not used. Such forecasts are (at best) educated guesses. This principle applies whether the opinions are expressed in words, spreadsheets or mathematical models. It also applies regardless of how much scientific evidence is possessed by the experts. Among the reasons for this are:

1. Complexity: People cannot assess complex relationships through unaided observations.
2. Coincidence: People confuse correlation with causation.
3. Feedback: People making judgmental predictions typically do not receive the explicit feedback
   they could use to improve their forecasting.
4. Bias: People have difficulty in obtaining or using evidence that contradicts their initial beliefs.
   This problem is especially serious for people who view themselves as experts.

Principle: Agreement among experts is weakly related to accuracy. This is especially true when the experts communicate with one another and when they work together to solve problems, as is the case with the IPCC process.

Principle: Complex models involving nonlinear relationships and interactions between variables are less accurate when there is uncertainty because errors multiply. For instance, the widely publicized 1972 forecasts by the Club of Rome of ecological and economic collapse proudly proclaimed, “in our model, about 100,000 relationships are stored in the computer.”4 However, the model made wildly inaccurate predictions, such as widespread famine and resource depletion by the year 2000.

Complex models tend to fit random variations in historical data well, but as a consequence of their complexity, their forecasts are less accurate than simpler models and result in misleading conclusions
about outcomes. Furthermore, when complex models are developed there are many opportunities
for errors, and the complexity makes errors more difficult to find. For example, long-term energy forecasts based on computer models failed to predict the ability of the U.S. economy to increase energy efficiency in response to oil embargos in the 1970s.5

Principle: Given even modest uncertainty, prediction intervals are enormous. Prediction intervals (ranges outside which outcomes are unlikely to fall) expand rapidly as time horizons increase. The longer the time from the date of the prediction to the date of the predicted event or result, the greater the likelihood of an unanticipated result (one falling outside the expected range of outcomes). This is true, for example, even when trying to forecast something as straightforward as automobile sales for General Motors over the next five years.

Principle: When there is uncertainty, forecasts should be conservative. Uncertainty arises when data contain measurement errors, when the series are unstable, when knowledge about the direction
of relationships is uncertain, and when a forecast depends upon forecasts of related (causal) variables. For example, forecasts of no change were found to be more accurate than trend forecasts for annual sales when there was substantial uncertainty in the trend lines.6

¹ J. Scott Armstrong, Long-Range Forecasting: From Crystal Ball to Computer (New York, N.Y.: Wiley-Interscience, 1985).

² J. Scott Armstrong, Principles of Forecasting (Norwell, Mass.: Kluwer Academic Press, 2001).

³ See forecastingprinciples.com, a site sponsored by the International Institute of Forecasters. A summary of the principles, currently numbering 140, is provided as a checklist in the Forecasting Audit software available on the site. The site is often updated in order to incorporate new evidence on forecasting as it comes to hand. J. Scott Armstrong, “Findings from evidence-based forecasting: Methods for reducing forecast error,” International Journal of Forecasting, Vol. 22, 2006, pages 583-598.

⁴ W. Ascher, Forecasting: An Appraisal for Policy Makers and Planners (Baltimore, Md.: Johns Hopkins University Press, 2005).

⁵ P. P. Craig, A. Gadgil and J. G. Koomey, “What Can History Teach Us? A Retrospective Examination of Long-Term Energy Forecasts for the United States,” Annual Review of Energy and the Environment, Vol. 27, 2002, pages 83-118.

⁶ This principle also implies that forecasts should revert to long-term trends when such trends have been firmly established, do not waver, and there are no firm reasons to suggest they will change. Finally, trends should be damped toward no
change as the forecast horizon increases. See S. P. Schnaars and R. J. Bavuso, “Extrapolation models on very short-term forecasts,” Journal of Business Research, Vol. 14, 1986, pages 27-36.