Global Warming: Experts’ Opinions versus Scientific Forecasts
Friday, February 01, 2008
by Kesten C. Green and J. Scott Armstrong
Table of Contents
Three Basic Forecasting Principles Violated by the Fourth Assessment Report
Some principles are so important that any forecasting process that does not adhere to them cannot produce valid forecasts. The following are three such principles, all of which are based on strong empirical evidence, and all of which were violated by the forecasting procedures described in Chapter 8 of the IPCC report.
Principle: Consider whether the events or series can be forecasted. This principle refers to whether a forecast would likely be more accurate than assuming things will not change. Predicting no change is to use a naïve forecasting method. It is appropriate to use a naïve method when knowledge is poor and uncertainty is high, as with climate.30 There is even controversy among climate scientists over something as basic as the current trend with respect to temperature. Temperature data is highly variable and cyclical, and proxy data (such as ice cores and tree rings) must be used to infer temperatures more than a few decades past. Whether a trend over time is determined to be rising, falling or stable depends largely on the beginning and endpoint chosen.31
Global climate is complex and scientific evidence on key relationships is weak or absent. For example, the IPCC holds that CO2 plays a significant causal role in determining global temperature, but a number of studies have presented evidence that causation generally works in the opposite direction (increases in atmospheric levels of CO2 are a result of higher temperatures) and that CO2 variation plays at most a minor role in climate change.32
“Temperature data is often incomplete and unreliable.”
Measurements of key variables such as local temperatures and a representative global temperature are contentious and subject to revision. In the case of modern measurements they must be adjusted for the changing distribution of weather stations and such complicating factors as the urban-heat-island effect. The interpretation of proxy data for ancient temperatures is often speculative.33 Finally, it is difficult to forecast the causal variables — for example CO2 and cloudiness.34
Although the authors of Chapter 8 claim that the forecasts of global mean temperature are well-founded, their language is imprecise and relies heavily on such words as “generally,” “reasonably well,” “widely,” and “relatively.” These terms indicate that the IPCC forecasts are uncertain, and the chapter makes many explicit references to uncertainty.35
In discussing temperature modeling, the authors wrote, “The extent to which these systematic model errors affect a model’s response to external perturbations is unknown, but may be significant,” and, “The diurnal temperature range… is generally too small in the models, in many regions by as much as 50 percent,” and “It is not yet known why models generally underestimate the diurnal temperature range.”36
Given the high uncertainty regarding climate, the appropriate naïve method for this situation would be the “no change” model since prior evidence suggests that attempts to improve upon the naïve model often increase forecast error. To reverse this conclusion, one would have to produce validated evidence in favor of alternative methods. Chapter 8 of the IPCC report does not provide this evidence. If long-term forecasting of climate is possible, it has yet to be demonstrated.
Principle: Keep forecasting methods simple. IPPC chapters and related papers leave the impression that climate forecasters believe that complex models are necessary for forecasting climate and that forecast accuracy will increase with model complexity. Complex methods involve such things as the use of a large number of variables in forecasting models and complex interactions between variables. Complex forecasting methods are only accurate when there is little uncertainty about relationships now and in the future, where the data are subject to little error, and where the causal variables can be accurately forecast. These conditions do not apply to climate forecasting. Thus, simple methods are recommended.
“The models are complex and produce conflicting results.”
The use of complex models when uncertainty is high conflicts with the evidence from forecasting research.37 For example, scientists Halmar Halide and Peter Ridd compared predictions of El Niño-Southern Oscillation events from a simple extrapolation of the time series with those from other researchers’ complex models.38 Some of the complex models were dynamic causal models incorporating laws of physics. In other words, they were similar to those upon which the IPCC authors depended. The simple model made more accurate three-month predictions than all 11 of the complex models. Every model performed poorly when forecasting further ahead.
Using complex methods prevents understanding of how forecasts were derived, makes criticism difficult, and error detection unlikely.
Principle: Do not use fit to develop the model. It is unclear to what extent the models described in the IPCC report are either based on, or have been tested against, sound empirical data.39 However, some statements were made about the ability of the models to fit historical data, after tweaking their parameters. Extensive research has shown that the ability of a model to reproduce historical data has little relationship to forecast accuracy.40 Fit can be improved by making a model more complex. The typical consequence of increasing complexity to improve fit, however, is to decrease the accuracy of forecasts.
Other Audit Results. The audit also found 12 “apparent violations.” These principles are areas where the authors had concerns over the coding or did not agree that the procedures clearly violated the principle. Finally, for many of the relevant principles, there was insufficient information to make ratings. Some of these principles might be surprising to those who are not familiar with forecasting research — for example: “Use all important variables (Principle 10.2).” Others are principles that any scientific paper should be expected to address, such as: “Use objective tests of assumptions.” And others are especially important to climate forecasting, such as: “Limit subjective adjustments of quantitative forecasts.”41
The number of violations of forecasting principles found by the audit confirms the survey finding described earlier that the IPCC authors were unaware of forecasting principles.42 Had they been aware of the principles, it would have been incumbent on them to present evidence to justify their departures from them. They did not do so. Because the forecasting processes examined in Chapter 8 overlook scientific evidence on forecasting, the IPCC forecasts of climate change are not scientific.
Climate change forecasters should use the readily available Forecasting Audit program to ensure that they are using appropriate forecasting procedures. Outside evaluators should also be encouraged to conduct audits. These reports should be made available to both study sponsors and the public by posting on an open Web site such as publicpolicyforecasting.com.