forecasts and outcomes, ska bur skis, a., & teitz, m. b. (2003)

8/2/2019 Forecasts and Outcomes, Ska Bur Skis, A., & Teitz, M. B. (2003)

http://slidepdf.com/reader/full/forecasts-and-outcomes-ska-bur-skis-a-teitz-m-b-2003 1/15

This article was downloaded by: [Computing & Library Services, University of Huddersfield]On: 06 May 2012, At: 04:15Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office:Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Planning Theory & Pract ice

Publication details, including instructions for authors and subscriptioninformation:ht tp:/ / www.t andfonline.com/ loi/ rptp20

Forecasts and outcomesAndrej s Skaburskis & Michael B. Teitz

Available online: 04 Jun 2010

To cite this art icle: Andrej s Skaburskis & Michael B. Teitz (2003): Forecasts and outcomes, Planning Theory Practice, 4:4, 429-442

To link to this article: htt p:/ / dx.doi.org/ 10.1080/ 1464935032000146309

PLEASE SCROLL DOWN FOR ARTICLE

Full terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditions

This article may be used for research, teaching, and private study purposes. Any substantialor systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, ordistribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that thcontents will be complete or accurate or up to date. The accuracy of any instructions, formulae,and drug doses should be independently verified with primary sources. The publisher shall notbe liable for any loss, actions, claims, proceedings, demand, or costs or damages whatsoever orhowsoever caused arising directly or indirectly in connection with or arising out of the use of thismaterial.

http://www.tandfonline.com/page/terms-and-conditions

http://dx.doi.org/10.1080/1464935032000146309

http://dx.doi.org/10.1080/1464935032000146309

http://www.tandfonline.com/page/terms-and-conditions

http://dx.doi.org/10.1080/1464935032000146309

http://www.tandfonline.com/loi/rptp20



Planning Theory & Practice, Vol. 4, No. 4, 429–442, December 2003

Forecasts and Outcomes

ANDREJS SKABURSKIS & MICHAEL B. TEITZ

ABSTRACT The forecasts used by planners often overstate the outcomes that are eventuallyobserved. Many forecasts prepared for planning purposes project changes that turn out to havebeen exaggerations of the changes that actually take place. This article explores some of thereasons forecasts of social changes may overstate the eventual outcomes and considers the role of interests as well as some of the technical factors that may encourage exaggeration andoverstatement. The nature of the models often used as a basis for preparing forecasts may lead tothe overstatements of impacts. The article concludes by recognizing the homeostatic nature of the

forces that forge city structures and presents some implications regarding power relations andethics.

Introduction

Official plans are often based on predictions about future environments and about theireffect on those environments. At times the predictions are made with reference toconventional wisdom, unquestioned belief, anecdotal evidence, limited information,wishful thinking or strategic myopia. At times they are produced by analysts thought-fully examining current conditions and extrapolating past trends with models supported

by theory identifying and explaining the key determinants. After thinking about the

forecasts that we have heard most about and by reflecting on the kind of predictions we,ourselves, would have made after hearing about new research findings, the strongimpression was formed that forecasts of change or of project, plan or policy conse-quences tend to be overstatements when compared to the eventual outcomes.1

This article draws on the authors’ North American experience and on an understand-ing of the analytical methods that are used in planning to explain why forecasts andpredictions of impacts are so often overstatements when compared with the eventualoutcomes. There is a look at the technical issues and factors that can guile us intoaccepting exaggerated trend extrapolations and depictions of their consequences(Skaburskis, 1995). Some of the overstated predictions are made knowingly and play arecognized and accepted role in discourse. Some are outright lies, as in the case of the

understated costs (the overstated net benefits) of public works projects (Flyvbjerg et al.,2002; Wachs, 1986, 1989, 1990). Forecasts that are recognized as being made for strategicpurposes we often accept without much concern as rhetoric intended to highlight claimsor encourage supporters and discourage opponents. Other forecasts are made by policyanalysts and planners who are sincerely trying to understand the underlying factorsdetermining trends and use this information to forecast the future conditions that

become the basis for planning today. Forecasts always accompany plans in the form of

Andrejs Skaburskis, School of Urban and Regional Planning, Policy Studies Building, Queen’s University, Kingston,Ontario, Canada K7N 3L6. Email: [email protected]

1464-9357 Print/1470-000X On-line/03/040429-14 2003 Taylor & Francis Ltd

DOI: 10.1080/1464935032000146309



430 A. Skaburskis & M. B. Teitz

predictions as to their consequences. Well-intentioned claims about future conditionsand outcomes may also end up as exaggerations or overstatements for reasons that maynot always be obvious.

Exaggeration may be built into a forecast as a result of the analyst’s or planner’sinclination to support a client’s interests, by institutional factors affecting the forecasters,

by errors in estimation, by limited knowledge of how the markets, political and social

processes function and react to fundamental changes or threats of change, and, also bythe way knowledge of the complex interactions among the factors that affect cities andsocial networks evolves through research and is promulgated through publications andconferences. The overstatements may be due to our not appreciating the underlyingcomplexity of social processes, due to unrecognized cycles that can bring a deviation

back to a long-run trend that is guided by deeper factors, or, due to social, political oreconomic feedback mechanisms or reactions that revert trends or ameliorates adverseconsequence. Overstatements in forecasts may be the result of the homeostatic nature ofsocial systems.

The general observation, ‘forecasts overstate outcomes’, may be due to our selectivememory; we may simply not have paid attention to the forecasts offering moderate

change and have forgotten the forecasts that matched the outcomes fairly well. Perhapsit is the exaggerations that catch our interest and motivate us in making plans andadvocating policy. But the biases due to our the nature of our memory are not thatimportant here because you and I may share the way we screen claims about outcomesand we may all recognize that most of our forecasts are overstatements. As a result, wemay without much thought be discounting most of the forecasts presented to us andtake them with a pinch of salt, or, at times, put technical analysis aside in favour ofconventional wisdom, or, perhaps seek agreements as to what the future will be likeamong interested parties that may turn out to be equally wrong or worse. However,some forecasts are not easily recognized as overstatements because we may be aware ofour attraction to exaggeration, and the overstatement may be due to the way we revealand organize causal interrelationships. An understanding of the underlying reasons why

‘technical’ forecasts can turn out to be overstatements or exaggerations may help us better assess the forecasts that are used in policy development and planning practice.

To start, it must be recognized that not all forecasts are exaggerations and not all typesof forecasts that are used in planning and policy works have consistently been overstate-ments. Population forecasts, for example, have been systematically wrong for longperiods of time as old assumptions were maintained past their useful life (assumptiondrag), or as the nature of what was thought of as a structural and lasting relationship

between factors changes (parameter drift) (Stoto, 1983). The errors produced by assump-tion drag and parameter drift have persisted long enough to produce populationforecasts for the USA that have been consistently wrong, but over the long-run have notsystematically over-stated or understated the eventual outcome. Keyfitz (1982) compared

more than a thousand population forecasts from around the world with their eventualoutcome and found no systematic error. Demographers, somewhat more removed frompolicy than planners, have been making serially correlated errors but not ones thatconsistently exaggerate trends. Of course demographers do not usually discuss theconsequences of their population projections and here is where the potential andpossibly the need for exaggeration emerge.

Predictions of effectiveness of city plans or of the consequences of changing economicand demographic conditions differ from national forecasts in a number of ways. First,scale issues are raised in the case of city planning because the subject is narrower andthe relevant geographic area much smaller. The effects of the substantive and spatial



Forecasts and Outcomes 431

differences are discussed later. Second, the research budgets and consulting fees avail-able for planning studies are much smaller, as is the knowledge base on which cityplanners can draw. Third, the consequences of the forecasts are more apparent to theaffected parties, allowing interests to play a larger role in their preparation and adoption.Fourth, local forecasts usually matter to the people whose behaviour or whose situationis being projected, with the result that predictions of large changes will be noticed and,

when undesirable, generate behavioural or policy changes that try to mitigate theadverse consequences.

Even when predictions are prepared with a sincere regard for truth, they will differfrom the eventual outcome because all relevant factors will not have been accounted forand changes in the way the factors affect outcomes could not have been foretold. Thisshould not be a major problem, as Jack Dykman used to explain in his lectures back inthe late 1960s. There is no point in trying to judge forecasts by their eventual correspon-dence to future conditions because they are intended for the present and should beassessed by how well they improve the decisions that are made today. Forecasts areintended to help make decisions today that will affect the future and the overstatementin the forecast may be its salient feature that draws attention to what really matters and,

thereby, stimulates imagination to help us appreciate a wider range of possible futurepossibilities. The hope is that forecasts are properly qualified because if they are not,then they can be lies. Of course, the qualifications must accompany their use at the timethat concrete actions are being planned and the consequences of the plans are beingadvertised to the affected people. This is not the case when the predicted ‘best’ scenariois accepted by policy advocates regardless of the alternative projections. City councillorsor planners tend to accept the middle projection using a symmetry kind of an argumentfor removing the uncertainty in depictions of the future. Analysts sometimes try toremove themselves from taking responsibility for their forecast’s consequences bywording the qualifications in a way that only other attentive analysts can appreciate.

We, as authors of this article, should qualify our conjecture that the forecasts that are

used in planning tend to exaggerating outcomes. Because our thoughts are based on ourown experience and observation not on scientific analysis, we may remember best theexaggerated forecasts and base our generalization on a selected and biased sample. Toaccept our conjecture erroneously or to unwittingly maintain the tendency of discount-ing all forecasts and predictions for being overstatements, even the conservative predic-tions that will end up as true statements, can lead to serious error. Our quite naturaladjustment to the usual bias in forecasts may make us less sensitive to the messagesconveyed by all forecasts; we may ignore the warnings in the predictions that are notgross overstatements, especially when they hard to accept. This could, for example,explain why so many people are so oblivious to the predicted consequences of globalwarming. We may, in fact, be discounting these forecasts because we are used to the

‘overstatement’ in most of the other predictions that we have been exposed too. We haveto recognize that not all forecasts are overstatements and, unfortunately, that we can notalways identify the ones that are true, a priori.

This article describes some of the reasons that exaggerations are introduced and theway that the modelling that underlies ‘technical forecasts’ can lead to overstatements ofoutcomes. The article offers possible reasons why we may be more inclined to accept anduse the forecasts that present exaggerated views of future changes. The reasons mayrelate to our own interests and to institutional concerns, to the way social, political andeconomic processes react to change, and to the way we gain knowledge of theseprocesses.




The Role of Interest

Proponents for a plan obviously gain from unfavourable forecasts of the conditions thatwould prevail if the plan is not implemented and favourable depictions of the plan’sconsequences. The process of commissioning a forecast or accepting a consultant’sforecast may favour exaggerated predictions, and because the clients will want the

strongest supporting arguments, interest will move the analysts and forecasters toexaggeration. Consultants are rewarded directly and indirectly for making favourableforecasts (Wachs, 1982). Political bias affects forecasters and forecast users. Exaggerationmay be due to more subtle factors and may be introduced through the use of hiddenassumptions underlying the forecast (Ascher, 1978). Again, political interests may keepthe assumptions implicit.

Forecasts may be intended to generate interest; self-defeating forecasts are by definitionoverstatements in comparison with the eventual outcomes and self-fulfilling forecastsmay have been prepared as exaggerations to lure the support that boosts the trend.Forecasts developed for advocacy should be overstatements because their message will

be heavily discounted in any case. The amount of exaggeration needed to move

constituencies and the amount by which the forecasts will be discounted depend on theculture within which they are made and on what is the norm for making and acceptingstatements about current conditions and future possibilities. The stereotypical Englishaudience in the UK may, for example, be moved by an understatement as much as anotheraudience by flamboyant exaggeration. In everyday life we can imagine cases in whichoverstatements are expected and anything less would have us back off—‘the fish is quitefresh’—while in other cases, such as a doctor’s prognosis, we expect no overstatement.

Personal interests may encourage us to exaggerate our research findings. The socialcontext of our work as analysts may encourages us to overstate the consequences of thetrends we are exploring. Who notices predictions of the status quo? Academics may bemore distant from policy interests but we still want our articles to show more rather thanless for obvious reasons. The detachment of academic pursuits may make the difference

between a projection and the eventual outcomes appear to be less important to theauthors. After all, someone else has the responsibility for the application and thequalification of results. Moreover, an exaggerated claim can be justified as helpingstimulate further inquiry, moving funding agencies and encouraging others to work onthe subject; recall the instance of ‘cold fusion’. In any case, scientific propositions are putforward with the understanding that they will be rejected by future research so whyworry about this article’s claim. Our contribution is temporary and, at the very best ourpredictions will have their fifteen minutes. The exaggeration will not matter in the longrun.

Analysts who do not recognize or publicize qualifications to forecasts will introduce bias in the presentation of their findings that is usually in the form of an overstatement.

Planners and city councillors, who select research results, disregarding or dismissing theconclusions and references they do not like, compound the error and add to the historyof plans and policies built on overstated predictions and forecasts. Interests aside, evensincere efforts to remain free from client interests can lead to exaggeration in forecastsand projections.

The Seduction of Outliers

A major potential source of overstatement in projections of the consequences of emerg-




ing conditions or trends can be found in the informal methods used to developgeneralizations. This type of error is due to using limited and highly skewed data orusing rare events as a basis for scenario building. Politicians often make this error andit abounds in the projections based on conventional wisdom. Newspapers publicize theexceptional cases, the outrageous events, and the anecdotes that draw people’s attentionto the dire consequences of current and impending conditions. Stories depicting extra-

ordinary events can be useful to planners and policy advocates by helping generate theemotive force that motivates the constituency that will pressure for change.2 Theoverstatements may be the result of blatant disregard for statistical rules when makinggeneralizations, or they may be due to the fears raised by the events that, in turn, focusour attention on subsets of the evidence that then define the backdrop used to interpretall the other related conditions and trends. In other words, we can be blinded byextraordinary situations, or events may depict the future as being saturated by theoutliers we want to avoid.

The drawing of unjustifiable inferences from anecdotal evidence may be motivated bya proponent’s lack of statistics, by commitment to prior beliefs, or by a public whoseopinion and voting patterns are moulded by the publicizing of rare but extraordinary

events. In some cases this bias can be justified by our felt conviction that even the fewcases, the anecdotal cases, should not have been allowed to happen and that policyshould have been in place that precludes their occurrence. In these cases, the relevantprojection may be the exaggerated one.

Mitigation and Remediation Policies

Threats of programme change or programme termination are typically met with exag-gerated predictions of dire consequences. Political reactions to the overstatements may

bring about changes that moderate the undesirable predictions. The closure of govern-ment programmes may raise pressures for other remediation actions that soften theimpact and prevent the dire predictions from coming true. The ending of the US Section236 housing subsidy programme, welfare reform, and base closures in the USA provideexamples of outcomes that turned out to be much milder than the predicted conse-quences. In some cases ameliorating policy reduces the impact, while in others, notably

base closures, the predicted catastrophe does not materialize due to the many otherfactors that come into play (Bradshaw, 1999). In these cases the forecasts are notnecessarily wrong as the predicted conditions might have materialized had policy notchanged.

Scale Differences, Catch 22 and Intuition

The methods used to study large regions or more general subjects cannot always beapplied to smaller regions or more narrowly defined subjects. The smaller the geo-graphic area or the more specific the subject, the less is the chance that influential factorscancel each other out, and the greater the chance that a smaller set of factors play a majorrole in determining future conditions. We may not know which of the many potentialfactors will turn out to be important. Not only do the many relevant factors have to be

better understood and explicitly accounted for, but also, because we do not know forsure which of the factors will turn out to be the most important, we want to haveknowledge about most of them. And here is the Catch 22—the smaller the expected




impact or scope or geographic area, the more difficult the forecasting task and thesmaller the budget for developing the forecast. But more, the narrower the subject, thesmaller is the existing knowledge base that can be drawn on to guide the analysis, tooffer theoretical models, to yield useable parameters and to provide precedents forchecking the validity of one’s work. Since there is less of a ‘technical’ basis fordeveloping forecasts, budget-constrained analysts are more likely to stay with the

methods that support the client’s preconceptions rather than challenge them while facedwith inadequate data, limited analysis and lack of supporting precedents. Since the clienthas a reason for commissioning the forecast, the scale effects are likely to encourageexaggeration.

Scale differences can also lead to exaggeration of the beneficial impacts of projects andplans when they involve the multiplication of small numbers by very large numbers thatare beyond our intuitive grasp. Surely every Californian would be willing to pay a dollara year to have the ‘option’ of boating on a new lake formed by a new hydro dam. Atthe time such claims were said to be made by the US Army Corps of Engineers, therewere 20 million Californians who might have been willing to pay a dollar per year forthe option, so there would be $20 million in annual benefits, which discounted at 4%

would yield a $500 million present value bonus in the benefit/cost analysis. This is nota lie in the sense that Flyvbjerg et al. (2002) point out, but the selective inclusion of adoubtful impact. In this case, the exaggeration is brought about in three ways. The firstis due to our not being able to gain a fair sense of the value of small numbers—onedollar or three dollars for a year’s worth of ‘benefits’, or are they worth only 50 cents?If we accept the benefit, then who is going to dispute nickels and dimes? But theconsequences of the choice are huge. The second source for exaggeration is due to nothaving a feel for very large numbers—what is $20 million really worth? Since we cannotassess the validity of the result obtained by multiplying 20 000 000 by 50 cents or by 3dollars, the range of apparently reasonable numbers may include both the dealmakerand breaker. Third, project proponents are not likely to look for the other subtle indirect

consequences that go against the project and their small size can help make themunforeseen. We may overlook the fact that 20 million people may also be willing to paya dollar to leave the unaltered wilderness to future generations. It is best to leave thesmall impacts affecting large populations to a subjective and qualitative assessment astheir quantification will tend to exaggerate the proponents’ claims.

Risk and Uncertainty

All possible outcomes of plans cannot be enumerated and the probability of attaining apossible outcome may depend on factors on which we can have no data. If we were ableto identify the possible outcomes and assign, at least, a subjective probability to each,

there would be no technical reason for overstated predictions of consequences. However,errors are made when statements suggesting that ‘such and such’ will be the case ‘if allgoes well’ or ‘if it works as expected’ become accepted as the expected impact of theplan. Not discounting for the possibility that a beneficial outcome may not be achievedleads to an overstatement of benefits. In addition, the benefits are exaggerated when wedo not also discount for the variation in possible outcomes. The exaggeration is due tothe expected value calculation not properly accounting for risk, or for the cost of havingvariability in possible outcomes. In such cases, the exaggeration usually results in anerroneous overstatement of benefits relative to costs. The common error or intention isin ignoring the risks associated with cost overruns (Flyvbjerg et al., 2002, p. 287).




Overstated forecasts can also be used properly to help explain the nature of theinherent uncertainty. ‘Worst–best’ analysis presents exaggerated brackets for the possibleoutcomes to inform decisions. The a fortiori case exaggerates the negative outcomes toshow that they may still be good enough to justify the plan. In these cases, the projectionexaggerates the understatement of the plan’s expected efficacy, or overstates the imped-iments.

Confounding Space and Time

An econometric forecast is typically based on measures of the relationships between aprojected time series and a set of determinants that are thought to be easier to predictor more useful as predictors. For example, it may be useful to make the forecast of aparticular situation conditional on changes in some other factors that can be affected bypolicy. Another example is offered by predictions based on demographic trends thatappear to be more reliable because demographic trends are slow to change. While it mayappear natural to develop predictions by looking only for past trends, the study ofdifferences across space can be useful in some cases. Most survey data is developed for

only one time period. The analyst may have to choose between using this data and usinginferior data that have been gathered at repeated points in time for some other purpose.Data that are repeatedly gathered will in most cases have changed and improved overtime limiting the value of comparisons with past data. Canadian census micro-data forthe more recent years include many more variables and more observations than themicro-data from past censuses and, most likely, they have a smaller error due toundercounting. Many of the relationships that can be studied by using only the mostrecent cross-sectional data cannot be examined by restricting the analysis to the variablescommon to all censuses. Cross-sectional data may allow more control variables andmore degrees of freedom and, thereby, reduce the bias in estimates due to omittedvariables and reduce their variance as a result of the larger number of cases. We may,

therefore, want to examine the relationship between spatial differences in income levels,housing prices and tenure distributions to improve our predictions, for example, of theeffect on home ownership patterns of future changes in housing prices and incomelevels. In doing so, we may make two types of errors that will lead to overstatedforecasts of change. One is due to the fact that the systems are not likely in equilibriumand the other is due to the confusion of age group differences with cohort changes.

Differences across space are often studied and the parameters are used to predictchanges over time. Meteorologists do this all the time. The prediction is based on theacceptance of a causal relation, for example, higher income prospects increase thepropensity of young people to leave home. We may now point to a region with lower

but increasing prospects and say that when its level reaches that of a higher income

region, its household formation rate will be the same as is currently in the higher incomeregion. The relationship describing differences across space will overstate the changesthat may occur over time unless adjustments are made for the time it takes for a factorto produce its full effect, for the market to achieve its equilibrium. Assessing the amountof time is impossible with cross-sectional analysis. Differences across cities and regionsmay have persisted for long periods of time, and the people in the different locationswill have adjusted to them in a number of ways: some economic, some social, culturalor institutional. Using an estimate of a parameter obtained through a cross-sectionalstudy directly in a forecasting model can lead to gross overstatements as illustrated ina study of headship rate changes by Skaburskis (1994).




Myers (1990, 1999) and Pitkin & Myers (1994) show the importance of distinguishing between differences across age groups and differences in an aging cohort. The Mankiw& Weil (1989) forecast of a meltdown in housing prices in the late 1990s due to the ageing

baby boomers was based on a prediction of a change in a cohort’s behaviour based onobservations of differences in the housing consumption of people of different ages.Assuming that current 50-year-olds will, 10 years later, be like the current 60-year-olds

ignores that fact that the histories of the two groups may differ in important ways. Thecurrent 60-year-olds may not have had the same opportunity to buy a home as theiryounger counterparts and are, therefore, more likely to be renters. This does not meanthat current 50-year-olds will give up their homes when they turn 60. The errorsintroduced by the cohort/age confusion can lead to understatements as well as overstate-ments, but the understatements are usually of little interest and we do not hear of them.

Problems with Specification Errors

Errors in estimation method and in the way in which underlying data is used may biasforecasts in ways that are not always detected (Alonso, 1968). Simple trend extrapolations

or more complex regression models using time series may yield biased projections as aresult of erroneous belief in the existence of a deterministic trend. The omitted factors thatform the error term in regressions may be serially auto-correlated leading to biasedestimates that overstate the true relationships. The variables describing the determinantsmay contain errors that are correlated with the outcome variable leading to biases andoverstated predictions. Errors in counting past housing starts, for example, are usuallydue to some starts not being counted and projections of these trends will usuallyunderstate future infrastructure requirements. However, when data gathering methodshave been improving over time then trend extrapolations will exaggerate the changes asthe slope of the trend line will be due to the increase in starts and, in part, to the extraunits found through the improvement in data gathering. In some cases the bias is

knowable and econometric methods have been developed to overcome or to reduce itssize. In other cases the conventional methods for making the corrections may lead towrong conclusions when the analyst does not recognize that corrected estimates show theshort-run consequences rather than the more moderate long- run effects of changes in theexplanatory factors.

Projections will be biased when the excluded factors are determined simultaneouslywith the series of interest and are correlated with their own past values. Seriallycorrelated error terms will lead to overstatements when the correlations are positive andunderstatements when negative (Johnston, 1972, pp. 307–311). Some series may notfollow a deterministic trend at all but may be the result of a random walk sequencethrough time. A set of random events (innovations) may affect the series and be

embedded in all future values. The number of immigrants in a country, for example,follows an upward trend line but the trend is meaningless in itself, as the changes aredetermined each year by policy and by the changing conditions in the source countries.Projections that do not recognize this type of condition3 will be based on spuriouscorrelations and assume a trend when none is present (Chin, 1991). The extrapolation ofthese false trends will predict a change when none is immanent.

Incomplete Models

All forecasts are based on assumptions about the invariants. Often forecasts of one set




of events are based on the belief that another set of events will not change, or changein a way we can predict with greater ease, and that we can specify how the two sets (orseries) relate to each other. Leaving variables or time-series out of the estimation modelsleads to biased estimates when these factors are also related to the series on which thepredictions are based. In practice, factors are always left out due to data constraints, dueto our ignorance, or due to the emergence of new factors that have not played a role in

the past. The problem of omitted variables typically yields estimated coefficients thatoverstate the effect of the series that is being extrapolated and on which the forecast is

based. The overstatement is due to the estimated coefficients including the effects thatare properly attributed to the omitted variables and emerging factors.

Although complex models may lead to error, simple models can be worse. Rentcontrol provides a good first example of the effects of relying on overly simplisticmodels. In the rent control debate, landlords have tended to use any argument to vilifythe policy, just as tenant advocates have done the reverse. However, most of theeconomists who have joined the opposition ranks did not do so out of obviousself-interest but because they held a particular view of housing markets that now isrecognized as having been overly simplistic. The economist’s comparative static models

show that a restriction on price leads to the loss of housing service through increasedrates of depreciation. The increased complexity of the ‘new’ models has increased thenumber of factors that can temper some of the worst consequences. The new modelsshow that moderate rent controls are not as bad as was thought by economists and thatthey can be ‘welfare enhancing’. While these findings have not changed the views ofmost economists, they suggest that we should be careful about relying on the back-of-the-envelope deductions when predicting the behaviour of complex systems.

In a related domain, Anas & Arnott (1993) show that increased subsidies to low-in-come housing suppliers can cause land values to rise and, thereby, increase the rate ofdemolition and replacement construction, which, in turn, reduces the total stock oflow-income housing. Their analysis suggests that the beneficial effects of housing

subsidies are not as great as they were generally thought to be as a result of this newlydiscovered side-effect. Incompleteness of the models, as mentioned earlier, is a source oferror, but attempts to develop complete econometric models to verify complex theoreti-cal propositions may also fail due to specification errors and the usual problemsassociated with the simultaneity of processes and the endogeneity of variables.

In addition, not only are the models depicting trend lines simplistic but they are alsoincomplete in the sense of focusing on one or two of the many strands that form ourhistory. Myers & Kitsuse (2000) explain:

Many different trends occupy the same historical time line. Examples includepopulation, housing, and employment growth trends; changing technology;

fluctuations in financial markets; and the rise and fall of political regimes. Ofcourse these parallel trends are not independent and are clearly linked. Butforecasts often address only a limited set of the possible historical trends,focusing on one part of the future to the exclusion of other factors. (p. 226)

Does incompleteness result in overstatements or does it just introduce random errors?The answer may depend on the subject being considered. In engineering, ignoring anentire line of consequence can have compounding effects that lead to collapse. Socialsystems appear to be more robust as institutions have been developed to stabilize thesystem and policy changes may explicitly aim to defeat the forecast. An example is




offered by US base closures with their predicted devastating consequences. The closuresinitiated a set of new behaviour patterns and had numerous side-effects within munic-ipalities that mitigated the predicted effects. Homeostasis will make simplistic modelsoverstate outcomes.

Ignored Cycles and HomeostasisAt least three factors associated with cycles can lead to erroneous forecasts. The first isdue to our tendency to ignore cyclic behaviour by getting caught up in the current trend,as in the recent technology stock market bubble. During upswings we may deceiveourselves by thinking that the good times are here forever. Another source of error isdue to the changes in the structure of the underlying factors that govern trends. The USDepartment of Housing and Urban Development once had econometric models to helppredict turnarounds in housing market conditions but these were discarded due to theirtendency to give analysts an unwarranted sense of confidence at a time when theconfidence was most unwarranted. Technical analysis may be most heeded at times ofhigh uncertainty, the times when changing structures make the technical forecasts

particularly unreliable. The third source of error may be due to the complex interactionof a series of nested cycles as described by Myers & Kitsuse (2000):

Distinct from the notion of parallel historical trends are the concepts of nestedcycles and embedded life-cycles. These are among the most difficult conceptsrelated to forecasting because they are key components lurking beneath thesurface of the historical time line. Myers and Kitsuse. (p. 226)

Social systems often appear to have a tendency to revert to some long establishedhistorical trend or direction. On the other hand, forecasts of change are often based onan observation of some more enticing, probably new or not fully studied, relationshipamong a set of factors. The forecasts resulting may accurately depict the new trend line

and point to its eventual target. Yet, the deviation is important, and then other factorsmay intervene to correct the change in direction. These factors may now command ourattention and become the subjects of study and the basis for new predictions. Examplesof such changes abound. Rising housing prices would discourage potential young home

buyers with limited incomes. But then rising prices change work behaviour thatincreases income. However, further price increases may make people think that owner-ship is unattainable and discourages work effort and the propensity to save for the downpayment. A downturn in home ownership might now be predicted. But further studyhas shown parents increasing their gifts to children to overcome the down paymenthurdle, and predictions of decline in ownership rates would have been overstatements.At each stage in the evolution of our knowledge of the tenure decisions of young people

we would have been making exaggerated predictions because we could not know aboutthe behavioural changes that compensated for the adverse consequences.In economic relationships, homeostasis is imposed by constraints on resource endow-

ments, technology and income distribution. Over-supply is met by dropping prices andthe movement of factors of production to establish equilibrium between demand andsupply. In other spheres of endeavour, a return to the main trend may be brought about

by deeply held values. The quest for home ownership is so fundamental a force that itovercomes the effect of other factors that would change the trend toward increasinghome ownership rates. Behavioural adjustments may be too complex to predict at theoutset or they may evolve in response to the situation. Deeply held values can make




forecasts of major changes into overstatements. Forecasting war or peace, however, isanother matter.4

Lies, Deceit and Wishful Thinking

Interests can also turn forecasts into lies as in the conclusions developed by the analysisof Flyvjerg et al. (2002). The authors explore alternative reasons for the systematicunderestimation of project costs and dismiss them by pointing to the consistency of theunderestimation across project type and location. Moreover, the errors have been madeover the last 70 years and because the forecasters have not been learning from pastmistakes, the authors conclude that they must be trying to deceive us:

The use of deception and lying as tactics in power struggles aimed at gettingprojects started and at making a profit appear to best explain why costs arehighly and systematically underestimated in transportation infrastructureplanning. (p. 290)

Forecasts may be prepared to deceive for the purpose of gain but we do not personallyknow analysts who would knowingly ‘cook’ the figures to make the forecast suit themaster. The persistence of understatements for 70 years tells not only of the forecasters’inability to learn but also of everyone else’s gullibility. Have the users and promulgatorsand clients of forecasts not been aware of the past record? Why would institutions acceptlies for so long? Did people really think that the Sydney Opera House would come inon budget? Or did we all agree to accept the deception and engage in wishful thinkingin order to make something that we really wanted happen? Is the exaggeration aconfidence trick, or is it made and agreed upon by all of us because we are really saying‘go with it, shoot the arrow, and let the calculations lie’? Or did we simply accept theconjecture that forecasts overstate outcomes and agree to go ahead with the project

anyway because we felt that the unquantifiable benefits were truly large? Our regret, inhindsight, may be that the overstatement was so large, but do Australians really regretthose dramatic sails in the harbour? Or would they have regretted more the decision thatwould most reasonably have been based on a fair prediction of costs?

Conclusions

Forecasts of change are often overstatements and, as dispassionate observers, we mightdiscount the claims and make corrections based on our assessment of the methods, onour knowledge of the forecasters, and, on the magnitude of the predicted change. Unlesswe are the intended users, we may just ignore the forecasts. Sometimes we are induced

to adjust our behaviour and make contingency plans. Before doing this, we usuallysubjectively discount the claims by amounts depending on our own interests and on ourunderstanding of the methods used to develop the forecast. The exaggerations maypresent more clearly the ideal we seek or the consequence we try to avoid, they maymake the planning venture appear worthwhile. They are at times the result of the zealthat is needed to make plans work and sometime they are outright lies to get a projectoff the drawing board. The exaggerated forecast can also be a plea to break from thestatus quo, to avoid just leaving a problem to market processes or to accept the usual orineffective agency response.

Exaggeration in forecasts, as has been seen, may have many causes, from the most




individual and subjective, to the most systemic and uncontrollable. Our assertion is notproven, but we would argue that there is enough evidence to suggest that it is aplausible hypothesis. But if it were so, why would that matter? For planners, surely theappropriate question must be whether it has implications for their behaviour and forpolicy choices. Are there some systematic lessons that might be drawn?

The most obvious point is that these forecasts are not neutral in their impact. The work

of critical theorists and advocates of communicative rationality, such as Forester (1989)or Healey (1996), suggests that information is asymmetric in social relations surroundingplanning. Those with less access to power and resources are likely both to have lessaccess to appropriate information, and to have less control of its creation and dissemi-nation. An effective element of a strategy for increasing transparency and equalizingaccess may well be to focus sharply upon the validity of forecasts and predictions. Thereis a tendency among advocacy groups to disdain careful quantitative analysis as simplyserving established power.5 However, there is also room for counter analysis, of whichthere is probably too little in public planning debates. Of course, this also requiresresources, which may be why more affluent communities are better able to defendthemselves in public argumentation than poor ones. However, the lack of resources can

be addressed if the will is there and if the utility of action is perceived. The question iswhether analysis is seen as potentially power enhancing. Information does conveypower, albeit of a limited form, and it can be used effectively.

A second lesson to be learned is that quantitative analysts and forecasters shouldpractice humility in their trade.6 The litany of error rehearsed above, whether of over orunder exaggeration, may be more than matched in other fields (Hendry & Ericsson, 2001;Klein, 2002), but that is no reason for forecasters in planning to feel comfortable. In truth,we are probably wrong most of the time, and that goes for designers as much as foranalysts. That we are not held to account more often may be more due to lack ofattention and the fact that new times bring new issues. Thus, questioning our resultsshould be an ethical imperative, no matter how uncomfortable or unprofitable it may be.

Forecasters are all too human and should not be expected to be inherently better thananyone else. Nonetheless, the consequences of their work are often so important to thelives of others that they should be held to a high standard. It is not enough to plead thatsomeone else will do it if we do not.

A third reason to think that forecast exaggeration is important concerns the health ofthe field of planning itself. Since the time of Patrick Geddes and ‘Survey before Plan’, itmight be argued that planners have had an implicit ethic that their ideas should begrounded in as accurate a depiction of reality as possible. Yet there has always also beena strong element of ambivalence about that relationship. It may well be that we should‘Make no little plans’, but perhaps we should also ‘Tell no big lies’. We like to think thatthe ethics of planning are such as to leave lying to other professions engaged in urban

development, but it is hard to deny that even if planners do not lie, they do deal inwishful thinking to a remarkable degree. Who has not seen a spurious forecast of transitridership in the service of reducing urban sprawl? Paying more attention to the qualityof our forecasts can keep us honest. But in a world in which the forces opposingplanning are so powerful, can we actually afford that? Perhaps the best that can be doneis to argue that somewhere in the system, someone should have the responsibility forpointing out error.

Finally, while we do not want to make overstatements by mistake or as efforts todeceive, we recognize their importance in motivation and pointing out the relevantaspects of the situations that gives rise to the need for planning. Without overstatements




in some forecasts, there may be little interest in planning. A forecast is not just aprojection laced with judgement, it is a statement, implicit or explicit, as to what shouldor should not happen. A forecast is fiction. The future has not happened. Forecasts canform an integral part of the rhetoric needed to develop and implement plans. They can

be the arrow shot into the future to define and then help direct our common quest. Butin launching that arrow, we must always be aware of where it may land.

Acknowledgements

Andrejs Skaburskis would like to thank the Social Science and Humanities ResearchCouncil for their grant supporting this work. We thank the journal’s referees for theirhelpful comments. All errors and opinions are ours.

Notes

1. Andrew Isserman’s (1984) distinction between projection and forecast and the role forecasts play inplanning as described by Myers & Kitsuse (2000) are recognized. We do not limit our observations to

forecasts that have discounted projections as a result of a judicious review. Forecasts used to promotepositions as well as the hypothetical projections one would make given a level of understanding are alsoconsidered.

2. Newspaper stories, for example, telling of a few huge rent increases may characterize the whole rentalmarket and help gel the support needed for implementing rent control policies. In Ontario, for example,it was the published stories of grandparents being evicted by horrendous rent increases in the early 1970sthat led the provincial Conservative Government to introduce rent controls soon after winning an electionwith the promise of no controls (Nash & Skaburskis, 1997).

3. This is the unit root condition. A time-series follows a random walk when its current value can beregressed on any polynomial of lagged values with a set of estimated parameters that have a root equalto one.

4. Only the system for which the forecast applies need be homeostatic Forecasts are accepted along with atacit recognition of their context. Forecasts are usually seen as assertions of our best belief about the futureand ‘projections’ are defined as the statements about the implication of trends and are, therefore, strictly

qualified by their assumptions. The difference in practice is usually one of explicit as opposed to implicitassumptions. The differences brought about by totally unforeseen consequences are not usually labelledas errors because the forecast is accepted in the context of a schema (Ascher, 1981, p. 256). We accept, atleast tacitly, the context within a forecast is seen to apply. As a result, the outcomes against which aforecast is compared are bound in ways that favour the proposition that they tend to overstate theoutcomes. Big discrepancies may disqualify a forecast as being irrelevant rather than wrong.

5. This was particularly troubling back in the late 1960s and 70s when we started to use computers whoseaccess was very limited and expensive. Planning students at Berkeley who were interested in analysis hada real conscience problem as we realized that our work would tend to support the ‘establishment’ at a timewhen it was bombing Cambodia. Being the ‘guerilla’ in the agency was not a real long-term option whenyou depend on multi-million dollar machines. Our first housing starts forecasts for HUD had to use theLawrence Berkeley Lab’s CDC 7600 computer, one of the most powerful in the country and one thatrequired security clearance to access. It was the computer that Dr Teller was said to be using to simulate

H-bomb blasts. The change in computing power and costs and the access to massive data files have beenliberating and the main constraint now is the interest, time and skills of the analysts.6. Humility may evolve by reflecting on our work and accepting the humiliation stemming from the errors.

It does not come by presenting a ‘humble’ face, or in the words of Hubert Benoit (1955):

Every effort to conquer humility can only result in a false humility in which I again exalt myselfegotistically by means of the idol that I have created for myself. It is strictly impossible for me toabase myself, that is for me to reduce the intensity of my claim to ‘be’. All that I can and should do,if I wish to escape definitively from distress, is to let myself be debased by the evidence … (hisemphasis, p. 228)

Unfortunately, consultants tend not to wait around for the humiliating evidence no matter how beneficialit may be.




References

Alonso, W. (1968) Predicting best with imperfect data, Journal of the American Institute of Planners, 34(4),

pp. 248–255.

Anas, A. & Arnott, R.J. (1993) Technological progress in a model of the housing-land cycle, Journal of Urban

Economics, 34(2), pp. 186–206.

Ascher, W. (1981) Forecasting potential of complex models, Policy Science, 13(3), pp. 247–267.

Ascher, W. (1978) Forecasting: An Appraisal for Policy-Makers and Planners (Baltimore, MD, John Hopkins Press).Benoit, H. (1955) The Supreme Doctrine: Psychological Studies in Zen Thought (Guildford and Kings Lynn, Biddles

Ltd).

Bradshaw, T.K. (1999) Communities not fazed: why military base closures may not be catastrophic, Journal of

the American Planning Association, 65(2), pp. 193–206.

Chin, M.D. (1991) Beware of econometricians bearing estimates: policy analysis in a ‘unit root’ world, Journal

of Policy Analysis and Management, 10, pp. 546–567.

Engelhardt, G.V. (1994) House prices and the decision to save for down payments, Journal of Urban Economics,

36, pp. 209–237.

Flyvbjerg, B., Holm, M.S. & Buhl, S. (2002) Underestimating costs in public works projects: error or lie? Journal

of the American Planning Association, 68(3), pp. 279–293.

Forester, J. (1989) Planning in the Face of Power (Berkeley, CA, University of California Press).

Healey, P. (1996) The communicative turn in planning theory and its implications for spatial strategy making,

Environment and Planning B: Planning and Design, 23(2), pp. 217–234.

Hendry, D.F. & Ericsson, N.R. (Eds) (2001) Understanding Economic Forecasts (Cambridge and London, MIT

Press).

Isserman, A. (1984) Projections, forecasts and plans: on the future of population forecasting, Journal of the

American Planning Association, 50(2), pp. 208–221.

Johnston, J. (1972) Econometric Methods (New York, McGraw-Hill).

Keyfitz, N. (1982) Can knowledge improve forecasts? Population and Development Review, 8(4), pp. 729–751.

Klein, L.R. (2002) The accuracy of economic prediction, in: P.K. Puttaswamaiah (Ed.) Paul Samuelson and the

Foundations of Modern Economics (New Brunswick, NJ and London, Transaction).

Mankiw, N.G. & Weil, D.M. (1989) The baby boom, the baby bust, and the housing market, Regional Science

and urban Economics, 10, pp. 139–162.

Myers, D. (1990) Housing Demography: Linking Demographic Structure and Housing Markets (Madison, University

of Wisconsin Press).

Myers, D. (1999) Cohort longitudinal estimation of housing careers, Housing Studies, 14(4), pp. 473–490.

Myers, D. & Kitsuse, A. (2000) Constructing the future in planning: a survey of theories and tools, Journal of

Planning Education and Research, 12(3), pp. 221–232.

Nash, C. & Skaburskis, A. (1998) Toronto’s changing rent control policy, in: W.D. Keating, M.B. Teitz & A.

Skaburskis Rent Control: Regulation and the Rental Housing Market, Chap. 12, pp. 169–192 (Rutgers, NJ, Center

for Urban Policy Research).

Pitkin, J.R. & Myers, D. (1994) The specification of demographic effects on housing demand: avoiding the

age-cohort fallacy, Journal of Housing Economics, 3(3), pp. 240–250.

Skaburskis, A. (1994) Determinants of Canadian headship rates, Urban Studies, 31(8), pp. 1377–1389.

Skaburskis, A. (1995) Resisting the allure of large projection models, Journal of Planning Education and Research,

14(3), pp. 191–202.Stoto, M.A. (1983) The accuracy of population projections, Journal of the American Statistical Association, 78(381),

pp. 13–20.Wachs, M. (1982) Ethical dilemmas in forecasting for public policy, Public Administration Review, 42(6),

pp. 562–567.Wachs, M. (1986) Technique vs. advocacy in forecasting: a study of rail rapid transit, Urban Resources, 4(1),

pp. 23–30.Wachs, M. (1989) When planners lie with numbers, Journal of the American Planning Association, 55(1),

pp. 476–479.Wachs, M. (1990) Ethics and advocacy in forecasting for public policy, Business and Professional Ethics Journal,

9(1–2), pp. 141–157.

forecasts and outcomes, ska bur skis, a., & teitz, m. b. (2003)

Documents