Evaluating density combinations– Forecasting Norwegian GDP in real-time
Knut Are Aastveit Karsten R. Gerdrup Anne Sofie JoreChristie Smith Leif Anders Thorsrud
Presentation at the 6th Colloquium on modern tools forbusiness cycle analysis: “The lessons from global economiccrisis”, 26 - 29 September 2010
Context - Norges Bank’s system of averaging models (SAM)
1. Norges Bank instituted a project in 2006 to improve itsshort-term forecasts through model combination
2. Bjørnland et al. (2009) show that model combination improveforecasts from individual models, and that model combinationout-performs Norges Bank’s own short-term forecasts forinflation
3. SAM is used in the monetary policy process to provide theBank with model-based forecasts for GDP and inflation
4. SAM until September 2009:I Selection of the 8 best models for each horizon based on quasi
out-of-sample point forecast performance (RMSFE)I Forecasts are combined in a linear pool using univariate,
horizon-specific weights based on mean squared errors
5. SAM from September 2009:I Simplify the model space by grouping similar models together.I Two step combination procedure.
Economics Department/Nowcasting Team
Motivation
I Forecast combinations may produce better forecasts thanselecting, ex ante, a single model (portfolio diversification,unknown instabilities, and biases in individual models, seeTimmermann (2006))
I A number of studies have found that forecast combinationusing time-varying recursive weights, based on historicalforecast performance, is an ineffective strategy for improvingpoint forecast accuracy, see among other Stock and Watson(2004) and Clark and McCracken (2010)
Economics Department/Nowcasting Team
Motivation
I Density forecasts provide policy-makers with a full impressionof forecast uncertainty
I Mitchell and Hall (2005) and Hall and Mitchell (2007) providea framework and justification for density combination
I Point forecasts are better seen as central points of ranges ofuncertainty
I Jore, Mitchell and Vahey (2010) examine density forecastsand conclude that adaptive weights improve simple weights
I Bache et al (2009) combines VAR and DSGE predictivedensities and Amisano and Geweke (2010) combines VAR,DSGE and Factor model predictive densities.
Economics Department/Nowcasting Team
Contribution of this paper
What we do
I We produce combined density forecasts for NorwegianMainland-GDP from a system of different model classes.
I We compare our forecast combination approach with variouscombinations procedures.
I We use Norwegian real-time data.
ResultsI We show that both the log-score for the predictive densities
and the RMSE from our approach outperforms alternativestrategies:
I Selecting the best model ex-anteI Give all models equal weights
Economics Department/Nowcasting Team
Outline
1. Modeling framework and data
2. Empirical exercise
3. Preliminary results
4. Concluding remarks
Economics Department/Nowcasting Team
Forecasts
I We adopt an “expert” combination approach
I We define i = 1, . . . , N experts, where each expert producesone of the N density forecasts
I A decision maker combines the densities from two or moreexperts based on the fit of the density forecasts over theevaluation period
Economics Department/Nowcasting Team
Forecast density combination
I The decision maker constructs the combined densities by alinear opinion pool method
p(Yτ,h) =N∑i=1
wi,τ,h g(Yτ,h | Ii,τ ), τ = τ , . . . , τ ,
where g(Yτ,h | Ii,τ ) are the h-step ahead forecast densitiesfrom individual model i, i = 1, . . . , N of a random variableY τ , (with realization yτ ), conditional on the information setIτ .
I wi,τ,h are a set of non-negative weights that sum to unity.
Economics Department/Nowcasting Team
Choosing the weights
I Recursive logarithmic score weights (RLSW)
I We use the log score to measure the fit of the experts’densities through the evaluation period
wi,τ,h =exp
[∑τ−hτ ln g(yτ,h | Ii,τ )
]∑N
i=1 exp[∑τ−h
τ ln g(yτ,h | Ii,τ )] , τ = τ , . . . , τ
I Similarities with Bayesian approachI Approximate predictive likelihood approachI Bayesian Model Averaging
Economics Department/Nowcasting Team
Two-stage modeling combination approach
Step 1
I Group all models into different model classes
I Models are combined within each model class using a linearopinion pool and recursive logarithmic score-based weights.
I Calculate the predictive densities for each model class
Step 2
I Combine the predictive densities from each model class intoone single combined density
I The different model classes are combined usingI Equal weightsI Recursive logarithmic score-based weights and linear opinion
poolI Optimal weights, see Hall and Mitchell (2007)
Economics Department/Nowcasting Team
Model Classes
I VARs - Univariate and Vector AutoregressionsI Univariate, bi-variate and tri-variate autoregressionI 4 different lag lengths, 3 different transformations and 3
different estimation periodsI Number of models: 144
I SURV - Leading indicator and survey modelsI Bi-variate VARs with surveys and GDP growthI Business tendency surveys, Consumer confidence surveys,
Regional network surveysI Number of models: 46
I FM - Dynamic factor modelsI Dynamic factor models using monthly informationI 4 different factor combinations, 4 different lag lengthsI Number of models: 16
Economics Department/Nowcasting Team
Empirical exercise
I Our recursive forecasting exercise is intended to mimic thebehavior of a policymaker forecasting in real-time.
I We use real-time vintage data for the Norwegian economy forall forecasts and realizations.
I Forecasts are performed at the last day of the quarter.
I Evaluation period is 2001q2 to 2010q1.
I We use both 5th release of GDP and the last available datavintage of GDP as “final” data for the evaluation.
I Compare the logarithmic score and RMSFE for 3 differentversions of the two-stage combination approach with
I Selection: Select the ex ante, best model measured bylogarithmic score.
I Equal weights: Pool all models together and give them equalweights.
Economics Department/Nowcasting Team
Norwegian Mainland-GDP. Quarterly growth. Percent
Economics Department/Nowcasting Team
Average recursive logarithmic scores - final vintage
(a) All Models (b) Model Classes
Economics Department/Nowcasting Team
Average recursive logarithmic scores - 5th release vintage
(c) All Models (d) Model Classes
Economics Department/Nowcasting Team
Recursive logarithmic scores ensembles - final vintage
Economics Department/Nowcasting Team
Average recursive logarithmic scores - final vintage
Economics Department/Nowcasting Team
Weights - Logarithmic Score - final vintage
Economics Department/Nowcasting Team
Weights - Logarithmic Score - 5th release
Economics Department/Nowcasting Team
PITS ensemble combination
Economics Department/Nowcasting Team
Concluding remarks
What are the advantages of the two stage densitycombination approach?
I Performs well when forecasting GDP Mainland-Norway, bothin terms of point and density forecasts.
I The clustering facilitates story-telling – linking forecasts toparticular data
I Make it computationally possible to calculate optimal weightsI Ongoing extensions:
I Explore the importance of new data releases in real-time forNorwegian and U.S. data
I Investigate the importance of data revisions
Economics Department/Nowcasting Team
Fanchart published after interest rate meeting
Economics Department/Nowcasting Team
Density forecasts and outcomes
Economics Department/Nowcasting Team