comparing forecast accuracy of different models for prices...

May 29, 2012

DETM/DIRI

The Economics and Econometrics of

Commodity Prices – 2012

João Victor Issler (FGV) and Claudia F. Rodrigues (VALE)

August 17, 2012

Comparing Forecast Accuracy

of Different Models for Prices

of Metal Commodities

Forecasting within a panel data framework

■ There are many ways to combine forecasts:

1. bias-corrected average forecast

2. simple average forecast

3. weighted average forecast based on MSE

■ Our aim:

evaluate these forecast combinations according to their predictability performance measured by the RMSE.

compare the predictability of: forecast combinations x “best-model” forecast (BIC).

■ According to the errors decomposition formula:

fhi,t = yt + κi + εi,t + ηt

■ i indicates a model; i = 1,..., N

■ fhi,t stands for forecasts of yt computed using conditioning sets

lagged h periods

■ κi time-invariant bias

■ εi,t idiosyncratic risk

■ ηt aggregate shock

Analyzing the forecast errors

ηt

κi Bias Idiosyncratic shock

εi,t


Aggregate shock

ηt


εi,t


Aggregate shock

Shocks that affect all forecasts

Errors of idiosyncratic nature Model misspecification or unknown risk function

ηt


εi,t


Aggregate shock

Shocks that affect all forecasts

Model misspecification or unknown risk function

Errors of idiosyncratic nature

Combine forecasts Estimate the bias

Cannot avoid it

The series

■ Target variable

■ the metals price index¹ (IMF)

■ Co-variates used in forecasting

■ US industrial production

■ Chinese industrial production

■ Primary metals coincident index (USGS²)

■ Leading index of metals price (USGS²)

■ VIX – a volatility index

■ US real effective exchange rate

■ S&P500

¹ an aggregation of weighted average of individual indices: copper, aluminum, iron ore, nickel, tin, zinc, lead and uranium. ² US Geological survey

The series

■ Correlations between the target series and each covariate in dlog:

US IP China IP Coincident

index

Leading

index VIX REER S&P500

Lag 1 0.13 0.26 0.16 0.25 -0.17 -0.15 0.11

■ Time span

■ 01/01/1980 to 01/05/2012

■ Monthly basis

■ T = 389

Models

■ AR and Garch

■ Only one co-variate at a time

■ VAR

■ Combination of covariates: C7,1 , C7,2 , C7,3 , C7,4

■ Distinct functional forms

■ level, log, dlog

■ Total number of models and hence forecasts in each time period

■ N = 318 – this is the number of forecasts in each cross section of the panel

40

80

120

160

200

240

280

1980 1984 1988 1992 1996 2000 2004 2008 2012

ind

ex

Sources: IMF and Vale

The metals price index from 1980 to 2012

Metals price index Real terms

40

80

120

160

200

240

280

1980 1984 1988 1992 1996 2000 2004 2008 2012

ind

ex


Estimation period Forecast period

Bias window


T1 T2

The bias-corrected forecast: learning about the model

40

80

120

160

200

240

280

1980 1984 1988 1992 1996 2000 2004 2008 2012

ind

ex



Bias window

T1+1 T2+1



40

80

120

160

200

240

280

1980 1984 1988 1992 1996 2000 2004 2008 2012

ind

ex



Bias window

T-h-R



40

80

120

160

200

240

280

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

In

dex


Bias-corrected 1-step ahead forecast

Metals price index

Real terms

0

40

80

120

160

200

240

280

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

In

dex



Bias-corrected 6-steps ahead forecast

Distribution of the bias

h=1 h=2 h=3

h=4 h=5 h=6

Bias

■ Conley significance test

Bias T-statistic P-value

1-step -0.92 -0.57 0.28

3-steps -2.93 -0.62 0.26

6-steps -6.08 -0.69 0.24

9-steps -9.54 -0.76 0.22

12-steps -12.94 -0.82 0.20

15-steps -16.25 -0.86 0.19

40

120

200

280

2004 2005 2006 2007 2008 2009 2010 2011 2012

BCAF - 1-step ahead

BCAF - 6 steps ahead

Metals price index

The outcome of BCAF


40

80

120

160

200

240

280

1980 1984 1988 1992 1996 2000 2004 2008 2012

ind

ex


Forecasting


309 obs 80 obs


T1

40

80

120

160

200

240

280

1980 1984 1988 1992 1996 2000 2004 2008 2012

ind

ex


Forecasting one-step ahead


309 obs 80 obs


T1




310 obs 79 obs


40

80

120

160

200

240

280

1980 1984 1988 1992 1996 2000 2004 2008 2012

ind

ex

40

80

120

160

200

240

280

1980 1984 1988 1992 1996 2000 2004 2008 2012

ind

ex



310 obs 79 obs



40

80

120

160

200

240

280

1980 1984 1988 1992 1996 2000 2004 2008 2012

ind

ex



388 obs 1 obs



40

80

120

160

200

240

280

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

In

dex


The outcome of forecasting N models 1-step ahead


40

80

120

160

200

240

280

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

In

dex


Comparison of 1-step ahead forecasts

Metals price index

Real terms

40

80

120

160

200

240

280

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

In

dex

Bias-corrected forecasts Forecasts

50

100

150

200

250

300

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

In

dex



The outcome of forecasting N models 6-steps ahead

0

40

80

120

160

200

240

280

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

In

dex


Forecasts 6-steps ahead

Metals price index

Real terms

0

40

80

120

160

200

240

280

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

In

dex

Forecasts Bias-corrected forecasts

40

80

120

160

200

240

280

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

ind

ex

Metals price index

AF 1-step ahead

AF 6-steps ahead


Average forecast 1-step ahead and 6-steps ahead


The weighted average forecast

■ The inverse of the MSE can be used to weight forecasts

■ The higher the error the lower the weight of the forecast

Wi = 1/MSEi

Σ 1/MSEi

40

120

200

280

2004 2005 2006 2007 2008 2009 2010 2011 2012

BCAF

WAF

AF

Metals price index

The outcome of distinct combined forecasts 1-step ahead


40

120

200

280

2004 2005 2006 2007 2008 2009 2010 2011 2012

BCAF

WAF

AF

Metals price index

The outcome of distinct combined forecasts 6-steps ahead


Bias

corrected

average

Average

forecast

Weighted

average

(MSE)

Best model

BIC

1-step ahead 11.3 11.0 11.4 11.6*

3-steps ahead 23.7 23.3 24.1 25.3*

6-steps ahead 36.6 37.3 38.6 43.0*

9-steps ahead 44.9 47.4 50.2 57.2*

12-steps ahead 49.7 53.5 59.1 67.7*

15-steps ahead 52.6 58.3 66.6 76.6*

The RMSE of different forecasts

(*) Applying the Diebold-Mariano test , we could not reject the hypothesis that the errors of BIC forecast are different from the errors of the best forecast combination, at 10% level.

Conclusions

■ Combining forecasts proved to be better for predicting the metals price index than any individual model

■ Among the combinations the “Average forecast” performed better in short horizons, while “Bias corrected forecast” outperformed for long horizons

Thank you!

comparing forecast accuracy of different models for prices...

Documents