working papers du statec - gouvernement luxembourgeois · for the main relationships whether the...

Les articles publiés dans la série "Économie et statistiques" n'engagent que leurs auteurs. Ils ne reflètent pas forcément les vues du STATEC et n'engagent en rien sa responsabilité.

99 Economie et Statistiques Working papers du STATEC

mars 2018

Auteur: Massimiliano Marcellino (Bocconi University)

An evaluation of the short-term forecasting properties of Modux

Abstract

The recent financial crisis has highlighted the difficulty of making reliable forecasts of economic and financial variables. Forecast errors are unavoidable since economic indicators are random variables. However, part of the errors are due to incorrectly specified models and/or assumptions. Hence, it is important to evaluate the properties of a given set of forecasts, trying to understand how reliable they are and to what extent they can be improved.

In this report we present an evaluation of the forecasts produced by Modux, a yearly macroeconometric model for Luxembourg including about 500 variables, with about 100 econometric equations for the key economic indicators, developed by Statec1.

1 I am grateful to Ferdy Adam and Michel Geller for documentation and many useful conversations about Modux, and for help with the data and the preparation of the empirical analysis for this report.

Economie et Statistiques Working papers du STATEC N° 99 mars 2018

3

.

1. Introduction

The recent financial crisis has highlighted the difficulty of making reliable forecasts of economic

and financial variables. Forecast errors are unavoidable since economic indicators are random

variables. However, part of the errors are due to incorrectly specified models and/or

assumptions. Hence, it is important to evaluate the properties of a given set of forecasts, trying

to understand how reliable they are and to what extent they can be improved.

In this report we present an evaluation of the forecasts produced by Modux, a yearly

macroeconometric model for Luxembourg including about 500 variables, with about 100

econometric equations for the key economic indicators, developed by Statec.

The model equations are mostly in error correction form, specified according to economic

theory and institutional considerations. They can be grouped into five main blocks:

determination of GDP, production factors (employment and capital stock), prices and wages,

population and unemployment, and public sector.

The structure of the model and the equation specification is rather standard, with some

specificities related to the particular characteristics of the Luxembourg economy, such as the

high degree of openness, the relevant role of the financial sector and the presence of a large

number of non resident and cross-border workers.

The model parameters are estimated by OLS equation by equation, using a two-step

procedure, where first the cointegrating (long-run) parameters are estimated and then the

resulting error correction terms are inserted in the equations for the variables in first differences

(which makes them stationary). The model is estimated using yearly data, with varying samples

for each equation, depending on data availability.

The model is used for:

• short term forecasting (1-2 year horizon)

• medium term forecasting (3-4 year horizon)

• evaluation of the effects of economic policy shocks and other structural measures


4

.

In the evaluation, we have first considered the in-sample properties of the main Modux

equations (Section 2). Then we have moved to the forecast evaluation, using the latest version

of the model (Section 3). Next, we have considered a real time evaluation, based on the actual

published Statec forecasts, focusing on GDP growth as a key economic indicator (Section 4).

Finally, we have summarized the main results and provided suggestions for further

developments (Section 5). An Appendix provides detailed results on the evaluation of the main

Modux equations.

2. In-sample evaluation of Modux

The auditing focused on three main aspects: first, the use as explanatory variables of

reasonable economic indicators based on economic theory and institutional considerations;

second, the statistical properties of the estimated equations (in particular the dynamic

specification); third, the effects of the crisis. With reference to the last point, we have assessed

for the main relationships whether the parameters of the long run and short run equations

remained relatively stable.

Overall, no major problems emerged from this in-sample evaluation: the explanatory variables

are carefully selected and the estimated coefficients in the short and long run relationships

have reasonable sign and size; residuals of most equations pass conventional diagnostic

checks for no serial correlation, homoscedasticity and normality; the model parameters are

rather stable or, when not, proper dummy variables are inserted in the specification to capture

the effects of structural breaks.

Additional details on this in-sample evaluation are presented in Appendix A.

In summary, the specification of most Modux equations is satisfactory, and we can move to

evaluate their forecasting performance.


5

.

3. Forecast evaluation of Modux, static simulation

We have focused on the evaluation of 1-step (1-year) ahead forecasts, due to the yearly

frequency and short sample available (typically, 1995-2016), and we have fixed the parameters

to the full sample values, again due to the short sample, and treated the future values of the

exogenous variables as known (static simulation).

We have first computed a set of relevant descriptive statistics on the resulting forecast errors.

The statistics include the mean error, the mean squared error and the mean absolute error

(MSE and MAE, both measures avoid compensation of positive and negative errors). We have

then graphed, for each variable, the actual values and the forecasts, and the forecast errors.

The previous analysis provides information on the absolute quality of the forecast. However, it

is also important to consider the relative performance with respect to standard time series

models. For this, the Modux forecasts for the various variables have been also compared with

the forecasts resulting from autoregressive models with two lags (AR(2)), generally a good

benchmark for macro variables, on the basis of the mean absolute or mean squared forecast

errors (MAE and MSFE, respectively).

Finally, as the evaluation sample is rather long and the crisis could have caused some

permanent shifts in economic relationships, we have also compared the MSE and MAE before

and after 2008, to understand whether the forecasting performance has remained stable or

improved/deteriorated.

The main results can be summarized as follows. First, for most variables the forecasts track

well the actual values and the forecast errors show no statistically significant bias. It should be

stressed that the actual forecasting performance of Modux could be worse, as the model

coefficients should be recursively estimated, the future values of the exogenous variables

should be forecasted, and the dependent variables themselves can be subject to revisions (all

features we cannot take into account due to the short sample available and lack of information

on the actual vintages used in each forecast round). In this sense, a positive outcome of the

static evaluation we have conducted is necessary but not sufficient for a good actual

forecasting performance, but at least this necessary condition is clearly satisfied.


6

.

Second, with respect to the AR benchmark, most Modux equations produce better forecasts in

terms of MSE and MAE.1 This provides further evidence on the proper dynamic specification of

the equations and on the relevance of the used explanatory variables.

Finally, the forecasting performance of most Modux equations after the crisis has improved,

particularly so for the demand components. This positive pattern is partly due to the ri-

specification of some of the Modux equation in the recent period

Detailed results on the forecast evaluation are presented in Appendix B, with a list of variables

in Appendix D and some illustrative figures in Appendix E.

4. An evaluation of historic forecasts

As a final step, the forecast evaluation is conducted in a fully real time context, based on the

forecasts that were actually published by Statec in the second part of each year for the

following year. We focus on GDP growth, as it is a key economic indicator and we could gather

data for a relatively long period, 1995-2015.

A first question is whether the forecasts were reliable in absolute terms. To answer this

question, in Figure 1 we plot the forecast errors in percentage points (pp) and their absolute

values. The largest episodes of over-prediction were in 2001 and 2008-09, not surprisingly,

while there were also some periods of noticeable under-prediction, in 1998, 1999 and 2000.

Overall, the average forecast error is 0.38pp (not statistically different from zero), and the mean

absolute forecast error is 2.07pp where, for comparison, the average growth rate over this

period is 3.81.

1 As the evaluation sample is short, we do not consider formal statistical tests of equal forecasting performance.


7

.

Figure 1: Actual GDP growth forecast errors (ERR_1_A) and absolute forecast errors

(ABS_ERR_1_A), in percentage points.

A second question is whether these forecast errors were somewhat predictable. It turns out

that a composite financial indicator (made up of variables such as stock price indexes and

bond spreads), building permits, and a combination of various surveys do have some

explanatory power for the forecast errors. These are all timely indicators often used in

nowcasting models, but not so much in theory based econometric models, as it is in general

rather difficult to justify them from high level economic theory. This result suggests that the

forecasts could be further improved, either by including some of the indicators in the model

underlying the forecasts, or by combining the model based forecasts with those resulting from

simple nowcasting models relying on these indicators.

Additional results are presented in Appendix C.

-6-4-20246

0 1 2 3 4 5 6

96 98 00 02 04 06 08 10 12 14

ERR_1_A ABS_ERR_1_A


8

.

5. Conclusions

This assessment of Modux is overall quite positive. The model is in general well specified and

produces reliable short term forecasts (in particular given how difficult it is to forecast macro

and financial variables in small and very open economies).

Among the possible extensions at the specification level, I would suggest to evaluate in more

details the role of sectorial and financial data, and of forward looking variables (such as

surveys, profit expectations, inflation expectations, etc).

At the estimation stage, it would be interesting to estimate jointly at least subsets of the model

equations, to see whether this improves estimation efficiency and to permit a joint dynamic

solution of the subset of equations. IV-type estimation could be also evaluated for equations

conditioning on possibly endogenous variables.

The short term forecasts could be also compared with other internal and external forecasts

(e.g., those by the BCL or DG ECCFIN). Methods for the construction of interval /density

forecasts based on bootstrapping could be assessed. And longer term forecasts could be also

evaluated, though for the moment the available short sample makes this difficult for most of the

modelled variables.

Conditional on the availability of time and resources, the development of a complementary

quarterly version of Modux could be a sensible strategy, even though the extent of the data

revisions and other possible measurement errors make this task very difficult for Luxembourg.

One option could be to develop as a starting point a very simplified version based on only a

small subset of the variables currently modelled in Modux, possibly integrated with the indicator

models. Ideally, this would require an additional dedicated person within Statec, plus external

specialized help in the first phase of the project (model specification, testing and evaluation).

Finally, I should mention that a continuous monitoring of the Modux performance is needed for

a reliable use, as for any other model. Hopefully, the steps that we have followed in the audit

will be also helpful for future periodic evaluations and modifications of Modux.


9

.

Appendix A: In-sample results

Variables Variable names AC HC NOR

cfin_r Cons. men. residents (vol.) 0.14 0.99 0.73xcomm_r Exp. serv. fin. (mia eur, vol.) 0.25 0.93 0.88xso_r Exp serv. n.-fin. (mia eur, vol.) 0.99 0.98 0.75xbmcd_r Exportations de biens, merchanting 0.16 0.57 0.14xbhormcd_r Exportations de biens, hors or et merchanting 0.35 0.82 0.84mbe_r Imp. biens energ. (vol. mia eur) 0.90 0.81 0.83mbo_r Imp. biens aut. (vol., mia eur) 0.18 0.96 0.07p_cfinhtva Prix cons. fin. men. HTVA (2005=1) 0.68 0.76 0.73

lbnq Emploi effectif sec. fin. 0.12 0.92 0.65lprvo Emploi effectif sec. priv. hors fin. 0.22 0.71 0.90p_vabbnq Prix VAB sec. fin. (2005=1) 0.31 0.97 0.73p_vabprvo Prix VAB priv. n.-fin. (2005=1) 0.04 0.95 0.88

1995-2016

GDP demand side

GDP supply side


stoxx50 Indice boursier zone Euro Stoxx50 (2001=100) 0.55 0.98 0.63msw_r Demande mondiale services (vol.) 0.19 0.01 0.40mbw_r Demande mondiale biens (vol.) 0.23 0.65 0.83

rcsnmprop Revenus de la propriete, SNM, (mia Eur) 0.15 0.99 0.61livpet Livraisons de produits pétroliers (L) 0.10 1.00 0.58imrwpmav Imp. sur les bénéfices des ménages, avances 0.54 0.85 0.86imrwpmsol Impots sol. ent. (mia eur) 0.28 0.91 0.67impmdivabo Imp. p/i taxe abo. (mia eur) 0.92 0.41 0.63

salmbnqhhce CSM sect. fin. heure hors cot. emp. (1000 Eur) 0.53 0.54 0.70salmprvohhce CSM heure priv. n.-fin. hors cot. emp. (1000 Eur) 0.02 0.82 0.80migrino Flux migratoires entrants (autres pays) (1000 pers.) 0.54 0.76 0.59frin Frontaliers entrants (1000 pers.) 0.01 0.47 0.08

1995-2016

International

Public income

Population


10

.

Note: The table reports for key variables (defined in the table in Appendix D) p-values from specification tests:

AC = Autocorrelation, Lagrange Multiplier Test

HC = Heteroscedasticity, White Test

NOR = Normality, Jarque-Bera Test

Appendix B: Out-of-sample results


capbmeq_r Stock cap. brut mach. & equip. (vol.) 1.00 0.85 0.77capbres_r Stock cap. brut res. (vol.) 0.23 0.93 0.93lprvo Emploi effectif sec. priv. hors fin. 0.17 0.81 0.95p_vabprvo Prix VAB priv. n.-fin. (2005=1) 0.07 0.93 0.87risknfc Prime de risque, credits aux entr. non fin. 0.12 0.95 0.58riskmen Prime de risque, credits hypothécaires 0.50 0.97 0.56crednfc Credit to non financial corporation 0.36 0.72 0.62credresmen Credit to residential households 0.17 0.50 0.98cfin_r Cons. men. residents (vol.) 0.79 0.95 0.48

1995-2016

Financial accerator related Variables

Variables Variable names ME BIAS (pval) MAE MSE MAE scaled to AR(2) MSE scaled to AR(2) MAE MSE

cfin_r Cons. men. residents (vol.) 0.03 0.77 0.42 0.30 0.48 0.22 1.10 1.25xcomm_r Exp. serv. fin. (mia eur, vol.) 0.42 0.13 0.95 1.35 0.20 0.05 1.42 1.80xso_r Exp serv. n.-fin. (mia eur, vol.) 0.86 0.27 2.96 3.72 0.51 0.22 0.65 0.45xbmcd_r Exportations de biens, merchanting 6.24 0.19 16.30 21.17 0.64 0.45 2.70 7.01xbhormcd_r Exportations de biens, hors or et merchanting 0.37 0.55 2.22 2.91 0.37 0.12 1.16 1.53mbe_r Imp. biens energ. (vol. mia eur) 0.25 0.74 2.72 3.65 0.52 0.28 1.44 2.59mbo_r Imp. biens aut. (vol., mia eur) 0.29 0.52 1.28 2.00 0.23 0.08 2.59 6.46p_cfinhtva Prix cons. fin. men. HTVA (2005=1) 0.02 0.82 0.35 0.45 0.32 0.10 1.47 1.48

lbnq Emploi effectif sec. fin. -0.27 0.55 1.84 2.15 0.69 0.42 1.20 1.29lprvo Emploi effectif sec. priv. hors fin. 0.07 0.55 0.44 0.54 0.37 0.11 0.68 0.58p_vabbnq Prix VAB sec. fin. (2005=1) 0.66 0.67 5.76 7.30 0.84 0.53 1.41 1.86p_vabprvo Prix VAB priv. n.-fin. (2005=1) 0.03 0.87 0.64 0.97 0.64 0.41 0.80 1.21

1995-2016 pre crisis / post-crisis

GDP demand side

GDP supply side


11

.

Note: The table reports for key variables (defined in the table in Appendix D) statistics related to the associated forecast errors:

ME = Mean Error

MAE = Mean Absolute Error

MSE = Mean Squared Error

MAE scaled to AR(2) = MAE of equation scaled to MAE of an in-sample AR(2)

MSE scaled to AR(2) = MSE of equation scaled to MSE of an in-sample AR(2)

Pre-crisis/post-crisis = Relation of the coefficient in time horizon 1995-2007 to 2008-2016


stoxx50 Indice boursier zone Euro Stoxx50 (2001=100) 2.86 0.20 8.76 10.67 0.65 0.42 0.97 1.15msw_r Demande mondiale services (vol.) 0.21 0.41 0.95 1.22 0.39 0.16 2.49 3.96mbw_r Demande mondiale biens (vol.) 0.41 0.07 0.91 1.11 0.23 0.04 1.28 1.52

rcsnmprop Revenus de la propriete, SNM, (mia Eur) 1.10 0.09 2.27 3.19 0.24 0.05 0.57 0.55livpet Livraisons de produits pétroliers (L) 0.22 0.44 1.04 1.30 0.23 0.05 0.78 0.87imrwpmav Imp. sur les bénéfices des ménages, avances 0.70 0.33 2.62 3.37 0.39 0.08 0.97 1.15imrwpmsol Impots sol. ent. (mia eur) 2.89 0.29 8.05 13.17 0.64 0.47 0.58 1.18impmdivabo Imp. p/i taxe abo. (mia eur) 1.20 0.28 4.37 5.40 0.52 0.23 1.00 1.09

salmbnqhhce CSM sect. fin. heure hors cot. emp. (1000 Eur) 1.73 0.00 1.88 2.32 0.68 0.62 1.26 1.43salmprvohhce CSM heure priv. n.-fin. hors cot. emp. (1000 Eur) -0.03 0.89 0.70 0.94 0.67 0.58 1.50 1.72migrino Flux migratoires entrants (autres pays) (1000 pers.) -0.52 0.61 3.84 4.90 0.69 0.46 1.53 1.83frin Frontaliers entrants (1000 pers.) 0.23 0.20 0.66 0.88 0.52 0.23 0.99 1.74


International

Public income

Population


capbmeq_r Stock cap. brut mach. & equip. (vol.) 0.11 0.42 0.40 0.27 0.09 0.01 1.41 1.78capbres_r Stock cap. brut res. (vol.) 0.02 0.32 0.08 0.01 0.04 0.00 0.71 0.49lprvo Emploi effectif sec. priv. hors fin. 0.08 0.41 0.35 0.19 0.12 0.02 1.05 1.24p_vabprvo Prix VAB priv. n.-fin. (2005=1) 0.03 0.88 0.65 1.02 0.36 0.15 0.59 0.61risknfc Prime de risque, credits aux entr. non fin. 0.00 1.00 0.16 0.04 0.74 0.40 0.70 0.53riskmen Prime de risque, credits hypothécaires 0.00 1.00 0.14 0.03 0.47 0.21 0.92 0.87crednfc Crédits aux sociétés non financieres 1.01 0.47 4.53 31.39 0.56 0.30 1.47 1.59credresmen Crédits aux ménages 0.48 0.24 1.25 3.86 0.41 0.22 1.23 3.61cfin_r Cons. men. residents (vol.) 0.03 0.76 0.40 0.27 0.17 0.04 0.90 0.71


Financial accerator related Variables


12

.

Appendix C: Explaining the historic forecast errors

Note: Regression of historic forecast errors on composite financial, housing and survey indexes

(plus intercept and 1999 dummy).

Note: Actual, fit and residuals from regression of historic forecast errors on composite financial,

housing and survey indexes.

Dependent Variable: ERR_1_AMethod: Least SquaresDate: 13/09/17 Time: 11:18Sample (adjusted): 1996 2015Included observations: 20 after adjustments

Variable Coefficient Std. Error t-Statistic Prob.

C 0.611034 0.443032 1.379210 0.1880AV_Q_D_FININDEX_A_L 5.468493 3.671028 1.489635 0.1570AV_A_D_HOUSING_A_L 1.167951 0.315697 3.699594 0.0021AV_Q_SURVEYDEXN... -1.156376 0.631649 -1.830726 0.0871

@YEAR=1999 6.367078 2.430613 2.619536 0.0193

R-squared 0.593703 Mean dependent var 0.362000Adjusted R-squared 0.485357 S.D. dependent var 2.626313S.E. of regression 1.884081 Akaike info criterion 4.317076Sum squared resid 53.24643 Schwarz criterion 4.566009Log likelihood -38.17076 Hannan-Quinn criter. 4.365670F-statistic 5.479698 Durbin-Watson stat 2.116216Prob(F-statistic) 0.006353

-4

-2

0

2

4-6 -4 -2 0 2 4 6

96 98 00 02 04 06 08 10 12 14

Residual Actual Fitted


13

.

Appendix D: List of variables under evaluation and their acronyms


14

.

Appendix E: Graphs of actual and forecast values for selected variables

Variablename in Modux: CFIN_R = Cons. men. residents (vol.)Residual-analysis P-Value:

Autocorrelation: 0.14Heteroscedasticity: 0.99

Normality: 0.73Bias: 0.77

Error-analysisMean Error: 0.0MASE: 0.4MAE: 0.4RMSE: 0.5

Error-RatiosMASE AR/EQ: 2.1MASE EQ before/after crisis: 0.5

Forecasts and observations (growth-rate)

-1.0

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

Observation Forecast equation Forecast AR

Variablename in Modux: livpet = Livraisons de produits pétroliers (L)Residual-analysis P-Value:






-10.0

-5.0

0.0

5.0

10.0

15.0

20.0



15

.

Variablename in Modux: xcomm_r = Exp. serv. fin. (mia eur, vol.)Residual-analysis P-Value:






-15.0

-10.0

-5.0

0.0

5.0

10.0

15.0

20.0

25.0


Variablename in Modux: mbo_r = Imp. biens aut. (vol., mia eur)Residual-analysis P-Value:






-25.0

-20.0

-15.0

-10.0

-5.0

0.0

5.0

10.0

15.0

20.0


Variablename in Modux: stoxx50 = Indice boursier zone Euro Stoxx50 (2001=100)Residual-analysis P-Value:






-40.0

-30.0

-20.0

-10.0

0.0

10.0

20.0

30.0

40.0

50.0



16

.

Appendix F: Published documentation about Modux Adam, Ferdy, 2004: Modelling a small open economy: What is different? The case of Luxembourg, available at: http://www.statistiques.public.lu/en/methodology/definitions/M/M odux/Modux.pdf Adam, Ferdy, 2007: Cahier de variantes Modux, Cahier économique du STATEC N° 104 Adam, Ferdy, 2010: Modelling aggregate migration and cross border worker flows, Économie et statistiques du STATEC, N° 35/2009 Glocker, Christian, 2016: Introducing a financial accelerator in Modux, Économie et statistiques du STATEC, N° 87/2016 Glocker, Christian, 2016: Introducing a financial accelerator for the housing sector in Modux, Économie et statistiques du STATEC, N° 95/2017

Variablename in Modux: riskmen = Prime de risque, credits hypothécairesResidual-analysis P-Value:






-1.0

-0.5

0.0

0.5

1.0

1.5

2.0


working papers du statec - gouvernement luxembourgeois · for the main relationships whether the...

Documents