ecotrim a program for temporal disaggregation of time series eurostat – unit c2 roberto barcellan
TRANSCRIPT
ECOTRIMA program for temporal disaggregation of time series
Eurostat – Unit C2Roberto Barcellan
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
2
Ecotrim for Windows
Ecotrim is a program developed by Eurostat, Directorate C, Economic and Monetary Statistics, Unit C2, Economic accounts .
Windows version: based on Visual Basic and C++
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
3
Forewords
The Ecotrim project has been developed by Eurostat since beginning of 90s
Several versions: GAUSS, Fortran, SAS, Windows The version 1.01 beta 3 currently available will be
the reference for this presentation Ecotrim for Windows is still a beta version A first version of the manual will be soon available For specific technical details related to methodology,
please refer to the literature mentioned in the supporting papers
Several users in Europe and outside Europe
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
4
Why Eurostat developed Ecotrim?
ESA 95 paragraph 12.04 “The statistical methods used for compiling quarterly accounts
may differ quite considerably from those used for the annual accounts. They can be classified in two major categories: direct procedure and indirect procedure.
... On the other hand, indirect procedures are based on temporal disaggregation of the annual accounts data in accordance with mathematical and statistical methods using reference indicators that permit the extrapolation for the current year.
The choice between the different indirect procedures must above all take into account the minimisation of the forecast error for the current year, in order that the provisional annual estimates correspond as closely as possible to the final figures. The choice between these approaches depends, among other things, on the information available at quarterly level”.
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
5
Temporal disaggregation
process of deriving high frequency data from low frequency data and, if available, related high frequency information
ECOTRIM supplies a set of mathematical and statistical techniquesto carry out temporal disaggregation
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
6
Temporal disaggregation techniques are a valid support in compiling short-term statistics (e.g. QNA):
Quarterly National Accounts (QNA)give a quarterly breakdown of the figures in the annual accounts
Flash estimatesuse the available information in the best possible way including, in the framework of a statistical model, the short-term available information and the low frequency data in a coherent way
Monthly indicators of GDPthe monthly estimates are derived from the available information respecting the coherence with quarterly data
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
7
Other short-term statistics:
Short-term industrial statistics Employment Money and banking statistics
in this presentation we focus on QNA
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
8
The present Windows version of the program supplies a range of techniques concerning:
temporal disaggregation of univariate time series by using or not related series and fulfilling temporal aggregation constraints (the methods that ECOTRIM offers, follow the mathematical approach and the optimal, in the least squares sense, approach);
temporal disaggregation of multivariate time series with respect of both temporal and contemporaneous aggregation constraints (in this case too ECOTRIM proposes both adjustment and optimal techniques, in the least squares sense);
forecasting of current year observations by using or not available information on related series.
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
9
Basic ideas - QNA (1)
Temporal disaggregation methods for compiling quarterly accounts are an integral part of the estimation approach.
Their use is more intensive or less intensive according to the main philosophy that characterises the system of quarterly accounts.
The use of mathematical and statistical methods do not necessarily imply a lack of basic information since these models can be used also to improve the quality of the quarterly figures.
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
10
Each series is linked to one or more available related quarterly series.
Due to differences in definition and coverage, the account indicators do not give the same value as the series to be estimated (such as in the direct approach)
Their movement can be used to recover the quarterly dynamics of the unknown aggregate.
Basic ideas - QNA (2)
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
11
Temporal and Accounting constraint
National accountants are often faced with the estimation of a set of quarterly series linked by some accounting relationship.
Temporal disaggregation methods can also be used in such cases, to give a solution consistent with both temporal and contemporaneous aggregation constraints.
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
12
Characteristics of temporal disaggregation methods (1)
a) The set of basic information should include statistical variables that are considered as good proxies of the aggregates that have to be estimated
b) All variables that have a high explanatory power with respect to a specific national accounts aggregate but which do not satisfy (a) have to be eliminated from the set of basic information (for example the interests rate for the estimation of GDP);
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
13
Characteristics of temporal disaggregation methods (2)
c) The statistical models need not to incorporate any relationships between the aggregates of quarterly accounts that imply economic hypotheses as for example, the relation between consumption and disposable income;
d) The set of basic information should only include variables associated with the economy of the country for which the quarterly accounts are compiled. This means that the information set is closed;
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
14
Selection of indicators
Choice at high frequency (movements)
Relationships and statistics available only at low frequency (link with the target series)
Experience
Ex-post analysis: statistics (available in Ecotrim), correlation between estimated and related series (levels and growth rates)
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
15
Basic principles
DistributionWhen annual data are either sums or averages of quarterly data (e.g., GDP, consumption, indexes and in general all flow variables and all average stock variables)
InterpolationWhen annual value equals by definition that of the fourth (or first) quarter (e.g., population at the end of the year, money stock, and all stock variables)
ExtrapolationWhen estimates of quarterly data are made when the relevant annual data are not yet available
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
16
time consistencyquarterly values have to match annual values (for example the sum of quarterly values of the GDP must be equal to the annual value):
accounting coherence
quarterly components of an account should respect the accounting constraints (for example, the sum of quarterly values of the GDP expenditure side components should be equal to the corresponding quarterly value of GDP):
Estimates have to be consistent and coherent
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
17
Methods that do not involve the use of related series
Smoothing procedures Time series methods
Basic ideas:
sufficiently smoothed path
coherence with temporal aggregation constraints
these methods can be used when there are serious gaps in basic information (only annual data are available)
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
18
Methods that make use of related series
The quarterly path is estimated on the basis of external quarterly information for logically and/or economically related variables.
quarterly information linked to the relevant variable of interest are used
sub-annual or short-term indicators
multivariate applications
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
19
Temporal disaggregation approaches According to the techniques, the accounting constraints and the different amount of basic information used, temporal disaggregation methods can be distinguished in:
Univariate Approach Multivariate ApproachSmoothing methods Two steps adjustment methodsTwo steps adjustment methods Regression based methodsTime series methods*Regression based methods
static models dynamic models*
* Not in Ecotrim Windows
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
20
Smoothing methods
They typically assume that the unknown quarterly trend can be conveniently described by a function of time such that the necessary condition of satisfying aggregation constraints and the desirable condition of smoothness are both met.
Generally these techniques estimate the quarterly figures by considering a "window" of annual values and a subset of the time series. Starting from these data, the techniques minimise the discrepancy between known annual values and quarterly estimated data.
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
21
Smoothing method within Ecotrim for Windows
Boot , Feibes e Lisman Minimise the sum of squared first differences
between successive disaggregated values (model FD)
Minimise the sum of squared second differences (model SD)
suitable for situation with lack of information they ensure interpolation estimates for the quarterly
breakdown use of all the information available and give
estimation for all the period considered no extrapolation and diagnostics or confidence bands
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
22
Two steps adjustment methods
They divide the process of estimation in two parts:
The first step in indirectly estimating quarterly accounts series is usually the conversion of quarterly indicators into quarterly series which are not consistent with the annual counterpart. We shall refer to this step as preliminary estimation.
At the second step, the preliminary estimates are then processed in order to fit the known annual series, using procedures that we shall refer to as adjustment.
In the multivariate case, the second step includes the fulfilment of the contemporaneous accounting constraints
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
23
Procedure of the two steps adjustment methods
Preliminary estimation: direct way, for example sample survey mathematical-statistical way, for example by using a
linear regression relationship between the annual accounts series and the annualised related indicators.
But the preliminary quarterly estimatesdo not generally satisfy the temporal aggregation
constraints.
Distribution of the annual discrepancy between the annual aggregate and the aggregated preliminary quarterly estimates
Fitting annual constraints and altering the quarterly path given by the preliminary estimates to the least extent possible.
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
24
Denton
Benchmarking
Movement preservation principle
AFD levels
PFD proportional levels
Weighted matrices
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
25
Time series methods(not in Ecotrim Windows)
– Wei and Stram (1990) and Al-Osh(1989)
– They are not currently implemented within Ecotrim for Windows but they are present in the Gauss version
– The advantage of this procedure is that they provide now-casts » during the year even if no related indicators are available
more sophisticated statistical smoothing methods they can be used in case of lack of information ARIMA model based techniques
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
26
Optimal statistical methods
they merge the steps of preliminary estimation and adjustment
one statistically optimal procedure
use of all the available information in the context of a regression model
the model involves annual information and quarterly related information
ensure the annual consistency
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
27
Chow and Lin solution
Chow and Lin (1971) worked out a least-squares optimal solution on the basis that a linear regression model involving the quarterly aggregate series and the related quarterly series will hold
natural and coherent solution to the extrapolation problem.
intensively used in National Statistical Institutes, especially in France, in Italy, Portugal, Belgium and Spain.
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
28
Optimal statistical methods (static models) within Ecotrim for Windows
Different versions of this technique have been developed according to the different hypotheses related to the structure of the error in the regression model. The stochastic error models usually considered when estimating quarterly accounts series are the following:
Model AR(1) Chow and Lin GLS (min SSR of Barbone and others 1981, max Log Bournay and Laroque, 1979);
Random walk model (Fernàndez, 1981);
Random walk-Markov model (Litterman, Min SSR and Max Log).
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
29
Statistics
Rhô
R-squared
Durbin-Watson
Probability of F
T-stat
Reliability indicators (lower value for the range between Min and Max)
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
30
Multivariate models
multivariate dimension
contemporaneous accounting constraints are introduced in the estimation step
temporal and accounting coherence
two approaches: multivariate benchmarking BLUE approach
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
31
Regression based methods for the multivariate approach
White noise Random Walk
No preliminary estimates fulfilling the annual constraint are requested
Here is an extension to the multivariate of the univariate approach
From the statistical point of view is better to use WN or RW but for the practical aspects Rossi and Denton ensure more coherence in terms of growth rates
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
32
Multivariate adjustment
A reasonable way to eliminate the discrepancy between a contemporaneously aggregated value and the corresponding sum of disaggregated preliminary quarterly estimates, consists in distributing such a discrepancy according to the weight of each single temporally aggregated series with respect to the contemporaneously aggregated one
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
33
Denton multivariate adjustmentDenton’s multivariate adjustment generalises the univariate procedure shown in the univariate case by taking into account some technical devices about (i) the treatment of starting values (Cholette, 1984, 1988) and (ii) the nature of the accounting constraints
Preliminary estimates fulfilling the annual constraint are not necessarily requested
Denton AFD Denton ASD Denton PFD Denton PSD
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
34
Rossi multivariate adjustment
Preliminary estimates fulfilling the annual constraint are requested
Rossi’s procedure can be viewed as a sub-case of Denton’s.
The estimated series are forced to satisfy the accounting constraint
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
35
Use of ECOTRIM
ECOTRIM is a program that supplies a set of mathematical and statistical techniques to carry out temporal
disaggregation.
Structured for Windows 95/98 and Windows NT Visual Basic and C++ User friendly
It can be used according to two different modes:
interactive mode batch mode
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
36
Interactive mode
Input session
Univariate methods
• Boot, Feibes and Lismann
• Denton
• AR(1)
• Fernàndez
• Litterman
Multivariate methods
• White noise
• Random walk
• Rossi
• Denton
Output session
Graphs and display
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
37
Batch mode
ECOTRIM performs temporal disaggregation of several jobs starting from a batch command file.
Batch mode is very useful when handling many series.
Batch command
file
Batch session
• univariate
• multivariate
Output
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
38
ECOTRIM: A guided ExampleThe compilation of the euro-area and EU quarterly accounts
Available dataSuppose that you have at your disposal a set of annual data composed by the series of GDP and main expenditure and output components:
Expenditure:households final consumption;government final consumption;gross fixed capital formation;changes in inventories;export;imports.
Output:agriculture, hunting, forestry and fish.;industry, including energy;construction;wholesale, retail trade; hotels and rest.;financial, real-estate, renting and business activities;other services activities;FISIM;taxes less subsidies on products.
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
39
Note that the annual GDP is unique and that the output approach and the expenditure approach are balanced.
Annual data cover the period 1991-2002.
In addition, Suppose that you have at your disposal a set of quarterly preliminary estimates/indicators to be used for estimating the GDP and the expenditure and output components on a quarterly basis preliminary
Quarterly indicators cover the period 1991Q1-2003Q2.
Unique GDP
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
40
Objectives
The objective of the exercise to obtain quarterly estimates of GDP and expenditure and output components that:
Fulfil the time consistency requirements: the sum of the four
quarters of a year is equal to the corresponding annual figure for each variable;
Fulfil the accounting requirements: the sum of the quarterly components is equal to the corresponding quarterly value for GDP both on the expenditure and output side.
The available quarterly preliminary estimates/indicators do not satisfy the temporal constraints and the accounting constraint.
They give an idea of the quarterly movements of the target variables but do not present the same level as the target variables.
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
41
The approach to the estimate of quarterly figures
The approach to the estimation of the quarterly figures is divided in two steps:
Estimate of each component on the expenditure and output side by respecting the time constraint (the sum of the quarter for the past year has to be equal to the corresponding annual value;
Balancing of the expenditure and output side.
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
42
The univariate methods used
Univariate estimates: univariate method of Chow and Lin;
The Chow and Lin method allows to obtain single estimates of each component that respect the annual constraint for the past years (1991-2002) and to obtained the estimates for the quarters in the current year (in the example, 2003Q2).
The main idea of the approach is that indicator and target variable satisfy a regression model that is valid both for annual and quarterly data, with the exception of the error structure. From the available annual figures the procedure derives the estimates of the parameters of the regression model. These parameters are then applied to the quarterly model to derive the quarterly figures, including the “extrapolation” for the quarters of the current year.
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
43
The forced multivariate adjustment
Balancing: multivariate Denton procedure.
The Denton multivariate method allows obtaining a balanced set of data that respect the accounting constraints for all the considered period and the annual constraints for the past years. This technique requires an input series that already fulfils the time consistency constraint.
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
44
Annual GDP - euro-zone
Euro-area, GDP, constant prices 1995
5 000 000.0
5 200 000.0
5 400 000.0
5 600 000.0
5 800 000.0
6 000 000.0
6 200 000.0
6 400 000.0
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
45
Quarterly indicator
Euro-zone, quarterly indicator
1 200 000
1 250 000
1 300 000
1 350 000
1 400 000
1 450 000
1 500 000
1 550 000
1 600 000
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
46
Annual data and indicator
Annual vs. indicator
1 200 000.0
1 250 000.0
1 300 000.0
1 350 000.0
1 400 000.0
1 450 000.0
1 500 000.0
1 550 000.0
1 600 000.0
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49
B1GM IND B1GM ANN
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
47
Final estimate
Annual vs. final
1200000.0
1250000.0
1300000.0
1350000.0
1400000.0
1450000.0
1500000.0
1550000.0
1600000.0
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49
B1GM ANN B1GM FIN
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
48
GDP, statisticsThe value of the parameter is: 0.702346Dependent variables 1
----------------------------------------------------------------------------------------------------------------------------------------------------------------Variable Estimate Std Error t-Stat
----------------------------------------------------------------------------------------------------------------------------------------------------------------CONSTANT -40845.3 6909.05 -5.91B1GM_KPM95E_QS 1.04 0 210.52
----------------------------------------------------------------------------------------------------------------------------------------------------------------Valid Cases 12 Degrees of freedom 10Total SS 1.08E+11 Residual SS 24402792R-Squared 1 Rbar-Squared 1STD error of est 1562.14 Log-likehood 110.79F(2,10) 44320.56 Probability of F 0.25Akaike Info Criterion 14.86 Heterosk. Condition number NDDurbin-Watson 1.87 Jarque-Bera normality stat. 0.32Box-Pierce statistic 0.08 Box-Pierce statistic 0.89Ljung Box Q-statistic 0.1 Ljung Box Q-statistic 1.23
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
49
Estimate, full outputE-B1GMFlowAR(1)MIN SSR________________________________________________________________________________
Date Val. Std. Reliab. Low HighDev. Ind.
199101 1275993 975.92 0.08 1273818 1278167199102 1279190 725.29 0.06 1277574 1280806199103 1278653 747.39 0.06 1276988 1280318199104 1290833 920.86 0.07 1288781 1292885199201 1310309 902.12 0.07 1308299 1312319199202 1300632 725.3 0.06 1299016 1302248
… … … … … …
199403 1324011 725.26 0.05 1322396 1325627200102 1557465 726.42 0.05 1555846 1559083200103 1560110 725.24 0.05 1558494 1561726200104 1558513 900.14 0.06 1556507 1560518200201 1563885 922.7 0.06 1561829 1565941200202 1571309 748.92 0.05 1569640 1572978200203 1574866 725.39 0.05 1573250 1576482200204 1575812 979.79 0.06 1573629 1577995200301 1576201 1360.96 0.09 1573169 1579233200302 1574970 1542.53 0.1 1571533 1578407
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
50
Batch fileDI="H:\...\Estimates_expenditure\ecotrim";DO="H:\...\Estimates_expenditure\ecotrim";FP="eur12_EXP_CON_KPM95E_qs.PRN";FR="eur12_EXP_CONDET_KPM95E_qs.PRN";FL="OUTPUT.LOG";OW="0";{ MET= 4 ; TA= 1 ; ORDER= 4 ; ("eur12_EXP_AGG_KPM95E_AN.PRN":1); ["eur12_EXP_REL_KPM95E_qs.PRN":1]; PARL=-.99; PARH=+.99;}
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
51
Scheme expenditure side
GDP(q1) = PC(q1) + GC(q1) + GFCF(q1) + CI(q1) + EXP(q1) - IMP(q1)
GDP(q2) = PC(q2) + GC(q2) + GFCF(q2) + CI(q2) + EXP(q2) - IMP(q2)
GDP(q3) = PC(q3) + GC(q3) + GFCF(q3) + CI(q3) + EXP(q3) - IMP(q3)
GDP(q4) = PC(q4) + GC(q4) + GFCF(q4) + CI(q4) + EXP(q4) - IMP(q4)
GDP(a) = PC(a) + GC(a) + GFCF(a) + CI(a) + EXP(a) - IMP(a)
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
52
Discrepancypreliminary vs. constraint
Discrepancies preliminary vs. constraint
-1 500.0
-1 000.0
-500.0
0.0
500.0
1 000.0
1 500.0
2 000.0
2 500.0
1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
53
Preliminary vs. final estimateHousehold consumption
Preliminary vs. final household consumption
700 000.0
750 000.0
800 000.0
850 000.0
900 000.0
950 000.0
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49
Series1 Series2
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
54
GVA construction
Preliminary vs. final
7400074500750007550076000765007700077500780007850079000
1999 2000 2001 2002
PREL RES
27 November 2003 OECD, Paris
Application of advanced temporal disaggregation techniques to economic statistics
55
For any information or question about Ecotrim and to obtain the latest releases related to the program,
please contact
Mr Roberto BARCELLANEUROPEAN COMMISSION
Statistical OfficeDirectorate C -Unit C2
Jean Monnet Building
BECH B3/398L-2920 LUXEMBOURG
Tel. (+352) 4301 35802Fax. (+352) 4301 33879
e-mail: [email protected]