04/05/2011 seasonal adjustment and demetra+ estp course eurostat 3 – 5 may 2011 dario buono and...

04/05/2011

Seasonal Adjustment and DEMETRA+

ESTP course

EUROSTAT 3 – 5 May 2011

Dario Buono and Enrico InfanteUnit B2 – Research and Methodology

© 2011 by EUROSTAT. All rights reserved. Open to the public

204/05/2011 ESTP course

Identification of type of Outliers

Calendar Effects and its determinants

X-12 ARIMA and TRAMO/SEATS

Session 2 – May 4th

Dario Buono, Enrico Infante

3

Outliers

Outliers are data which do not fit in the tendency of the Time Series observed, which fall outside the range expected on the basis of the typical pattern of the Trend and Seasonal Components

Additive Outlier (AO): the value of only one observation is affected. AO may either be caused by random effects or due to an identifiable cause as a strike, bad weather or war

Temporary Change (TC): the value of one observation is extremely high or low, then the size of the deviation reduces gradually (exponentially) in the course of the subsequent observations until the Time Series returns to the initial level

Level Shift (LS): starting from a given time period, the level of the Time Series undergoes a permanent change. Causes could include: change in concepts and definitions of the survey population, in the collection method, in the economic behavior, in the legislation or in the social traditions

04/05/2011 ESTP course Dario Buono, Enrico Infante

4

Outliers


Type of Outliers

Additive Outlier Temporary Change Level Shift

0,98

1

1,02

1,04

1,06

1,08

1,1

1,12

1,14

1,16

jan.98 jan.99 jan.00 jan.01 jan.02 jan.03 jan.04 jan.05 jan.06



AO - Estate agency activity

Outliers



Outliers

TC – Business machine



Outliers

LS – Tobacco



Outliers

Additive Outliers Unusual high or low singular values in the data series

Transitory ChangesTransitory changes in the trend, followed by slow comebacks to the initial tendency

Level ShiftClear changes of the trend

Assimilated to the IrregularComponent

Assimilated to the TrendComponent



TRAMO & X-12 ARIMA use regARIMA models where Outliers are regressors

Xt is the raw series, Zt is the “linearized” series (corrected from Outliers and other effects)

Zt follows an ARIMA(p,d,q):

where is the rupture date, is the rupture impact and (B) models the kind of rupture (its “shape”)

Outliers

Tttt

tt

BZX

BZB

)(

)()(



Outliers

Automatic and iterative detection and estimation process are available in TRAMO and X-12 ARIMA

date

Tt

TtTt if0

if1

AO 1)( B LS

BB

11

)(

TC )10(

11

)(

B

B



Outliers

The smoothness of series can be decided by statisticians and the policy must be defined in advance

Consult the users This choice can influence dramatically the credibility Outliers in last quarter are very difficult to be identified Some suggestions:

– Look at the growth rates– Conduct a continuous analysis of external sources to

identify reasons of Outliers– Where possible always add an economic explanation – Be transparent (LS, AO,TC)



Calendar Effects

Time Series: usually a daily activity measured on a monthly or quarterly basis only– Flow: monthly or quarterly sum of the observed variable– Stock: the variable is observed at a precise date (example:

first or last day of the month) Some movements in the series are due to the variation

in the calendar from a period to another– Can especially be observed in flow series– Example: the production for a month often depends on the

number of days



Calendar Effects

Trading Day Effect– Can be observed in production activities or retail sale– Trading Days (Working Days) = days usually worked

according to the business uses– Often these days are non public holiday weekdays

(Monday, Tuesday, Wednesday, Thursday, Friday)– Production usually increases with the number of working

days in the month



“Day of the week” effect– Example: Retail sale turnover is likely to be more

important on Saturdays than on other weekdays Statutory (Public) Holidays and Moving Holidays

– Most of statutory holidays are linked to a date, not to a day of the week (Christmas)

– Some holidays can move across the year (Easter, Ramadan) and their effect is not completely seasonal

Months and quarters are not equivalent and not directly comparables

Calendar Effects



French Calendar

Calendar Effects – Trading Days

Q1 Q2 Q3 Q4 Monday 12.55 11.01 12.86 12.70 Tuesday 12.75 12.82 12.87 12.72 Wednesday 12.75 12.83 12.85 12.73 Thursday 12.73 11.85 12.89 12.69 Friday 12.76 12.82 12.83 12.75 "Trading days" 63.54 61.32 64.30 63.59 Saturday 12.76 12.79 12.86 12.68 Sundays 12.89 13.00 13.14 13.14 Public holidays 0.93 3.68 1.42 2.14 Number of trading days in:

1999 63 62 65 64 2000 65 60 63 63 2001 64 60 64 64 2002 63 60 65 63 2003 63 60 64 64



Country Q1 Q2 Q3 Q4 Country Q1 Q2 Q3 Q4 AT 62.88 60.44 65.02 61.46 IT 62.17 62.73 65.02 62.90 BE 63.59 61.44 64.32 63.59 LT 59.15 63.56 64.68 63.95 BU 63.02 61.95 64.34 63.61 LU 63.59 60.73 65.02 63.61 CY 61.29 60.59 65.02 62.83 LV 63.39 61.27 65.73 62.83 CZ 63.59 62.68 63.63 62.12 MT 61.07 62.12 63.61 62.88 DE 60.72 59.70 65.17 61.40 NL 63.59 61.44 65.73 64.29 DK 63.02 59.09 65.73 63.20 PL 63.59 61.88 65.17 63.07 EE 62.40 62.21 65.32 63.27 PT 63.49 61.07 65.02 62.20 ES 62.78 63.51 65.02 62.22 RO 63.73 63.02 65.73 63.59 FI 62.32 60.50 65.73 62.51 SE 62.78 60.83 65.73 62.15 FR 63.59 60.82 64.32 63.59 SI 62.15 61.27 65.02 61.44 GR 61.29 61.32 65.02 63.56 SK 62.63 61.93 62.93 62.17 HU 62.90 62.41 65.00 63.56 UK 62.05 62.37 64.70 63.73 IE 62.59 62.49 64.73 63.29

Average number of Working Days




56

58

60

62

64

66

68

Q1 Q2 Q3 Q4

AT BE BU CY CZ DE DK EE ES FI FR GR HU IE IT LT

LU LV MT NL PL PT RO SE SI SK UK

Average number of Working Days




Calendar Effects can be partly considered seasonal:– The number of Trading Days is almost always smaller

in February than in March– Some months may have more public holidays (ex: May

in France), and therefore less trading days than others Thus, we can:

– Either estimate the non seasonal Calendar Effects– Make the correction of both Seasonal and Calendar

Effects Spectra of the “Weekday” series: number of Mondays,

Tuesdays, …, Friday per month




Average length of a year: 365.25 days

Average length of a month:

Average number of weeks per month:

4375.30

12

25.365

357.125 3482.0 alias 3482.47

4375.30




Spectra of Weekdays series




Example: wear



Hypothesis: for the most common models, it is assumed that the effect of a day is constant over the span of the series

Basic model:

– Ni,t = # of Mondays (i=1), …, Sundays (i=7) in month t

i is the effect of day i (ex: average production for a Monday (i=1))

ti

itit NX

7

1




In some cases (particular activities) you could have to make a distinction between working and non working Mondays, Tuesdays etc.

You can of course adapt the model to other more simple or more complex cases

ti

fiti

i

nfitit NNX

7

1

7

1




Certain groupings could be more sensible than others:– Public holidays– Non working weekdays– Trading days and weekends– Etc.

Much better to use contrasts (W. Bell):

where: is the average effect of a day (mean of the and )– mt is the length of month t

ttti

fiti

it

nfitit mNNNNX

))(())(( 7

6

1

6

17




The variables take into account a part of Seasonality: the length of the month is seasonal, the White Monday is always a Monday etc.

If we want to estimate the non seasonal Calendar Effects, we can remove from the variables their long-term average

In this case, the explanatory variables do not have any Trend or Seasonality. Thus, the dependant variable must be stationary, an Irregular Component for example (X-11), or a regARIMA model should be used

X-12 ARIMA and TRAMO-SEATS




Coffee-Break!!!

We will restart in 15 minutes



Calendar Effects – Moving Holidays

The date of some public holidays (usually religious events) moves across the year and can affect different months or quarters according to the year

Examples: Ramadan, Chinese New Year, Easter (Catholic or Orthodox) etc.

The effect of the event could be gradual and could affect the days before or after the event itself: Chocolate sales around Easter

We focus in the following on the treatment of Easter but the methodology and the models can be easily adapted




Catholic Easter– Sunday Easter can fall between March 22 (last time: 1818,

next time: 2285) and April 25 (last time: 1943, next time: 2038) i.e. during the first or second quarter

Orthodox Easter– Between April 4 (last time: 1915, next time: 2010) and May

9 (last time before 1600, next time: 2173) i.e. always during the third quarter

Easter affects the production of many economic sectors:– Statutory holidays (transportation, hotel frequentation),

consumption habits (chocolate, flowers, lamb meat, etc.)




Catholic Easter dates (1900-2099)

0

1

2

3

4

5

6

7

8

9

10

23/3 26/3 29/3 1/4 4/4 7/4 10/4 13/4 16/4 19/4 22/4 25/4




Catholic and Orthodox Easter dates (1800-2200)




Usually models are based on the number of days “affected” by Easter in the various months (March or April for Catholic Easter)– Immediate Impact

– Gradual Impact: effect on the n days preceding Easter Easter can affect the w preceding days; the impact can therefore

affect 2 months The impact is supposed constant on the w days




The 2 Easter models are estimated using a regARIMA model:

Where:– Xt represents a regressor modeling Calendar Effects, Outliers

(AO, LS, TS) and other user-specified effects including Easter Effects

– Dt represents a priori adjustments such as leap year, strikes, etc.




The w-day period, during which the activity is affected, does NOT include the Sunday Easter. The activity level remains constant on the w-day period

Period m-1 Period m Event

9 8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7




If i denotes the year and j the month, let us note and the number of the w days falling in month j of year i

The regressor associated to this model is:

The regressor values are 0 except for February, March and April depending on the value of w

251 w ijn

w

nwX ij

ij




A general constant effect model


9 8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7




A general linear effect model


9 8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7 8



Orthodox Easter falls more in April than in May (83% versus 17%)

Catholic Easter falls more in April than in March (78% versus 22%)

There is a “Seasonal Effect” one can correct by removing the long-term monthly average:

jij XwX




Calendar Effects

Definitions:– The Seasonally Adjusted series is logically

defined as the raw series from which the Seasonality (St) has been removed

– The “Working Day” adjusted series is defined as the raw data from which the Calendar Effects (TDt, MHt) have been removed

– This definition can vary if you include or not the Calendar Effects

ttttttt IMHTDOSTCX



X-12 ARIMA VS TRAMO/SEATS

Seasonal Adjustment is usually done with an off-the-shelf program. Three popular tools are: – X-12 ARIMA (Census Bureau)

– TRAMO/SEATS (Bank of Spain)

– DEMETRA+ (Eurostat), interface X-12 ARIMA and Tramo/Seats

X-12 ARIMA is Filter based: always estimate a Seasonal Component and remove it from the series even if no Seasonality is present, but not all the estimates of the Seasonally Adjusted series will be good

TRAMO/SEATS is model based method variants of decomposition of Time Series into non-observed components



X-12 ARIMA

Auto-projective Models

What is f ?– Under certain hypothesis (Stationary), it is possible to find

an “ARMA function” which gives a good approximation of f– These models usually depend on a few number of

parameters only– Box et Jenkins methodology allows to estimate the

parameters and gives quality measures of the adjustment

),.....,,,( 321 ttttt XXXfX



X-12 ARIMA

AutoRegressive Model of order p AR(p):

Moving Average Model of order q MA(q):

tttp

p

tptpttt

XBXBBB

XXXX

)().....1(

or

.....

221

2211

ttq

qt

qtqtttt

BBBBX

X

)().....1(

or

.....

221

2211



X-12 ARIMA

ARMA(p,q):– Corner method, triangle method, ODQ etc.

ARIMA(p,d,q): – Non stationary Time Series, with a Trend

SARIMA(p,d,q)(P,D,Q)S:

– Non stationary Time Series, with Trend and Seasonality

tt BXB )()(

ttd BXBB )()()1(

tstsDsd BBXBBBB )()()()()1()1(


43

X-12 ARIMA

A regARIMA model is a regression model with ARIMA errors. When we use regression models to estimate some of the components in a Time Series, the errors from the regression model are correlated, and we use ARIMA models to model the correlation in the errors. ARIMA models are one way to describe the relationships between points in a Time Series

Besides using regARIMA models to estimate regression effects (such as outliers, Trading Day, and Moving Holidays), we also use ARIMA models to forecast the series. Research has shown that using forecasted values gives smaller revisions at the end of the series



X-12 ARIMA

X-12 ARIMA runs through the following steps:1. The series is modified by any user defined prior adjustments2. The program fits a regARIMA model to the series in order to

detect and adjust for outliers and other distorting effects to improve forecasts and Seasonal Adjustment

3. It detect and estimates additional component (e.g. Calendar Effects) and extrapolate forward (forecast) and backwards (backcast)

4. The program then uses a series of Moving Averages to decompose a Time Series into three components. It does this in three iterations, getting successively better estimates of the components. During these iterations extreme values are identified and replaced

5. In the last step a wider range of diagnostic statistics are produced, describing the final Seasonal Adjustment, and giving pointers to possible improvements which could be made



Moving Averages

A Moving Average of order p+f+1 and coefficients {i} is defined by:

The value at date t is therefore replaced by a weighted average of p past values, the current value and f future values

Examples: simple moving averages of order 3

fi

piitit XMX

X3

1

X3

1

11-t

12-t

ttt

ttt

XXMX

XXMX Asymmetric

Symmetric



Moving Averages

Let us suppose the following decomposition model:

One can want remove the Seasonality and the Irregular by using a Moving Average:

So, we want to cancel some frequencies!!

tttt STCX

00

tttt

tttt

TCMSMTCM

STCMMX



Moving Averages

We would like to find a Moving Average which preserve the Trend and removes both Seasonality and Irregular– Example (preservation of constants): let us define a Series

and a Moving Average M

The coefficients must sum to 1

Property: if you want to preserve polynomials of degree d, the Moving Average coefficients must verify:

aX t

1et

fi

pii

fi

piitt

fi

pii

fi

piitit aaXXMaXXM

fi

pii

kfi

pii 0i d1,2,.....,k and 1



A Filter is a weighted average where the weights sum to 1 Seasonal Filters are the filters used to estimate the

Seasonal Component. Ideally, Seasonal Filters are computed using values from the same month or quarter, for example, an estimate for January would come from a weighted average of the surrounding Januaries

The Seasonal Filters available in X-12 ARIMA consist of seasonal Moving Averages of consecutive values within a given month or quarter. An n x m Moving Average is an m-term simple average taken over n consecutive sequential spans

X-12 ARIMA



An example of a 3x3 filter (5 terms) for January 2003 (or Quarter 1, 2003) is:

2001.1 + 2002.1 + 2003.1 + 2002.1 + 2003.1 + 2004.1 +

2003.1 + 2004.1 + 2005.1 9

An example of a 3x5 filter for January 2003 (or Quarter 1, 2003) is:

2000.1 + 2001.1 + 2002.1 + 2003.1 + 2004.1 + 2001.1 + 2002.1 + 2003.1 + 2004.1 + 2005.1 + 2002.1 + 2003.1 + 2004.1 + 2005.1 + 2006.1 15

X-12 ARIMA



X-12 ARIMA

Trend Filters are weighted averages of consecutive months or quarters used to estimate the trend component

An example of a 2x4 filter (5 terms) for First Quarter 2005:

2004.3 + 2004.4 + 2005.1 + 2005.2 2004.4 + 2005.1 + 2005.2 + 2005.3

_______________________________________

8

Notice that we are using the closest points, not just the closest points within the First Quarter like with the Seasonal Filters above

Notice also that every quarter has a weight of 1/4, though the Third Quarter uses values in both 2004 and 2005


X-12 ARIMA

Keep in mind that the data used in the Seasonal and the Trend Filters can go back several years. Let's look at an example using X-12 ARIMA's seasonal Moving Average Filters: if the last point in the series is January 2006, and you're using 3x5 Seasonal Filters, the value at January 2006 will effect the estimates for Januaries in 2004, and 2005. You can see the value for January 2003 in the equations below

The 3x5 filter for January 2004:

2001.1+2002.1+2003.1+2004.1+2005.1

2002.1+2003.1+2004.1+2005.1+2006.1

2003.1+2004.1+2005.1+2006.1+2007.1

_____________________________________________________________

15


X-12 ARIMA

Step 1.1: Initial Trend Estimate

Compute a centred 12 term (13 term) Moving Average as a first estimate of the Trend:

The first Trend estimation is always 2x12 or 2x4 Step 1.2: Initial Seasonal-Irregular component or “SI Ratio”

The ratio of the original series to the estimated trend is the first estimate of the de-trended series:

6556)1(

24

1

12

1

12

1

12

1

24

1 tttttt YYYYYT

)1()1(

t

tt T

YSI )1()1(

ttt TYSI

53

)1(24

)1(12

)1()1(12

)1(24

)1(

9

1

9

2

9

3

9

2

9

1ˆ tttttt SISISISISIS

)1(624

1)1(512

1)1(512

1)1(624

1

)1()1(

ˆˆˆˆ

ˆ

tttt

tt

SSSS

SS

24

ˆ

12

ˆ

12

ˆ

24

ˆˆ

)1(6

)1(5

)1(5

)1(6)1(1 tttt

tt

SSSSSS

)1()1(

t

tt S

YA )1()1(

ttt SYA


X-12 ARIMA

Step 1.3: Initial Preliminary Seasonal Factor

5 term weighted Moving Average (3x3) is calculated for each month of the Seasonal-Irregular ratios (SI) to obtain preliminary estimates of the Seasonal Factors:

Step 1.4: Initial Seasonal Factor

Crude “unbiased” seasonal from step 1.3 via centering:

Step 1.5: Initial Seasonal Adjustment

54

H

Hjjt

Hjt AhT )1()12()2(

)12( Hjh jj hhHjH ,

)2()2(

t

tt T

YSI )2()2(

ttt TYSI


X-12 ARIMA

Step 2.1: Intermediate Trend

Calculate an intermediate Trend (Henderson) of length 2H+1 for data-determined H

where are the (2H+1) term Henderson weights

Step 2.2: Final SI Ratios

Calculate the de-trended series from Henderson trend:

55

)2(36

)2(24

)2(12

)2()2(12

)2(24

)2(36

)2(

15

1

15

2

15

3

15

3

15

3

15

2

15

1ˆ tttttttt SISISISISISISIS

)2(624

1)2(512

1)2(512

1)2(624

1

)2()2(

ˆˆˆˆ

ˆ

tttt

tt

SSSS

SS

24

ˆ

12

ˆ

12

ˆ

24

ˆˆ

)2(6

)2(5

)2(5

)2(6)2()2( tttt

tt

SSSSSS

)2()2(

t

tt S

YA )2()2(

ttt SYA


X-12 ARIMA

Step 2.3: Preliminary Seasonal Factor

Calculate final “biased” seasonal factors via a “3x5” Seasonal Moving Average:

Step 2.4: Seasonal Factor

Calculate final “unbiased” seasonal factors via centering:

Step 2.5: Seasonal Adjustment

56

H

Hjjt

Hjt AhT )2()12()3(

)3(

)2()3(

t

tt T

AI )3()2()3(

ttt TAI


X-12 ARIMA

Step 3.1: Final Trend

Final Trend from a Henderson Trend Filter determined from data:

Step 3.2: Final Irregular

Final Irregular Factors as ratios between the Seasonally Adjusted series from Stage 2 and the final Trend from Step 3.1:


X-12 ARIMA

Estimated decomposition

)3()2()3(tttt ISTY

)3()2()3(tttt ISTY


X-12 ARIMA

The first Trend estimation (in Stage 1) is always 2x12 or 2x4

The Henderson Trend Filter choices (Step 2.1, 3.1) are based on noise-to-signal ratios, the size of the irregular variations relative to those of the Trend and labelled I/C in the X-12 ARIMA output– If I/C<1, the 9 term Henderson Filter (H=4) is used;

otherwise, in Stage 2, the 13 term filter (H=6) is used– In Stage 3, the 13 term Filter is used when 1<I/C<3.5, but

the 23 term Henderson Filter (H=11) is used when I/C≥3.5

59

I / S Seasonal Moving

Average

0<I / S<2.5 3x3

3.5<I / S<5.5 3x5

6.5<I / S 3x9


X-12 ARIMA

The criterion for selection of the seasonal Moving Average is based on the global I/S, which measures the relative size of irregular movements and seasonal movements averaged over all months or quarters. It is used to determine what seasonal Moving Average is applied using the following criteria:

The global I/S ratio is calculated using data that ends in the last full calendar year available

If 2.5<I/S<3.5 or 5.5<I/S<6.5 then the I/S ratio will be calculated using one year less of data to see if the I/S ratio than falls into one of the ranges given above. The year removing is repeated either until the I/S ratio falls into one of the ranges or after five years a 3x5 Moving Average will be used


TRAMO/SEATS

The objective of the procedure is to automatically identify the model fitting the Time Series and estimate the model parameters. This includes:– The selection between additive and multiplicative model types

(log-test)

– Automatic detection and correction of outliers, eventual interpolation of missing values

– Testing and quantification of the Trading Day effect

– Regression with user-defined variables

– Identification of the ARIMA model fitting the Time Series, that is selection of the order of differentiation (unit root test) and the number of autoregressive and Moving Average parameters, and also the estimation of these parameters



TRAMO/SEATS

The application belongs to the ARIMA model-based method variants of decomposition of Time Series into non-observed components

The decomposition procedure of the SEATS method is built on spectrum decomposition

Components estimated using Wiener-Kolmogorov Filter SEATS assumes that:

– The Time Series to be Adjusted Seasonally is linear, with normal White Noise innovations

– If this assumption is not satisfied, SEATS has the capability to interwork with TRAMO to eliminate special effects from the series, identify and eliminate outliers of various types, and interpolate missing observations

– Then the ARIMA model is also borrowed from TRAMO



TRAMO/SEATS

The application decomposes the series into several various components. The decomposition may be either multiplicative or additive

The components are characterized by the spectrum or the pseudo spectrum in a non-stationary case:– The Trend Component represents the long-term development of

the Time Series, and appears as a spectral peak at zero frequency. One could say that the trend is a cycle with an infinitely long period

– The effect of the Seasonal Component is represented by spectral peaks at the seasonal frequencies

– The Irregular Component represents the irregular White Noise behaviour, thus its spectrum is flat (constant)

– The Cyclic Component represents the various deviations from the trend of the Seasonally Adjusted series, different from the pure White Noise



First SEATS decomposes the ARIMA model of the Time Series observed, that is, identifies the ARIMA models of the components. This operation takes place in the frequency domain. The spectrum is divided into the sum of the spectra related to the various components

Actually SEATS decides on the basis of the argument of roots, which is mostly located near to the frequency of the spectral peak– The roots of high absolute value related to 0 frequency are assigned to

the Trend Component– The roots related to the seasonal frequencies to the Seasonal

Component– The roots of low absolute value related to 0 frequency and the cyclic

(between 0 and the first seasonal frequency) and those related to frequencies between the seasonal ones are assigned to the Cyclic Component

– The Irregular Component is always deemed as white noise

TRAMO/SEATS


64

Questions?


04/05/2011 seasonal adjustment and demetra+ estp course eurostat 3 – 5 may 2011 dario buono and...

Documents

time series

enrico infantediagram31111

thdario buono

enrico infanteoutliersoutliers

permanent change

wartemporary change

given time period

enrico infanteunit b2