NNational ational TTransfer ransfer AAccountsccounts
Friedman's Friedman's SuperSmoother: Special SuperSmoother: Special
Notes on SpanNotes on SpanIVAN MEJIA GUEVARAIVAN MEJIA GUEVARA
CONSEJO NACIONAL DE POBLACIONCONSEJO NACIONAL DE POBLACIONCEPAL/IDRC PROJECTCEPAL/IDRC PROJECT
Honolulu, June 5 2008Honolulu, June 5 2008
National Transfer Accounts
Seoul’s Meeting: SmoothingSeoul’s Meeting: Smoothing►Some problems identified in the workshop:Some problems identified in the workshop:
Young ages.Young ages. Many profiles are inherently not Many profiles are inherently not smooth at early ages smooth at early ages
Old ages.Old ages. The age profile estimates at old The age profile estimates at old ages are often based on relatively few ages are often based on relatively few observationsobservations
Over smoothing.Over smoothing. There is a natural tendency There is a natural tendency to over-smooth the data. The danger is that to over-smooth the data. The danger is that real and important fluctuations may be real and important fluctuations may be concealed. Smoothing too little is better than concealed. Smoothing too little is better than smoothing out real changes. smoothing out real changes.
National Transfer Accounts
Seoul’s Meeting: SmoothingSeoul’s Meeting: Smoothing►Some problems identified in the workshop:Some problems identified in the workshop:
National Transfer Accounts
Seoul’s MeetingSeoul’s Meeting► For most, but not all purposes, For most, but not all purposes,
SUPSMU with a relatively narrow SUPSMU with a relatively narrow bandwidth (0.05 or less) appears to be bandwidth (0.05 or less) appears to be a relatively reliable methoda relatively reliable method
National Transfer Accounts
Alternative Method using Alternative Method using SUPSMUSUPSMU► This approach attempts to deal with the This approach attempts to deal with the
problem of selecting the most problem of selecting the most appropriate bandwidth (span)appropriate bandwidth (span)
► The method might also be effective to The method might also be effective to deal with the problem of over smoothingdeal with the problem of over smoothing
►The method proposed relies on:The method proposed relies on: The use of statistical criteriaThe use of statistical criteria Semi-automatic selection of bandwidthSemi-automatic selection of bandwidth
National Transfer Accounts
Method DescriptionMethod Description1.1. Definition of a metric must be defined Definition of a metric must be defined
in order to measure the difference in order to measure the difference between actual (unsmoothed) values between actual (unsmoothed) values ((AAtt) and output (smoothed) values () and output (smoothed) values (FFtt))
2.2. Specification of span optionsSpecification of span options3.3. Compute the metric(s) defined in 1Compute the metric(s) defined in 14.4. Apply the Majority Voting Scheme Apply the Majority Voting Scheme
(MVS) as a selection criterion(MVS) as a selection criterion
National Transfer Accounts
1. Definition of a metric1. Definition of a metric
n
ttt FA
nMSEErrorSquareMean
1
21)(
n
ttt FA
nMAEErrorAbsoluteMean
1
1)(
n
t t
tt
AFA
nMAPEErrorPercentageAverageMean
1
1)(
National Transfer Accounts
2. Specification of span 2. Specification of span OptionsOptions Consider several span options from a Consider several span options from a
list previously specified by the user. list previously specified by the user. For example, the list {0.035, 0.04, For example, the list {0.035, 0.04, 0.045, 0.05, 0.055, 0.06, 0.065, 0.07, 0.045, 0.05, 0.055, 0.06, 0.065, 0.07, 0.075, 0.08, 0.09, 0.1, ‘cv’}0.075, 0.08, 0.09, 0.1, ‘cv’}
Users are free to choose the range Users are free to choose the range that they consider more appropriate that they consider more appropriate for their particular situationfor their particular situation
National Transfer Accounts
3. Compute the metric(s) 3. Compute the metric(s) defined in 1.defined in 1.► Compute the MSE, MAE and MAPE for Compute the MSE, MAE and MAPE for
every span in the list specified in 2.every span in the list specified in 2.
National Transfer Accounts
4. Use of MVS as a selection 4. Use of MVS as a selection criterioncriterion
National Transfer Accounts
ImplementationImplementationR code: R code: ******SUPSMU*************SUPSMU*******aloha<-read.csv("earnings.csv", header=T)aloha<-read.csv("earnings.csv", header=T)t1<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.035")t1<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.035")t2<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.04")t2<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.04")t3<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.045")t3<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.045")t4<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.05")t4<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.05")t5<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.055")t5<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.055")t6<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.06")t6<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.06")t7<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.065")t7<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.065")t8<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.07")t8<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.07")t9<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.075")t9<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.075")t10<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.08")t10<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.08")t11<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.09")t11<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.09")t12<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.1")t12<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.1")t13<-supsmu(aloha$age, aloha$income, aloha$sample, span = "cv")t13<-supsmu(aloha$age, aloha$income, aloha$sample, span = "cv")write.csv(c(t1,t2,t3,t4,t5,t6,t7,t8, t9,t10,t11,t12,t13), "smoothed_earnings")write.csv(c(t1,t2,t3,t4,t5,t6,t7,t8, t9,t10,t11,t12,t13), "smoothed_earnings")
National Transfer Accounts
ImplementationImplementationExcel Template: Excel Template:
National Transfer Accounts
DiscussionDiscussion► Ad hoc is used to break ties where MVS Ad hoc is used to break ties where MVS
cannot be appliedcannot be applied► This method could be effective in practice This method could be effective in practice
for some profilesfor some profiles► A lot of time is saved by (semi) A lot of time is saved by (semi)
automatically selecting the span in this wayautomatically selecting the span in this way► This method might be no appropriate when This method might be no appropriate when
the researcher cares more about some parts the researcher cares more about some parts of the age profile than others or when of the age profile than others or when profiles present a lot of sharp elbows profiles present a lot of sharp elbows
National Transfer Accounts
Example 1Example 1
National Transfer Accounts
Example 2Example 2
National Transfer Accounts
WikiWiki http://www.schemearts.com/proj/nta/web/nta/show/Documents/Smoothing/Friedman's%20SuperSmoother/Special%20Notes%20on%20Span