Marcello D’Orazio (madorazi@istat.it)

UNECE - Work Session on Statistical Data EditingLjubljana, Slovenia, 9-11 May 2011

Statistical Matching and Imputation of Survey Data with the Package “Statmatch” for the Environment

What is Statistical Matching?UNECE

Work Session on Statistical Data Editing

Ljubljana, 9-11 May 2011

Statistical Matching (data fusion o synthetic matching) consists in a series of methods to integrate two or more data sources referred to the same target population.

Basic SM framework:

Y X

source A

X Z

source B

1. X variables are in common

2. Y and Z are NOT jointly observed

3. The chance of observing the same unit in A and B is close to zero

Objectives of Statistical Matching

micro: derive a “synthetic” data-set with X, Y and Z

A filled-in with Z Y X Z

macro: estimation of parameters: correlation coef. ( )

or frequenciesYZ

zZ,yYN

UNECEWork Session on

Statistical Data Editing

Approaches

Objectives SM Parametric Nonparametric Mixed

Macro

Micro

Ljubljana, 9-11 May 2011

“StatMatch” provides R functions to apply some Statistical Matching methods

Generalization and optimization of the code provided with the monograph about SM by D’Orazio et al. (2006).

The first version of StatMatch (version 0.4) released on CRAN (Comprehensive R Archive Network) in 2008.

In the beginning of 2011 the version 1.0.1 has been released; this version present a significant improvement of the functionalities of the previous version (0.8 released in 2009).

http://cran.at.r-project.org/web/packages/StatMatch/index.html

Package available for: MS Windows (32 and 64 bit), Linux, Mac

The package StatMatch for the R environmentUNECE

Work Session on Statistical Data Editing

Ljubljana, 9-11 May 2011

Five main groups of functions:

functions to perform nonparametric SM at micro level by means of hot deck imputation (NND.hotdeck, RANDwNND.hotdeck, rankNND.hotdeck);

a function to perform mixed SM at macro or micro level for continuous variables (mixed.mtc);

functions to integrate data from complex sample surveys through calibration of weights as proposed by Renssen (1998) (harmonize.x and comb.samples);

functions to explore uncertainty on the contingency table YxZ (Frechet.bounds.cat and Fbwidhts.by.x);

other functions to compute distances (gower.dist and maximum.dist), to create the synthetic data set (create.fused), etc.

Functions in StatMatchUNECE

Work Session on Statistical Data Editing

Ljubljana, 9-11 May 2011

NND.hotdeck() nearest neighbour distance hot deck:- many distance functions- imputation classes- constrained or unconstrained

RANDwNND.hotdeck() random hot deck and some variants- random hot deck in classes- random hot deck in “moving” classes- it is possible to use weights

rankNND.hotdeck() nearest neighbour with distance computed on the percentage points ofthe empirical cumulative distributionfunction of X

SM via hot deck imputationUNECE

Work Session on Statistical Data Editing

Ljubljana, 9-11 May 2011

mixed.mtc() mixed SM methods for continuousvariables:

consist in two steps:

1) fits regression models (regression) Y vs. X and Z vs. X

2) fills A with units chosen by means of constrained distance hot deck computed on intermediate and live values of Y and Z

- two methods to estimate regression parameters: (ML and

Moriarity&Scheuren, 2001)- possibility of introducing auxiliary

information about the correlation coef.

between Y and Z

Mixed SM methodsUNECE

Work Session on Statistical Data Editing

Ljubljana, 9-11 May 2011

Renssen’s (1998) approach based on a series of calibration steps of the survey weights of A and B, and if available C (C may contain Y and Z or X, Y and Z)

harmonize.x() harmonizes the joint/marginal

distribution of X variables in A and B

comb.samples() estimates the contingency table Y vs. Z using available auxiliaryinformation in C (when available):- Conditional Independence Assum.- incomplete two way stratification- synthetic two way stratification

SM of data from complex sample surveys UNECE

Work Session on Statistical Data Editing

Ljubljana, 9-11 May 2011

Frechet.bounds.cat() to derive the uncertainty bounds forfrequencies in the contingency table Y vs. Z, starting from the marginal tables X vs. Y and X vs. Z

Fbwidths.by.x() explores how the various possiblesubsets of the X variables contributein reducing the uncertainty on thecells of Y vs. Z

Exploring uncertainty due to SM basic frameworkUNECE

Work Session on Statistical Data Editing

Ljubljana, 9-11 May 2011

Computational Efficiency

Hot deckmethods

StatMatch Function

#Matchvars

#Imp.

class.

Processtime

(secs)

Notes

UNconstrained NND

NND.hotdeck 4 36 1282 dist.fun=”Gower”

Constrained NND

NND.hotdeck 4 36 1446dist.fun=”Gower”constr.alg=”rela

x”

Random hot deck

RANDwNND.hotdeck 4 36 1936dist.fun=”Gower”cut.don="exact“

k=10

Artificial data: A contains 14,000 obs.; about 54,000 obs. in B. PC with CPU Pentium IV 3GHz, 3GB RAM, MS Windows XP Prof. (SP 3; 32bit)

All the functions in StatMatch are based on R code and there are no calls to other external code (compiled C or Fortran):

“Interpreted languages (Matlab, R, Python, Lisp) are fun ... but slow. Compiled languages (machine code, assembly, FORTRAN, C, Java) are fast… but are work (= no fun)” Mizera (2006)

UNECEWork Session on

Statistical Data Editing

Ljubljana, 9-11 May 2011

Warning!

“Although abusing R was not proved to be addictive,

it should be noted that it often leads to harder stuff”

Mizera (2006)

Thank You for Your attention!

Ljubljana, 9-11 May 2011

Some References

D'Orazio, M. (2009). StatMatch: Statistical Matching. R package version 1.0.1.http://CRAN.R-project.org/package=StatMatch

D’Orazio, M., Di Zio, M., and Scanu, M. (2006) Statistical Matching: Theory and Practice. Wiley and Sons, Chichester.

Mizera, I. (2006) “Graphical Exploratory Analysis Using Halfspace Depth”. Presentation at “useR!, The R User Conference 2006”, Wien, 15-17 June 2006.

Moriarity C., Scheuren F. (2001) “Statistical matching: a paradigm for assessing the uncertainty in the procedure”. Journal of Official Statistics, 17, 407–422.

Renssen, R.H. (1998) “Use of Statistical matching techniques in calibration estimation” Survey Methodology, 24, pp. 171-183.

UNECEWork Session on

Statistical Data Editing

Ljubljana, 9-11 May 2011