Methods and software for editing and imputation: recent advancements at

Istat M. Di Zio, U. Guarnera, O. Luzi, A.

ManzariISTAT – Italian Statistical Institute

UN/ECE Work Session on Statistical Data Editing

Ottawa, 16-18 May 2005

Outline• Introduction• Editing: Finite Mixture Models for

continuous data• Imputation: Bayesian Networks for

categorical data • Imputation: Quis system for continuous

data• E&I: Data Clustering for improving the

search of donors in the Diesis system

Recent advancements at IstatIn order to reduce waste of resources and to disseminate best practices, efforts were addressed in two directions:–identifying methodological solutions for some common types of errors

–providing survey practitioners with generalized tools in order to facilitate the adoption of new methods and increase the processes standardization

EditingIdentifying systematic unity

measure errors (UME)

A UME occurs when the “true” value of a variable Xj is reported in a wrong scale (e.g. Xj ·C, C=100, C=1,000, and so on)

Finite Mixture Models of Normal Distributions

Probabilistic clustering based on the assumption that observations are from a mixture of a finite number of populations or groups Gg in various proportions g

Given some parametric form for the density function in each group maximum likelihood estimates can be obtained for the unknown parameters

Finite Mixture Models for UME Given q variables X1,.., Xq, the h = 2q possible

clusters (mixture components) correspond to groups of units with different subsets of items affected by UME (error patterns)

Assuming that valid data are normally distributed and using a log scale, each cluster is characterized by a p.d.f. fg(yy;t)MN(g,) , where g is translated by a known vector and is constant for all clusters

Units are assigned to clusters based on their posterior probability g (yi; )

Model diagnostics used to prioritise units for manual

check Atypicality Index: allows to identify outliers

w.r.t. the defined model (e.g. units possibly affected by errors other than the UME)

Classification probabilities g (yi; ) allow to identify possibly misclassified units. They can be directly used to identify misclassifications that are possibly influential on target estimates (significance editing)

Main findingsFinite Mixture Modelling allows multivariate and not hierarchical data analyses. Costs for developing ad hoc procedures are saved

Finite Mixture Modelling produces highly reliable automatic data clustering/error localization

Model diagnostics can be used for reducing editing costs due to manual editing

The approach is robust for moderate departures from normality

The number of model parameters is limited by the model constraints on and

ImputationBayesian Neworks for categorical variables

The first idea of using BNs for imputation is by Thibaudeau and Winkler (2002)

• Let C1….,Cj be a set of categorical variables having each a finite set of mutually exclusive states

• BNs allows to represent graphically and numerically the joint distribution of variables:

– A Bn can be viewed as a Directed Acyclic Graph, and– an inferential engine that allow to perform inferences

on distributions parameters

Graphical representation of BNs

To each variable C with parents Pa (Cj) there is attached a conditional probability P(C|Pa (Cj))

BNs allow to factorize the joint probability distribution P(C1,...,Cj) of so that

P(C1….,Cj)=Πj=1,nP(Cj|Pa(Cj))

BN’s and imputation: method 1

1.Order variables according to their “reliability”

2.Estimate the network conditioned on this order

3.Estimate the conditional probabilities for each node according to (2)

4.Impute each missing item by a random draw from its conditional prob. distribution

BNs and imputation: methods 2/3

In a multivariate context is more convenient to use not only information coming from parents, but also from the children. This can be done by using Markov Blanket (Mb):Mb(X)= Pa(X)+Ch(X)+Pa(X Children)In this case for each node the conditional probabilities are estimated w.r.t. its Mb

Main findings BNs allow to express the joint probability

distributions with a dramatic decrease of parameters to be estimated (reduction of complexity)

BNs may estimate the relationships between variables that are really informative for predicting values

Parametric models like BNs are efficient in terms of preservation of joint distributions

The graphical representation facilitates modelling BN’s and hot deck methods have the same

behaviour only in the case that the hot deck is stratified according to variables explaining exactly the missing mechanism

ImputationQuis system for continuous

variablesQuis (QUick Imputation System) is a SAS generalized tool developed at Istat to impute continuous survey data in a unified environmentGiven a set of variables subject to non response, different methods can be used in a completely integrated way: Regression Imputation via EM algorithm Nearest Neighbour Donor Imputation (NND) Multivariate Predictive Mean Matching (PMM)

Regression imputation via EMIn the context of imputation, the EM algorithm is used for obtaining Maximum Likelihood estimates in presence of missing data for the parameters of the model assumed for the dataAssumptions MAR mechanism Normality

Regression imputation via EM Once ML estimates of parameters have been obtained, missing data can be imputed in two different ways:

directly through expectations of missing values conditional on observed ones (predictive means)

by adding a normal random residual to the predictive means (i.e. drawing values from the conditional distributions of missing values)

Multivariate Predictive Mean Matching (PMM)

Let Y =(Y1,...Yq) be a set of variables subject to non responseML estimates of the parameters of the joint distribution of Y are derived via EM

For each pattern of missing data ymiss, the parameters of the corresponding conditioned distribution are estimated starting from (sweep operator)

For each unit ui the predictive mean based on estimated parameters is computed

For each unit with missing data, imputation is done using the nearest donor w.r.t. the predictive mean

The Mahalanobis distance is adopted to find donors

Data clustering for improving the search for donors in the Diesis

system• The DIESIS system has been developed at ISTAT

for treating the demographic variables of the 2001 Population Census

• Diesis uses both the data driven and the minimum change approach for editing and imputation

• For each failed household, the set of potential donors contains only the nearest passed households

• The adopted distance function is a weighted sum of the distances for each demographic variable over all the individuals within the household

The in use approach for donor search

• For each failed household e, the identification of potential donors should be made by searching within the set of all passed households D

• When D is very large, as in the case of a Census, the computation of the distance between each e and all dD (exhaustive search) could require unacceptable computational time

• The in use sub-optimal search consists in arresting the search before examining the entire set D according to some stopping criteria. This solution does not guarantee the selection of the potential donors having actual minimum distance from e

The new approach for donor search

• In order to reduce the number of passed households to examine, the set of passed households D is preliminarily divided into smaller homogeneous subsets {D1, …, Dn} (D1 …Dn=D,)

• Such subdivision is obtained by solving an unsupervised clustering problem (donor search guided by clustering)

• The search for the potential donors is then conducted, for each failed household e, by examining only the households within the cluster(s) more similar to e

Main findings The donor search guided by clustering reduces computational times preserving the E&I quality obtained by the exhaustive search

The donor search guided by clustering increases the proportion of actual minimum distance donors selected with respect to the sub-optimal search (this is especially useful for households having uncommon structure for which few passed households are generally available)