Bayesian Methods to Handle Missing Data in
High-Dimensional Data Setsusing Factor Analysis Strategies
Thomas R. BelinUCLA Department of Biostatistics
Juwon SongUniv. of Texas-M.D. Anderson Cancer Center
Jianming Wang Medtronic Inc.
Introduction
General Problem: Incomplete high-dimensional longitudinal data
• A large number of variables
• A modest number of cases
• With missing values
• Initially consider cross-sectional data, then consider longitudinal structure
Multiple imputation
Rationale : Useful framework for representing uncertainty due to missingness
Requires imputations to be “proper”
Advice : include available information to the
fullest extent possible (Rubin 1996 JASA)
- avoid bias in the imputation
- make assumption of “ignorable” missing data
more plausible
Overparameterization concerns
With modest sample size and large number of variables, even a simple model can be overparameterized
Example : 50 variables 5049/2=1225
correlation parameters in multivariate normal model with general covariance matrix
Analysis often proceeds based on arbitrary choice of variables to include or exclude
Alternative modeling strategies
Address inestimable or unstable parameters by :
• deleting variables • using proper prior distribution -ridge prior for multivariate normal
(MVN) model (Schafer 1997 text) • restrictions on covariance matrix (common factors in MVN model)
Factor model for incomplete multivariate normal data
Idea : ignore factors corresponding to small
eigenvalues
Notation:
Y : np data matrix with missing items
Z : nk unobserved factor-score matrix,
where k p
(Yi Zi): iid (p+k)-variate normal distribution
Zi N(0, Ik), i.e., assuming orthogonal factors
Factor model for incomplete multivariate normal data (cont’d)
Model:
Yi = + Zi + i , for i=1, 2, ... , n,
where is 1p mean vector,
is kp factor-loading matrix,
and i N( 0, 2 ),
where 2 = diag(12,2
2,…, p2)
Model fitting
Gibbs sampling : based on assumed factor structure (i.e., k known), draw:
(a) mean vector
(b) factor loadings
(c) uniqueness
(d) factor scores
(e) missing items
Details of model fitting
• Can use weakly informative prior for uniqueness terms j
2 to avoid degenerate variance estimates• Can use either noninformative or weakly
informative priors for means and factor loadings • Used transformations to speed convergence • Multiple modes possible (Rubin and Thayer 1982,
1983 Psychometrika), so simulate multiple chains• Monitor convergence (Gelman and Rubin 1992
Statistical Science)
Simulation evaluations
Evaluate bias, coverage when model is correct, overparameterized, or underparameterized
n p # true factors # assumed factors
100 100 5 5, 10
10 5, 10
500 100 5 5, 10
10 5, 10
Simulation factor structure
Example: Each item loads on one factor
0.8 0.8 0 0 0 0 0 0 0 0
0 0 0.8 0.8 0 0 0 0 0 0
0 0 0 0 0.8 0.8 0 0 0 0
0 0 0 0 0 0 0.8 0.8 0 0
0 0 0 0 0 0 0 0 0.8 0.8
Simulation details
Also considered hypothetical scenario where items load on two factors
200 replications for each combination of simulation conditions
- error standard deviation of 1.5% for 95% coveragePercentage of missing data ranged from 5-25% for each variableThree missing-data mechanisms (MAR where available-case
analysis might do well, MAR where available-case analysis not expected to do well, and non-ignorable where method appropriate under MAR might do well)
Simulation results: Factor model, cross-sectional mean
Factor model performs well when model correct or overparameterized (coverages range from 93% - 97%)
Factor model coverage is below nominal level when model underparameterized (coverages range from 86% - 93%)
Simulation results: Other methods, cross-sectional mean
MVN frequently fails to converge with n=100 without ridge prior
MVN with ridge prior has good coverage (94% - 98%), interval widths typically wider than for factor model (2-16% wider on average, depending on details such as missing data mechanism)
Available-case analysis performs poorly (coverages ranging from 37% - 88%)
Simulation study based on observed covariance matrix
Generate multivariate normal data (200 replicates, SE = 1.5% for 95% coverage statistics) with mean and covariance fixed at published values from Harman (1967) study of 24 psychological tests on 145 school children
Number of factors not known in advanceConsider 4, 5, 7 factors following earlier analysis Also consider 11 factors based on cumulative
variance explained exceeding 80% and desire not to underparameterize model
Simulation results: psychological testing example
Coverage rates: 4-factor model: 93% - 95% 5-factor model: 93% - 96% 7-factor model: 93% - 95% 11-factor model: 93% - 95% MVN model: 94% - 95% Available-case analysis: 12% - 84%Interval widths for MVN model within 5% of factor
model widths, usually within 1%
Application: Emergency room intervention study
Specialized emergency room intervention vs. standard emergency room treatment for 140 female adolescents after suicide attempt
Twenty-seven outcomes measured at baseline, 3, 6, 12, 18 months + many baseline characteristics
Most vars 5-25% missing, some 50-60% missingMain interests: - effectiveness of emergency room intervention - whether baseline psychological impairment is
related to outcomes over time
Factor model for emergency room intervention study
135 variables, including 27 longitudinal outcomes Longitudinal outcomes: measures at different time
points treated as separate variablesAssume 30 factors: - explained about 80% of the variation - simulation analysis: insufficient number of factors can cause serious bias - with 27 longitudinal outcomes, general enough to
allow each longitudinal variable to represent a separate factor
Emergency-room intervention study: evaluations, results
After imputation, related longitudinal outcomes to baseline predictors using SAS PROC MIXED
Compared imputation under factor model with growth-curve imputation strategy developed by Schafer (1997 PAN program)
No substantial differences seen in significance tests for intervention effect
Some sensitivity seen in significance of impairment effect, intervention and impairment interactions
Imputation for longitudinal data
PAN (Schafer, 1997): Using Multivariate Linear Mixed-effect Model (MLMM)
• Appropriate for multivariate longitudinal data or clustered data
• Imputation by multivariate linear mixed-effect model
txm txp pxm txq qxm txm Assume and
i i i i iY X Z
( ) ~ (0, )Vi N ( ) ~ (0, )V
i N
Challenge with MI using PANMI under PAN can be over-parameterized easily• Example: 15 variables collected longitudinally
five times, modeled with 2 random effects in PAN• # of parameters in , random effects:
15*31/2=465• # of parameters in , error terms: 15*16/2=120• Total # of parameters: 585• Parameter reduction seems sensible when number
of cases is modest, e.g. 300
Potential solution to over-parameterization
If those 15 variables feature sizable correlations, they could be viewed as measuring 3-5 underlying factors.
Strategy:• Reduce the dimension of the problem by factor
analysis• Model the estimated factor scores by a MLMM• Factor structure reflects cross-sectional
correlations among variables measured at the same time; MLMM reflects longitudinal correlations
Ordinary factor analysis modelFactor analysis model
where and Because we often assume Also assume that is of full rank(Seber, 1977)
, 1, 2,...,i i iY f i n
~ (0, )i fff N ~ (0, )i N
1/ 2 1/ 2 * *( )T T TYY ff ff ff
~ (0, )if N I
Ordinary factor analysis model (continued)
Identifiability• Solution invariant under orthogonal
transformation
• Common restrictions
which is equivalent to k(k-1)/2 restrictions• Identifiable if
1 * *i i i i iY TT f f
1T Diagonal
21[( ) ( )] 0
2p k p k
Generalizing factor analysis model
• Standardization of factor scores presents challenge for generalizing factor analysis model to longitudinal setting
• Idea: Use “error-in-variables” representation of factor model
Error-in-variables factor model
• Error-in-variables model (Fuller, 1987)
Interpretation: If we partition into and let
, ,
and ,
Then
0 1
0i i iY fI
iY 1
2
i
i
Y
Y
2i i iY f u 1 0 1i i iY f e
ii
i
e
u
1 0 1
2 0i i
i ii i
Y eY f
Y uI
Error-in-variables factor model (continued)
• Covariance matrix of Y is
• The total # of distinct parameters is
which is exactly the same as the ordinary model with the additional k(k-1)/2 restrictions used to avoid indeterminacy
• No additional restrictions necessary
1 1
T
YY ffI I
1 1( ) ( 1) ( 1)
2 2p k k p k k p pk k k
A Longitudinal Factor Analysis model
• Extending Error-in-variables Model to LFA
11 01 11
12
121 02 12
222
0 11
2
0 00
0 00
0 00
i
i
ii
ii i
it
t tit
it
Y
Y IY
YY
Y Y I
YY
IY
1
1
12
22 0 1
i
i
ii
ii i i
it
it
it
e
uf
ef
u f
fe
u
Aspects of LFA model
• The # of factors is the same on each occasion, but the factor loadings and factor scores may change
• No constraints on covariance structure of the
• The unique-component vectors are uncorrelated with the factors both within and across occasions.
• The unique-component errors are uncorrelated within occasion and across occasions
if
Advantages of LFA model
Advantages of this LFA model:• Identifiability problem can easily be handled • Preserves the mean structure and covariance
structure, making the study of elevation change and pattern change simultaneously possible
• Can incorporate linear mixed-effect model structure for longitudinal data
• Can incorporate baseline covariates
Implementation
• Use data augmentation (I-step: linear regressions, P-step: analog to ML for multivariate normal with complete data)
• Assume conjugate forms (normal, inverse Wishart) for prior distributions for parameters, assume relatively diffuse priors that still produce proper posteriors
• Conditional distributions all in closed form
Evaluations
We generated 100 data sets with from a MVN
with mean
and variance
for i=1,2,…,350, p=15 measurements, k=5 factors at t=5 time points, has dimension (15x5)x1=75x1
1 1[( ) ( ) )]T Ti k i k tZ I Z I I
0 1( )RViX
iY
iY
Simulation design
• X incorporates intercept, 3 continuous variables, 1 binary variable and time
• Z allows for random intercepts, slopes•
( reflects small to moderate covariate effects for predicting factor scores and a linear trend in factor scores)
0.3 (0.5) / 2 6
/ 0.5 6rc
Bern for r
c for r
6 5
Simulation design (continued)
(to avoid singular factor loading matrix)
• Missingness introduced using MAR mechanism (a series of binary draws with probabilities depending on observed values)
• ( , and incorporate relative variances, covariance describing unique variance, common variance among factor scores, and variance of random effects
• Simulation SE 95% of coverage statistics with 100 replicates=0.0218, margin of error=0.0427
2( ) 5diagnal 20
( )5rc
if r c
if r c
4
1rc
if r c
if r c
( ) 1/ 6500rc
r c
The mean of , which (averaged across simulation replicates) was missing on 27% of individuals
49Y
AnalysisMethod
M.C.Average
M.C.S.E.
Average 95% Interval length
Actual 95% Coverage
True value 17.074
All data 17.078 0.426 1.677 98%
Available data 18.854 0.530 2.091 7%
5 imputations 17.072 0.567 2.231 96%
Simulation when number of factors is correctly specified
The mean of , a variable which is missing 100% of the time (i.e. a variable not measured at a given time point)
66Y
AnalysisMethod
M.C.Average
M.C.S.E.
Average 95% Interval length
Actual 95%
Coverage
True value 20.8195
All data 20.7955 0.5128 2.0170 94%
Available data
-- -- -- --
5 imputations
20.7678 0.6503 2.5554 95%
Simulation when number of factors is correctly specified
The mean of (average missingness rate=27%)49Y
AnalysisMethod
M.C.Average
M.C.S.E.
Average 95% Interval length
Actual 95% Coverage
True value 17.074
All data 17.078 0.4263 1.677 98%
Available data 18.854 0.5304 2.091 7%
F=5 (true number) 17.072 0.5672 2.231 96%
F=6 17.055 0.4873 1.9153 94%
F=4 17.612 0.5962 2.3429 89%
F=3 17.663 0.6213 2.4410 86%
Simulation when number of factors is incorrectly specified
The mean of , which has a 100% missingness rate66Y
AnalysisMethod
M.C.Average
M. C.S. E.
Average 95% Interval length
Actual 95% Coverage
True value 20.8195
All data 20.7955 0.5128 2.0170 94%
Available data -- -- -- --
F=5(true number)
20.7678 0.6503 2.5554 95%
F=6 20.9565 0.7161 2.8142 94%
F=4 20.6473 1.1139 4.3780 91%
F=3 20.4091 1.2484 4.9060 83%
Simulation when number of factors is incorrectly specified
Example using LFA: oral surgery studyRandomized study of two oral surgery treatments
(MMF, RIF) with longitudinal follow-up of quality-of-life (GOHAI) and psychological outcomes
Hierarchical growth-curve model using WINBUGS:
, if RIF , if MMF
0 1 ( ) ,ij i i ij ijY t t
0 00 01 0 ,i i iS
1 10 11 1i i iS
~(0,) ijNV 20 0~ (0, )i N 2
1 1~ (0, )i N
1iS 0iS
Findings of interest
• Difference in average intercept, average slope between RIF and MMF ( , ) significant under MI (NORM or LFA) analysis, not under available-case analysis
• Different interpretations emerge from MI analysis (RIF starts lower, ends with comparable values)
• Compared to MI using NORM, MI using LFA has 17%-34% narrower interval estimates for parameters
1101
Summary and future research
Summary• Factor-analysis methods provide flexible framework for
addressing incomplete high-dimensional longitudinal data
Ongoing and future research• Rounding continuous to binary imputations• Determining number of factors• Robustness of methods to normality assumption• Can the parameters in LFA be estimated by EM or
related methods?• Comparisons with IVEWare and related methods, hot
deck approaches
Goal
To develop general-purpose multiple imputation procedures appropriate for high-dimensional data sets
• Cross-sectional
• Longitudinal
Simulation missing data mechanisms
M1 (MAR): First 99 variables MCAR, missingness on last variable according to logistic regression on other 99 with normally distributed coefficients
M2 (MAR): First 99 variables MCAR, missingness on last variable according to logistic regression on other variables included in same factor with half-normal distributed coefficients
M3 (nonignorable but “close” to MAR): Missingness on each variable depends on two other variables in overlapping manner
Simulation results: simple regression coefficient
Factor model: coverages 93% - 98% when model correct or overparameterized, 19% - 80% when model underparameterized
MVN model: Frequently fails to converge with non-informative prior, coverages 91% - 99% with ridge prior
Available-case analysis: coverages range from 44% - 100%
Equivalence of two factor analysis models
1 1 12 2
2 2
11 1 2
22
*1 *1
2
***11 1 2
2
****11
( )
( )
( )0
0
i i
i
k
i
k
i
k
i
k
f f
f
f
f
f
One can write:
Incorporating multivariate linear mixed-effect model for factor scores
• Rearrange in a matrix form
Then can be modeled by
txk txm mxk txq qxk txk
We assume that the t rows of are iid
and . Thus
1
2
TiT
ii
Tit
f
ff
f
if
if
i i i i if X Z
i (0, )N
( ) ~ (0, )Vi N
1
2 ~ (( ) , ( ) ( ) ))
i
i V Ti i k i k t
it
f
fN X Z I Z I I
f
Modified LFA with covariates
• Combining the LFA with the linear mixed-effect model, we obtain
11 01 11
12
21 02 12
22
0 11
2
0 00
0 0[(0
0 00
i
i
i
i i i
t tit
it
Y
Y I
Y
Y Y XI
Y
IY
1
1
2
2) ]
i
i
iRV RV
i i i i
it
it
e
u
e
Z u
e
u
Analysis Method
Available Case Analysis
Multiple Imputation Using NORM
Multiple Imputation Using LFA
Estimate Posterior Mean
95% CI Posterior Mean
95% CI Posterior Mean
95% CI
Beta00 28.55 (26.24, 30.92)
29.30 (26.35, 32.33)
28.90 (26.45, 31.20)
Beta01 -0.29 (-4.67, 4.05)
-4.24 (-7.18, -1.44)*
-3.93 (-5.72, -1.95)*
Beta10 7.07 (4.78, 9.24)*
6.15 (1.90, 9.79)*
6.57 (2.24, 9.34)*
Beta11 1.86 (-2.42, 5.96)
2.72 (0.20, 5.38)*
2.69 (0.92, 5.02)*
*p<0.05.
Linear growth curve model estimates: Available-case analysis, MI using NORM, MI using LFA