survey participation, nonresponse bias, measurement error bias, … · 2017-06-11 · 6...
TRANSCRIPT
1
Survey Participation, Nonresponse Bias,
Measurement Error Bias, and Total Bias
Kristen OlsonProgram in Survey Methodology
University of MichiganDC-AAPOR Workshop on
Nonresponse Bias in Household SurveysMarch 30, 2007
2
Outline
• Summary of POQ article– Motivation– Main findings– Limitations
• New research related to nonresponse bias
3
Motivation• Long time hypothesis about reluctant respondents or
otherwise difficult respondents giving error-filled answers.
• Mixed evidence about this– Studies based on record checks imply that this is true; studies
based on covariance measures not as clear.
• Most of the studies ignore the reduction of nonresponse error compared to potential increases in measurement error.
• Basic question: Does including reluctant respondents increase the bias of an estimate?
4
Main Findings• The relationship between nonresponse propensity and
nonresponse bias is both statistic-specific and specific to the type of nonresponse.
• The nonresponse / measurement error trade-off is also statistic specific.
• There is a difference between a variable in a data set and a statistic calculated using that variable.– Different functions of a variable may pose different nonresponse
/measurement properties.
• In these statistics, using this propensity model, the overall reduction in bias tended to outweighed a net increase in either error source.
5
Nonresponse bias is statistic specific and varies by type of nonresponse.
Mean Length of Marriage
100
105
110
115
120
125
130
135
140
Truth
Nonco
ntacts
Contac
tedNon
interv
iews
Interv
iewed
Mon
ths
Mean Time Since Divorce
10
15
20
25
30
35
40
45
50
55
Truth
Nonco
ntacts
Contac
ted
Nonint
erview
sInt
erview
ed
Mon
ths
Divorce Date – Marriage Date Start of Field Period - Divorce Date
6
Relationship between nonresponse propensity and nonresponse bias varies across types of nonresponse.
120
125
130
135
140
145
150
High 4 3 2 Low
Contact Propensity Stratum
Cum
ulat
ive
Mea
n Le
ngth
of M
arria
ge
Records Target
120
125
130
135
140
145
150
High 4 3 2 Low
Cooperation Propensity Stratum
Cum
ulat
ive
Mea
n Le
ngth
of M
arria
ge
Records Target
7
Relationship between nonresponse propensity and nonresponse bias varies across statistics.
46
48
50
52
54
56
58
High 4 3 2 Low
Cooperation Propensity Stratum
Cum
ulat
ive
Tim
e S
ince
Div
orce
Records Target
120
125
130
135
140
145
150
High 4 3 2 Low
Cooperation Propensity Stratum
Cum
ulat
ive
Mea
n Le
ngth
of M
arria
ge
Records Target
8
Nonresponse / measurement error properties are statistic-specific.
Mean Length of Marriage
100
105
110
115
120
125
130
135
140
Truth
Nonco
ntacts
Contac
tedNon
interv
iews
Interv
iewed
Report
s
Mon
ths
Mean Time Since Divorce
42
44
46
48
50
52
54
56
58
Truth
Nonco
ntacts
Contac
ted
Nonint
erview
sInt
erview
edRep
orts
Mon
ths
9
Measurement error bias of a statistic increases for some, but not all, statistics.
120
125
130
135
140
145
150
High 4 3 2 Low
Cooperation Propensity Stratum
Cum
ulat
ive
Mea
n Le
ngth
of M
arria
ge
Records Survey reportsTarget
46
48
50
52
54
56
58
High 4 3 2 Low
Cooperation Propensity StratumC
umul
ativ
e Ti
me
Sin
ce D
ivor
ce
Records Survey reportsTarget
10
Statistics sharing variables can differ in their nonresponse / measurement error properties.
Time Since Divorce
0%
20%
40%
60%
80%
100%
High 4 3 2 Low
Cooperation Propensity
Absolute ME Bias Absolute NR Bias
Length of Marriage
0%
20%
40%
60%
80%
100%
High 4 3 2 Low
Cooperation Propensity
Absolute ME Bias Absolute NR Bias
Divorce Date – Marriage Date Start of Field Period - Divorce Date
11
Absolute total bias may be reduced.
Length of Marriage
02468
1012141618
High 4 3 2 Low
Cooperation Propensity
Absolute ME Bias Absolute NR Bias
Time Since Divorce
0
1
2
3
4
5
6
7
8
High 4 3 2 Low
Cooperation Propensity
Absolute ME Bias Absolute NR Bias
12
Main Findings• The relationship between nonresponse propensity and
nonresponse bias is both statistic-specific and specific to the type of nonresponse.
• The nonresponse / measurement error trade-off is also statistic specific.
• There is a difference between a variable in a data set and a statistic calculated using that variable.– Different functions of a variable may pose different nonresponse
/measurement properties.
• In these statistics, using this propensity model, the overall reduction in bias tended to outweighed a net increase in either error source.
13
Limitations
• Specification of the propensity model– Can we do any better than “it depends?”
• Only looked at the survey mean. What about other statistics?
• Measurement error analysis looked at for the “complete cases.” – This confounds a number of processes; these
need to be teased apart to really understand what’s happening at the unit level
14
New Research on Nonresponse Bias
• Examining nonresponse propensity and nonresponse bias in observational studies– What does “it depend” on?
• Who is being asked to participate? • The statistics of interest?• The recruitment process?
– Mechanisms and nonresponse bias of a statistic
– Survey recruitment process and nonresponse bias
15
Can we gain insights into nonresponse bias by speculating on what mechanisms might affect particular statistics?
• Specify proxy indicators for a mechanism: – At home patterns
• Age, gender, kids, education, ecological measures on commuting time, working at home, mode of transportation to work
– Social isolation• Age, gender, age*gender, kids, ecological measures on single
person households, persons living below the poverty line
• Relate to the Y variables– Expect at home patterns to be related to length of marriage
• Age is the common cause: older people are more likely to be at home, and have longer marriages
– Not clear direction for length of marriage and social isolation • Are people who were in longer marriages less isolated because
they were part of a longer union? Are they more isolated becausetheir ties are now broken?
16
Can we gain insights into nonresponse bias by speculating on what mechanisms might affect particular statistics?
No
NoYes
Mixed
Yes
No
Number of marriages
No
NoYesYes
No
Yes
Age at marriage
NoNoNoPositive affect toward sponsor
YesNoYesDiscretionary TimeNoNoNoSocial CohesionYesNoMixedSocial Isolation
Cooperation ModelsNoNoNo
Access impediments
YesNoYesAt home patternsContact Models
Age at divorce
Months since
divorceLength of marriage
17
Can we gain insights into nonresponse bias by speculating on what mechanisms might affect particular statistics? (part 2)
No
NoYes
Mixed
Yes
No
Number of marriages
No
NoYesYes
No
Yes
Age at marriage
NoNoNoPositive affect toward sponsor
YesNoYesDiscretionary TimeNoNoNoSocial CohesionYesNoMixedSocial Isolation
Cooperation ModelsNoNoNo
Access impediments
YesNoYesAt home patternsContact Models
Age at divorce
Months since
divorceLength of marriage
18
New Research on Nonresponse Bias
• Examining nonresponse propensity and nonresponse bias in observational studies– What does “it depend” on?
• Who is being asked to participate? • The statistics of interest?• The recruitment process?
– Mechanisms and nonresponse bias of a statistic
– Survey recruitment process and nonresponse bias
19
Survey recruitment process and nonresponse bias
• Traditional views of survey nonresponse look at nonresponse at the end of a survey.– Deterministic view – nonresponse reflects
classes of respondents and nonrespondents.– Stochastic view – everyone has a probability
of participating in a survey.• Traditional observational examinations of
nonresponse propensity look primarily at respondent or ecological characteristics.
20
Survey recruitment process and nonresponse bias
• We do many things as survey organizations to gain contact and cooperation.– Incentives, mode switches, advance letters,
persuasion letters, call timing, etc.– Everything that we do is designed to change
people’s likelihood of survey participation.
21
Survey recruitment process and nonresponse bias
• Survey participation is a process– To fully understand how and why people participate in
surveys, need to account for the process• What was done to the sample unit, When was it done, Who
was it done to– Where the process ends affects the final adjustment
model.– Where the process ends affects the observed
nonresponse bias.
• We need a new way of thinking about survey participation.
22
Dynamic Response Propensities
• Sample units have a propensity to respond each time they are attempted.
• Response propensity at time t is determined by sample unit characteristics and recruitment protocol decisions at time t and at prior times.– As additional levels of effort are exerted, the sample unit’s
propensity changes.– Each sample unit has a vector of response propensities.
• Call this vector the sample unit’s “dynamic response propensities.”– This is essentially the assumption underlying responsive designs
(Groves and Heeringa, 2006).
23
Dynamic Response Propensities
Interview p(1)
Interview p(2)
Interview p(3)
Interview p(4)
Noninterview q(1)
Noninterview q(2)
Noninterview q(3)
Noninterview q(4)
Call 1
Call 2
Call 3
Call 4
Adapted from Brand and Xie, 2006
24
Implications of Dynamic Response Propensities for Nonresponse Bias
• becomes
• That is, the nonresponse bias of the (unadjusted) mean changes over the life course of a survey.– Not only because the average response propensity is
changing, but because the covariance term is changing
( )r pyBias y pσ=
( )rt p yt tBias y pσ=
25
Estimating Dynamic Response Propensities
• How to estimate?– Could estimate separate logistic regressions
for each call, changing the risk set for each model
– Instead of lots of logistic regressions, make a few proportionality assumptions (which can be tested).
• Use discrete time hazard model
26
Correlation between pt and Length of Marriage
-0.1
-0.05
0
0.05
0.1
0.15
1 2 3 4 5 6 7 8 9 10
Call
Cor
r(p,Y
)
Conditional Correlation
27
Implications for Measurement Error
• Different mechanisms for survey participation vary in their implications for measurement error.
• Different components of the recruitment process may affect measurement error –does it matter when these components are implemented?
