nonresponse rates and nonresponse bias in surveys
DESCRIPTION
Nonresponse Rates and Nonresponse Bias In Surveys. Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA Emilia Peytcheva University of Michigan, USA. - PowerPoint PPT PresentationTRANSCRIPT
Nonresponse Rates and Nonresponse Bias In Surveys
Robert M. GrovesUniversity of Michigan and Joint Program in Survey Methodology, USA
Emilia PeytchevaUniversity of Michigan, USA
Funding from the Methodology, Measurement, and Statistics Program of the US National Science Foundation, Grant 0297435
Four Mutually-Problematic Observations
1. With 100% response rates probability sampling offers an inferential paradigm with measurable uncertainties for unbiased estimates
2. Response rates are declining3. Keeter et al. (2000), Curtin et al. (2000),
Merkle and Edelman (2002) show no nonresponse bias associated with varying nonresponse rates
4. Practitioners are urged to achieve high response rates
Result: Confusion among practitioners
Assembly of Prior Studies of Nonresponse Bias
• Search of peer-reviewed and other publications• 47 articles reporting 59 studies • About 959 separate estimates (566
percentages)– mean nonresponse rate is 36%– mean bias is 8% of the full sample estimate
• We treat this as 959 observations, weighted by sample sizes, multiply-imputed for item missing data, standard errors reflecting clustering into 59 studies and imputation variance
Percentage Absolute Relative Bias
mean sample full unadjusted the is y
mean respondent unadjusted the is y where
n
r
n
nr
y
yy )(*100
Percentage Absolute Relative Nonresponse Bias by Nonresponse
Rate for 959 Estimates from 59 Studies
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80Nonresponse Rate
Per
cen
tag
e A
bso
lute
Rel
ativ
e B
ias
Conclusions from 959 Estimates
• Examples of large nonresponse bias exist
• Variation in nonresponse bias lies mostly among estimates within the same survey
• The nonresponse rate by itself is not a good predictor of nonresponse bias
• [Note: We cannot infer from the scatterplot about what would happen within a study if response rates were increased]
Thinking Causally About Nonresponse Rates and Nonresponse Error
• Key scientific question concerns mechanisms of response propensity that create covariance with survey variable
where is the covariance between the survey
variable, y, and the response propensity, p
• What mechanisms produce the covariance?
pyyE yp
nr
)(
yp
Alternative Causal Models for Studies of Nonresponse Rates and Nonresponse Bias
2. Common Cause Model
Z
P Y
4. Nonresponse-MeasurementError Model
P
Z
P Y
X
1. Separate Causes Model
Y
Y P
3. Survey VariableCause Model
5. Nonresponse Error Attenuation Model
Y P
Y* Y*
Types of Hypotheses about Influences on Nonresponse Error
• Influences on response rates– urbanicity, gender, age– topic of survey, population’s interest in topic– mode of data collection– prenotification, incentives
• Sponsorship– prior involvement of population with sponsor– government vs. other sponsor
• Types of measures– attitudinal vs. behavioral– questions related to topic of survey vs. others
• Type of statistic– means on counts/continuous variables vs. percentages– differences of subclass means
Types of Hypotheses about Influences on Nonresponse Error
• Influences on response rates– urbanicity, gender, age– topic of survey, population’s interest in topic– mode of data collection– prenotification, incentives
• Sponsorship– prior involvement of population with sponsor– government vs. other sponsor
• Types of measures– attitudinal vs. behavioral– questions related to topic of survey vs. others
• Type of statistic– mean vs. percentages– differences of subclass means
Attributes ofSurveys
Attributes ofEstimates
Exploratory Analysis in Two Steps
Examine only estimates that are percentages, using standardized values of the percentages
• Step 1: pooling all 566 estimates, examine
, difference of respondent and nonrespondent means
• Step 2: separating the estimates by their survey’s response rate, examine
, nonresponse bias
mr yy
)( nr yy
Respondent-Nonrespondent Functions on Standardized Percentage Estimates by
Type of Population
0
0.05
0.1
0.15
0.2
0.25
Specific General
Pro
po
rtio
n o
f S
tan
dar
d D
evia
tio
n
)( mr yy
diff=.075ste=.033
Respondent-Nonrespondent Functions on Standardized Percentage Estimates by
Type of Population
0
0.05
0.1
0.15
0.2
0.25
Specific General
Pro
po
rtio
n o
f S
tan
dar
d D
evia
tio
n
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
<23% 23-38% 39%+
Specific General
)( mr yy )( nr yy
diff=.075 diff= .013 .0013 .045ste=.033 ste= .0092 .0058 .020
Respondent-Nonrespondent Functions on Standardized Percentage Estimates by
Mode
0
0.05
0.1
0.15
0.2
0.25
Self-Administered Interviewer-Administered
Pro
po
rtio
n o
f S
tan
dar
d D
evia
tio
n
00.010.020.030.04
0.050.060.070.080.09
<23% 23-38% 39%+
Self-Administered
Interviewer-Administered
)( mr yy )( nr yy
diff=.040 diff= .00091 .0012 .025ste=.024 ste= .0011 .0055 .011
Respondent-Nonrespondent Functions on Standardized Percentage Estimates by
Involvement with Sponsor
0
0.05
0.1
0.15
0.2
0.25
Involvement withSponsor
No Involvement withSponsor
Pro
po
rtio
n o
f S
tan
dar
d D
evia
tio
n
00.010.020.030.040.050.060.070.080.09
<23% 23-38% 39%+
Involvement with Sponsor
No involvement with Sponsor
)( mr yy )( nr yy
diff=.052 diff= .017 .0014 .029ste=.022 ste= .0079 .0063 .010
Respondent-Nonrespondent Functions on Standardized Percentage Estimates by
Type of Measure
0
0.05
0.1
0.15
0.2
0.25
Behavioral Attitudinal
Pro
po
rtio
n o
f S
tan
dar
d D
evia
tio
n
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
<23% 23-38% 39%+
Behavioral Attitudinal
)( mr yy )( nr yy
diff=.14 diff= .084 ---- .070ste=.022 ste= .0074 ---- .016
Respondent-Nonrespondent Functions on Standardized Percentage Estimates by
Statistic’s Relevance to Topic
0
0.05
0.1
0.15
0.2
0.25
Not Relevant Relevant
Pro
po
rtio
n o
f S
tan
dar
d D
evia
tio
n
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
<23% 23-38% 39%+
Not Relevant Relevant
)( mr yy )( nr yy
diff=.006 diff= .000 -.0014 .014ste=.023 ste= .0074 .0091 .025
Respondent-Nonrespondent Functions on All Estimates by Type of Estimator
0
1
2
3
4
5
6
7
Mean Percentage
No
nre
spo
nse
bia
s
0
0.5
1
1.5
2
2.5
3
3.5
4
<23% 23-38% 39%+
Mean Percentage
)( mr yy )( nr yy
diff=4.37 diff= 1.07 1.38 2.14ste=0.72 ste= 0.38 0.39 0.51
Do Differences of Subclass Means have Lower Nonresponse Bias?
• When estimating subclass differences, we hope that nonresponse biases of the two estimates cancel
• 120 reported estimates of subclass means and their differences
• Only 45 of them have bias of the differences of subclass means lower than average bias of the two subclass means– this comports with only 45 having two subclass
means with biases of the same sign
0
2
4
6
8
10
12
14
16
0 2 4 6 8 10 12 14 16
Absolute Value of Bias of Subclass Mean
Ab
solu
te V
alu
e o
f B
ias
of
Dif
fere
nce
of
Tw
o
Su
bcl
ass
Mea
ns
Absolute Value of Bias of Difference of Subclass Mean by Absolute Value of Subclass Mean
Types of Hypotheses about Influences on Nonresponse Error
• Influences on response rates– urbanicity, gender, age– topic of survey, population’s interest in topic– mode of data collection– prenotification, incentives
• Sponsorship– prior involvement of population with sponsor– government vs. other sponsor
• Types of measures– attitudinal vs. behavioral– questions related to topic of survey vs. others
• Type of statistic– mean vs. percentages– differences of subclass means
Five Summary Statements1. Large nonresponse biases exist2. Most variation in nonresponse biases lie among
estimates in the same survey3. Some types of surveys are more susceptible to biases
(e.g., interviewer-administered, studies of population without prior involvement with the sponsor, general population surveys)
4. Some types of estimates seem more susceptible to biases (e.g., measures of attitudes, percentages, (maybe) estimates related to survey topic)
5. Differences of subclass means do not tend to have lower biases that the individual subclass means
Note: preliminary and exploratory analysis; there is much more to do