nonresponse bias correction in telephone surveys using census geocoding: an evaluation of error...
TRANSCRIPT
Nonresponse Bias Correction in Telephone Surveys Using Census Geocoding:
An Evaluation of Error PropertiesPaul Biemer
RTI International and
University of North Carolina
Andy PeytchevRTI International
2
Estimating the Population Mean in an RDD Survey
Respondents {R} Nonrespondents
TOTAL SAMPLE
Nonrespondents {NR}
{ } { } { } { }R R NR NRy p y p y
mean of NRs is unknown
3
Methods for Adjusting and Evaluating for RDD Nonresponse
Very limited information on NRs in RDD surveys Post-stratification adjustments are the norm
Effectiveness at reducing bias is questionable at best Bias is sometimes evaluated using
Maximum followup effort approaches Only evaluates reduction in bias due to slight elevations in response rates
Comparison to external gold standard estimates Limited in scope to a few characteristics
Census block group geocoding Error properties are largely unknown
The focus of this research
4
Census Geocoding (CG) Method
Obtain the addresses for nonrespondents (50-60% “success” rate)
Geocode addresses Link to census aggregate data
Address matched: link unit to census block group (CBG) via geocode
Exchange matched: link to census tract (CT) via telephone number
Substitute the corresponding CBG or CT mean for the nonrespondent’s characteristic
5
Estimating the Population Mean in an RDD Survey
Respondents {R} Nonrespondents
TOTAL SAMPLE
Nonrespondents {NR}
{ } { } { } { }R R NR NRy p y p y
mean of NRs is unknown
6
Impute Nonrespondent Characteristics from Census Aggregate Data
Respondents {R} Nonrespondents
TOTAL SAMPLE
Nonrespondents {NR}
{ } { } { } { }NR RR NRp y py y
Obtained from NRs CBG or CT
7
Questions Addressed by this Research
What is the bias in ?
Is a valid estimate of the bias in the
unadjusted (or post-stratified) estimator of the mean?
Does the CG method provide useful data for modeling
response propensity?
The first two questions will be addressed in today’s
presentation.
y
{ }ˆ
RB y y
y
{ }ˆ
RB y y
8
Decomposition of the Bias in
Respondents Nonrespondents
Correctly matched
addresses
Incorrectly matched
addresses
Correctly matched
exchanges
Incorrectly matched
exchanges
TOTAL SAMPLE
{ }CA { }IA { }CE { }IE
{ }CA { }IA { }CE { }IE
y
Size
Expected Difference
9
{ } { } { }
{ } { } { }
{ } { } { }
{ } { } { }
CA CA CA
IA IA IA
CE CE CE
IE IE IE
y y
y y
y y
y y
{ } { }
{ } { }
{ } { }
{ } { }
( )
( )
( )
( )
CA CA
IA IA
CE CE
IE IE
E p
E p
E p
E p
10
Components of the Bias in
{ } { } { } { } { } { }
{ } { } { } { }
NR NR CA CA IA IA
CE CE IE IE
E y y
{ } { } { }
Bias( ) ( )
NR NR NR
y E y y
E y y
y
where
11
Components of the Bias in
{ } { } { } { } { } { }
{ } { } { } { }
NR NR CA CA IA IA
CE CE IE IE
E y y
{ } { } { }
Bias( ) ( )
NR NR NR
y E y y
E y y
y
wherecorrect CBG match incorrect CBG match
correct CT match incorrect CT match
12
Estimation of the Bias Components
National Comorbidity Survey Replication (NCS-R) National probability sample of 18+ in households Face to face survey with 71% response rate All addresses were geocoded
CG was applied to 8,178 responding hh’s that provided a telephone number (88% of NCS sample)
CG bias components estimated based on 41% response rate (response after 3 callbacks)
Sensitivity analysis based on three response rates: 2 callbacks 26% response rate 3 callbacks 40% response rate 5 callbacks 60% response rate
13
Why is it reasonable to use a face to face survey to evaluate the CG bias in an RDD survey?
The nonresponse mechanism is not a critical factor in the assessment of the CG bias.
A survey with a relatively high response rate is needed to evaluate the bias.
Addresses are known for all sample members and can therefore be geocoded to their correct CGs.
Sensitivity analysis can be performed to assess the effect on CG bias of increasing response rates.
14
Weighted Respondent Mean, True Mean, and CG Imputed Mean for Available Characteristics
Whit
eBlac
kAsia
n
Oth
erHisp
Male
Female
1 pe
rs H
H
2+ p
ers
HH
< 18
HH
Oth
er H
H
Age 1
8–24
Age 2
5–34
Age 3
5–49
Age 5
0–59
Age 6
0+
< $15
,000
$15K
- $3
0K
$30K
–$50
K
$50K
–$75
K
≥ $
75K
0
10
20
30
40
50
60
70
80
Respondent Mean True Mean Imputed Mean
15
Weighted Respondent Mean, True Mean, and CG Imputed Mean for Available Characteristics
Whit
eBlac
kAsia
n
Oth
erHisp
Male
Female
1 pe
rs H
H
2+ p
ers
HH
< 18
HH
Oth
er H
H
Age 1
8–24
Age 2
5–34
Age 3
5–49
Age 5
0–59
Age 6
0+
< $15
,000
$15K
- $3
0K
$30K
–$50
K
$50K
–$75
K
≥ $
75K
0
10
20
30
40
50
60
70
80
Respondent Mean True Mean Imputed Mean
16
Whit
eBlac
kAsia
nOth
erHisp
Male
Female
1 HH
1+ H
H
1+ c
hild
No ch
ild18
-24
25-3
435
-49
50-5
960
+<15
k
15K-3
0K
30k-
50k
50k-
75k
75k+
-50.00%
-40.00%
-30.00%
-20.00%
-10.00%
0.00%
10.00%
20.00%
26% RR
41% RR
60% RR
Demographic Characteristics
by Response RateRelbias( | ) Relbias( | )R Ry y y y
RTI International
17
Average Estimates of for {s} = {CA}, {IA}, {CE}, and {IE}
{ } { }s s
{ } { }s s
Bias Component
{CA} {IA} {CE} {IE}0.00%
0.10%
0.20%
0.30%
0.40%
0.50%
0.60%
0.70%
0.80%
0.90%
1.00%
26% RR
41% RR
60% RR
(Percentage points)
RTI International
18
Average Relative Size of the Bias Components
{ }s { }| |s26% RR 41% RR 60% RR 26% RR 41% RR 60% RR
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
40.00%
45.00%
50.00%
{CA} {IA} {CE}
{IE}
19
Conclusions
Bias in the CG estimates of NR bias is unacceptably large race, age, and income were the most biased
Major source of bias {IE} followed by {CA} (surprisingly) Approximately 75% of the cases fall into these subsets
Correctly matching to CBGs reduces the bias, but minimally Biases tend to build across components rather than netting
out. Increasing the survey response rate reduces bias in the CG
approach; relative importance of each component is stable
20
Next Steps
Further characterize the CG bias by its components Consider the use of CBG and CT information obtain from
the CG method for: modeling of response propensities adjusting for nonresponse bias
