issues to be investigated - stanford university
TRANSCRIPT
![Page 1: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/1.jpg)
Issues to be Investigated
• What response rates are being achieved in thebest surveys today by the news media andgovernment contractors?
• When lower response rates occur, why?
• What can be done to elevate response rates?
• How accurate are the best surveys in terms ofunweighted demographics?
• How much can demographic accuracy beincreased by increasing response rates?
![Page 2: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/2.jpg)
Outline of Today’s Presentation
Part I: What Response Rates Are Being Obtained?
Part II: Are Response Rates Related to Survey Characteristics
Part III: Are Response Rates Related to the Demographic Representativeness of National Samples?
![Page 3: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/3.jpg)
Our Data Collection
• We contacted 11 top organizations– News media
– Government contractors
• Requested information:– National and state-wide surveys
– General population, RDD
– Frequencies for AAPOR disposition codes
– Unweighted demographic frequencies
– Information about survey administration
![Page 4: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/4.jpg)
Data Received
• 60 surveys
• 53 suitable for AAPOR response rate calculator– 4 surveys - special populations– 1 survey - disposition information not provided– 2 surveys - list samples
![Page 5: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/5.jpg)
• Organization– 56% from media organizations
– 44% from government contractors
• Geographic area– National: 55.5%
– One state: 22.2%
– Region: 18.5%
– City: 1.9%
![Page 6: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/6.jpg)
Calculating Response Rates Numerator
Completes Completes + PartialsDenominator
Known eligible + Unknown eligibility RR1 RR2
Known eligible + e(Unknown eligibility) RR3 RR4
Known eligible RR5 RR6
![Page 7: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/7.jpg)
Estimating e
• AAPOR Response Rate Calculator:
Known eligible
Known eligible + Known ineligible
• A conservative estimate of e
• Keeter et al.: .2
![Page 8: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/8.jpg)
Response Rates (National Surveys)
Media Government Contractors
Min Max Mean Min Max Mean
RR1 .04 .37 .17 .19 .64 .36
RR2 .04 .38 .17 .20 .53 .36
RR3 .05 .39 .22 .28 .70 .46
RR4 .05 .40 .22 .29 .70 .46
RR5 .09 .51 .32 .38 .77 .60
RR6 .09 .51 .33 .39 .87 .62______________________________________________________________________________________________
e .26 .83 .52 .32 .48 .42______________________________________________________________________________________________
Media N=20; Government Contractors N=7
![Page 9: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/9.jpg)
RR3 (Media) (National)
0
1
2
3
4
5
6
7
0-
.025
.076-
.100
.151-
.175
.226-
.250
.301-
.325
.376-
.400
.451-
.475
.526-
.550
.601-
.625
.676-
.700
.751-
.775
.826-
.850
.901-
.925
.976-
1.00
Response Rate
Fre
qu
en
cy
![Page 10: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/10.jpg)
RR3 (Government Contractors) (National)
0
1
2
3
4
5
6
7
0-.10 .11-.20
.21-.30
.31-.40
.41-.50
.51-.60
.61-.70
.71-.80
.81-.90
.91-1.00
Response Rates
Fre
qu
ency
![Page 11: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/11.jpg)
Estimating Contact Rates• Contact rate 1:
Successful contacts (interviews and noninterviews) Successful + Non-contacted + Unknown eligibility contacts households
• Contact rate 2:
Successful contacts (interviews and noninterviews) Successful + Non-contacted + e(Unknown eligibility) contacts households
• Contact rate 3:Successful contacts (interviews and noninterviews)
Successful + Non-contacted contacts households
![Page 12: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/12.jpg)
Estimating Cooperation Rates• Cooperation rate 1: Completed interviews Successful contacts• Cooperation rate 2: Completed interviews + Partials Successful contacts• Cooperation rate 3: Completed interviews Completed interviews + Partials + Refusals• Cooperation rate 4: Completed interviews + Partials Completed interviews + Partials + Refusals
![Page 13: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/13.jpg)
________________________________________________
Media Government Contractors
Min Max Mean Min Max Mean
Contact .40 .79 .59 .55 .90 .70
Rate #2
Cooperation .11 .57 .37 .46 .85 .64
Rate #2
________________________________________________
Media N=20; Government Contractors N=7
Contact and Cooperation Rates (National)
![Page 14: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/14.jpg)
Contact Rate 2 (Media) (National)
0
1
2
3
4
5
0-.025
.076-.100
.151-.175
.226-.250
.301-.325
.376-.400
.451-.475
.526-.550
.601-.625
.676-.700
.751-.775
.826-.850
.901-.925
.976-1.00
Contact Rates
Fre
qu
en
cy
![Page 15: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/15.jpg)
Contact Rate 2 (Government Contractors)(National)
0
1
2
3
0-.10 .11-.20
.21-.30
.31-.40
.41-.50
.51-.60
.61-.70
.71-.80
.81-.90
.91-1.00
Contact Rates
Fre
qu
ency
![Page 16: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/16.jpg)
Cooperation Rate 2 (Media) (National)
0
1
2
3
4
5
6
7
0-.025
.076-.100
.151-.175
.226-.250
.301-.325
.376-.400
.451-.475
.526-.550
.601-.625
.676-.700
.751-.775
.826-.850
.901-.925
.976-1.00
Cooperation Rates
Fre
qu
en
cy
![Page 17: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/17.jpg)
Cooperation Rate 2 (Government Contractors)(National)
0
1
2
3
0-.10 .11-.20
.21-.30
.31-.40
.41-.50
.51-.60
.61-.70
.71-.80
.81-.90
.91-1.00
Cooperation Rates
Fre
qu
ency
![Page 18: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/18.jpg)
Correlations Among Rates(Media and Government Contractors)
(Media)
RR3 COOPR2 CONTR2
RR3
COOPR2 .90**
CONTR2 .37+ .31
N=27 **p<.01 +p<.10
![Page 19: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/19.jpg)
• Field period: 3 to 245 days; mean = 45 days
• Interview length: 3.5 to 34 minutes mean = 17.5 minutes
• Advance letter sent: 9.3%
• Incentives offered: 7.4%
• Refusal conversions: 79.6%
Part II: Survey Characteristics
![Page 20: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/20.jpg)
• Messages left– Never: 72.2%– Sometimes: 9.3%– Always: 1.9%
• Blocks of telephone numbers used– All blocks: 11.1%– All working blocks: 13.0%– Blocks with at least 2 listed numbers: 44.4%– Blocks with at least 3 listed numbers: 13.0%
![Page 21: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/21.jpg)
Correlations of Response Rate 3with Survey Characteristics (All Surveys)
RR3 N
Field period (days) .37** 52
Interview length (minutes) -.49** 43
Advance letter sent .45** 53
Incentive Offered .38** 53
Refusal conversions attempted .02 53
When messages were left .60** 45
Number of listed telephone .28+ 44 numbers in block required for inclusion
![Page 22: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/22.jpg)
Part III: Demographic Representativeness• 21 National General Population Surveys of
Adults 18+– Excluded surveys with oversamples
– Excluded surveys of special populations
– 19 media surveys and 3 government contractor surveys
• Unweighted Demographics vs. CPS Data– Gender
– Age
– Race
– Education
– Household Income
• Compute Average Absolute Difference
![Page 23: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/23.jpg)
Gender
Average Observed
CPS Sampling Error Discrepancy
Male 48.1%
2.9% 1.4%Female 51.9%
Females over-represented in 68% of surveys.
![Page 24: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/24.jpg)
Age Average Observed
CPS Sampling Error Discrepancy
18-25 14.8%26-35 18.6%36-45 21.4% 2.2% 2.5%46-55 18.4%56-65 11.7%66 + 15.1%
18-25 under-represented in 86% of surveys.
![Page 25: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/25.jpg)
Race
Average Observed
CPS Sampling Error Discrepancy
White 82.9%
2.0% 3.1%African- 11.6%American
African-Americans under-represented in 95% of surveys.
![Page 26: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/26.jpg)
Education Average
Observed CPS Sampling Error Discrepancy
Less than high school 16.9%
High school graduate 31.8%
Some college 27.0% 2.2% 5.6%
College graduate 16.4%
Post-graduate degree 7.9%
Less than HS under-represented in 100% of surveys,
Post-graduate over-represented in 100% of surveys.
![Page 27: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/27.jpg)
Household Income
Average Observed
CPS Sampling Error Discrepancy
Less than $30,000 28.1%
$30,000—50,000 21.2% 2.5% 3.7%$50,000—75,000 20.2%
Over $75,000 30.5%
Over $75,000 under-represented in 94% of surveys.
![Page 28: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/28.jpg)
Correlation of Response Rate 3with Demographic Discrepancies
____________________________________________________________________
Demographic Discrepancy in Terms of: _____________________________________________________
Race Age Gender Education Income____________________________________________________________________
RR3 .01 -.80** -.05 -.74** -.16
N 21 22 20 22 18____________________________________________________________________
![Page 29: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/29.jpg)
Average Absolute Age Discrepancy
0
2
4
6
8
10
Low High
Response Rates
Av
era
ge
Ab
so
lute
D
isc
rep
an
cy
![Page 30: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/30.jpg)
Average Absolute Education Discrepancy
0123456789
10
Low High
Response Rates
Ave
rag
e A
bso
lute
D
iscre
pan
cy
![Page 31: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/31.jpg)
Conclusions
• Response Rates varied a great deal– Media: .05 to .51
– Government Contractors: .19 to .87
• Contact Rates: .6 to .7 on average
• Cooperation Rates: .4 to .6 on average
![Page 32: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/32.jpg)
• Response rate strongly correlated withcooperation rate
• Response rate more weakly correlated withcontact rate
What Causes Higher Response Rates?
![Page 33: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/33.jpg)
Higher Response Rates for
• Longer field period
• Shorter interview
• Sending an advance letter
• Offering an incentive
• Leaving messages
• Using blocks with more listed phonenumbers
![Page 34: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/34.jpg)
• Gender and age discrepancies in the rangeof sampling error.
• Slightly larger error for race and incomethan would be expected based on samplingerror only.
• More error for education than samplingerror alone.
![Page 35: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/35.jpg)
Higher Response Rate Yields
• Smaller discrepancies for education and age
• No impact on discrepancies for gender,race, and income
![Page 36: Issues to be Investigated - Stanford University](https://reader031.vdocument.in/reader031/viewer/2022020702/61fb109f2e268c58cd59bd32/html5/thumbnails/36.jpg)
The Bottom Line
• Higher response rates can be achieved byspending considerably more money andtime on data collection.
• Even surveys with relatively low responserates have excellent demographicrepresentativeness.
• Achieving higher response rates will onlyimprove demographic representativenessslightly.