the effect of survey mode on nonresponse bias and … · 2015. 1. 28. · iab project team s....

The Effect of Survey Mode on Nonresponse Bias

and Measurement Bias. A Validation Approach

Antje Kirchner

Barbara Felderer

AAPOR, Boston

May17th, 2013

2 / 15 Research Interests Study Details Results Conclusions

Research Gap

Only a few web survey mode comparison studies use validation data …

aggregate level: Sanders et al. 2007 (voting behavior); Stephenson/Crête 2011

(voting behavior); Vannieuwenhuyze et al. 2010 (political attitudes and voting)

individual level: McCabe et al. 2002 (alcohol & drug use); Sax et al. 2003 (university

issues); Kreuter et al. 2008 (academic performance); Dillman et al. 2008 (long

distance service); Sakshaug et al. 2010 (academic performance)

And those typically focus on …

specific subpopulations, i.e. limited generalizability.

mean statistics.

only one error source.

and neglect the analyses of costs per error source.


Research Question

1) How do nonresponse bias and measurement bias compare across survey

modes and how do they interact?

Disentangling:

nonresponse bias

measurement bias

total bias

for socio-demographic and substantive variables across modes using

validation data from the sampling frame.

2) How does adjusting for nonresponse (e.g. raking or post-stratification)

affect total bias?

3) What cost savings can be achieved through the use of web surveys

compared to the telephone survey?


Study Details

Survey Data CATI Web

IAB Project Team S. Eckman, B. Felderer, A. Kirchner, F. Kreuter, J. Sakshaug

Project Title “Work and Consumption in Germany”

Stratified SRS Target persons who are either

1) Unemployed

2) Receipt of Welfare Benefits

3) At least two employment spells

Gross Sample N = 12,400 N = 11,838

Completes N = 2,400

(RR1=19.35%)

N = 1,082

(RR1= 9.01%)

Fieldwork 08/2011-10/2011 02/2012-04/2012

Advance Letter Paper-based Paper-based (tel. option)

3€ incentive (Amazon voucher)

reminder 2 weeks later

Questionnaire Employment Histories

Add. Experiments Filter Format

Consent

MTMM etc.

Filter Format

Consent

Incentive Experiment


Study Details

Administrative data from the registers of the Federal Employment

Agency (FEA) comprising employer reports to the social security

system

e.g. to administer benefit claims and payment

includes information on age, sex, education, status of benefit receipt or

income

Assumption of gold standard, i.e. true parameters (means and

distribution), of those information known to be

relevant to payments (pensions, welfare etc.)

accurate and complete


Bias Assessment: Analyses

Administrative Data Survey Data

Population

Sample

Respondents Respondents Item Respondents

Nonresponse Measurement

Bias Bias

Total Bias

respsvyy ,respadmy ,

fsadmy ,


Items under Study

Characteristics Statistics Nonresponse

Bias

Measure-

ment Bias

Total

Bias

Socio-

Demography:

Gender Proportion

Age Mean/

Distribution

Nationality Proportion Measurement inequality of

survey and admin. data Education Proportion

Substantive:

Current Employment (Overall: Other, Marginal and Regular

Employment)

Proportion

Current Receipt of

Unemployment Benefits II (UB II)

Proportion No survey data

Past Receipt of Unemployment

Benefits II (UB II)

Proportion

Labor Income from Current Job Mean/

Distribution


Socio-Demographic Variables


Relative Total Bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

Female

Mean Age (Years)

-60 -40 -20 0 20

Socio-Demography: Relative Total Bias (in %)

CATI Web


Substantive Variables


Relative Total Bias (except for UB II: only preliminary for web!)

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

Regular empl.

Marginal empl.

Mean Inc. (Euro)

Past UB II

-50 0 50

Substantive Items: Relative Total Bias (in %)

CATI Web


Distributions


Relative Total Bias: Income Distributions (only preliminary for web)

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

Mean Inc. (Euro)

Low Inc.

Middle Inc.

High Inc.

-40 -20 0 20 40

Income: Relative Total Bias (in %)

CATI Web

Measurement bias differs significantly (Two-sample Kolmogorov-Smirnov test: CATI p < 0.01; web p < 0.05) LINK


Conclusions

Nonresponse bias:

in telephone mode largely driven by noncontact

larger for web compared to telephone

Measurement error bias:

for sensitive and socially desirable items smaller in web compared to

telephone

Total bias:

Larger total bias in web compared to telephone

Driven mostly by nonresponse bias

Reinforced by (smaller) measurement error


Limitations and Future Work

Limitations:

Measurement equivalence survey and administrative data

New data release in summer

Stratified SRS

Future work:

Inclusion of measures of overall representativity: R-Indicators

Adjustment for Nonresponse and Sample Composition

Pooling the data

Costs per error source LINK

www.iab.de

[email protected]

Thank you.


Bibliography

Dillman, D.A., Phelps, G., Tortora, R., Swift, K., Kohrell, J., Berck, J. & B.L. Messer (2009) Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Social Science Research 38: 1-18.

Duffy, B., Smith, K., Terhanian, G. & J. Bremer (2005) Comparing data from online and face-to-face surveys. International Journal of Market Research 47(6): 615-639.

Kreuter, F., Presser, S. & R. Tourangeau (2008) Social desirability in CATI, IVR, and web surveys. The effects of mode and question sensitivity. Public Opinion Quarterly 72(5): 847-865.

Lugtig, P., Lensvelt-Mulders, G.J.L.M., Frerichs, R. & A. Greven (2011) Estimating nonresponse bias and mode effects in a mixed-mode survey. International Journal of Market Research 53(5): 2-16.

McCabe, S.E., Boyd, C.J., Couper, M.P., Crawford, S. & H. D’Arcy (2002) Mode effects for collecting alcohol and other drug use data: Web and U.S. mail. Journal of Studies on Alcohol 63(3): 755-761.

Sakshaug, J., Yan, T. & R. Tourangeau (2010) Nonresponse error, measurement error, and mode of data collection. Tradeoffs in a multi-mode survey of sensitive and non-sensitive items. Public Opinion Quarterly 74(5): 907-933.

Sanders, D., Clarke, H.D., Stewart, M.C. & P. Whiteley (2007) Does mode matter for modeling political choice? Evidence from the 2005 British election study. Political Analysis 15: 257-285.

Sax, L.J, Gilmartin, S.K. & A.N. Bryant (2003) Assessing response rates and nonresponse bias in web and paper surveys. Research in Higher Education 44(4): 409-432.

Stephenson, L.B. & J. Crête (2011) Studying political behavior: A comparison of internet and telephone surveys. International Journal of Public Opinion Research 23(1): 24-55.

Vannieuwenhuyze, J., Loosveldt, G. & G. Molensberghs (2010) A method for evaluating mode effects in mixed-mode surveys. Public Opinion Quarterly 74(5): 1027-1045.


6 Appendix: Response Rates

Sample CATI Web

Gross Sample 12,400 11,836

Net Sample (w. tel. nb. / correct address)

9,332 10,525

Completes 2,400 1,082

Response Rate (RR1) 19.35 % 9.01 %


Appendix: Stratification

BEH

2 employers

Empl.

last 10 years

LEH (UB I)

since 1/1/2001

LHG (UB II)

spell in 2010


Appendix: Sample Composition

Stratum CATI Web Total

UB I %

3,600 29.03

4,277 36.13

7,876 32.50

UB II %

3,800 30.65

4,218 35.63

8,017 33.08

Empl. %

5,000 40.32

3,343 28.24

8,342 34.42

Total %

12,400 100.00

11,838 100.00

24,235 100.00


What We (Think) We Know

(Validation) Studies* Comparing Web Mode to…

* non-exhaustive overview

** also considering causal inference, not just assessment of proportions

CATI: Computer Assisted Telephone Interview NR: Nonresponse (Bias)

IVR: Interactive Voice Recognition ME: Measurement (Error/Bias)

CAPI: Computer Assisted Personal Interview TE: Total (Survey) Error/Bias

Modes Item NR ME TE

w.

Valid.

Data

Aggregate Level (Frame Data, other External

Data such as Gov. Data, Large

Nat. Surveys, ‘Comparative

Sample’ etc.)

Sanders et al. 2007**

Stephenson/Crête 2011**

Vannieuwenhuyze et al.

2010

CAPI

CATI

CAPI, CATI

Voting Behavior

Voting Behavior

Political Variables (voting etc.)

Individual Level (Frame Data)

McCabe et al. 2002

Sax et al. 2003

Kreuter et al. 2008

Dillman et al. 2009

Sakshaug et al. 2010

Mail

Mail

CATI, IVR

CATI, IVR, Mail

CATI, IVR

Alcohol & Drug Use

University Issues

Academic Performance

Long Distance Service

Academic Performance

w/o.

Valid.

Data

Experimental

Approach (Mode Switch)

Dillman et al. 2008

…

s.a.

…

s.a. s.a.

s.a.

s.a.

Quasi-Exp. Approach/

Stat. Modeling (Prop. Score Matching, Test-

Retest, Weighting etc.)

Lugtig et al. 2011

Duffy et al. 2005

…

CATI

CAPI

…

Environ. knowledge

Voting, Activism, Cholesterol


Administrative data

Nonrespondents: date of last interview

Respondents: date of interview

(web: income and employment data only available of two months

prior to survey, i.e. Dec. 2011)

Standard Errors

Bootstrap: with 5000 reps (for relative bias),

C.I. according to quantiles (0.025, 0.975)

Total Bias

unadjusted for nonresponse

Analyses: Assumptions


Nonresponse Bias

1

Measurement Bias

2

Total Bias

3

Bias Estimates for Mean Statistics

100*)(

)(.

100*)(

)(.

100*)(

)(.

,

,,

,

,,

,

,,

fsadm

fsadmrespsvy

te

respadm

respadmrespsvy

me

fsadm

fsadmrespadm

nr

y

yyybiasrel

y

yyybiasrel

y

yyybiasrel

data frame :admdata;survey :svysample full :fs ;respondent :resp


Implications for Survey Costs

CATI

Web with mailing

Web w/o mailing

Costs per interview (completed)

completed interviews

Costs

pe

r in

terv

iew

(com

ple

ted

)


ncrefnr

ncrefnr

ncrefnr

ncrefnr

ncrefnr

ncrefnr

ncrefnr

Female

Mean Age (Years)

Edu. unknown

Edu. low. sec.

Edu. up. sec.

Edu. tert.

Nationality

-50 0 50 100 150

Socio-Demography: Relative Nonresponse Bias (in %)

CATI Web

4 Results: Relative Nonresponse Bias

(by type of nonresponse: noncontact and refusal)

nc: noncontact

ref: refusal

nr: nonresponse


nc

ref

nr

nc

ref

nr

nc

ref

nr

nc

ref

nr

nc

ref

nr

nc

ref

nr

Employed:

Regular empl.

Marginal empl.

Mean Inc. (Euro)

Currently UB II

Past UB II

-40 -20 0 20 40

Substantive Items: Relative Nonresponse Bias (in %)

CATI Web



nc: noncontact

ref: refusal

nr: nonresponse


nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

Mean Age (Years)

<=30

30-39

40-49

50-59

>=60

-60 -40 -20 0 20 40

Age: Relative Total Bias (in %)

CATI Web



nc: noncontact

ref: refusal

nr: nonresponse


nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

Mean Inc. (Euro)

Low Inc.

Middle Inc.

High Inc.

-40 -20 0 20 40

Income: Relative Total Bias (in %)

CATI Web



nc: noncontact

ref: refusal

nr: nonresponse


Relative Total Bias: Age Distributions

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

nonresponse

meas. bias

tot. bias

Mean Age (Years)

<=30

30-39

40-49

50-59

>=60

-60 -40 -20 0 20 40

Age: Relative Total Bias (in %)

CATI Web


Measurement Bias: Age Distributions

Two-sample Kolmogorov-Smirnov test:

D = 0.000, p-value = 1 D = 0.000, p-value = 1 alternative hypothesis: two-sided alternative hypothesis: two-sided


Measurement Bias: Income Distributions (only preliminary for web)

Two-sample Kolmogorov-Smirnov test:

D = 0.0721, p-value < 0.05 D = 0.0642, p-value < 0.01 alternative hypothesis: two-sided alternative hypothesis: two-sided


Relative Nonresponse Bias


Absolute Nonresponse Bias


Relative Total Bias


Absolute Total Bias


1 The Question

Disentangling

nonresponse bias and measurement bias

comparing web and telephone mode

using validation data,

ideally from the sampling frame

(government data, large nat. surveys, etc.)

the effect of survey mode on nonresponse bias and … · 2015. 1. 28. · iab project team s....

Documents