mode effects in social surveys a mixed-mode experiment linked to the safety monitor ger linden,...

Mode effects in social surveysA mixed-mode experiment linked to the Safety Monitor

Ger Linden, Leanne Houben, Barry Schouten (Statistics Netherlands)

Mode effects in social surveys – taskforce SASU, Feb 18 2

Summary

• The impact of the survey mode• Do we expect mode effects in surveys on safety?• How to deal with mode effects?• How to measure mode effects?• Relevance to SASU and ESS


The impact of the survey mode

Data collection steps:

1. Persons need to be reached (coverage)

2. Persons need to respond (response)

3. Persons need to provide valid answers (measurement)

Representative



Coverage and response are usually measured in terms of coverage rate and response rate. Coverage and response form the “selection-effect”.

ModeCoverage Response Total

Face-to-face 100% 65% 65%

Telephone 70% 65% 45%

Web 85% 30% 25%

Web + paper 100% 35% 35%

Registered phone and web populations are known to be different from overall population.



Differences in measurement as a consequence of response styles per mode, “pure mode-effect” or “measurement effect”.Response styles identified through model of Tourangeau & Rasinski:

InterpretationInformation

retrievalJudgement &

processingReporting

Satisficing

Social desirability


The impact of the survey mode – an example

Items (categories 1 – 5)Estimate phone –

estimate web

Perception of economic development over past year +0.406

Expected economic development for coming year +0.573

Perception of change personal finances over past year +0.364

Expected change personal finances for coming year +0.508

Expected change in consumption: durable goods +0.531

Pilot study Survey on Consumer Satisfaction – web and telephone


Do we expect mode effects in safety surveys?

Literature has shown that selection effects are mode-specific for general demographics and socio-economic background.

Specific selection effects in safety surveys:• Persons interested in the topic• Persons afraid to let unknown persons in the house• Persons that do not pick up unknown phone numbers• Persons that experience feelings of unsafety



Safety Monitor

LFS Health Survey

Consumer Satisfaction

Socially desirable answering X (X) X X

Memory effects (under-reporting) X X X -

Memory effects (over-reporting) X (X) X -

Memory effects (telescoping) X - X -

Non-differentiation X - - X

Random Response X - - X

Acquiescence X (X) - X

Order effects (primacy, recency) X - X X

No opinion X (X) (X) X



Consequences of mode effects: • Impact on comparability of statistics over time • Impact on comparability between publication domains

For example: Young persons prefer self-assisted modes and these modes lead to more satisficing to batteries of questions on perception of safety or victimisation.


How to deal with mode effects?

Adapt methodology:• Data collection strategy and choice of modes• Questionnaire design• Estimation strategy

General rule: avoid contrasts between modes

Effectiveness of methodology depends on selection and pure mode effects, i.e. we need to disentangle the mode effects!


How to measure mode effects?

Experimental design linked to Safety Monitor

Sample = 8000 persons

CAPI CATI CAWI PAPI

CAPI + CATI

Golf 1

Golf 2

Response Nonresponse

Principle: Selection effects and pure mode effects are measured relative to face-to-face interviews



Wave 1:• Random assignment to survey mode• Questionnaire is Safety Monitor with small modifications• Regular data collection strategy



Wave 2:• Almost full sample is observed face-to-face• Questionnaire repeats main wave 1 questions• Additional questions about general survey attitudes, attitudes towards politics and survey design features.

Wave 1 mode effects are measured and disentangled by weighting wave 1 to wave 2.


Analyses and main results

Main result:Size of relative selection effects and pure mode effects of telephone, web and paper for key Safety Monitor, European Social Survey and LFS statistics.

Derived results:• Relation nature of question and type of population to mode effects• Relation between selection and pure mode effect• Impact on latent factors and scales deduced from Safety Monitor• Evaluation of regular estimation strategies

Recommendations for data collection strategy, questionnaire design and estimation strategy


Planning

March to June 2011: data collectionJuly to August 2011: preparation of data analysis filesStarting September 2011: Analyses of experimental data

Collaboration with Utrecht University until 2013


Relevance to EU survey on safety

Safety Monitor contains questionnaire modules that are very closely related to the new EU survey.

Experiment provides useful insight into the expected impact of the survey mode on several safety topics including some topics that will be part of the EU survey.

Suggested extensions of study:• EU funding to detail analyses to EU relevant topics• Conduct similar experiments in a few other EU countries, e.g. through an ESSnet

mode effects in social surveys a mixed-mode experiment linked to the safety monitor ger linden,...

Documents

pure mode effects

consequences of mode

mode experiment

specific selection effects

pure modeeffect

safety surveys

perception of safety

time impact