WP3 Task3

Fieldwork Monitoring

1

Johanna Bristle

1st DASISH Quantitative WorkshopMannheim, 13 Dec 2012

Fieldwork MonitoringPlans and Goals

Outline

�Description of WP3 Task 3

�Components of a fieldwork monitoring system:

1) Tool (data collection, sample management)

2) Data (paradata, indicators)

3) Monitoring (reports, intervention strategies etc)

2

Description WP3.3

�Objectives:

“To build up and computerize a fieldwork monitoring system

for cross-national surveys which can centrally manage their distributed fieldwork, helping to optimize response rates and data quality” (DOW-283646).

�Tasks and Deliverables:

3

�Tasks and Deliverables:

1. D3.6: Design of a standardized SMS (June 2013)

2. Preparing paradata set (June 2013)D3.7: Report on keystroke analysis (Dec 2014)

3. D3.8: Fieldwork monitoring application (Dec 2014)

Survey Agency

SD

CentERdata

CAPI

SMS

FTP

FTP

Dataflow for Monitoring in SHARE

MEA

Report

FTP

4

Interviewer Laptop

…Interviewer

Laptop

SMSTechnical Exports/Flat Files

KeystrokesSurvey Agency

Country Team

Area Coordinator

MEA

Tool: Standardized SMS

�Goal: Having a transportable, standardized Sample

Management System (SMS)

�Starting Point: SHARE‘s SMS (programmed by CentERdata)

�Steps:�Steps:

1. Adapting SHARE‘s SMS for the ESS and other surveys

• Requirement Needs Survey: Input from surveyagencies/country teams

• Requirements from a coordination point of view

2. Input from other decentralized, cross-national surveys

3. Demo: „How an SMS app might look like“

5

SHARE’s SMS

Tasks:

• Manage fieldwork and sample

• Document contact attempts

• Assign eligibility

• Conduct coverscreen

• Start CAPI-interviews

User:

• Interviewer, installed on laptop

• Need training (at TTT and NTS)

Benefits:

• Data on fieldwork is available during fieldwork

• Fieldwork procedure is comparable across all SHARE-countries

Requirement Needs Survey

�Topics:

�Agency procedures: field force, current system of sample management, training

�Technical requirements: availability of equipment(laptop/tablets/smartphones), technical expertise

�Monitoring: quality checks, procedures�Monitoring: quality checks, procedures

�Perspective of the country teams

�Coordination point of view:

�Standardization vs. decentralized autonomy

�Data collection mode: electronically

7

SMS on a tablet advice

�Draft Demo: SMS app programmed by CentERdata

�http://cdata3.uvt.nl/crs

�One ESS-country already uses an �One ESS-country already uses an app for contact protocols (Spain)

8

For discussion later:

What are your experiences on sample management tools in decentralized surveys?

Data: Paradata set

Type of paradata Concepts/Variables Release

SHARE Keystrokes& time stamps

Call records(from SMS)

Length of interview/module/itemTimes of changing an answer

For each contact attempts/Contact-mode, date, time-result

No

No

9

Neighborhoodinformation(interviewerobservation)

-resultReason for refusal

building type, size of city, stairs toentrancePresence of children, evidence fordisability, intercom

Most

Source: Stoop et al. 2010

Data: Paradata set

Type of paradata Concepts/Variables Release

ESS Time stamps

Call records

(contact protocols)

Length of interview

For each contact attempts/Contact-mode, date, time-resultReason for refusal, age and gender

Partial

Yes

10

Neighborhoodinformation

(interviewerobservation)

Reason for refusal, age and genderfor each refusing person

NUTS, vandalism, littering, buildingtype, physical state of building)

Yes

Monitoring: Components

�SHARE:

�Biweekly monitoring reports based on paradata (send tosurvey agencies, country teams, area coordinators)

�Final compliance profiles

�Quality checks (by core team, country teams and surveyagencies)agencies)

�ESS:

�Country teams monitor and report to assigned core teammember on a (bi-)weekly basis, they report to Sally

�Projection of fieldwork based on completed interviews

�Internal quality checks by core team

11

Survey Agency CentERdata MEA

CAPI

SMS

FTPSD

FTP

FTP

ReportMonitoring filesSMS exports

Statistics

Monitoringreport

Monitoring Output in SHARE

12

Interviewer Laptop

SMSTechnical Exports/Flat Files

KeystrokesSurvey Agency

Country Team

Area Coordinator

MEA

…Interviewer

Laptop

SMS exports

What MEA reports

�MEA gives a report on fieldwork progress every other week

�Separate figures for panel & refreshment sample

�Separate figures for HH & individual level

�Total numbers & ratios:

�Contact attempts�Contact attempts

�Contacts

�Cooperation

�Refusals

�Response/Retention

13

System output

� “Statistics“ tab and export (in SD at survey agency)

� Gives (final) household state of assigned hh / interviewer

� Gives latest individual event codes of detected respondents in assignedhh / interviewer

� Monitoring files provided by Centerdata

14

� Additional info on interviewer performance

� SMS exports (excel sheets)

� Contains all information loaded into / updated by / generated by SMS / SD

� Basis for monitoring reports by MEA

Example 1 (Source: SMS)

60

80

100ATBfBnCHCZDEDKEE

% of HH with at least 1 contact attempt

C O

U N

T R

I E

S

0

20

40

Nov11

Nov26

Dec10

Jan0

7Ja

n21

Feb04

Feb18

Mar

04M

ar18

Apr01

Apr15

Apr29

May

13M

ay27

Jun1

0Ju

n24

EEESFRgFRiHUITNLPTSE

Report 7: Data available to MEA by 24 Jun 2011.

C O

U N

T R

I E

S

Example 2 (Source: keystrokes)

60

80

leng

th in

sec

onds

Time needed for Reading out Item SN001

0

20

40

leng

th in

sec

onds

DE Bn NL FRg DK EE ES CH AT CZ PT IT HUexcludes outside valuesReport 2: Data available to MEA by 22 March 2011.

C O U N T R I E S

Survey Agency CentERdata MEA

CAPI

SMS

SD

FTP

FTP

Report

Questions and Comments

FTP

Interviewer Laptop

SMSTechnical Exports/Flat Files

KeystrokesSurvey Agency

Country Team

Area Coordinator

MEA

…Interviewer

Laptop

17

References

�Stoop, Ineke, Hideko Matsuo, Achim Koch and Jaak Billiet (2010): Paradata in the European Social Survey: Studying Nonresponse and Adjusting for Bias. Section on Survey Research Methods – JSM 2010, p.407-421.

18

APPENDIX

19

SHARE‘s Data Output andhow it is used forhow it is used for

Fieldwork Monitoring

20

Monitoring SHARE

Data Sources

MEA:

• Biweekly reports

• Final report on fieldwork

• Interested in country level

• SMS Files• Monitoring Files

(from CentERdata)• Keystrokes

Country Team:

• Quality checks (surveillance of SA)

Survey Agencies (SA):

• Quality checks on interviewers (mostly

personal contact by phone and email)

• Interested in interviewer level

• Monitoring Files (from CentERdata)

• SMS Files (some)• Keystrokes (some)

Gross sample(drawn/preloaded)

Sample Distributor

SMS client/ Agency

Country teamsMEA

- Monitor fieldwork- Produce response/

retention rates- Release data- Consolidated gross

sample- Preload

- Monitor fieldwork- Payment of agency

SMS client/ CAPI on laptop

Interviewer

CentERdata

SMS/CAPI output- SMS data

- Coverscreen- Fieldwork

- Keystroke data- CAPI data

- Manage contacts- Conduct interviews

- Preload

- Build software- Set up SD with gross

samples- Technical support- Process uploaded data- Data extraction- Translation process

- Manage interviewers- Monitoring fieldwork- billing

SMS data

Household level file

• Panel/refresher HH

• Screening Information

• Coverscreen

• Area Information

Respondent level file

• Demographics

• Eligibility

• Info on CAPI-interview

• Info from Coverscreen

• Contact Information• Participation in previous waves• Event Codes (Intw. Completed,

Refusal, etc.)

• Interviewer Identification

Calculations for Monitoring:• Completed Interviews• Refusal Analysis• All Monitoring Files from

CentERdata

Monitoring Files (from CentERdata)

Fieldwork Indicators for Monitoring at MEA:• % of Contacted Households• Active Interviewers• Progress Rate (for Panel)• Participation Rate at a given point in time• Weekday and Time Slot of Contact

Keystroke data

Calculations for Monitoring at MEA:• Length of interview/module/item• Quality check on reading out items

Example monitoring mea with sms files

Refusals

15%

20%

25%

0%

5%

10%

AT Bf Bn CH CZ DE DK ES FR GR IT NL PL SE

no time too old no interest other

Source: MEA Monitoring report for May 15, 2009.

C O U N T R I E S