survey at the crossroads: implementing electronic data
TRANSCRIPT
![Page 1: Survey at the crossroads: implementing electronic data](https://reader031.vdocument.in/reader031/viewer/2022012213/61df5b8521cad421585ffc8b/html5/thumbnails/1.jpg)
Survey at the crossroads: implementing electronic data collection on a large infrastructure survey in Kenya
3MCII, Chicago
Sarah M. Hughes, Mathematica Policy Research
July 27, 2016
![Page 2: Survey at the crossroads: implementing electronic data](https://reader031.vdocument.in/reader031/viewer/2022012213/61df5b8521cad421585ffc8b/html5/thumbnails/2.jpg)
22
“Collecting good data is one of the most powerful tools to end extreme poverty.”
Jim Yong Kim 2015
source: World Bank, 2015
![Page 3: Survey at the crossroads: implementing electronic data](https://reader031.vdocument.in/reader031/viewer/2022012213/61df5b8521cad421585ffc8b/html5/thumbnails/3.jpg)
33
Kenya Municipal Program baseline study
• World Bank-sponsored 2012-2013• Assessed infrastructure, living conditions and
municipal service delivery• 15 municipalities• Two-stage stratified cluster sampling design• 2,087 enumeration areas• Full electronic listing of ~153,000 households• 30 minute tablet-programmed interviews• 14,581 households
![Page 4: Survey at the crossroads: implementing electronic data](https://reader031.vdocument.in/reader031/viewer/2022012213/61df5b8521cad421585ffc8b/html5/thumbnails/4.jpg)
44
Challenges: outdated sample frames, rapidly urbanizing areas, refugees and
internal displacement
![Page 5: Survey at the crossroads: implementing electronic data](https://reader031.vdocument.in/reader031/viewer/2022012213/61df5b8521cad421585ffc8b/html5/thumbnails/5.jpg)
55
Solutions: paper and digitized maps
Overlay of satellite images for pilotPaper maps for main study
![Page 6: Survey at the crossroads: implementing electronic data](https://reader031.vdocument.in/reader031/viewer/2022012213/61df5b8521cad421585ffc8b/html5/thumbnails/6.jpg)
66
Solutions: GIS and cluster guides
• “List and Go” with overnight sample selection
• Selected cases assigned to interviewer tablets
![Page 7: Survey at the crossroads: implementing electronic data](https://reader031.vdocument.in/reader031/viewer/2022012213/61df5b8521cad421585ffc8b/html5/thumbnails/7.jpg)
77
Challenges: uneven interviewer experience, incentives to cut corners
![Page 8: Survey at the crossroads: implementing electronic data](https://reader031.vdocument.in/reader031/viewer/2022012213/61df5b8521cad421585ffc8b/html5/thumbnails/8.jpg)
8
Solutions: interviewer training
• Administer informed consent correctly
• Listen actively
• Use Show Cards
• Probe when necessary
• Read the question as it is written
• Read with correct voice inflection and tone
• Use question pacing
• Take and use instructions given
• Navigate through the tablet
• Know the subject matter
• Establish eye contact and build rapport
• Manage multiple tasks
• Effectively gain cooperation
• Apply professional interviewer skills
• Assure the respondents of confidentiality of their info
• Effectively manage difficult respondents
• Read interviewer instructions
• Take instruction
Rate interviewers on ability to:
How would you rate trainee’s overall performance?
![Page 9: Survey at the crossroads: implementing electronic data](https://reader031.vdocument.in/reader031/viewer/2022012213/61df5b8521cad421585ffc8b/html5/thumbnails/9.jpg)
99
Solutions: CAPI questionnaire
• Review item non-response rates– Don’t know– Refuse– Not applicable
• Check for unusual filter question patterns
• Check for valid phone numbers
• Check for duplicate phone numbers
• Create consistency checks
• Review duration by interviewer, by supervisor
![Page 10: Survey at the crossroads: implementing electronic data](https://reader031.vdocument.in/reader031/viewer/2022012213/61df5b8521cad421585ffc8b/html5/thumbnails/10.jpg)
1010
Solutions: CAPI sample management
• Track by interviewer:– Location of listed household vs. interviewed household (GIS)– Field result and final result– Contact attempts- time, day, date, fieldwork period, outcome
• Track by supervisor team:– Completes per day– Refusal rates– Location of clustered nonresponse
• Frequent feedback to supervisors and separate data quality team conducting call-backs
![Page 11: Survey at the crossroads: implementing electronic data](https://reader031.vdocument.in/reader031/viewer/2022012213/61df5b8521cad421585ffc8b/html5/thumbnails/11.jpg)
1111
Results
• Listing task: – Efficient use of CAPI for
enumeration and selection– Little time to review contact
results
• Interviewing task: – GPS comparison unwieldy
and too prone to error for use during field
– No variation in item nonresponse
![Page 12: Survey at the crossroads: implementing electronic data](https://reader031.vdocument.in/reader031/viewer/2022012213/61df5b8521cad421585ffc8b/html5/thumbnails/12.jpg)
1212
Results
• Duplicate phone numbers appeared in the response data
• Post data collection review of filters didn’t reveal obvious problems
• Durations were suspicious, but callbacks didn’t reveal falsification
![Page 13: Survey at the crossroads: implementing electronic data](https://reader031.vdocument.in/reader031/viewer/2022012213/61df5b8521cad421585ffc8b/html5/thumbnails/13.jpg)
1313
Mean duration of interview
0
5
10
15
20
25
30
35
40
45
1 2 3 4 5 6 7 8 9 10 11 12
Mea
n Du
ratio
n in
Min
utes
Data collection collapsed into 3-week periods
Mean duration of interview
Kenyan ElectionEnd Phase 1
![Page 14: Survey at the crossroads: implementing electronic data](https://reader031.vdocument.in/reader031/viewer/2022012213/61df5b8521cad421585ffc8b/html5/thumbnails/14.jpg)
1414
Conclusions
• Paradata can be useful for improving fieldwork, but a thorough review requires substantial labor and budget
• “Proof” of unacceptably poor data quality or faked cases very difficult to ascertain
Further research needed:• Need to test whether poor adherence to survey
methods, such as not reading questions precisely as they are written, affects estimates