a comparison of survey reports obtained via standard questionnaire and event history calendar
DESCRIPTION
A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar. Jeff Moore, Jason Fields, Joanne Pascale, Gary Benedetto, Martha Stinson, and Anna Chan U.S. Census Bureau American Association for Public Opinion Research May 14-17, 2009. Overview. Background: - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/1.jpg)
A Comparison of Survey Reports Obtained via Standard
Questionnaire and Event History Calendar
Jeff Moore, Jason Fields, Joanne Pascale,Gary Benedetto, Martha Stinson, and Anna Chan
U.S. Census Bureau
American Association for Public Opinion ResearchMay 14-17, 2009
![Page 2: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/2.jpg)
Overview Background:
- SIPP; SIPP “re-engineering”- event history calendar (EHC) methods
Goals and Design of the SIPP-EHC Field Test
Preliminary Results
Summary / Conclusions / Next Steps
![Page 3: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/3.jpg)
SIPPSurvey of Income and Program Participation
- income/wealth/poverty in the U.S.; program participation dynamics/effects- interviewer-administered; longitudinal- panel length = 3-4 years
Key Design Feature:- 3 interviews/year, 4-month reference pd.
![Page 4: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/4.jpg)
SIPP Re-EngineeringImplement Improvements to SIPP
- reduce costs- reduce R burden- improve processing system- modernize instrument- expand/enhance use of admin records
Key Design Change:- annual interview, 12-month reference
pd., event history calendar methods
![Page 5: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/5.jpg)
EHC InterviewingHuman Memory
- structured/organized- links and associations
EHC Exploits Memory Structure- links between to-be-recalled events
EHC Encourages Active Assistance to Rs- flexible approach to help elicit an
autobiographical “story”
![Page 6: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/6.jpg)
Evaluations of EHC Methods Many EHC vs. “Q-List” Comparisons
- various methods- in general: positive data quality results
BUT, Important Research Gaps- data quality for need-based programs?- extended reference period?
![Page 7: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/7.jpg)
Field Test Goals & DesignBasic Goal:Can an annual EHC interview collect data of comparable quality to standard SIPP?
Basic Design:EHC re-interview of SIPP sample households
![Page 8: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/8.jpg)
Design Details (1)
Sample:SIPP 2004 panel interview cases
- reported on CY-2007 in waves 10-12
EHC re-interview in 2008, about CY-2007
![Page 9: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/9.jpg)
![Page 10: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/10.jpg)
Design Details (2)
SIPP Sample Cases in Two Sites- Illinois (all)- Texas (4 metro areas)
N = 1,096 Wave 10-11-12 Addresses(cooperating wave 11 households)
IL: 487TX: 609
![Page 11: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/11.jpg)
Design Details (3)
EHC Questionnaire- paper-and-pencil- 12-month, CY-2007 reference period- subset of SIPP topics (“domains”)- month-level detail
Sample of Addresses, Not People- post-interview clerical match to SIPP
![Page 12: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/12.jpg)
Design Details (4)
1096 initial sample addresses
Outcomes:
- 935 household interviews (91%)
- 1,922 individual EHC interviews (99%)
- 1,658 EHC Rs matched to SIPP (86%)
FINAL ANALYSIS SAMPLE: 1,620
![Page 13: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/13.jpg)
Primary EvaluationCompare SIPP and EHC Survey Reports
- same people- same time period- same characteristics
Differences Suggest Data Quality Effects
(later: use administrative records for a moredefinitive data quality assessment)
![Page 14: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/14.jpg)
Main Research Questions1. Are responses to Qs about government
programs and other characteristics affected by interview method (SIPP vs. EHC)?
2. Does the effect of interview method vary across calendar months (especially early in the year vs. late in the year)?
![Page 15: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/15.jpg)
Preliminary Results3 Government “Welfare” Programs: Food Stamps
Supplemental Security Income (SSI)Women Infants & Children (WIC)
4 Other Characteristics:MedicareSocial Securityemploymentschool enrollment
![Page 16: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/16.jpg)
Results in ContextAlmost All SIPP and EHC Reports Agree
- all characteristics, all months
- in general: 97-98% likelihood that a respondent’s SIPP and EHC reports will agree
- worst case (employment): 92-94%
Disagreements are RARE EVENTS
![Page 17: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/17.jpg)
Results Summary3 Patterns:
1. EHC = SIPP All Year equivalent data quality
![Page 18: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/18.jpg)
0.0%
2.5%
5.0%
7.5%
10.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
Analysis Summary
- no “main effect” for method
- no significant method difference in any month
![Page 19: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/19.jpg)
WIC (Illinois Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
0.0%
2.5%
5.0%
7.5%
10.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
Analysis Summary
- no “main effect” for method
- no significant method difference in any month
![Page 20: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/20.jpg)
Results Summary3 Patterns:
1. EHC = SIPP All Year SSI; WIC (IL)
2. EHC < SIPP All Yearreduced EHC data quality, but not due to longer recall period
![Page 21: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/21.jpg)
15.0%
17.5%
20.0%
22.5%
25.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
MEDICARE -- % Covered in Each Month of CY2007 According to the SIPP and EHC Reports
Analysis Summary
- significant “main effect” for method
- method difference (SIPP > EHC) is constant across months
![Page 22: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/22.jpg)
SOCIAL SECURITY -- % Covered in Each Month of CY2007 According to the SIPP and EHC Reports
15.0%
17.5%
20.0%
22.5%
25.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
Analysis Summary
- significant “main effect” for method
- method difference (SIPP > EHC) is constant across months
![Page 23: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/23.jpg)
WIC (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
0.0%
2.5%
5.0%
7.5%
10.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
Analysis Summary
- significant “main effect” for method
- method difference (SIPP > EHC) is constant across months
![Page 24: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/24.jpg)
FOOD STAMPS (Illinois Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
0.0%
2.5%
5.0%
7.5%
10.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
Analysis Summary
- significant “main effect” for method
- method difference (SIPP > EHC) is essentially constant across months
![Page 25: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/25.jpg)
Results Summary3 Patterns:
1. EHC = SIPP All Year SSI; WIC (IL)
2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL)
3. EHC < SIPP, Early in the Year OnlyEHC data quality may suffer due
– to longer recall period
![Page 26: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/26.jpg)
FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
0.0%
2.5%
5.0%
7.5%
10.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
Analysis Summary
- no significant “main effect” for method
- BUT significant variation by month --
JAN-MAY: SIPP > EHC
later months: no difference (reversal?)
![Page 27: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/27.jpg)
EMPLOYMENT -- % Working for Each Month of CY2007 According to the SIPP and EHC Reports
57.5%
60.0%
62.5%
65.0%
67.5%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% W
orki
ng
SIPP % w orking
EHC % w orking
Analysis Summary
- significant “main effect” for method (SIPP > EHC)
- BUT significant variation by month --
JAN-AUG (SEP): SIPP > EHC
later months: no difference
![Page 28: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/28.jpg)
5.0%
7.5%
10.0%
12.5%
15.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% E
nrol
led
SIPP % enrolled
EHC % enrolled
SCHOOL ENROLLMENT -- % Enrolled in Each Month of CY2007 According to the SIPP and EHC Reports
Analysis Summary
- no significant “main effect” for method
- BUT significant variation by month
JAN-APR: SIPP > EHC
JUN-JUL: SIPP < EHC
AUG-DEC: no difference
![Page 29: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/29.jpg)
Field Test Overall SummarySuccessful “Proof of Concept”
Overwhelming Finding: SIPP-EHC Agreement
Valuable Lessons to Inform Next Test- larger, broader sample- “correct” timing of field period- automated questionnaire
Specific Data Comparisons are Instructive
![Page 30: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/30.jpg)
Results ImplicationsPattern 1. EHC = SIPP All Year
SSI; WIC (IL)
No evident problems; no reason for concern about data quality in a 12-month EHC interview
![Page 31: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/31.jpg)
Results ImplicationsPattern 2. EHC < SIPP All Year
Medicare; Social Security; WIC (TX); Food Stamps (IL)
Problems with data quality in the EHC treatment, but probably not due to recall length
- less effective screening questions (no D.I.; fewer probes; no local labels)- different definitions
Likely fixes in CAPI
![Page 32: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/32.jpg)
Results ImplicationsPattern 3. EHC < SIPP, Early in the Year Only
Food Stamps (TX); employment;school enrollment
Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year
Additional research:- why these characteristics?- understand Field Test time lag effects
![Page 33: A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar](https://reader035.vdocument.in/reader035/viewer/2022062816/568156de550346895dc484b5/html5/thumbnails/33.jpg)
.