experiences in implementing a responsive collection design … experiences in implementing a... ·...
TRANSCRIPT
Éric Joyal
System Team Leader – CAI Development
Statistics Canada
IBUC 2015, April 2015
Experiences in Implementing
a Responsive Collection
Design (RCD) for Blaise CATI
Social Surveys
2015-04-27 1 Statistics Canada • Statistique Canada
Overview
Introduction
Pre-Responsive Design Tools
Overall Responsive Strategy
Highlights
Lessons Learned
2015-04-27 2 Statistics Canada • Statistique Canada
Introduction
Harder to reach household/respondents
• Downward trend in response rates
• Increase in number of attempts
• Increase in no-contact rates
• Increase in tracing rates for longitudinal
and targeted respondent surveys
2015-04-27 3 Statistics Canada • Statistique Canada
Introduction
Need to find more efficient ways for managing
collection
• Researches and Analysis
Paradata
Collection cycle activity
• Improvement Strategies
2015-04-27 Statistics Canada • Statistique Canada 4
Pre-Responsive Design Tools
Cap on Calls
Time slices Strategies
Stratum Prioritization
Etc.
Need for a plan...
2015-04-27 5 Statistics Canada • Statistique Canada
Responsive Collection Design (RCD)
Responsive Design (RD) is an adaptive approach that uses the information available prior to and during data collection to adjust collection strategy for the remaining in-progress cases
2015-04-27 6 Statistics Canada • Statistique Canada
• Opportunity to control quality, cost and
productivity, based on responding potential
of in-progress cases, survey mode and
interaction between surveys
RCD History
Two experiments 2009-2010
• Households and the Environment Survey (HES)
• Survey of Labour and Income Dynamics (SLID)
Current projects
• Canadian Communities Health Survey (CCHS)
• General Social Survey (GSS)
• Households and the Environment Survey (HES)
• Canadian Tobacco, Alcohol and Drugs Survey (CTADS)
• Etc.
2015-04-27 7 Statistics Canada • Statistique Canada
RCD Strategy
Planning – (Pre-collection)
Initial Data Collection Phase
Responsive Design phase-in 1
Responsive Design phase-in 2
2015-04-27 8 Statistics Canada • Statistique Canada
Regular Management Special Survey Specific
(Z Groups)
INTER_E
INTER_F
SI_IP
PM
REFUS_E
REFUS_F
TRA_DEP_E
TRA_DEP_F
Z01
Z02
Z03
...
Z98
Z99
Overall RCD Strategy – Planning
Analysis of previous data collection cycle
Data collection phases and strategies
(Including staffing plans and propensity
model)
Active management tools and reports
New key indicators and communication plan
2015-04-27 9 Statistics Canada • Statistique Canada
Improvement Strategies
Time Slices Adaptive Strategies
• Based on historical data
Survey of Labour and Income Dynamics (SLID)
National Population Health Survey (NPHS)
2015-04-27 10 Statistics Canada • Statistique Canada
2015-04-27 Statistics Canada • Statistique Canada 11
Improvement Strategies
Time Slices Reactive Strategy
• Based on number of attempts
• Based on household composition
Monitor key indicators to identify start of RCD
phase-in 1
Response rate
Productivity
Cost (proportion of budget spent)
Responding potential of in-progress cases
Use strategic improvement opportunities
previously identified (e.g. new time slice strategy
and intermediate cap)
Overall RCD Strategy - Initial data
collection phase
2015-04-27 12 Statistics Canada • Statistique Canada
RCD Dashboard
2015-04-27 13 Statistics Canada • Statistique Canada
ROResponse
Rate
Cond
1Initial Current
Cond
2
% of
Budget
payroll
hours
Cond
3
% of
Budget
System
Time
Cond
4
% of regular
In-progress
cases
Cond
5
Avg. # of calls
for regular in-
progress
cases / cap
Cond
6
Sum of
conditions
EDM 52.2% 1 64.3% 41.6% 1 56.6% 1 62.4% 1 28.3% 0 7.2 0 4
HFX 52.6% 1 63.6% 45.5% 1 62.8% 1 69.8% 1 27.0% 0 7.7 0 4
SHER 46.3% 0 66.9% 58.3% 1 49.8% 0 50.7% 0 34.5% 0 3.4 0 1
STURG 64.2% 1 74.4% 46.6% 1 67.2% 1 65.6% 1 13.9% 1 11.3 1 6
TOR 52.6% 1 65.7% 53.6% 1 58.0% 1 55.4% 1 27.9% 0 6.8 0 4
WIN 62.4% 1 69.9% 46.8% 1 69.8% 1 69.0% 1 12.5% 1 13.0 1 6
Response Rate Budget and Cost Propensity of In-Progress CasesProductivity
(Average over last 5
• Decision based on survey progress in terms of response
rate, productivity, proportion of budget spent (cost) and
responding potential of in-progress sample
Overall RCD Strategy – Phase-in 1
Categorize and prioritize cases to improve overall
response rates
Probability of completion (propensity) – logistic regression model
(sampling frame and sequence of calls information)
2015-04-27 14 Statistics Canada • Statistique Canada
Monitor key indicators to identify start of RD phase-
in 2
Overall RCD Strategy – Phase-in 2
Prioritize cases to improve sample
representativity
• Priority to domain of interest with lower
response rates
• National versus regional objectives
2015-04-27 15 Statistics Canada • Statistique Canada
RCD Strategy
2015-04-27 16 Statistics Canada • Statistique Canada
Planning
Phase
RCD Phase 2 - to
improve
representativeness
No contact (after first 5 calls)
All No Contact High Priority
Sample Regular
cases High Probability
Special Cases (e.g. refusal, tracing)
Miscellaneous Miscellaneous (All
other in-progress and
special cases)
Initial Collection Phase RCD Phase 1 - to improve
response rate
Intermediate cap on calls
Highlights
Productivity higher for RCD
• Less time spent for same result
Sample representativity improved
More homogenous distribution of calls
• Less cases reaches the Cap on calls
2015-04-27 17 Statistics Canada • Statistique Canada
Lessons Learned
RCD concepts and technical issues
Communication – Active Management
• Required on-going and well planned
communication
2015-04-27 18 Statistics Canada • Statistique Canada
Lessons Learned
Staff
• Interviewer staffing required guidelines
• New analytical skills required
2015-04-27 19 Statistics Canada • Statistique Canada
Limited impact
Questions
2015-04-27 20 Statistics Canada • Statistique Canada