a strategy for prioritising non-response follow-up to reduce costs without reducing output quality...
Post on 01-Jan-2016
219 Views
Preview:
TRANSCRIPT
A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without
Reducing Output Quality
Gareth James Methodology Directorate
UK Office for National Statistics
Outline of presentation
• Introduction response-chasing in ONS business surveys
• Understanding non-responseeffects, patterns and reasons
• Strategy for response-chasingscoring methods – current investigations and future strategies
2
Introduction
• Non-response … the failure of a business to respond in part or full to a survey. Effect on:
– bias and standard error, – perception of output quality, – business behaviour
Improve response rates by:– better questionnaire design, sample rotation rates, …– response-chasing - necessary, but expensive
• Quality improvements and efficiency targets– effective targeting needed
3
Current practice at ONS
• Use of % targets (mainly counts, occasionally other variables)
• Written reminders to all. Then targeted phone calls, … could lead to enforcement
• Businesses identified as ‘key’ (by survey area) chased intensively first
• After ‘keys’, principle to chase large-employment businesses next
• Methods differ between surveys
4
Current practice at ONS
• Areas for improvement:– Methods for ‘key’ businesses:
make more consistent, transparent, scientific
– Effective use of response-chasing tools– Team structure and knowledge
(Area undergoing restructure)
• Efficiency initiatives– save resources: some changes already implemented– effects being monitored; evaluation needed
5
Efficiency initiatives – removal of second reminders
Stocks Inquiry (50-99 sizeband)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%0
10
20
30
40
50
60
70
80
90
10
0
Days since despatch
Res
pons
e ra
te
With 2nd reminder Without 2nd reminder
6
UNDERSTANDING NON-RESPONSE
Patterns of non-response
• Industrial sector - identified those with lower response rates (e.g. catering, hotels)
• High correlation between industry response rates at early and final results
• Size of business – larger businesses take longer to respond. Chasing strategy ensures responses are received later though
8
Intensive Follow-Up (IFU) exercise
• Dual aims:– to estimate non-response bias (work in progress – see final
paper)– to establish reasons for non-response and (later) cost response-
chasing
• Used the Monthly Inquiry into the Distribution and Services Sector (MIDSS):
– dedicated team for the IFU– contacted c.600 non-responders per month in chosen industries– businesses to receive up to 5 phone calls– reason for initial non-response; nature of call; length of call
9
IFU results – returned data
• c.80% of all businesses selected for IFU returned questionnaire, but
• many businesses returned questionnaire just after deadline – no call needed!
• Only c.60% of those contacted returned questionnaire
10
IFU results – reasons for non-response
Reason for initial non-response
Number who gave a reason
Returned data after IFU calls
Still didn’t return data
after IFU calls
Forgot, missed date
667 77% 23%
Too busy, too low priority
361 67% 33%
Actively decided not to
67 33% 67%
11
BUILDING A RESPONSE-CHASING STRATEGY
Dealing with businesses that don’t respond
• Aim to make response-chasing more efficient
• Create a scoring system to prioritise/categorise non-responders
• Focus on reducing non-response bias
13
Estimation in ONS business surveys
We impute/construct where there is non-response.
Then estimate totals as
where
*y i i
i S
t w y
* if
ˆ if i R
ii NR
y i Sy
y i S
14
Bias in ONS business surveys
• Total potential non-response bias (= total imputation error) given by
• We will concentrate on
(i.e. the absolute error of imputation for each business)
ˆi i ii S
w y y
ˆi i iw y y
15
Scoring - principles
• Reduce imputation error by attempting to predict
(Large value means increased risk if business is imputed – therefore target these)
• May also wish to score to encourage good response behaviour from businesses – e.g. new-to-sample
• Need a system that is easy to use and justify.
ˆ| |i i iw y y
16
Scoring methods
• (McKenzie) Calculate imputation error from previous returns; then rank into deciles: 0, 1, …, 9.
(Smallest – Largest)
New-to-sample or long-term non-responders = 10
Tested on MIDSS in 2001-2; implementation issues
• (Daoust) Calculate weighted contribution to estimates – categorise into 3 groups for follow-up
• New investigations with adapted methods
17
Current investigations in MIDSS
• Predict imputation error in monthly turnover (= y)
– Various predictors available– Rank businesses then group – No imputation score?
Use stratum average.
• Assess actual error against predicted.
1. imputation error(t-1)
2. register turnover
3. ( ,imp.error(t-1),
imp.error(t-2), ...,
reg. turnover ,
reg. employment , ...)
i
i i
i
i
i
POSSIBLE PREDICTORS
w
w
f w
18
Results (5 groups)
ˆR
i i ii S
w y y
Actual
Score Imputation error
4 88
3 8
2 3
1 1
0 << 1
• Percentage of within each priority score group
19
Results
ˆR
i i ii S
w y y
Actual Weighted
prediction
Score Imputation error
Previous imp. error
4 88 73
3 8 12
2 3 10
1 1 3
0 << 1 2
• Percentage of within each priority score group
19
Results
ˆR
i i ii S
w y y
Actual Weighted
prediction
Score Imputation error
Previous imp. error
Register turnover
4 88 73 68
3 8 12 15
2 3 10 8
1 1 3 5
0 << 1 2 4
• Percentage of within each priority score group
19
Results
ˆR
i i ii S
w y y
Actual Weighted
prediction
Unweighted prediction
Score Imputation error
Previous imp. error
Register turnover
Register employment
Register employment
4 88 73 68 42 40
3 8 12 15 20 15
2 3 10 8 11 12
1 1 3 5 9 18
0 << 1 2 4 18 15
• Percentage of within each priority score group
19
Conclusions
• Significant gains available in response chasing
Future plans:• Refinements to scores:
– optimum predictor– individual adjustments (e.g. long-term non-responders)– overall or by separate industry groups?– multivariate surveys
• Dynamic updating of scores• Live testing
20
References
• Daoust, P., (2006), 'Prioritizing Follow-Up of Non-respondents Using Scores for the Canadian Quarterly Survey of Financial Statistics for Enterprises', Conference of European Statisticians
• McKenzie, R., (2000) 'A Framework for Priority Contact of Non Respondents', Proceedings of the Second International Conference of Establishment Surveys
21
top related