![Page 1: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/1.jpg)
A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without
Reducing Output Quality
Gareth James Methodology Directorate
UK Office for National Statistics
![Page 2: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/2.jpg)
Outline of presentation
• Introduction response-chasing in ONS business surveys
• Understanding non-responseeffects, patterns and reasons
• Strategy for response-chasingscoring methods – current investigations and future strategies
2
![Page 3: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/3.jpg)
Introduction
• Non-response … the failure of a business to respond in part or full to a survey. Effect on:
– bias and standard error, – perception of output quality, – business behaviour
Improve response rates by:– better questionnaire design, sample rotation rates, …– response-chasing - necessary, but expensive
• Quality improvements and efficiency targets– effective targeting needed
3
![Page 4: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/4.jpg)
Current practice at ONS
• Use of % targets (mainly counts, occasionally other variables)
• Written reminders to all. Then targeted phone calls, … could lead to enforcement
• Businesses identified as ‘key’ (by survey area) chased intensively first
• After ‘keys’, principle to chase large-employment businesses next
• Methods differ between surveys
4
![Page 5: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/5.jpg)
Current practice at ONS
• Areas for improvement:– Methods for ‘key’ businesses:
make more consistent, transparent, scientific
– Effective use of response-chasing tools– Team structure and knowledge
(Area undergoing restructure)
• Efficiency initiatives– save resources: some changes already implemented– effects being monitored; evaluation needed
5
![Page 6: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/6.jpg)
Efficiency initiatives – removal of second reminders
Stocks Inquiry (50-99 sizeband)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%0
10
20
30
40
50
60
70
80
90
10
0
Days since despatch
Res
pons
e ra
te
With 2nd reminder Without 2nd reminder
6
![Page 7: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/7.jpg)
UNDERSTANDING NON-RESPONSE
![Page 8: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/8.jpg)
Patterns of non-response
• Industrial sector - identified those with lower response rates (e.g. catering, hotels)
• High correlation between industry response rates at early and final results
• Size of business – larger businesses take longer to respond. Chasing strategy ensures responses are received later though
8
![Page 9: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/9.jpg)
Intensive Follow-Up (IFU) exercise
• Dual aims:– to estimate non-response bias (work in progress – see final
paper)– to establish reasons for non-response and (later) cost response-
chasing
• Used the Monthly Inquiry into the Distribution and Services Sector (MIDSS):
– dedicated team for the IFU– contacted c.600 non-responders per month in chosen industries– businesses to receive up to 5 phone calls– reason for initial non-response; nature of call; length of call
9
![Page 10: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/10.jpg)
IFU results – returned data
• c.80% of all businesses selected for IFU returned questionnaire, but
• many businesses returned questionnaire just after deadline – no call needed!
• Only c.60% of those contacted returned questionnaire
10
![Page 11: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/11.jpg)
IFU results – reasons for non-response
Reason for initial non-response
Number who gave a reason
Returned data after IFU calls
Still didn’t return data
after IFU calls
Forgot, missed date
667 77% 23%
Too busy, too low priority
361 67% 33%
Actively decided not to
67 33% 67%
11
![Page 12: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/12.jpg)
BUILDING A RESPONSE-CHASING STRATEGY
![Page 13: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/13.jpg)
Dealing with businesses that don’t respond
• Aim to make response-chasing more efficient
• Create a scoring system to prioritise/categorise non-responders
• Focus on reducing non-response bias
13
![Page 14: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/14.jpg)
Estimation in ONS business surveys
We impute/construct where there is non-response.
Then estimate totals as
where
*y i i
i S
t w y
* if
ˆ if i R
ii NR
y i Sy
y i S
14
![Page 15: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/15.jpg)
Bias in ONS business surveys
• Total potential non-response bias (= total imputation error) given by
• We will concentrate on
(i.e. the absolute error of imputation for each business)
ˆi i ii S
w y y
ˆi i iw y y
15
![Page 16: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/16.jpg)
Scoring - principles
• Reduce imputation error by attempting to predict
(Large value means increased risk if business is imputed – therefore target these)
• May also wish to score to encourage good response behaviour from businesses – e.g. new-to-sample
• Need a system that is easy to use and justify.
ˆ| |i i iw y y
16
![Page 17: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/17.jpg)
Scoring methods
• (McKenzie) Calculate imputation error from previous returns; then rank into deciles: 0, 1, …, 9.
(Smallest – Largest)
New-to-sample or long-term non-responders = 10
Tested on MIDSS in 2001-2; implementation issues
• (Daoust) Calculate weighted contribution to estimates – categorise into 3 groups for follow-up
• New investigations with adapted methods
17
![Page 18: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/18.jpg)
Current investigations in MIDSS
• Predict imputation error in monthly turnover (= y)
– Various predictors available– Rank businesses then group – No imputation score?
Use stratum average.
• Assess actual error against predicted.
1. imputation error(t-1)
2. register turnover
3. ( ,imp.error(t-1),
imp.error(t-2), ...,
reg. turnover ,
reg. employment , ...)
i
i i
i
i
i
POSSIBLE PREDICTORS
w
w
f w
18
![Page 19: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/19.jpg)
Results (5 groups)
ˆR
i i ii S
w y y
Actual
Score Imputation error
4 88
3 8
2 3
1 1
0 << 1
• Percentage of within each priority score group
19
![Page 20: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/20.jpg)
Results
ˆR
i i ii S
w y y
Actual Weighted
prediction
Score Imputation error
Previous imp. error
4 88 73
3 8 12
2 3 10
1 1 3
0 << 1 2
• Percentage of within each priority score group
19
![Page 21: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/21.jpg)
Results
ˆR
i i ii S
w y y
Actual Weighted
prediction
Score Imputation error
Previous imp. error
Register turnover
4 88 73 68
3 8 12 15
2 3 10 8
1 1 3 5
0 << 1 2 4
• Percentage of within each priority score group
19
![Page 22: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/22.jpg)
Results
ˆR
i i ii S
w y y
Actual Weighted
prediction
Unweighted prediction
Score Imputation error
Previous imp. error
Register turnover
Register employment
Register employment
4 88 73 68 42 40
3 8 12 15 20 15
2 3 10 8 11 12
1 1 3 5 9 18
0 << 1 2 4 18 15
• Percentage of within each priority score group
19
![Page 23: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/23.jpg)
Conclusions
• Significant gains available in response chasing
Future plans:• Refinements to scores:
– optimum predictor– individual adjustments (e.g. long-term non-responders)– overall or by separate industry groups?– multivariate surveys
• Dynamic updating of scores• Live testing
20
![Page 24: A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National](https://reader036.vdocument.in/reader036/viewer/2022062321/56649ece5503460f94bdaf57/html5/thumbnails/24.jpg)
References
• Daoust, P., (2006), 'Prioritizing Follow-Up of Non-respondents Using Scores for the Canadian Quarterly Survey of Financial Statistics for Enterprises', Conference of European Statisticians
• McKenzie, R., (2000) 'A Framework for Priority Contact of Non Respondents', Proceedings of the Second International Conference of Establishment Surveys
21