the application of selective editing to the ons monthly business survey emma hooper office for...
TRANSCRIPT
The application of selective editing to the ONS Monthly Business Survey
Emma Hooper
Office for National Statistics
Overview
1. Editing at the ONS
2. Monthly Business Survey (MBS)
3. Application of selective editing to MBS
4. Quality indicators
5. Implementation and post-implementation
Editing at the ONS
• 2008 project reviewed editing processes for Office for National Statistics (ONS) business surveys
• New selective editing methodology for ONS short-term business surveys
• Mix of selective editing and traditional manual micro editing was previously used
Surveys using selective editing
• Tested and implementing selective editing for the Retail Sales Inquiry– methodology developed with assistance from
Pedro Silva (University of Southampton)
• MBS selected as second survey to test and implement selective editing on
• Currently investigating using Selekt for Annual Business Inquiry
Monthly Business Survey
• Launched in January 2010, it brings together existing short-term surveys that cover different sectors of the economy
• Old selective editing methodology used edit rules, those units that failed an edit rule would have a selective editing score calculated
• New selective editing methodology to run on live MBS data from summer 2010
Editing processes for MBS
1. Edit rule checks
2. Automatic editing
3. Selective editing
4. Macro editing
Check recordswith unit score greater than
threshold
Check aggregated data
Check £000s errorand components
Check for validdates
Selective editing for MBS
• Target units that have significant effect on key estimates by domain (input/output group) if not edited
• Calculate item score for each unit and key variable – turnover, export turnover, new orders (monthly)
and total employment (quarterly)
• Predictor for true value– previous edited value (else use register value for
turnover or employment, or pseudo-imputed value for export turnover or new orders)
Item score
1
ˆ100
ˆ
t t tij ij ijt
ij tjd
a z yscore
T
1
sample design weight for variable ,unit at time
unedited variable value for unit at time
ˆ predicted variable value for unit at time
ˆ previous period's total v
tij
tij
tij
tjd
a j i t
z j i t
y j i t
T ariable estimate for domain j d
Unit score
• Combine item scores into single unit score using average of item scores
• Units ranked according to their unit score– if score for a unit is above threshold then that
units responses are sent for manual editing– units with scores below threshold are not
manually checked
tiu
Thresholds
• Thresholds set for each key domain to reduce editing costs without impacting quality
• Quality indicators used to compare thresholds
• 41 periods of data used, should ensure robustness of results
Absolute relative bias
• Absolute relative bias aims to control the residual bias left in the domain estimates after editing
ˆ ˆ| |td
t t t t t tjd ij ij ij i d jd
i s
ARB w z y I u c T
, unit at time
estimation weight for variable , unit
sample at time in domain
edited v
at time
equal to 1
alue for va
if the uni
riable
t score for unit at time is less
ti
td
tj
j
i i t
w j i
s t d
t
t
y
I i
j
than threshold for domain c d
Savings
• Savings measure the change in the number of units that will be manually micro edited
t tt d dd t
d
trad selectSavings
trad
the number of units failing at least one traditional edit rule at time in domain
the number of units with a unit score above the threshold at time in domain .
td
td
trad t d
select t d
Absolute relative bias
1 2 3 4 5 6 7 8 9 10
0. 000
0. 500
1. 000
1. 500
2. 000
AbsBias1
cut off
Savings
1 2 3 4 5 6 7 8 9 10
0. 00
25. 00
50. 00
75. 00
100. 00
RelSavings_Score1
cut off
Quality indicators
• Aimed to keep ARB below 1%, ARB levels showed large improvement compared to bias left after current micro editing
• Overall savings in the number of units being edited of around 40% in non-employment months
• Overall savings of 55% (MPI sectors) and 15% (MIDSS sectors) in employment months
Current edit rule method
0
5
10
15
20
25
30
35
40
45%
Edit failure rate Edit change rate
New selective editing method
0
5
10
15
20
25
30
35
40
45%
Edit failure rate Edit change rate
Implementation and limitations
• Selective editing is carried out via a module in the in-house built Common Software system
• The module is currently– restricted to 5 item scores– restricted to combining the item scores as a mean
or maximum– restricted to only using variables already available
in the system for use in calculating predicted values
– not able to use current edit rules to calculate an edit-related score
Following implementation
• Need to monitor the thresholds, ideally through editing a small sample of those that aren’t being selectively edited
• This would enable us to estimate the bias left in the estimates and adjust the thresholds accordingly
• Continue testing these methods for other ONS business surveys, more efficient editing will result in a better quality editing process