local and global scores in selective editing

1

Local and Global Scores in Selective Editing

Dan Hedlin

Statistics Sweden

2

Local score

• Common local (item) score for item j in record k:

• wk design weight

• predicted value

• zkj reported value

j standardisation measure

jkjkjkkj zyw ~~

kjy~

3

Global score

• What function of the local scores to form a global (unit) score?

• The same number of items in all records

• p items, j = 1, 2, … p

• Let a local score be denoted by kj

• … and a global score by kg γ

4

Common global score functions

In the editing literature:

• Sum function:

• Euclidean score:

• Max function: kjj

max

p

jkj

1

2

p

jkj

1

5

• Farwell (2004): ”Not only does the Euclidean score perform well with a large number of key items, it appears to perform at least as well as the maximum score for small numbers of items.”

6

Unified by…

• Minkowski’s distance

• Sum function if = 1

• Euclidean = 2

• Maximum function if infinity

1

1

;

p

jkjkg γ

1

7

• NB extreme choices are sum and max

• Infinite number of choices in between = 20 will suffice for maximum unless

local scores in the same record are of similar size

8

Global score as a distance

• The axioms of a distance are sensible properties such as being non-negative

• Also, the triangle inequality

• Can show that a global score function that does not satisfy the triangle inequality yields inconsistencies

lklk ggg γγγγ

9

• Hence a global score function should be a distance

• Minkowski’s distance appears to be adequate for practical purposes

• Minkowski’s distance does not satisfy the triangle inequality if < 1

• Hence it is not a distance for < 1

10

Parametrised by

• Advantages: unified global score simplifies presentation and software implementation

• Also gives structure: orders the feasible choices…from smallest: = 1…to largest: infinity

11

• Turning to geometry…

12

Sum function = City block distance

p = 3, ie three items

13

Euclidean distance

14

Supremum (maximum, Chebyshev’s) distance

15

Imagine questionnaires with three items

1k

Record k2k

3k Euclidean distance

16

17

The Euclidean function, two items

A sphere in 3DThreshold

Threshold

18

The max function

A cube in 3D Same threshold

19

The sum function

An octahedron in 3D

20

21

• The sum function will always give more to edit than any other choice, with the same threshold

22

Three editing situations

1. Large errors remain in data, such as unit errors

2. No large errors, but may be bias due to many small errors in the same direction

3. Little bias, but may be many errors

23

Can show that if…1. Situation 32. Variance of error is

3. Local score is

• Then the Euclidean global score will minimise the sum of the variances of the remaining error in estimates of the total

2~kjkjkj zyVar

jkjkjkkj zyw ~~

24

Summary

• Minkowski’s distance unifies many reasonable global score functions

• Scaled by one parameter• The sum and the maximum functions are

the two extreme choices• The Euclidean unit score function is a good

choice under certain conditions