Quality assurance surveys and program administration – when

accuracy really counts Stephen Horn

Contributed paper Q2008

outline• Quality assurance surveys in context

– Regulating risk– Fraud and error in social security systems– Measuring payment accuracy

• Random Sample Surveys– Design– Quality assurance– Estimation & Reporting

• Issues– Exploiting an Admin frame – Kalman filtering – SE estimation of variables with highly skewed distributions– Performance measures [kpis, correctness/accuracy, latency measures]– Audit oversight – the construction of official statistics

Cost of Compliance Customer Attitude to Compliance Compliance Strategy Debt Recovery Strategy

Reparation sought through the courts

FaCSIA Fraud and Compliance Response

Those customers whose deliberate and organised non-compliance results in a significant attack on the integrity of the payment

Fraud Investigation/Prosecution Action/Criminal Conviction

Those who deliberately not comply

Targetted Compliance Activities e.g. Data Matching, Compliance Reviews

Those who do not take reasonable care to comply

Those who do not take reasonable care to comply

Those who do not take reasonable care to comply

Those who get it right

Small Debt Waiver: <$50

Adapted from Braithwaite Regulation Model

Those who try to comply but make honest mistakes/do not understand their requirements

Education Programmes e.g. Real Estate and Overseas Pensions publicity campaigns

Targetted Prevention e.g. Service Profiling

legislation

policies

PROGRAMS

rules

OUTLAYS

Audit

appropriationPortfolio depts

DELIVERY AGENCY

CUSTOMER BASEPopulation

Finance

eligibility

objectives

$entitlementcircumstances

Financial accountability

Social accountability

Politicalaccountability

Accountability for Social Security payments

RSS Design . compliance history . policy priority . accuracy requirement . budgeted funds . risk analysis

customer population

Sample size Stratification Clustering

Sample allocation to strata

Sample allocation to area

Service Level

Agreement

Exclusions & substitutions

Field

. Allocation to trimester

. Refreshment

Refresh instrument

Sample Selection

Check sample

Quality assurance - field Validation – 10%sample

Prelim TRI data Validation checks

Progressive release

Final clearance TRI data

RSS Estimation & Release

. Actuals

. Customer population

Admin Source data

Program view

KPI Latency series BPA Reporting – Pay accuracy

Debt series

Correctness series

RSS SAMPLE SPECIFICATION 2008-09 - DISABILITY SUPPORT PENSION STRATUM ID POP Mar08 SAMPLE SKIP

BY TRI MESTER

D01 22,140 <50yrs 128 172 43D02 47,143 50+yrs 187 252 62D03 33,402 <50yrs 158 212 53D04 84,984 50+yrs 252 338 84D05 35,302 <35yrs 162 218 54D06 54,720 35-49yrs 202 271 67D07 90,896 50+yrs 260 349 87D08 54,656 <35yrs 202 271 67D09 74,531 35-49yrs 236 316 79D10 93,920 50+yrs 265 355 88D11 52,644 198 266 66D12 42,384 178 239 59D13 46,629 165 283 55D14 46,736 165 283 55

3000TOTAL 780,088 2757

Note: sample is allocated using Caroll formula (that is proportional to the square root of the stratum population)The allocation of sample between nonremote and remote areas is proportional to respective customer populations

Nonremote

Max rate

Partnered

Female

Male

Unpartnered

Female

Male

Reduced rate

STRATIFICATION

PartneredUnpartnered

RemotePartnered

Unpartnered

STRATIFICATION & SAMPLE ALLOCATION

AGE PENSION RANDOM SAMPLE 2006-2007, ALLOCATION TO AREAS

Allocation of Stratum Sample to Areas - Nonremote Sample Stratum sample Code AAB AAC AAE AAF AAG AAI AAK AAL AAM AAS AAX ABD AWAnr NR1 22 37 21 20 39 32 36 39 38 40 40 14 27 407 NR2 13 20 12 12 24 23 23 24 21 23 25 6 16 242 NR3 23 36 20 20 37 30 35 37 37 39 38 14 26 392 NR4 18 26 16 16 32 29 29 29 27 30 32 8 20 313 NR5 23 35 22 23 37 34 34 33 32 37 34 12 27 383 NR6 25 35 25 27 44 49 41 39 37 43 46 13 30 453 NR7 13 27 16 14 24 25 20 23 21 26 21 9 16 256 NR8 13 21 14 13 25 25 20 23 19 23 24 7 14 242 NR9 12 30 21 16 32 31 37 29 29 36 40 11 32 355 NR10 6 15 11 8 18 21 22 16 16 18 23 5 17 198 NR11 12 29 20 16 30 29 35 27 27 35 38 11 31 340 NR12 8 20 14 11 24 26 28 20 20 24 29 7 22 253 NR13 9 21 15 13 25 27 26 18 19 25 25 8 23 253 NR14 11 21 15 15 31 34 32 22 23 28 33 8 25 297 NR15 7 15 10 9 15 17 16 13 13 18 16 5 15 169 NR16 7 15 9 9 18 19 18 15 14 17 19 5 14 179 Area Total 223 405 260 455 451 451 408 394 461 483 143 358 4,733

SAMPLE DESIGN – REMOTE CLUSTER FORMATION

Area NE Queensland: Location of Customers

46804701

4740

48144870

1

1000

4600 4650 4700 4750 4800 4850 4900

Postcode (modal postcodes highlighted)

Custo

mers

(log s

cale)

Age Pension Carer Allowance Carer Payment

CLUSTERING METHODOLOGY

SAMPLE SURVEY PROCESS MAP – RSS REVIEWS

RANDOM REVIEW RESULTS SYSTEM

•

SURVEY-ADMIN DATA LINKAGE MAP

DATASET 1 – CUSTOMER LINK IRS TO ADMIN DATA

DATASET 3 – DEBT ANALYSIS LINKED TO DMIS

DATASET 4 – VARIABLE RESULTS FOR RRRS

INDEBTEDNESS (attributable to current year) TO OUTLAY: MAJOR PAYMENTS, 2003-200612Month forward moving aggregates, 1SE Error Bars

0.00%

1.00%

2.00%

3.00%

4.00%

5.00%

6.00%

AGE DSP NSA PPP PPS YAL

INDEBTEDNESS BY PROGRAM

AGE Pension Latent indebtedness by recency of debt, 2003-2006 12Month forward moving aggregates, 1 SE Error bar

0.00%

0.50%

1.00%

1.50%

2.00%

2.50%

3.00%

3.50%

20024

20031

20032

20033

20034

20041

20042

20043

20044

20051

20052

20053

20054

20061

20062

20063

20064

Deb

t to

Out

lay

new overlapping legacy All debts Current FY

LATENT INDEBTEDNESS – RECENCY OF DEBT

AGE PENSION: PREVENTION KPI - DEBT LATENCY 2003-07Source RSS, 1 SE Error Bar

0.00%

0.10%

0.20%

0.30%

0.40%

0.50%

2002

-03

2003

-04

2004

-05

2005

-06

2006

-07

Financial Year (4 Qtr Rolling Aggregates)

Deb

t to

Out

lay

(%)

NEW DEBTS Lower 95% CI Upper 95% CI

DEBTS – KEY PREVENTION INDICATOR

Age Pension: Average Debt Size, Latent and Raised Series, 1999-200712 Month Moving Average, 1SE Error Bar

0

50

100

150

200

250

300

350

1999

420

004

2001

420

024

2003

420

044

2005

420

064

2007

4

Leading quarter

Ave

rage

Deb

t Siz

e ($

)

Raised Series (new debts only) Latent Series (new debts only)

LATENT AND RAISED SERIES – AVERAGE DEBT SIZE

TABLE 2

($m) RSE(%) ($m) RSE(%) ($m) RSE(%) ($m) RSE(%)

2002-03 353.7 20% 65.1 35% 193.3 20% 95.4 18%2003-04 342.0 18% 68.0 45% 215.3 13% 65.1 15%2004-05 507.7 13% 102.8 37% 306.8 9% 90.3 15%2005-06 538.6 17% 87.1 44% 321.4 17% 123.5 13%2006-07 470.9 3% 95.9 22% 264.1 2% 107.7 6%

2002-03 2.00% 20% 0.37% 35% 1.09% 20% 0.54% 18%2003-04 1.82% 18% 0.36% 45% 1.15% 13% 0.35% 15%2004-05 2.59% 13% 0.52% 37% 1.56% 9% 0.46% 15%2005-06 2.62% 17% 0.42% 44% 1.56% 16% 0.60% 12%2006-07 2.17% 3% 0.44% 21% 1.22% 2% 0.50% 6%

Data source: RSS 2003 to 2008 rounds; Standard Errors calculated using the GREGWEIGHT macro, weighted residual methodProgram AU_0308all, Feb15 2008 [AGE+DSP only]

Upward Variation

Inaccuracy ($m)

Inaccuracy to Outlay ($)

AGE PENSION: Inaccuracy of paymentby by nature of adjustment, 2003-2007

Financial YearGross mispayment Cancellation&suspension Downward Variation

PAY INACCURACY – LATENT MEASURES

AGE Pension: Payment Inaccuracy, 2003-2007

0.00%

0.50%

1.00%

1.50%

2.00%

2.50%

3.00%

3.50%

2002

-03

2003

-04

2004

-05

2005

-06

2006

-07

Mis

paym

ent t

o O

utla

y (%

)

All Mispayment (gross) Cancellation and suspension Downward Variation Upward Variation

PAYMENT INACCURACY – MOVING AGGREGATES

POINT PAYMENT ACCURACY, LATENT MEASURE 2003-2007, BY MAJOR BENEFIT TYPE

Financial year

Point estimate RSE(%)

%of All Customers

Cancellations & Suspensions

Downward Variation

Upward Variation

2002-03 1,337,552 2.91% 72.5% 68.3% 76.6% 0.5% 15.6% 11.2%2003-04 1,296,035 1.23% 69.4% 67.7% 71.1% 0.7% 17.8% 12.1%2004-05 1,230,452 0.82% 64.8% 63.7% 65.8% 0.7% 21.2% 13.3%2005-06 1,259,238 3.56% 65.7% 61.1% 70.3% 0.7% 21.8% 11.8%2006-07 1,254,123 2.50% 64.6% 61.5% 67.8% 0.8% 22.3% 12.3%

2002-03 555,194 2.73% 83.1% 78.6% 87.5% 2.8% 8.7% 5.2%2003-04 584,698 1.34% 84.6% 82.4% 86.8% 1.5% 8.0% 5.9%2004-05 604,071 1.09% 84.7% 82.9% 86.5% 2.5% 8.0% 4.8%2005-06 643,804 1.91% 87.5% 84.3% 90.8% 3.3% 6.3% 2.9%2006-07 701,017 0.55% 92.5% 91.5% 93.5% 1.3% 4.5% 1.7%

2002-03 456,154 3.00% 82.4% 77.6% 87.3% 10.2% 4.5% 2.9%2003-04 408,188 1.04% 77.9% 76.3% 79.5% 9.0% 6.1% 7.0%2004-05 437,154 0.99% 87.9% 86.2% 89.6% 6.6% 3.7% 1.8%

2002-03 127,156 1.30% 65.7% 64.0% 67.3% 11.4% 15.2% 6.9%2003-04 113,457 3.60% 61.2% 56.9% 65.6% 10.3% 18.1% 10.3%2004-05 119,072 2.53% 66.9% 63.6% 70.2% 8.7% 16.4% 8.0%

2002-03 366,291 2.60% 83.1% 78.9% 87.3% 3.3% 9.7% 3.9%2003-04 348,666 1.86% 77.2% 74.4% 80.0% 4.5% 12.5% 6.1%2004-05 390,267 2.11% 84.9% 81.4% 88.4% 4.7% 6.5% 3.9%

2002-03 314,313 1.42% 83.5% 81.2% 85.9% 9.0% 4.7% 2.2%2003-04 320,956 1.17% 86.9% 84.9% 88.9% 8.1% 3.6% 1.4%2004-05 328,210 0.93% 91.0% 89.4% 92.7% 5.3% 2.2% 1.5%

Source: Random Sample Surveys 2002-203 to 2006-07

NSA

PPP

PPS

YAL

Customers Accurately paidCustomers inaccurately paid by type (% all

customers)

AGE PENSION

DISABILITY SERVICES PENSION

95% Conf Bounds on accuracy

POINT PAYMENT ACCURACY

Customers paid accurately (to all customers) by Payment Type

Latent measures, 2003-2007

50%

55%

60%

65%

70%

75%

80%

85%

90%

95%

100%

Cus

tom

ers

paid

acc

urat

ely

to to

tal c

usto

mer

s

Disability Support pension Age pension NSA PPP PPS YAL

CUSTOMERS PAID ACCURATELY

MOVING AGGREGATE