On Tap: Developments in Statistical Data Editing at

Statistics New ZealandPaper by Allyson Seyb, Felibel Zabala and Les

Cochran

Presented by Felibel ZabalaSept 2012

Aim of paper

To describe the latest developments in Statistics New Zealand’s economic and household processing platforms

2

Strategic Developments

Statistics 2020 Te Kāpehu Whetū - Recent actions

Reduction in number of tools

Use of Colectica to centralise storage of all information of Statistics NZ’s outputs

Establishment of processes to research and introduce new standard methods and tools

3

Strategic Developments (cont’d)

International collaboration

Regular bilateral and trilateral meetings with various national statistical offices

Involvement in the Statistical Networks on Confidentiality and the Industrialisation of Editing

Continued investigation on the use of SELEKT

Evaluation of SAS2Argus

4

Platforms

Need for new infrastructure to produce statistics that are fit for purpose in a cost and effective way

A platform is a logical cluster of functionality that enables components to be put together to provide a complete end-to-end system

5

System 1

System 2

System 3

System 4

System 5

System 6

System 7

System 8

System 9

System 10

System 1

System 2

System 3

System 4

System 5

etc

System 6

System 7

System 8

System 9

System 10

System 11

System 12

System 13

System 14

System 80

etc

System 60

System 1

System 2

System 3

System 4

System 5

System 6

System 7

System 8

System 9

System 10

etc

System 50

System 1

System 2

System 3

System 4

System 5

System 6

System 7

etc

System 25

Use standard tools

Use for• Surveys and

censuses• Administrative

sourced data• Mixed sources

Use SAS for processing and analysis

6

Five main platforms

Need Develop & Design Build Collect Process Analyse Disseminate

Collection DisseminationMicro-economic

Household

National Accounts

Household platform

Is a second generation platform with the design informed by an evaluation of the interim platform

Processes and in the future analyse three social surveys and their supplements

Uses standard tools to load, code, micro-edit and finalise a unit record dataset

Processes a Blaise-based survey

7

Uses a mix of shared and specific systemsThe Household Platform

Portal

Process Configure

Setup

View

Code

Search

Edit

View Diary

Workflow

SAS

Execute

SaveLoad

Load Diary

File

Save Diary

Select

Admin

StatisticalToolbox

GREGWt

X12

Core EditsCore DVs

Derivations

CANCEIS

Edits

Screens Tasks Rules

Edit Diary

Info

DataMetadata

DataMetadata

Config.

Config.

Classifications & Stds

Metadata

Survey Specific

Shared across Social

Shared across Stats

Other Data

Configure

Paradata

Treat Estimation

Extract Setup

Surv. Inst.

Paradata

Core Questions

Format of micro-level data

9

The Household platform

Survey cycle code

ID Repeat Variable name

Value

HLFS 107 Person1 11 Age 15

HLFS 107 Person1 11 Sex M

HLFS 107 Person2 11 Sex F

HLFS 107 Household1 11 Household composition

Multi-person

Micro-economic Platform

Previously referred to as BESt platform

Processes and analyses economic surveys and administrative data collections

Also has elements of ‘Develop and Design’, ‘Build’, ‘Collect’ (from general Business Process Model)

Has a user configurable workflow

Allows incremental statistical maintenance and uses standard tools for common processes

10

Uses a mix of shared and specific systemsThe Micro-economic platform

Business Intelligence

View

Configure process

Metadata

Manual edit (generic)

Manual edit

SAS

Load metadataLoad data

Workflow

StatisticalToolbox X12

EditsDerivations

Configuration

Banff

Screens Tasks Rules

Reporting

Info

DataMetadata

DataMetadata

Configuration

Configuration

Classifications & Stds

Metadata

Survey Specific

Shared across Micro Economics

Shared across Stats

Other Data

Process trace

Run process Extract

Trace Reporting

Process trace

System Admin View Respondent

Build cubesGeneric E&I

Sample selectWrite-back

ImputationCode

EstimationTreat

Allocate

Challenges and lessons learnt

Balancing generic and specific needs

Determining process elements suitable to be implemented as common services

Moving from a process culture to a more constructive innovative culture

Key enabler of the culture change - the adoption of an agile project management approach for IT development projects

12

Moving forward

Review of Statistics NZ’s generic Business Process Model

Implementation of a framework to measure and report on the benefits achieved with recent and on-going developments

Transformation of the organisation’s data collection processes

13