quality assessments of statistical production processes in eurostat pierre ecochard and małgorzata...

Quality Assessments of Statistical Production Processes in Eurostat

Pierre Ecochard and Małgorzata Szczęsna

pierre.ecochard@ec.europa.eu, malgorzata.szczesna@ec.europa.eu

Quality assessment in the ESS: 2005-2007

6

5

3

0

14

12

4

3

10

11

16

15

0

1

7

11

quality reports?

quality indicators?

self-assessments?

quality audits?

All

Most (≈25 to 75%)

Some (< 25%)

None

“Over the last three years (2005-2007), how many of your statistical processes have been monitored using:”

Quality assessment in the ESS: 2008-2010

10

9

4

0

18

17

14

13

2

4

8

12

0

0

3

4

quality reports?

quality indicators?

self-assessments?

quality audits?

All

Most (≈25 to 75%)

Some (< 25%)

None

“Over the next three years (2008-2010), how many of your statistical processes do you plan to monitor using:”

Audits and self-assessments in the ESS

Number of NSIs conducting:

Audits SAs

5 NSIs 9 NSIs 10 NSIs

7 NSIs

The European StatisticsCode of Practice

The Eurostat Quality Assurance Framework

DocumentationMeasurement

Evaluation Conformity

•Quality reports•Quality indics•Process descriptions•Process variablesetc

Quality Assessments Labelling

Background

What is Eurostat Quality Assessment?

A systematic review and evaluation of all stages of the statistical production process with the use of the DESAP-

based Checklist

IT conditions; Management, planning and legislation; Staff, work conditions and competence

User needs

Data

collection Validation Confidentiality

Dissemination

Documentation

Follow-up

Assessment Outputs (1)

Assessment Report

Assessment Outputs (2)

Coherence General coherence

Timeliness Timeliness of final publication

Accuracy Overall accuracy

Relevance User satisfaction

Accessibility and clarityOverall quality of metadata

Comparability

Comparabilityover time

Comparabilityacross countries

0

1

2

3

4

5

Assessment Diagram

Assessment Outputs (3)

Highlight of good practice

Categories of Eurostat assessments

Self-Assessment

Supported Self-Assessment

Peer Review Rolling Review

Process 1Process 4

Process 2

Process 3

Characteristics:- Periodicity- Legal Basis- Output- ESTAT intervention

Similiarity: DESAP-based Checklist, outputsDifference: extent of external interventation in a review

Process 5

Office-wide implementation plan

Approach piloted in two domains in 2007 Most of the statistical process will benefit from an

assessment within a three-year period 2008-2010 33 reviews planned for 2008:

– 14 Supported Self-Assessments– 14 Self-Assessments– 4 Rolling Reviews– 1 Peer Review

Follow-up report to Eurostat management by the end of 2008

Benefits of quality assessment

For production teams: An opportunity for a

systematic analysis of the production process

Identify and prioritize improvement actions

Spread and benefit from the Good Practice

For Eurostat: Identify horizontal

problematic issues Foster standardisation of

statistical processes Support resource allocation,

planning and programming Show quality commitment

Feedback

The general workflow works well, the Checklist is flexible and the assessments are considered useful by domain managers

Heads of Unit should be involved earlier The diagram can be a red herring: it should be

used with caution The ownership of the results should be made very

clear More assessment should involve an external expert

Keys for success in implementing quality assessments

Top management commitment

Middle management acceptance

Sound communication

Long term perspectives

Implementation and fine-tuning in pilot projects

Standardised use of methods

Clear responsibilities and ownership

Sufficient resources allocated for the assessments