Eurostat

Process Quality – tools for measuring and improving

Mária Dologová Statistical Office of the Slovak Republic

ESTP course on Quality Management in Statistical Agencies – Introductory course

Statistics Finland, Helsinki 16-19 May 2017

THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION

Eurostat

2

Contents

1. Introduction

2. Process quality, process variables

3. Self-assessment / DESAP

4. Audits

5. Benchmarking - Benchlearning

6. Brainstorming - Brainwriting

Eurostat

3

Introduction

Eurostat

4

Product

- one form of output

Critical product characteristics

- characteristics that mostly enhance the customer

satisfaction, characteristics the customer is sensitive about

Key process variables

- variables that mostly affect the critical product

characteristics and related costs

Eurostat

5

Process quality process variables

step by step, example

Partly based on: Handbook on improving quality by

analysis of process variables

Eurostat

6

1. Subprocess:

. 2. Product /output: . (aim of the process)

3. Customer:

4. Critical product . characteristic:

Interviewing

Punctuality, timeliness

accuracy, accessibility

complete, correct accessible in DB at required time

subject matter unit (int.)

data collected in the field stored in central DB

For the selected process – identify critical product characteristics

Eurostat

addressing RepU .......

filling in Q ...................

check 1 ...

recording ....................

coding ........................

check 2 ...

transfer to gestor .....

creation of regional file

check 3 ...

transfer to server .....

check 4 ...

storing in DB ............

+

+

-

-

+

-

-

-

-

-

-

-

re-addr.

re-filling

re-check

re-rec.

re-coding

-

-

-

-

-

-

-

- Availability of RepU

- Q of interviewer

- Methodology

- Technology

- Technology

ideal state:.. ?

Describe process flow, identify critical aspects improvements should focus on

Eurostat

8

Sampling

Questionnaire

registers

wording sensitivity length, ...

methodology interviewer

Selection

Training

Q of training Q of materials

Critical product

characteristic

technology

Interview method

PAPI, -TI CAPI, -TI

Application sw

Brainstorming,

brainwriting

Determine key process variables useful tool: Cause-effect diagram

availability of RepU

Confidence in:

interviewer NSI

Interv. timing

NSI info pre-info on survey

Ineligibility Response Delay from ... rate rate timetables 1. Stream – PV:

Eurostat

9

Determine measurements for key process variables

Set up system for monitoring, measuring Monitor, measure

Measurements; unit; target: responses; %; min x%

breakdown acc. to nonresp. type; %; max x%

Monitoring, measur. system: regular reporting to manager on nonconformity

setting up early warning system

defining action scenario: action XY if ...

Monitor, measure: record data

PV: Response rate

etc.

Eurostat

10

Control Chart of Consistency Rate

0.88

0.89

0.90

0.91

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

01

-Ju

l-2

00

1

01

-Au

g-2

00

1

01

-Se

p-2

00

1

02

-Oct-

20

01

02

-No

v-2

00

1

03

-De

c-2

00

1

03

-Ja

n-2

00

2

03

-Fe

b-2

00

2

06

-Ma

r-2

00

2

06

-Ap

r-2

00

2

Scanning End Date

Ra

te

Consistency

Lower bound

Upper bound

Centre Line

... specify what accounts for majority of nonconformities

... measure the dominant source of nonconformity to see whether the process is stable ... if not, find root cause, take action stable process

LFS - 1st Quarter 2003

0100200300400500600700800

Ineligibleunits

Temporary awayunits

RefusalsBehaviour of non-response

N

um

be

r o

f c

as

es

0%

20%

40%

60%

80%

100%

c

um

ula

tiv

e %

706

54%

Useful tool –

Pareto chart

Stability of process variables (of process)

Capability of process

Analyse data

Useful tool –

Control chart

capable process (able to meet customer requirements

or limits for the process)

... if the process is reasonably stable determine limits of the expected process variation evaluate the capability of the process

Eurostat

11

Formulate, implement improvement actions Report

Improvement actions - Interviewing

Implement improvements, evaluate effectiveness, report

Reduce number of ineligible units (improve quality of

registers, sampling) by x%;

Resp.: xx; DL: xx

... etc.

1. stream - PV:

2. stream:

Improve pre-info on survey (use local info channels)

Change survey timing (move to evening hours, ...);

Resp.: xx; DL: xx

... etc.

Eurostat

12

Self-assessment

- DESAP -

Eurostat

13

Self-assessment - comprehensive and systematic review of a process (statistical) and its outputs against a model or a framework, undertaken by the staff of the production unit

- outputs: improvement actions, identification of good practices

- example - checklists:

DESAP - The European Self-assessment Checklist for Survey Managers

ESPAC - The Eurostat Statistical Processes Assessment Checklist

ISPAC - The International Statistical Processes Assessment Checklist

- part of Intermediate / Advanced package within ESS QAF

- main element of EFQM Excellency Model, CAF

Eurostat

14

DESAP The European Self Assessment Checklist for Survey Managers

process oriented generic checklist for a systematic quality assessment of surveys in the ESS

designed as an assessment tool for survey managers

applies to individual statistics collecting micro data

compliant with the ESS quality criteria

based on questions with numerous response categories, assessment questions, and open questions

outputs: improvement actions, good practices comparisons over time thus:

helpful in decision making re resource allocation training

Eurostat

15

Process oriented generic checklist – it covers the value adding process ... from users and their needs ... to improvements

I. Decision to undertake a survey

II. Survey design

III. Data collection

IV. Data capture, data processing

V. Data analysis & output quality

VI. Documentation, disemination

VII. Improvement cycle

Eurostat

16

1

2

3

4

5

Section II/6

overcoverage

Section II/7

undercoverage

Section II/8

misclassification

Section IV/4

necessity of

editing

Section V/6

CVs for key

variables Section V/15 unit

unit non-

response

Section V/21

time lag reference

period/

publication

Section V/25

comparability

over time

Section V/18

item non-

response

Section V/27

coherence for

results of different

frequencies

Section V/24

comparability across

non-geographical

domains

Section V/28

Coherence within

the same socio-

economic area

Section V/2

user satisfaction Section V/3

overall

relevance

Section V/22

Publication

dates kept?

Quality profile

of the statistics

Assessment diagram linked to assessment questions

Eurostat

17

DESAP - other versions

DESAP condensed – covers questions only pertinent to assessment diagram i.e. questions relating to Relevance, Accuracy, Timeliness and Punctuality, Comparability, Coherence

DESAP-E programme – electronic version of DESAP - excel application draws automatically the result diagram developed by Statistics Lithuania

Eurostat

18

Audits

Eurostat

19

ISO 19011:2011 Guidelines for

auditing management systems Audit

General - systematic, independent and documented process for obtaining audit evidence (records, ...) and evaluating it objectively to determine the extent to which the audit criteria (policies, procedures, requirements) are fulfilled; conducted by people from outside the audited subject

***

Statistical process audit - comprehensive and systematic review of a statistical process and its outputs against a model or a framework; conducted by people from outside the production unit

- part of Intermediate / Advanced package within ESS QAF

- obligatory element of QMS acc. to ISO 9001,

- conducted after the self-assessment within EFQM Excellency Model, CAF

Eurostat

20

Internal audit - conducted by the organisation itself for internal purposes

External audit - conducted by interested party (partner, user, ...)

- or by external independent organisation (certification -, supervisory -, re-certification audit; EFQM audit)

Outputs - suggestions for improvement actions, - for corrective and preventive actions

- good practices

Eurostat

Main players, responsibilities - lead auditor (can not audit the own process)

- auditor („-“) - technical expert - Q manager - audited party

Programming - Annual programme of audits (purpose of audits, audited subjects (process, ...), teams of auditors, date)

Conduct of audit - Plan of audit - Tasks of lead auditor, auditor, technical expert - Role of Q manager - Report from audit, end of audit

21

Eurostat

Reporting to top management - Annual report on audits conducted

(summary of findings, - of suggestions for improvement, corrective,

preventive actions)

Auditors level control - Auditors knowledge & skills

(formal training, training on the job, card of personal development,

appraisal)

22

Eurostat

23

Benchmarking - Benchlearning

Eurostat

24

Benchmark - performance level recognised as the standard of excellence for a specific process; standard for comparison

Benchmarking - ‘to make comparison with others‘ making comparisons with other organisations (industry best / good practice) and then learning the lessons that those comparisons reveal)

(source : European Benchmarking Code of Conduct)

In practice, it encompasses: - regularly comparing aspects of performance (functions or processes) with organisations which are considered to be good practitioners; - identifying gaps in performance; - seeking approaches to bring about improvements in performance; - following through with implementing improvements; and - following up by monitoring progress and reviewing the benefits.

- obligatory element of EFQM Excellency Model (and element of ES Code of Practice)

Eurostat

25

Benchlearning - „benchmarking focusing on the learning aspects“ - learning how to improve through sharing knowledge, information, and sometimes resources, it is recognised to be an effective way of introducing organisational change; it reduces risks, is efficient and saves time.

- obligatory element of CAF (as benchmarking focusing on learning from others)

Best / Good practice - superior performances, methods or approaches that lead to exceptional achievement - „best practice“ is a relative term and sometimes indicates innovative or interesting business practices, which have been identified through benchmarking - since one cannot be sure that there is not a better practice, it is preferable to talk about “good practice”

Eurostat

26

Brainstorming - Brainwriting

Eurostat

27

Brainstorming - team working tool to generate spontaneously ideas verbaly without constraints in a short period of time - used before problem solving, decision making, scenario analysis - important rules = generate as many ideas as possible, don´t criticise, welcome unusual ideas, combine, improve ideas

Brainwriting - similar to brainstorming – but generating ideas

in written form useful with a group of people who are reticent and would be unlikely to offer many ideas in an open group session

Due to advantages:

Brainwriting = more effective than Brainstorming

at stimulating production of creative, usefeul ideas

Eurostat

28

Topics for group work

1. For the selected subprocess (or part of it): - formulate aim of the subprocess (or of part of it) - suggest the way (the basic approach) how you would identify key process variables (Slide 9)

2. Have you implemented or do you intend to implement internal audits in your organisation? What are (could be) the benefits and what are (could be) the problems?

3. Select one section of the DESAP checklist and conduct selfassessment for the survey of your choice, describe results and main problems you had during selfassessment