czech statistical office 1 the quality metadata system in the czech statistical office work session...

CZECH STATISTICAL OFFICE

www.czso.cz

1

The Quality Metadata SystemThe Quality Metadata SystemIn the Czech Statistical OfficeIn the Czech Statistical Office

Work Session on Statistical Metadata (METIS)Work Session on Statistical Metadata (METIS)

10 -12 March 2010 , Geneva

Czech Statistical OfficeCzech Statistical Office

Jitka Prokop


www.czso.cz

2

ContentContent

1. Background

2. Quality Metadata System

3. Quality Monitoring

4. Quality Assessment

5. Lessons Learned

6. Conclusions


www.czso.cz

3

1. Background1. BackgroundSMS sub-systemsSMS sub-systems

SMS-Quality

SMS-Variables

SMS-Respondents SMS-Classifications

SMS SMS-Statistical Tasks

SMS-Dissemination

SMS-Users

SMS-Time series


www.czso.cz

4

1. Background1. BackgroundLinks between SMS subsystemsLinks between SMS subsystems

SMS-CLASS

SMS-VAR

SMS-TASKS

SMS-QUALITY

SMS-USERS

SMS-DISSEM

SMS-RESP

SMS-SERIES

Statistical task - a set of statistical activities needed to fulfil a user’s request for statistical information. It can be composed of one or more statistical surveys or similar statistics.


www.czso.cz

5

2. Quality Metadata System2. Quality Metadata SystemSMS-QUALITY ArchitectureSMS-QUALITY Architecture

Based on • The ESS concept

• Quality criteria (e.g. relevance, accuracy, etc.)• Quality and performance indicators (respondent

burden, cost)

• The European Self-Assessment Checklist for Survey Managers (DESAP)

• Statistical Business Process Model

Applicable for different types of statistics (in some extent).


www.czso.cz

6

2. Quality Metadata System2. Quality Metadata SystemQuality Form Map (QFM)Quality Form Map (QFM)

• Model of Q-attributesQ-attributes in defined structure.Approved by the top management.Includes all proposed Q-attributes.Different types of statistics to be covered.

• SW application for QFM• Defines Q-attributes and/or links into other SMS subsystems• Enables inputs of values manually• Provides views into DWH

• Values of Q-attributes are stored in DWH


www.czso.cz

7

2. Quality Metadata System2. Quality Metadata SystemStructure of the QStructure of the QF F MapMap

….. …. …….

Quality Form Map

Section n-1 Section 1 Section n Section 2

Q-attribute 1.1

Q-attribute 1.2

Q-attribute 1.3

Q-attribute 1.n

Q-attribute 2.1

Q-attribute 2.2

Q-attribute 2.3

Q-attribute 2.n

Q-attribute n-1.1

Q-attribute n-1.2

Q-attribute n-1.3

Q-attribute n-.1.n

Q-attribute n.1

Q-attribute n.2

Q-attribute n.3

Q-attribute n.n

Section Group 2 Section Group 1 Section Group n

level A/B/C

level A/B/C

level A/B/C

level A/B/C

level A/B/C

level A/B/C

level A/B/C

level A/B/C

level A/B/C

level A/B/C

level A/B/C

level A/B/C

level A/B/C

level A/B/C

level A/B/C

level A/B/C


www.czso.cz

8

2. Quality Metadata System2. Quality Metadata SystemExample of Example of QF MapQF Map components components

Group of Sections Sections Q-Attributes

Design of statistical survey & Relevance

Cooperation with users

Testing of questionnaire

• Number of responses in tests of questionnaire – internal, ministries, respondents.

• Number of testing cycles.• Sensitive questions in the questionnaire.

Training of staff • Scheduled and actual dates• Punctuality

Feedback form users & their requirements


www.czso.cz

9

2. Quality Metadata system2. Quality Metadata systemLevels of Q-attributesLevels of Q-attributes

1. Most stable Q-attributes • Relates to the whole statistical task, survey, phase of process• e.g.: Key users, Information on training of staff, Methodology..

2. Q-attributes related to processing in a concrete period• e.g.: Unit response rate, Extent of sample and frame, Punctuality

3. Key statistical variable • In generally defined breakdown• e.g.: Coefficient of variation, Item response rate

Levels of Q-attributes relate to content and stability of values.


www.czso.cz

10

3. Quality Monitoring3. Quality MonitoringAspectsAspects

• Quality monitoring coverscovers• CollectionCollection of data / metadata• CalculationsCalculations of Q-attributes• ChecksChecks with expected values / scales (future; comparisons)

• Quality monitoring is based onbased on• Input variablesInput variables, incl. „history of changeshistory of changes“• Variables specially designed for quality monitoringVariables specially designed for quality monitoring

• Related to questionnaire or interview• Related to concrete variables

• ResultsResults of Q-attributes• During a sub-process / phase of statistical process• At the end of the sub-process / phase• After finishing the whole statistical process


www.czso.cz

11

• CollectionCollection of input data for calculation of Q-attributes

• Calculated Q-attributesCalculated Q-attributes (by SW) e.g.• Unit response rateUnit response rate for the populationpopulation• Unit response ratesUnit response rates for individual stratastrata

3. Quality Monitoring3. Quality Monitoring During During running of running of sub-processsub-process


www.czso.cz

12

3. Quality Monitoring3. Quality Monitoring After the end sub-processAfter the end sub-process

• Based on input variablesinput variables• Editing and imputation rates• Distinction of inputs, corrections and deletes• Recognition of most edited items

• Based on specific quality variablesspecific quality variables• Way (mode) of data collection• Type of contact with respondents• Aspects of individual imputations (e.g. with or without

respondent, technical mistakes)


www.czso.cz

13

3. Quality Monitoring3. Quality Monitoring After the end of the process or After the end of the process or

After the end of one run of the processAfter the end of one run of the process

• Indicators on accuracy

• Coefficient of variation• Item response rates• Final unit response rates• Imputation rates• Revisions

• Timeliness, Punctuality, Comparability, Coherence


www.czso.cz

14

4. Quality Assessment4. Quality Assessment The Quality Assessment GuidanceThe Quality Assessment Guidance (QAG) (QAG)

• Purpose, applicability• Support for management of process• Support for high level decisions• Support for quality reporting• Self-assessment• Auditing

• Levels - Assessment of • Quantitative and qualitative results of Q-attributes• Statistical survey / statistical task


www.czso.cz

15

4. Quality Assessment4. Quality AssessmentWaysWays of Assessment

• Categorial assessment

• Averages of individual results

• Textual assessment and summary • Expert commentaries (suggested issues in QAG)

• Strong aspects

• Weaknesses

• Proposal of concrete actions, priorities


www.czso.cz

16

4. Quality Assessment4. Quality Assessment Levels of Levels of Q Q Assessment of Q criteriaAssessment of Q criteria

Structure of QA follows the ESS quality criteria and covers the following levels:

a. Key statistical variable statistical variable in particular breakdowns.

b. Statistical variable as an aggregateaggregate (average) of the breakdowns or statistical survey.

c.c. SetSet of similar indicators, quality of sub-process, quality sub-criterion or criterion.

d. The audited statisticsaudited statistics as the whole.


www.czso.cz

17

Suggestions ->

Collection ->

Calculations ->

Assessment ->

Feedback and actions

5.Lessons learned5.Lessons learned1. Links to the phases of the SBP1. Links to the phases of the SBP

Specify needs

Design

Build

Collect

Process

Analyse

Disseminate

Archive

Evaluate

Metadata on Quality

setting monitoring assessment


www.czso.cz

18

5. Lessons Learned5. Lessons LearnedCross-sectional aspects and Time coordinationCross-sectional aspects and Time coordination

2.2. Cross-sectional aspectsCross-sectional aspects• Ensure applicability for differentdifferent types of statistictypes of statisticss• Avoid duplicitiesduplicities, arrange linkslinks with other SMS subsystems• Involve expertsexperts:

• SMS-Quality project team• quality methodologists• subject-matter statisticians• ICT experts, members from other project teams

3.3. Time coordination Time coordination • Design and implementation of SMS-QUALITY and other SMS

subsystems to be mutually coordinated – Committee for the mutually coordinated – Committee for the Redesign of SIS and SMSRedesign of SIS and SMS


www.czso.cz

19

5. Lessons Learned5. Lessons Learned4.4. SMS-Quality project teamSMS-Quality project team

• Appointed by the top-management

• Suggests schedule of the SMS-QUALITY activities

• Regularly reports to the top-management

• Designed QF-map (Architecture of SMS-QUALITY) =>

• Proposes SW application (content, functions) incl. updates

• Coordinates implementation and cooperation among activities of involved experts


www.czso.cz

20

5.Lessons Learned5.Lessons Learned 5. Role of subject-matter statisticians5. Role of subject-matter statisticians

and methodologists and methodologists (a) (a)

Testing phaseTesting phase• Quality methodologists

• Define Q-attributes and explanatory notes

into SW application• Test the SW applications• Administrate SMS-Quality, QF Map, QAG• Provide scripts for calculations of quality indicators• Suggest scales for quality assessment

• Subject-matter statisticians• Provide data for tests• Adjust scales for quality assessment• Manage routine quality monitoring, assessment


www.czso.cz

21

Full implementationFull implementation

• Q-methodologists manage• Quality methodology updates (according to the ESS

development)• Support subject-matter statisticians• Administration of SMS-QUALITY, updates

• Subject-matter statisticians manage• Collection of Q-attributes (automatic or manually)• Quality assessment, self-assessment• Approval of results of assessment

5.Lessons Learned5.Lessons Learned 5. Role of subject-matter statisticians5. Role of subject-matter statisticians

and methodologists and methodologists (b) (b)


www.czso.cz

22

5.Lessons Learned5.Lessons Learned6. Compliance with ESS quality framework6. Compliance with ESS quality framework

• SMS-QUALITY should be designed as a flexible tool to ensure easy methodology updates, taking into account development on the ESS level.

• Application software should have necessary flexibility concerning collection, monitoring and assessment procedures.


www.czso.cz

23

6. Conclusions6. Conclusions

• Development of SMS-QUALITY has been scheduled with highest priority for the next two years.

• Further progress in development of SMS-QUALITY will depend • On available human and financial resources due to fact the

same experts are involved in real production of quality reports and in development of SMS-QUALITY.

• On progress of other SMS subsystems.

• Development of quality methodology on both national and international levels shall be taken into account.


www.czso.cz

24

Thank you for your attentionThank you for your attention..

[email protected]

czech statistical office 1 the quality metadata system in the czech statistical office work session...

Documents