quality reporting within the eurostat and the ess metadata systems
DESCRIPTION
Quality reporting within the Eurostat and the ESS metadata systems August Götzfried and Håkan Linden Eurostat Unit B6: Reference databases and metadata. Current situation. - PowerPoint PPT PresentationTRANSCRIPT
1
Quality reporting within the Eurostat and the ESS metadata systems
August Götzfried and Håkan Linden
Eurostat Unit B6: Reference databases and metadata
2
Current situation
Within the European Statistical System (ESS) reporting on statistical data quality exists in many statistical domains….
3
… BUT :
– Quality reports do not exist for all statistical processes within the ESS;
– No homogeneity between the different report structures used for data quality reporting;
– Not all the quality related information is made publicly available;
– No common and standard IT infrastructure is used within the ESS;
The new Eurostat vision: “Improving the production method of EU statistics” requires an improvement action.
Problem statement
4
Progress made since 2008
2008: introduction of the Euro SDMX Metadata Structure (ESMS) at Eurostat and the ESS for the production and dissemination of reference metadata (Commission Recommendation 498/2009)
01/2009: release of the new version of the ESS quality reporting documents:
• ESS Standard for Quality Reports (ESQR)• ESS Handbook for Quality Reports (EHQR)
Detailed requirements following the European Statistics Code of Practice ESS Quality and Performance Indicators (QPI’s) defined
03/09: EP/Council Regulation 223/2009 Article 12 defining the quality criteria to be reported
2009/2010: Development and deployment of the Eurostat Metadata Handler with:
• EMIS: production and dissemination of ESMS files at Eurostat • National Reference Metadata Editor (NRME): production, transmission and
dissemination of national ESMS files
5
ESMS and ESQR
ESMS is more oriented to the USERS of statistics
to understand the statistical data released there is no need for too detailed information on data quality 21 SDMX cross domain concepts used
ESQR is more oriented to the PRODUCERS of statistics
to monitor the quality of the statistics produced in detail concentrating on the main quality concepts (being also part of the ESS Statistics Regulation No 223/2009)
However, there is information on quality criteria which is common to both ESMS and ESQR.
6
ESMS and ESQR: the starting point
XI. Conclusions
X. Confidentiality,
IXI. Performance, Cost and Respondent
Burden
VIII. Assessment of User needs and
Perceptions
VII. Trade-offs between Output Quality Components
I. Introduction to the Statistical Process and
Its Outputs
XI. Conclusions
X. Confidentiality,
IXI. Performance, Cost and Respondent
Burden
VIII. Assessment of User needs and
Perceptions
VII. Trade-offs between Output Quality Components
I. Introduction to the Statistical Process and
Its Outputs
VI. COMPARABILITYand COHERENCE
V. ACCESSIBILITYand CLARITY
and PUNCTUALITY
IV. TIMELINESS
III ACCURACY
II. RELEVANCE
14. Accuracy and reliability
7. Confidentiality
20. Statistical processing
13. Relevance6. Institutional mandate
19. Data revision12. Quality management
5. Reference period
18. Cost and burden11. Accessibility of documentation
4. Unit of measure
17. Coherence10. Dissemination format
3. Statistical presentation
16. Comparability9. Frequency of dissemination
15. Timeliness and punctuality
8. Release policy
14. Accuracy and reliability
7. Confidentiality
20. Statistical processing
13. Relevance6. Institutional mandate
19. Data revision12. Quality management
5. Reference period
18. Cost and burden11. Accessibility of documentation
4. Unit of measure
17. Coherence10. Dissemination format
3. Statistical presentation
16. Comparability9. Frequency of dissemination
15. Timeliness and punctuality
8. Release policy
ESQR ESMS
TIMELINESS
COMPARABILITY
2. Metadata update
1. Contact
ACCESSIBILITYCOHERENCE
CLARITY
ACCURACY
RELEVANCE
ACCURACY
21. CommentTransparency, Security
7
The new ESQRS Based on the ESQR, a new report structure - the ESS
Standard for Quality Reports Structure: ESQRS - was created for harmonising the reporting on statistical data quality within the ESS.
The ESQRS is using the main statistical data quality criteria as listed in EP/Council Regulation 223/2009 and as being part of the ESMS and details them further :
• Relevance• Accuracy• Timeliness and Punctuality• Accessibility and Clarity• Comparability• Coherence
A subset of the Quality Performance Indicators (QPI’s) is also covered in the new ESQRS.
8
ESQR and ESQRS
XI. Conclusions
X. Confidentiality,
IXI. Performance, Cost and Respondent
Burden
VIII. Assessment of User needs and
Perceptions
VII. Trade-offs between Output Quality Components
XI. Conclusions
X. Confidentiality,
IXI. Performance, Cost and Respondent
Burden
VIII. Assessment of User needs and
Perceptions
VII. Trade-offs between Output Quality Components
ESQR ESQRS
Transparency, Security
III ACCURACY
IV. TIMELINESSand PUNCTUALITY
V. ACCESSIBILITYand CLARITY
VI. COMPARABILITYand COHERENCE
II. RELEVANCE
I. Introduction to the Statistical Process
and Its Outputs
IntroductionII IntroductionII
Relevance (user needs and perceptions)
III Relevance (user needs and perceptions)
III
AccuracyIV AccuracyIV
Timeliness and punctualityV Timeliness and punctualityV
Accessibility and clarityVI Accessibility and clarityVI
ComparabilityVII ComparabilityVII
CoherenceVIII CoherenceVIII
ContactI ContactI
CommentIX CommentIX
9
The ESQRS
Concept nameESQRS Concept nameESQRS
Average size of revisions (A8)III.3.7.3
Data revision – policyIII.3.7.1
Data revision - practiceIII.3.7.2
Imputation rate (A6)III.3.4.1
Item response rate (A5)III.3.3.3
Unit response rate (A4)III.3.3.1
Edit failure rate (A3)III.3.2.1
Rate of overcoverage (A2)III.3.1.1
Seasonal adjustment III.3.8
Data revisionIII.3.7
Mistakes (A7)III.3.6
Model assumptionsIII.3.5
Processing errorsIII.3.4
Non response errorsIII.3.3
Measurement errorsIII.3.2
Coverage and other frame errorsIII.3.1
Non-sampling errorIII.3
Coefficient of variation (A1)III.2.1
Sampling errorIII.2
Overall accuracyIII.1
AccuracyIII
Average size of revisions (A8)III.3.7.3
Data revision – policyIII.3.7.1
Data revision - practiceIII.3.7.2
Imputation rate (A6)III.3.4.1
Item response rate (A5)III.3.3.3
Unit response rate (A4)III.3.3.1
Edit failure rate (A3)III.3.2.1
Rate of overcoverage (A2)III.3.1.1
Seasonal adjustment III.3.8
Data revisionIII.3.7
Mistakes (A7)III.3.6
Model assumptionsIII.3.5
Processing errorsIII.3.4
Non response errorsIII.3.3
Measurement errorsIII.3.2
Coverage and other frame errorsIII.3.1
Non-sampling errorIII.3
Coefficient of variation (A1)III.2.1
Sampling errorIII.2
Overall accuracyIII.1
AccuracyIII
IntroductionI IntroductionI
Rate of available statistics (R1)II.3.1
CompletenessII.3
User satisfaction survey – dateII.2.2
User satisfaction index (US1)II.2.1
User satisfactionII.2
User needsII.1
Relevance (user needs and perceptions)
II
Rate of available statistics (R1)II.3.1
CompletenessII.3
User satisfaction survey – dateII.2.2
User satisfaction index (US1)II.2.1
User satisfactionII.2
User needsII.1
Relevance (user needs and perceptions)
II
Quality documentationV.7
Metadata –completeness (AC3)V.6.1
On-line database – accesses (AC2)
V.3.1
Publications – number (AC1)V.2.1
PublicationsV.2
On-line databaseV.3
Micro-data accessV.4
OtherV.5
News releaseV.1
Documentation on methodologyV.6
Accessibility and ClarityV
Quality documentationV.7
Metadata –completeness (AC3)V.6.1
On-line database – accesses (AC2)
V.3.1
Publications – number (AC1)V.2.1
PublicationsV.2
On-line databaseV.3
Micro-data accessV.4
OtherV.5
News releaseV.1
Documentation on methodologyV.6
Accessibility and ClarityV
Concept nameESQRS Concept nameESQRS
Punctuality – publication (T3)IV.2.1
PunctualityIV.2
Timelag – final results (T2)IV.1.2
Timelag – first results (T1)IV.1.1
TimelinessIV.1
Timeliness and PunctualityIV
Punctuality – publication (T3)IV.2.1
PunctualityIV.2
Timelag – final results (T2)IV.1.2
Timelag – first results (T1)IV.1.1
TimelinessIV.1
Timeliness and PunctualityIV
Coherence – cross domainVII.1
Length of comparable time series (CC1)
VI.2.1
Assymetries for statistics mirror flows (CC2)
VI.1.1
CoherenceVII
Coherence – internalVII.2
Coherence with other statisticsVII.1.3
Coherence – National AccountsVII.1.2
Coherence –subannual and annual statistics
VII.1.1
Comparability – domainsVI.3
Comparability - over timeVI.2
Comparability – geographicalVI.1
ComparabilityVI
Coherence – cross domainVII.1
Length of comparable time series (CC1)
VI.2.1
Assymetries for statistics mirror flows (CC2)
VI.1.1
CoherenceVII
Coherence – internalVII.2
Coherence with other statisticsVII.1.3
Coherence – National AccountsVII.1.2
Coherence –subannual and annual statistics
VII.1.1
Comparability – domainsVI.3
Comparability - over timeVI.2
Comparability – geographicalVI.1
ComparabilityVI
= Concepts in common with ESMS
10
The ESMS and the ESQRS
The metadata produced in the ESMS and ESQRS need to be kept consistent. The ESQRS is based on the ESQR, but not taking up all the chapters contained in the latter one.
The information in the ESQRS is more detailed compared to the information on statistical data quality contained in the ESMS.
ESQRS reports deeper in terms of data quality compared to the ESMS
11
The ESMS and the ESQRS
Accuracy and reliability Non- sampling error
Description:Accuracy:closeness of computations or estimates to the exact or true values that the statistics were intended to measure.Reliability: closeness of the initial estimated value to the subsequent estimated value.
ESMS
Description:Error in survey estimates which cannot be attributed to sampling fluctuations.
Non- response errorNon- response error Unit response rateUnit response rate Formulae unit resp. rateFormulae unit resp. rate
Description:The difference between the statistics computed from the collected data and those that would be computed if there were no missing values.
Description:The ratio of the number of units for which data for at least some variables have been collected to the total number of units designated for data collection.
Description:
Ex. calculation formluae for un-weighted unit response rate.
Accuracy Non- sampling error
Description:Accuracy:closeness of computations or estimates to the exact or true values that the statistics were intended to measure.
Description:Error in survey estimates which cannot be attributed to sampling fluctuations.
ESQRS
12
ESS Guidelines
The guidelines for quality reporting from ESS Handbook for Quality Reports (EHQR) are already used in the “ESS Guidelines” for ESMS.
These guidelines will be further used in the ESQRS in order to provide detailed guidelines for 6 different statistical processes:
• Sample survey• Census• Statistical Process using Administrative Sources• Statistical Process involving Multiple Data Sources• Price or other Economic Index Process• Statistical Compilation
13
The underlying IT infrastructure
The Eurostat Metadata Handler as IT tool for the production, transmission and dissemination of the ESQRS metadata.
National Metadata Editor
National Metadata Editor
EMISEMIS
RAMONRAMON CODEDCODED
Eurostat Metadata Handler
Common user interface
Eu
ro S
DM
X
Reg
istr
y
14
The ESQRS Statistical Business Process
NationalMetadata
Editor
NRME Database Eurostat Website
NationalESQRS
NationalESQRS
National and
Eurostat ESQRS
eDamis
PRODUCTION TREATMENT AND ANALYSIS
DISSEMINATION
NATIONAL STATISTICAL AUTHORITY
EUROSTAT
15
Production and dissemination of metadataat national and European level
EU Member States Eurostat
ESQRS - Production of national ESQRS - Transmission of national ESQRS to
Eurostat- Dissemination of national ESQRS
(if decided so)
- Production of the Eurostat ESQRS based on the national ESQRS
- Dissemination of the Eurostat ESQRS if decided so
- Checking and dissemination of the national ESQRS (dissemination only if decided so)
ESMS - Production of national ESMS files- Transmission of national ESMS
files to Eurostat- Dissemination of national ESMS
files (if decided so)
- Production of the Eurostat ESMS files
- Dissemination of the Eurostat ESMS files
- Checking and dissemination of the national ESMS files (dissemination only if decided so)
16
Summary
New reporting structure for quality related metadata has been created: the ESQRS.
The ESQRS is based on the existing EU legislation and documentation for data quality in the ESS.
The quality indicators contained in the ESQRS allow the harmonised measurement/ monitoring of the statistical data quality within and across statistical processes.
The ESS quality reporting will successively be converted into the ESQRS by the use of the National Reference Metadata Editor
The ESQRS needs to be further promoted and communicated
within and beyond the ESS.