statistical data editing - unece work session – oslo 24-26 september 2012 proposal of a revised...
TRANSCRIPT
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
Proposal of a revised approach for data validation within the European Statistical System
Michel HENRARDEuropean Commission – EUROSTAT
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
The current situation
Validation process : 2 separated steps
– Member States
– Eurostat
Double work ? Zero Work ? Ping-Pong ?
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
The current situation
Harmonisation
– Validation procedures• Harmonised?• Documented ?• Formally agreed? • Coordinated ?
– Validation rules• Common syntax ?• Formalised ?
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
The current situation
Harmonisation of software solutions
– Standard building blocks within Eurostat• Partial degree on integration• Many ad-hoc solutions
– High number of supported solutions• Does not help streamlining of production processes• High cost of development and maintenance• Common validation solutions within the ESS ?
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
The current situation
Compliance monitoring
– Assessment of data quality
• Subjective interpretation• Not structured
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
Yielding efficiency gains
Vision Infrastructure Project on Validation (end 2010)
– Validation solutions that can be used both by the Member States & Eurostat
– Validation as soon as possible
– Definition of the roles in the validation chain
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
Project proposals
Better & standard documentation of the validation process
– Should be part of the general documentation of the complete production system
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
Project proposals
Agreed & standard description of checking rules
– Common language (syntax)
– Guidelines for the selection of the validation rules
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
Project proposals
Availability of common software solutions
– Common standard IT tools and associated services
– Automation of procedures within the ESS
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
Project proposals
Clear & agreed distribution of responsibilities
– Validation where it can be done in the most efficient way
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
How ?
Process Description
– Systematic standard description of• The data validation• The validation rules using a common syntax
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
How ?
Validation review with Member States
– Gradual introduction of objective measurement of data quality
– Clear responsibility for each validation rule based on efficiency considerations (the sooner the better)
– Calendar of introduction of the new mechanism
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
How ?
Integration of validation in compliance assessment procedures
– Based on the agreed set of validation rules
– Reports on validation problems in the statistical domains
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
How ?
Improve the coherence of validation solutions within Eurostat
– Taxonomy of validation rules– Standard description of the validation rules (common
language/syntax) – Good practices– Current validation building blocks adapted to the users’
requirements
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
How ?
Provision of standard solutions to be shared in the ESS
– Deployment of a standardized validation language
– Adapting the IT tools to be shared in Eurostat & to be offered to Member States
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
What the VIP on Validation does provide ?
Templates & Guidelines
– Documentation of the validation process
– Error messages and reporting
– Selection criteria of validation rules ensuring minimum quality standard
– Attribution of responsibility of validation tasks
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
What the VIP on Validation does provide ?
Inventory of validation rules used in Eurostat
Typology of validation rules
Human understandable language for validation rules
User requirements & functional specifications to adapt the IT tools to the needs identified
Statistical data editing - UNECE work session – OSLO 24-26 September 2012
Q&A
Thank you for your attention