Quality Reporting at SORS – Experiences and Future

Perspectives

Rudi Seljak, Tina OstrežStatistical Office of the Republic of Slovenia

The summary of the presentation

• History of quality reporting at SORS • Feedback from the producers of quality reports • Expanding of the area of quality reporting • Future plans• Conclusions

Quality reporting at SORS

• The systematic work on quality reporting at SORS began in 2003.

• On the basis of the “European” methodological documents our own quality reporting framework was developed.

• Cornerstones of the framework:– The standard template with prescribed structure for quality

reporting– The list of standard quality indicators

• The first quality reports were prepared in 2004 and 2005 → just for internal use.

• In 2006 the first publicly available reports were disseminated.

Quality reporting at SORS cont’d

• Two types of Quality reports are disseminated:

Standard quality reports• Produced every five years• Exhaustive document →

covering all aspects of quality• Available only in Slovene

language

http://www.stat.si/metodologija_porocila-standardna.asp

Annual quality reports• Produced every year• Short document → mostly

based on quality indicators• Translated also to English

http://www.stat.si/eng/metodologija_porocila-kakovost.asp

Procedure of preparation

• At the beginning of the year the domains, for which the reports should be prepared, are identified.

• The workshop with the subject-matter statisticians is organized:– The theoretical and practical views of quality assessment are

presented

– The timetable is set up • During the preparation of the reports further assistance

is provided by the coordinators.• When the report is written, it is reviewed by the

coordinators before the final version is accomplished.

Survey among producers

• A small-scale survey was conducted among the persons involved in QR preparation in order to:– get the general feedback (burden, usefulness,…)– get the useful information for future planning

• The survey took place in December 2009.• From around 40 standard quality reports prepared by

that time, we obtained answers for 32 domains.

Main findings of the survey

• The perceived burden due to the preparation of standard quality reports is high, but varies a lot: 8-200 hours (average: 46 hours).

• For annual quality reports: 1 – 50 hours (avarage:17 hours)

• Double work in the case where the QR from Eurostat is demanded was pointed out → degree of comparability of two concepts sometimes found very low

• Also some positive feedback were provided:– The organization of work was found very good– The preparation has forced them to think about their own work in

a new way– Deeper insight into the data they are preparing

Expanding the area of quality reporting

• The concepts set up in 2003 mainly targeted the “classical” statistical surveys.

• Use of registers and different administrative sources is very popular in practice of SORS → need for adjusted concepts clearly pointed out.

• The template as well as the list of quality indicators should be adjusted.

• Also the need for adjustments for some specific domains (price statistics, national accounts) was pointed out.

Quality assessment and administrative data

• Some distinctive characteristics of quality assessment when admin data are used: – The data were not collected for the statistical purposes →

suitability for statistical purpose must carefully be studied → relevance dimension becomes a crucial part of the assessment.

– The administrative data are determined by the legal world → danger of describing the world as it is “de-iure” instead of as it is “de-facto” → consideration of this problem should be put in the quality report.

– The data collection process is dislocated from the statistical process → often wrongly assumed that data are free of measurement (as well as non-response, coverage) errors. To assess these errors close cooperation with the data provider is needed.

Plans for expansion of the quality framework

• The adjusted quality concepts for the surveys which use administrative data or combine different sources:– Adjusted (more flexible) template for quality reporting– Adjusted list of standard quality indicators– Planed to be developed in 2010

• Quality reporting concepts for registers and large administrative data sources (shared by several surveys):– New template should be developed– The standard list of quality indicators will probably essentially be

adjusted– Planed to be developed in 2011 and implemented in 2012

More plans for the future

• The quality reports should be accomplished for around 50 additional statistical surveys.

• The further harmonization with the Eurostat concepts should be carried on.

• Improvements in the availability of the quality reports on the website are planned.

• The completion of the database of quality indicators and its inclusion in the process of quality reports preparation is one of the main tasks in the following years.

Conclusions

• SORS publicly disseminates quality reports from 2006 on.

• Two types of reports are produced: standard quality reports and annual quality reports.

• The survey among the producers of the reports showed that the perceived burden of the preparation is quite high but also some positive consequences were pointed out.

• The main task for the future is to adjust the concepts for all different kind of surveys and to ease the burden of the preparation.