a basis for better decision making: improvements to quality reviews and reporting catherine bremner...
TRANSCRIPT
A Basis for Better Decision Making: Improvements to Quality Reviews and Reporting
Catherine Bremner and Sarah Tucker
Office for National Statistics
Motivation
• Quality Centre is Centre of methodological expertise in statistical quality and respondent burden
• Working over the past 12 months to streamline current quality initiatives and drive innovation
Outline
• Introduction to quality reporting• Quality reporting developments• Using quality reports to drive quality
improvements• The new quality review process at ONS
Quality Reporting Overview
• Why do we report on Quality?
• Give users enough information to decide on suitable uses for the data
• Meet obligations under Code of Practice • Meet requirements from Eurostat
Quality and Methodology Information
• Static quality information• Updated on annual basis or when there are
major methodological changes• Includes:
• history of output • details of statistical process • reports against European Statistical System
Quality Dimensions
• Extensively quality assured
Quality and Methodology Information
Quality and Methodology Information
• Current format 5 years old – time to review • Changes to website – accessibility• More information on user personas - Inquiring
Citizen, Data Forager and Expert Analyst• Need to change the way we present the data
– content unlikely to change much as meets our obligations
Reviewing Presentation of Quality Information
• Aims to extend the reach of quality information to user personas – alongside current QMIs
• Investigations currently focussing on information that users need so they can’t misuse data
• First steps: consulting with data producers, methodologists and other stakeholders
Reviewing Presentation of Quality Information
• One size does not fit all• Pool of suggested pieces of information to
choose from• Fit together to give picture of how not to
misuse data• Held focus groups/workshops and individual
meetings to gain thoughts on what should be considered for inclusion
Internal Consultation Ongoing – Points Raised
• Four focus groups have been held - 2 each for output managers and quality assurers
• Individual meetings with output managers and other stakeholders
• Some themes are coming through consistently: uncertainty – is it an estimate etc, periodicity, coherence, accuracy and health warnings.
• As expected, emphasis is different across type of outputs – admin data vs. Survey
Next steps
• Compiling a shortlist of these vital pieces of information
• Work with data producers to produce pilot• User test• Review results• Propose action
Quality Reviews at ONS
• Previously carried out Annual Quality Reviews
• Made use of Quality, Methodology and Harmonisation Tool (QMHT)
• Responding to feedback, developed a new process during summer 2014
Regular Quality Reviews (RQR)
• Face to face meeting between Output manager, Deputy Director, G6/G7 Methodologist and Quality Centre representative
• Makes use of existing information from Value Engineering and Quality and Methodology Information (QMI) reports, which are discussed at the meeting
• Quality Assurance (DD) walk-through incorporated into the meeting
Regular Quality Reviews (RQR)
• G6/G7 methodologist produces bespoke recommendations, which will be presented at Management/Project boards
• Output reviewed once every 3 years• Supports the Code of Practice • Piloted on 3 outputs during summer 2014• Process rolled out in November 2014; 13
reviews conducted to date
Why are RQRs required?
• Support requirements of the Code of Practice• Assesses the output• Adds value to output through bespoke
recommendations• Allows recommendations to be addressed,
improving the output
What do we expect?
• What we already require - QMI, Quality Assurance (DD) walkthrough, Quality checklist
• Quality Centre collate all the relevant documents e.g. QMI, Value Engineering, Assessment Reports to be reviewed
• Output Manager, DD, and team members (as required) to meeting
Recommendations – Adding Value
• Aim to write up within two weeks of the meeting
• Assigned a RAG status• Prioritised from a Methodological and ONS
view• Timetable of when/how recommendations are
dealt with depend on what they are• Range from a quick win – update QMI or
longer term plan of getting work onto business plan
• Role for Quality Centre in monitoring
Relevance for GSS
• Regular Quality Review approach is a tool that could be used by GSS
• Expected that senior statistician could carry out review
• Alternatively, support from Methodology Advisory Service
• The approach will be included in the GSS quality resources
Contacts
• Quality Reporting Sarah Tucker
email: [email protected]
• Regular Quality Reviews
Catherine Bremner, email: [email protected]