institut de la statistique du québec (isq) surveys: compendium of good practices and quality...
TRANSCRIPT
Institut de la statistique du Québec (ISQ) surveys: compendium of good practices and quality self-assessment checklist
European Conference on Quality in Official Statistics
Rome, July 8-11, 2008
Louise BourqueFrance Lapointe
Institut de la statistique du Québec
2
Presentation outline
• Context
• Compendium of good survey practices– Development
– Content
• Survey quality self-assessment checklist– Objectives
– Development
– A few observations by users (pre-test)
• Challenges in applying the tools
• Conclusion
3
Context
• The Institut de la statistique du Québec (ISQ)– Its enabling legislation
“... to provide reliable and objective statistical information” “... in such a manner that comparisons in and outside Québec
may be made”
– Organization of work A number of units work together in conducting a survey
4
Context (cont.)
• Specific work dealing with quality incorporated into everyday activities – Variation according to the project and unit (neither systematic
nor harmonized)
• The ISQ has adopted an Integrated Quality Management Framework– General Quality Management Policy (November 2005)
5
Context (cont.)• General definition of quality
Quality is composed of all the characteristics that affect the capacity to meet the needs of users
• Six dimensions of quality– Relevance– Reliability and objectivity– Comparability– Timeliness– Intelligibility– Accessibility
6
Context (cont.)
• The policy refers to many documents (policies, procedures, programs, reports, strategic declarations, etc.)
• More specifically, survey quality assurance tools– Compendium of good practices
– Self-assessment checklist
7
Compendium of good survey practices
• Objective– Promote the use of good practices in conducting surveys, at
all stages, in order to ensure the quality of the results
• Development– Multidisciplinary team
▪ Bringing together all the entities involved in the carrying out of surveys
▪ Mandated by top manager▪ Work directed and monitored by a senior management
committee
8
Compendium of good survey practices (cont.)– Inventory of reference practices
Statistics Canada, Statistics Finland ... Division of the survey process into 16 stages
– Observation of ISQ practices 10 or so surveys selected as representative of diversity Application of a common observation checklist
o Adaptation of the European DESAP checklist
– Comparison ISQ practices vs. reference practices Choice of selected practices
9
Compendium of good survey practices (cont.)• Content of a section related to a stage
– Introduction Overview of the work to be done and links with quality dimensions
– Definitions (non-specialized reader)
– Principles (reader knows how a survey is conducted) General quality assurance policy directions
– Good practices (specialized reader) with ratings(1) Essential practices(2) Highly recommended practices(3) Recommended practices
10
Self-assessment checklist
• Objectives– Survey self-assessment and quality measurement according to
the six quality dimensions selected
– Checklist at the time of design, and throughout the conducting of a survey
– Verification of the suitable application of the compendium’s good practices
– Training for new employees
– Etc.
11
Self-assessment checklist (cont.)
• Development– Multidisciplinary team
– In three stages Adapt the DESAP checklist for observation of practices at the
ISQ (done) Review adaptation following use for observation (pre-test) with
the comments received (in progress) Ensure consistency with the compendium's good practices (in
future)
12
Self-assessment checklist (cont.)
• A few observations further to the pre-test– Logical structure of the questionnaire
– Recognized usefulness of the checklist
– High response burden Long questionnaire Complex questions, difficult to interpret
– Questionnaire format to be revised
13
Tool application challenges
• The tools must ...– Be in keeping with the ISQ’s reality– Gain the approval of the various entities
• The tools must be developed ...– At the least cost
– While ensuring that everyday activities continue
14
Tool application challenges (cont.)
• Tool introduction must ...
– Ensure the tools’ appropriation and use by employees
• Tool use must not ...– Be too constraining
– Require too many resources
15
Tool application challenges (cont.)
• General approach selected – Not to reinvent the wheel
– Actively involve the units concerned
– Be guided and supported by senior management
– Inform and train the staff
– Incorporate means of monitoring application
16
Conclusion• Current and future work ...
– Compendium of good practices Information and training activities for employees (in progress)
– Finalize the self-assessment checklist (in progress)
– Develop and adopt a survey quality policy (in future) Monitor the use of the compendium and checklist
