data collection and processing, data quality management seminar on global assessment yalta, 23 - 25...
TRANSCRIPT
![Page 1: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/1.jpg)
Data collection and Data collection and processing, data qualityprocessing, data quality
Management seminar on global assessment
Yalta, 23 - 25 September 2009Session 10
1
Heinrich Brüngger
10
![Page 2: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/2.jpg)
Part I: Data collection and processing
Part II: Confidentiality (with emphasis on data collection and processing)
Part III: Data quality (quality management)
2
![Page 3: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/3.jpg)
Nature of decisions on Nature of decisions on methods of data collection methods of data collection and processingand processingUnder professional independence (FP
2), therefore within the statistical system without outside interference once the decision about the “what” to measure in broad terms have been made
Decisions about what official statistics should measure may involve the political level, and have to involve discussions with a broad range of users
3
![Page 4: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/4.jpg)
Professional Professional independenceindependence
The “how” part of statistical activities has to focus exclusively at obtaining the most reliable representation of the phenomenon of the real world with the given resources
Scientific methods, international standards, and empirically established good practices are the best guide for professional independence
The statistical law may require the adoption by government in the case of statistical surveys because of explicit legitimacy in the interaction with respondents and the burden caused to respondents, but governments should not interfere in methodological issues
4
![Page 5: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/5.jpg)
Main data sources (FP 5)Main data sources (FP 5)Official statistics has to have the
right to use three types of sources:◦Statistical surveys: data collection
from individuals, households, corporate and unicorporated businesses, and public entities outside the own government structure, for the exclusive purpose of statistics (most of which official statistics)
◦Administrative records: secondary use of data collected primarily for administrative purposes about individuals or private businesses by other parts of the same government structure
5
![Page 6: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/6.jpg)
Main data sources (ctd.)Main data sources (ctd.)
◦Secondary use of environment and territorial observation and monitoring data collected by specialised government agencies
For statistical surveys, the statistical law applies to all phases of the statistical activity; for the other two sources, primary data collection has to be based on other laws, and the statistical legislation becomes only applicable once data are handed over to a producer of official statistics
6
![Page 7: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/7.jpg)
Statistical programmesStatistical programmesIf the law requires input-oriented
programmes, a clear distinction has to be made between statistical surveys, where the statistical law and the statistical programmes are the legal basis for data collection, and the second and third data source, where the primary data collection is for another purpose and has therefore to have a different legal basis
7
![Page 8: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/8.jpg)
Minor data sources for Minor data sources for national official statisticsnational official statisticsOfficial statistics from a partner
country or from an international organisation (especially for international flows)
Registers in the hand of specialised private companies (as one of the sources used to keep statistical registers updated)
8
![Page 9: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/9.jpg)
Statistical surveysStatistical surveys
Exhaustive vs. sample surveys (mixed forms possible)
Mail, face to face interview, telephone interviews, internet (mixed forms possible)
Compulsory participation of the respondent vs. right to decline participation (mixed forms possible)
Recall and follow-up policy for non-response, depending on type of survey
9
![Page 10: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/10.jpg)
Response burdenResponse burdenResponse burden has to be
assessed in advance and taken into account:◦all new surveys have to be tested first
(pilots)◦all respondents have to be informed
about the purpose and legal basis of the survey, and especially about the confidentiality measures
◦forms and questions have to be understandable, non-intrusive, and make answers possible for him/her from memory or from existing material (businesses) 10
![Page 11: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/11.jpg)
Response burden (ctd.)Response burden (ctd.)◦first reminders have to be
proportionate ◦response rates have to be closely
monitored
The NSI has also to act as custodian for the respondents; users will tend to minimise the issue of response burden
11
![Page 12: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/12.jpg)
Surveys (ctd.)Surveys (ctd.)Information given by a respondent in
statistical surveys have no legal implications for him; they can (and should) therefore be checked and when necessary altered (edited) by the NSI
In many cases, the characteristics of interest to users cannot be taken over directly into questionnaires, but have to be compiled as derived characteristics using several questions
12
![Page 13: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/13.jpg)
Non-response (valid for all Non-response (valid for all data sources)data sources)The NSI has to develop a policy for
treating general and item non-response, which has to be subsequently fine-tuned and documented for each survey
Imputations for missing (or unusable) information is legitimate, but a careful balance has to be found between “not enough” and “too much” (risk of introducing biases)
13
![Page 14: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/14.jpg)
Administrative sourcesAdministrative sourcesThe statistical office must have the
legal and de facto right to receive, for its tasks, regularly and on an ad hoc basis, microdata sets from other ministries and public entities taken from their administrative sources
This does not mean that direct identifiers have to be included in all cases, but the possibility should not be entirely excluded
14
![Page 15: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/15.jpg)
Administrative sources Administrative sources (ctd.)(ctd.)Statistical producers have the right to
alter (edit an impute) the administrative data received from other ministries to improve compatibility with statistical definitions and classifications
Data received in this way should never be given back to the data owner or transferred by the statistical producer to a third party for administrative purposes
15
![Page 16: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/16.jpg)
Relationship between data Relationship between data sourcessourcesBefore considering introducing a new
survey, or expanding an existing one, for new information needs, other possible data sources (especially administrative) have to be examined for their potential
The possibility of combining questions about different areas in the same survey is limited: the survey system has to be structured carefully
16
![Page 17: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/17.jpg)
Relationship between data Relationship between data sources (ctd.)sources (ctd.)Administrative sources are in constant
change, and some risk of disappearing completely
The final decision-making about administrative sources is outside the statistical system. Therefore, the NSI has the responsibility to constantly monitor (if possible anticipate) and react to planned changes that may affect the use of administrative sources for official statistics positively or negatively
17
![Page 18: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/18.jpg)
Relationship between data Relationship between data sources (ctd.)sources (ctd.)
NSIs have to avoid considering data sources, and especially surveys, as stove-pipe type of parallel operations, but rather as a system of interrelated operations with n:m relationship between data sources and results
Very often, the best estimates are based on a judicious combination of sources, combining the individual strengths and reducing the individual weaknesses
Statistical registers of good quality are a cornerstone for a good survey system, and for matching different sources
18
![Page 19: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/19.jpg)
Relationship between data Relationship between data sources (ctd.)sources (ctd.)When results on the same issue are
published from different sources separately (i.e. without prior integration), an accounting relationship between the results should be established, with the components quantified if possible
Otherwise, only the most complete aggregate should be released as results of official statistics in absolute terms
19
![Page 20: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/20.jpg)
Confidentiality (FP 6)Confidentiality (FP 6)
Statistical confidentiality aims at protecting the privacy of individual units about which data are collected and processed. It has two components:◦Producers of official statistics do not disclose, either directly or indirectly, characteristics about protected units to any third party in such a way that any user might derive additional information about a protected unit 20
![Page 21: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/21.jpg)
FP 6 (ctd.)FP 6 (ctd.)Producers of official statistics use data about
protected individual units only for statistical purposes (official statistics in the first place)
The statistical system, and in the first instance the NSI, have to make sure in an effective a credible way that the two elements of the confidentiality principle are strictly followed in all parts of the production and dissemination processes, and that confidential data are stored in a safe way
21
![Page 22: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/22.jpg)
Confidentiality in data Confidentiality in data collectioncollection
◦Legal situation in countries may differ with respect to: Protected units: physical persons, private
households, private businesses (whether legal persons or unincorporated enterprises) should be fully protected; government units as institutions cannot invoke statistical confidentiality; public enterprises in a competitive market are a borderline case
All characteristics treated alike, or some considered to be free (e.g. for legal persons), or particularly sensitive (higher degree of protection, e.g. about health, crime, ethnicity)
22
![Page 23: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/23.jpg)
Confidentiality in data Confidentiality in data collection (ctd.)collection (ctd.)
◦Data collection in statistical surveys: replies of respondents should go back as directly as possible to the statistical producer. No other government agency should be involved as intermediary (regional level!)
◦Data processing: except in the context of statistical registers, identifiers of units should be separated from the context variables by the statistical producer at an as early stage as possible, and either stored separately or destroyed
23
![Page 24: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/24.jpg)
Confidentiality in data Confidentiality in data collection (ctd.)collection (ctd.)
◦Data security: final (edited) microdata sets have to be stored safely, and access limited to those within the NSO who use the data regularly. No direct access from outside the NSO should be possible. For files with identifiers (other than statistical registers), each access has to be documented.
◦Data security, update and access regulations for statistical registers have to be worked out carefully in a separate book of rules.
24
![Page 25: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/25.jpg)
Quality management: Quality management: transparency as prerequisitetransparency as prerequisiteAll decision on methods
(collection/processing/dissemination) have to be documented and made accessible for the public (FP 3)
Quality management is addressed by several principles of the EU Code of Practice
Quality management refers to products, processes and staff
25
![Page 26: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/26.jpg)
Quality management: Quality management: productsproductsProducts to be regularly assessed
with pre-defined quality criteria (and rejected if minimal quality level for fitness of use not met):◦Final micro-data sets from a data
collection instance◦Results of official statistics prior to
releaseProducts to be assessed occasionally
◦Questionnaires for surveys◦Dissemination products
26
![Page 27: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/27.jpg)
Products to be releasedProducts to be releasedAn important element of
professional independence is the right of the NSI not to release official results if pre-defined minimal quality criteria are not met; this has to be a decision by the chief statistician
This does not preclude the use and release of such results as one input among others into a broader (e.g. geographical) aggregate
27
![Page 28: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/28.jpg)
Quality management: Quality management: processesprocessesProcesses should be defined in
terms of who decides on what and when within the statistical system, and who has to be consulted and informed; they have to be documented for mainly internal use
Processes should be defined for each phase in a generic statistical business model (from design to archiving)
28
![Page 29: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/29.jpg)
Processes (ctd.)Processes (ctd.)It is important to define the system of
processes and responsibilities in such a way that the tendency for creating “black boxes” by responsible organisational units is offset. Stove-pipes (same organisational unit responsible for all phases of the production process for a given activity such as survey) are problematic also from this point of view
29
![Page 30: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/30.jpg)
Processes (ctd.)Processes (ctd.)The process definitions will be
different between on-going activities (with room for small improvements), and major redesigns/innovations, and between production activities ending in the release of results of official statistics and infrastructure activities such as dissemination or training
30
![Page 31: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/31.jpg)
Cross-cutting evaluationsCross-cutting evaluationsAn overall evaluation of the portfolio
of activities and the extent to which the needs of users are met is necessary when the preparation for the next multi-annual programme starts
Extensive user consultations are necessary, and external expertise (such as in a GA) may help in identifying and prioritising gaps to be addressed in the next programme
31
![Page 32: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/32.jpg)
Tendency in organisational Tendency in organisational structures of NSIstructures of NSIFirst-level division by broad subject
areas used to be predominantRecent shift to functional break-
down at first level (data collection; processing; dissemination) for various reasons, among others improved quality management
Senior management has to provide incentives for quality management leading to measurable improvements
32
![Page 33: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/33.jpg)
Staff managementStaff managementQuality management has to become
an almost automatic part of the activity for each staff member
Regular training on management issues should include a substantial part on quality management
Regular assessment of the individual performance of each staff member can be very instrumental, if not implemented as a bureaucratic routine, but as open dialogue towards improvements
33
![Page 34: Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10](https://reader035.vdocument.in/reader035/viewer/2022070413/5697bfde1a28abf838cb202e/html5/thumbnails/34.jpg)
Thank you for your attention
34