pilot census in poland some quality aspects geneva , 7-9 july 2010
DESCRIPTION
Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010. Janusz Dygaszewicz Central Statistical Office POLAND. Data processing infrastructure. XML. Registry 1. CAXI. TXT. Questionaries. Registry 2. ETL Tools. Operational Microdata Base. Analitycal Microdata Base. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/1.jpg)
Pilot Census in PolandSome Quality Aspects
Geneva, 7-9 July 2010
Janusz DygaszewiczCentral Statistical Office
POLAND
![Page 2: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/2.jpg)
2
XML
TXT
Registry 1Registry 1
Metadata serverMetadata server
Operational Microdata
Base
Operational Microdata
Base
Registry 2Registry 2
Registry nRegistry nAnalitycalMicrodata
Base
AnalitycalMicrodata
Base
ETL ToolsETL
Tools
Portal
CAXI
Data processing infrastructure
XML
FilesStatistical
FilesGolden Record
Metadata MetadataMetadata
SDMX
Questionaries
![Page 3: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/3.jpg)
Key elements of census process in terms of census quality • Census planning - scope of census,• Data sources,• Data collecting,• Data storing,• Data processing,• Development of census results,• Dissemination of census results,• Census Metadata System.
Census Quality
3
![Page 4: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/4.jpg)
CENSUS PLANNING
4
![Page 5: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/5.jpg)
Census planning Quality aspects: relevance, accuracy, costs including the burden on respondents, information security
• Determining the data scope defined in Act including:• Compliance with needs of domestic and
EU users,• Quality of data source,• Coherence and comparability of results
from census 2011 and 2002,
Census Quality
5
![Page 6: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/6.jpg)
DATA ACQUISITION
6
![Page 7: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/7.jpg)
7
XML
TXT
Registry 1Registry 1
Metadata serverMetadata server
Operational Microdata
Base
Operational Microdata
Base
Registry 2Registry 2
Registry nRegistry nAnalitycalMicrodata
Base
AnalitycalMicrodata
Base
ETL ToolsETL
Tools
Portal
CAXI
Data acquisition
XML
FilesStatistical
FilesGolden Record
Metadata MetadataMetadata
SDMX
Questionaries
![Page 8: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/8.jpg)
Files format:• Flat files,• XML files,• Local Databases XML files integration,
Data acquisition
8
![Page 9: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/9.jpg)
Data acquisition - Portal
9
![Page 10: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/10.jpg)
Datasources Quality aspects: accuracy, timeliness and punctuality, comparability and coherence, costs including the burden on respondents, information security• Assessment of data sources quality for census:
• analyses of methodological compliance of concepts definitions from registers with those adopted in statistics and the UNECE and EUROSTAT Recommendations for the 2010 Censuses on Population and Housing,• developing methodology for compliance
analyses,• constructing the IT system PiK for describing,
comparing and assessing coherence level,
Census Quality – data acquisition
10
![Page 11: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/11.jpg)
Registers• developing methodology for assessing the
quality: dimensions, quality indicators,• evaluation and description of sources
quality,• MATRIX that represents the possibility of
obtaining the values for the census from registers:• census variable compliance indicators
(methodology compliance indicator), • register suitability indicators (population
coverage indicator for data from the register),
Census Quality – data acquisition
11
![Page 12: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/12.jpg)
Data sets• developing methodology for assessing
the quality,• evaluation and description of data sets
quality,• developing methodology for improving
source data sets quality – rules for: standardization, normalization, de-duplication, editing, imputation, calibration
Census Quality – data acquisition
12
![Page 13: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/13.jpg)
CENSUS FRAME PREPARATION
13
![Page 14: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/14.jpg)
Citizens, buildings and dwelling list preparing,
Citizens, buildings and dwelling list and statistical data integration,
Census Frame preparing.
Census Frame preparation
14
Goal Frame Preparation,
Random Sample preparation,
![Page 15: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/15.jpg)
Quality of Census Frame
15
Census frame pre-census revision - checking in field by enumerators
Census frame preparation – validation and updating in counties,
![Page 16: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/16.jpg)
Enumerator tracking
![Page 17: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/17.jpg)
![Page 18: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/18.jpg)
18
![Page 19: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/19.jpg)
19
![Page 20: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/20.jpg)
20
![Page 21: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/21.jpg)
21
![Page 22: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/22.jpg)
22
![Page 23: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/23.jpg)
Census Completeness Monitoring
![Page 24: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/24.jpg)
24
![Page 25: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/25.jpg)
TRANSFORMATION TO STATISTICAL REGISTER
25
![Page 26: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/26.jpg)
26
XML
TXT
Registry 1Registry 1
Metadata serverMetadata server
Operational Microdata
Base
Operational Microdata
Base
Registry 2Registry 2
Registry nRegistry nAnalitycalMicrodata
Base
AnalitycalMicrodata
Base
ETL ToolsETL
Tools
Portal
CAXI
Source data collection and preparation
XML
FilesStatistical
FilesGolden Record
Metadata MetadataMetadata
SDMX
Questionaries
![Page 27: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/27.jpg)
Registers loading into data laboratory envroiment,
Denormalization,
Standarization,
Deduplication,
Validation,
Data completion,
Vocabulary validation and automatic correction,
Statistical files (register) generation,
Source data collection and preparation
27
![Page 28: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/28.jpg)
Collecting dataQuality aspects: accuracy, costs including the burden on respondents, information security
• Collecting data from information systems• Central registers,• Distributed registers,
• format / file structure (XSD schemas),• data transfer platform,• application for encrypted data transfer,• application for validation and data set control
Census Quality – collection and preparation
28
![Page 29: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/29.jpg)
Data loading to Operational Microdatabase,
Validation
Manual and automatic correction (cleaning),
Deduplication,
Variables calculating,
Source data loading and correction
29
![Page 30: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/30.jpg)
30
XML
TXT
Registry 1Registry 1
Metadata serverMetadata server
Operational Microdata
Base
Operational Microdata
Base
Registry 2Registry 2
Registry nRegistry nAnalitycalMicrodata
Base
AnalitycalMicrodata
Base
ETL ToolsETL
Tools
Portal
CAXI
CAxI
XML
FilesStatistical
FilesGolden Record
Metadata MetadataMetadata
SDMX
Questionaries
![Page 31: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/31.jpg)
•CAII - Computer Assisted Internet Interview,•CAPI - Computer Assisted Personal Interview,•CATI - Computer Assisted Telephone Interviewing.
CAxI
CAxI
31
CAXI
![Page 32: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/32.jpg)
• Collecting data from respondents: CAII, CAPI, CATI;• CAxI input validation:
• Numerical data validation (answers within boundaries)• Cross question arithmetical validation• Hints and automatic answer completion• Dictionaries and drop down menus
• CAxI logical validation: • Answers determined by questions• Cross question logical validation• Data collection logical paths
Census Quality – data collection by electronic questionare
32
![Page 33: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/33.jpg)
Data storingQuality aspects: information security
• Data storing in Operational Microdata Base,• Notification of Operational Microdata Base
to registration by General Inspector for Protection of Personal Data,
Census Quality
33
![Page 34: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/34.jpg)
GOLDEN RECORD,
34
![Page 35: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/35.jpg)
35
XML
TXT
Registry 1Registry 1
Metadata serverMetadata server
Operational Microdata
Base
Operational Microdata
Base
Registry 2Registry 2
Registry nRegistry nAnalitycalMicrodata
Base
AnalitycalMicrodata
Base
ETL ToolsETL
Tools
Portal
CAXI
Golden Record generation
XML
FilesStatistical
FilesGolden Record
Metadata MetadataMetadata
SDMX
Questionaries
![Page 36: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/36.jpg)
36
XML
TXT
Registry 1Registry 1
Metadata serverMetadata server
Operational Microdata
Base
Operational Microdata
Base
Registry 2Registry 2
Registry nRegistry nAnalitycalMicrodata
Base
AnalitycalMicrodata
Base
ETL ToolsETL
Tools
Portal
CAXI
Export to Analitycal Microdata Base
XML
FilesStatistical
FilesGolden Record
Metadata MetadataMetadata
SDMX
Questionaries
![Page 37: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/37.jpg)
Integration with Census Frame and CAxI data,
Validation,
Correction,
Operational Imputation,
Transfer proper values to Golden Record,
Golden Record generation
37
Registers 1..n
CAxI
Golden Record
OMB Layers
![Page 38: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/38.jpg)
Transition Tables Preparing,
Golden Records anonymisation,
Transfer to Analitycal Microdatabase,
Export to Analitycal Microdata Base
38
![Page 39: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/39.jpg)
Data processingQuality aspects: accuracy
• Developing quality indicators for data sets at each stage of data processing and the procedures for calculating their value,
• Developing procedures for bringing data from administrative sources to full compliance or minimum discrepancy with appropriate methodology adopted in statistics,
• Developing procedures for normalization, editing of data sets from the administrative systems, including the imputation of data (administrative data sets),
• Developing procedures for synchronization of data from administrative systems,• Developing rules for linking data from different administrative systems,• Developing rules for linking data from administrative systems with data from CAII, CAPI, CATI,• Developing rules for calculation of Golden Record census variables,• Developing rules for anonymisation of Golden Record census data.
Census Quality
39
![Page 40: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/40.jpg)
ANALITYCAL MICRODATABASE
40
![Page 41: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/41.jpg)
41
XML
TXT
Registry 1Registry 1
Metadata serverMetadata server
Operational Microdata
Base
Operational Microdata
Base
Registry 2Registry 2
Registry nRegistry nAnalitycalMicrodata
Base
AnalitycalMicrodata
Base
ETL ToolsETL
Tools
Portal
CAXI
Analitycal Microdata Base
XML
FilesStatistical
FilesGolden Record
Metadata MetadataMetadata
SDMX
Questionaries
![Page 42: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/42.jpg)
Analitycal Microdata Base - process
42
Process
data
Load dat a and m et adat aI nt egrat e dat aCl assi f y and code dat aEdi t and val i dat e dat aI m put eD er i ve new var i abl esWageAggregat eCreat e fil es
Analyse
Disse
minate
Archive
Manage metainformation
Manage quality
![Page 43: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/43.jpg)
Functionality
43
AdministrationInformation
Security Management
Data Processing
Information Analisys
Requirement and Product Management
Dissemination
Metadata
Quality Management
Analitycal Microdatabase
![Page 44: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/44.jpg)
Development of census resultsQuality aspects: relevance, accuracy, comparability and coherence
• Developing rules for missing data completion - imputation and calibration,• Developing rules for creating derived objects - creation of new objects
(households, families),• Developing a model / method of data estimation with the use of the data
from administrative systems and sample surveys,• Developing rules for calculating data outputs.
Census Quality
44
![Page 45: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/45.jpg)
DISEMINATION
45
![Page 46: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/46.jpg)
Dissemination of census resultsQuality aspects: relevance, timeliness and punctuality, accessibility and clarity, comparability and coherence, information security
• Designing Analitycal Microdata Base features including compliance with users needs, accessibility and clarity of census data.
Census Quality - disemination
46
![Page 47: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/47.jpg)
METAINFORMATION MANAGEMENT
47
![Page 48: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/48.jpg)
48
XML
TXT
Registry 1Registry 1
Metadata serverMetadata server
Operational Microdata
Base
Operational Microdata
Base
Registry 2Registry 2
Registry nRegistry nAnalitycalMicrodata
Base
AnalitycalMicrodata
Base
ETL ToolsETL
Tools
Portal
CAXI
Metadata server
XML
FilesStatistical
FilesGolden Record
Metadata MetadataMetadata
SDMX
Questionaries
![Page 49: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/49.jpg)
Metainformation management
49
Metainformation
Definition
BussinesReferencial
Conceptual Methodical Quality
Structural
Technical
System
Postprocessing
![Page 50: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/50.jpg)
Census Metadata SystemQuality aspects: accessibility and clarity
• Developing quality indicators at each stage of census and the procedures for calculating their value.
Census Quality – metainformation
50
![Page 51: Pilot Census in Poland Some Quality Aspects Geneva , 7-9 July 2010](https://reader035.vdocument.in/reader035/viewer/2022081520/568151c8550346895dbffea0/html5/thumbnails/51.jpg)
51
POLAND