model of transformation administrative data to statistical data

Model of transformation administrative data to statistical data

Data used in Population and Housing Census 2011

– examples

Janusz Dygaszewicz and Paweł MurawskiCentral Statistical Office

POLAND

Outline

1. Purpose of the work on administrtive sources

2. Data quality3. Extract data4. Transform data5. Summary

Data Owners:• Ministry of Finance,• Ministry of Interior and Administration,• Ministry of Justice,• Agricultural Social Insurance Fund,• National Health Fund,• Agency for Restructuring and Modernisation of Agriculture,• Agricultural and Food Quality Inspection,• Agency for Geodesy and Cartography,• State Fund for Rehabilitation of Disabled Persons,• County Offices,• Commune Offices,• Regional Offices,• Telcoms,• Energy Suppliers,• Office For Foreigners,• Social Insurance Institution,• Housing Managers,

Registers - data acquisition

3

Purpose of the work on administrative data

Obtaining a sufficiently complete data set –subjective and objective completeness corresponding to classification standards, definitions and basic categories, and thus the effective use of administrative data

Data quality-measures-

1. Measuring the quality of administrative registers– timeliness of data– methodological compatibility– completeness– identification standards used in the registry– usefulness– compatibility of data in administrative sources to data obtained in the

study/survey

2. Measuring the quality in processing of data registers– excessive coverage error rate– incomplete coverage error rate– subjective indicator of completeness– objective indicator of completeness– imputation rate– data correction index– integration data from various sources index

Extract data consolidation data from various source systems; different data

format, extract data into the production environment based on the SAS

software, converting data into one format that is suitable for processing – SAS

tables, validate of imported data structure is an integral part of this process .

Extract data-examples-

Register/System Name CentralY/N

Relational Y/N Data format

PESEL General electronic system of Population Register Y N TXTKEP Register of the National Taxpayers Y N TXTGZM Community Registers of Residence N Y SQL ServerPIT Personal income tax register Y N TXTSI MS Ministry of Justice system Y N XLSPOBYT Foreigners evidence system Y Y SQL ServerZUS CRPS Central Register of Contribution Payers Y N TXTZUS CRU Central Register od Insured Persons Y N TXTZUS SER Pension insurance system Y N TXTKRUS Agricultural Social Insurance Fund System Y N XMLCWU NFZ Central Register of Insured Y N TXTPFRON State Fund for Rehabilitation of Disabled Persons Y N TXTARiMR Agency for Restructuring and Modernisation of Agriculture

systemY Y XLS

EPN Property Tax Records N Y SQL Server

Transform data

Data processing in the production environment consisting of:• profiling – create a raport on the data quality,• unification/standardization of data,• parsing (separation) or combining variables,• standardization with schemes,• conversion,• validation,• deduplication,• data integration.

Transform data- profiling-

Transform data- standardization and parsing examples-

Incorrect data format Format after standardization

1985-02-21 198502211985.02.21 198502211985 02 21 19850221

Voivodeship City Street Place of birthMAZOWQIECKIE WARZSWA ul. DŁUGA LONDYN - ANGLIAMAZPWOECKIE WARS-AWA Ulica DŁUUGA LONDYN – WLK BRYTANIAZAZOWIEVCKIE AWRSZAWA DLUGAA LONDYN/CHELSEA

MZAOWIECIE WARSZAAAWA DŁUGA (ul.) LONDYN BRIDGE

Voivodeship City Prefix Street Place of birth

MAZOWIECKIE WARSZAWA UL DŁUGA LONDYN

Transform data- schemes examples-

Transform data- exemples: report data cleaning -

Description Before cleaning After cleaning

Group of variables Variable TotalInorrect

TotalInorrect

total incorrect In % total

incorect In %

Address of permanent residence

COMMUNITY 4320724 428469 9,92% 4316061 72797 1,69%CITY 4353209 207399 4,77% 4352983 43086 0,99%STREET 3514154 573899 16,34% 3440932 125392 3,65%PREFIX 0 0 - 108551 0 0,00%

Address of residence


Corresponding address


Personal Data NAME 4355764 7208 0,17% 4355757 5534 0,13%

Transform data- conversion: gender variables

famale

male

F

M1

2

Transform data- conversion: marital status variable-

Family benefits system

501 – unmarried woman

504 – married (F)505 – divorced (M)506 – divorced (F)507 – widow508 – widower509 – non-formalized (M)510 – non-formalized (F)

511 – separated

Foreigners evidence system

$BD – no data

MTA – married (F)PNA – unmarried womenRWA – divorced (F)RWY – divorced (M)WDA – widowWDC – widower

WNA – single (F)

WNY – single (F)

Statistical standard

2 unmarried woman

4 married (F)5 divorced (M)6 divorced (F)7 widower8 widow9 unidentified10 separated (M)11 separated (F)

12non-formalized relationship (M)

13non-formalized relationship (F)

14 single (M)

15 single (F)

Community Registers of Residence

2 – unmarried woman

4 – married (F)5 – divorced (M)6 – divorced (F)7 – widower8 – widow

no data – empty field

123456789

914265873

1514

21345687

1213

10/11

3 married (M)

503 – married (M)

ZNY - married (M)

3 – married (M)

1 – bachelor

KWR – bachelor

502 – bachelor

1 bachelor

Transform data-validation-

checking the data, correcting abnormal values, according

to the algorithms prepared by methodologists,

eventual exclusion from further processing records which improvement is impossible.

Transform data- deduplication -

removal of repeated units,requires detailed analisys, including

alalysis of legal actsindividual for each register, result of deduplication – one record with

all the possible and unique information.

Transform data-expamle of deduplication process-

imie1_GZM nazwisko1_GZM plec_GZM pesel_GZM adr_ulica_GZM adr_nr_dom_GZM adr_nr_lok_GZM data_zam_od_GZM JAN KOWALSKI M 00000000001 PUŁAWSKA 4 37 20070303ANNA MALINOWSKA K 00000000002 ANDERSA 7 20010205ADAM PIOTROWSKI M 00000000003 FILTROWA 2 45 20090101JAN KOWALSKI M 00000000001 PUŁAWSKA 4 37 19840712




ANNA MALINOWSKA K 00000000002 ANDERSA 7 20010205ADAM PIOTROWSKI M 00000000003 FILTROWA 2 45 20090101

imie1_GZM nazwisko1_GZM plec_GZM pesel_GZM adr_ulica_GZM adr_nr_dom_GZM adr_nr_lok_GZM data_zam_od_GZM ANNA MALINOWSKA K 00000000002 ANDERSA 7 20010205ADAM PIOTROWSKI M 00000000003 FILTROWA 2 45 20090101JAN KOWALSKI M 00000000001 PUŁAWSKA 4 37 20070303JAN KOWALSKI M 00000000001 PUŁAWSKA 4 37 20070303

Transform data-data integration-

process of selection of the best, most current and correct value of several or a dozen of registers

Used to create a statistical register, which will be available for use by analysts.

Transform data-intergation process – scheme-

A Register

B Register

C Register

ONE ID

MULTIPLE IDENTIFIRES

ALTERNATIVE LINKING KEYS

DATA INTEGRATION

LINKING

SELECTING

ALGORYTHMS

SELECTING THE BEST VALUES

DATA COMPLETENESS

STATISTICAL REGISTER

REGISTER OF REFERENCE

kraj_ur_kod_KEP # not null

msce_ur_kod_POBYT # not null

kraj_ur_kod_GZM # not null

Transform data-data integration: example of algorythm

FALSE

FALSE

TRUE

TRUE

TRUE

Kraj_ur_kodselectkraj_ur_kod_GZM

selectkaj_ur_kod_POBYT

selectkraj_ur_kod_KEP

Data integration-example of process-

Summary Common difficulties: - poor quality data, missing values, duplicates, - conflicting data,- technical: size of the registers, time-consuming

process.Benefits: - obtain relevent, useful, accurate data- improve the quality of the output data. - selection of the best variables from multiple registers,

Thank you for your attention

www.stat.gov.pl

model of transformation administrative data to statistical data

Documents

data profiling

complete data

data integration

extract dataexamples

data qualitymeasures

transform data exemples

extract dataconsolidation

registers data acquisition3