streamlining the data compilation and dissemination at ilo department of statistics
DESCRIPTION
After many years of operating with an old system for the compilation and dissemination of labor statistics, the need to streamline processes and have new tools in the ILO Department of Statistics was clear. High maintenance costs, low coverage and problems of comparability between data were some of the most important gaps that determined the urgent need to redesign the system. The project to redesign the department’s approach included not only the development of new applications using updated and appropriate tools to achieve the required functionality, but procedures that could be automatized and allowed to have an auxiliary system for monitoring the flow of information to assist in the task of data collection. One aspect that was emphasized from the beginning of the new project was the adoption of every possible standard, so as to increase the chance of interaction with our partners. Thus, the process follows the recommendations of GSBPM, development tools from the Oracle suite (a “de facto” standard) are used, and the means of collection are based on Excel, XML and SDMX (coming soon). The new process for data compilation and dissemination is built on four main ideas: - The broadening of the ways of interaction with the countries for data collection; - The full automation of computerized procedures, so as to enable Statistical Assistants to engage more efficiently in non-computerized activities; - The systematization of the consistency and correction procedure regardless of the way the data was received; and - The ability to know when and why (or why not) data from the countries is arriving, thus knowing how much information is to be included in a publication. These simple ideas once implemented, would enable the new ILOSTAT database to have a better response rate from the countries, reduce the delay of the information received and improve the overall quality of the data published. The challenge was to achieve these objectives while simultaneously reducing the TCO of the system. This paper describes the set of new processes and the IT tools developed so far (as well as those forthcoming), to optimize the data compilation and dissemination at the ILO Department of Statistics.TRANSCRIPT
STREAMLINING DATA COMPILATION AND
DISSEMINATION ILO Department of Statistics
Edgardo [email protected]
Why?◦ High maintenance costs◦ Low coverage ◦ Problems of comparability between data
What?◦ Applications◦ Procedures
Objectives◦ Better response rate◦ Reduced delay in disseminating◦ More indicators covering the DW agenda◦ Improve overall data quality
Introduction
The old system◦Process overview◦IT considerations
The new approach◦Process overview◦IT considerations
Conclusions
Agenda
The old systemProcess Overview
Paper or faxquestionnaire
SAS Database
SAS Database(Replica)
YEARBOOK Publication
Automatic Upload(per sheet basis)
Format & totals check
Manual Inputby Stat. Assistant
Full DB Consistency check
Automatic printout generation
OPTIONAL Editingby Stat. Assistant
Weeklyupdate
LABORSTA Website
YEARBOOK CD
No Answer
Manual data grabbingby Stat. Assistant
Internet
Excel questionnaires(9 books per country)
Manual consistency No procedure to re-contact non-
responding countries Thousands of footnotes, dozens
of them meaning exactly the same
“Source & Methods” metadata collected as text documents
Veracity relayed on huge workload
Declining response rate Delayed publication release High direct costs Overburden to countries
The old systemIT considerations
Paper or faxquestionnaire
SAS Database
SAS Database(Replica)
YEARBOOK Publication
Automatic Upload(per sheet basis)
Format & totals check
Manual Inputby Stat. Assistant
Full DB Consistency check
Automatic printout generation
OPTIONAL Editingby Stat. Assistant
Weeklyupdate
LABORSTA Website
YEARBOOK CD
No Answer
Manual data grabbingby Stat. Assistant
Internet
Excel questionnaires(9 books per country)
Data stored in a hierarchical database
Time consuming «per sheet» upload procedure
Unfriendly consistency checking program turned useless
Data with errors not marked, could be published
No solution for “false positives” Command line editor Disconnected dissemination tools Manual workflow management High maintenance costs
The new approachProcess Overview
Country-centric approach Broader interaction with countries
◦ Off-line x-Questionnaire (Excel)◦ e-Questionnaire◦ EDI (SDMX)
Controlled vocabulary footnotes system
Error-free data passed to the dissemination database
New ILOSTAT website integrated to Department of Statistics’ and ILO’s IKMG
Paper or fax
questionnaire
Data CollectionDatabase
Web Database(Replica)
YEARBOOK Publication
Full AutomaticUpload
Manual Inputby Stat. Assistant
Consistency check
Automatic printout generation
Editingby Stat. Assistant
Weeklyupdate
LABORSTA Website(with dynamic charts and maps)
NO
YEARBOOK CD
Excel questionnaires(1 simplified book)
SDMX fileE-Questionnaires
(on line via internet)
Full AutomaticUpload
Stat. AssistantE-mail or phone call
Country counterparts
Selected data collection mode
CORRECT?
Main Database
YES
No Answer Received
Error Report
Data Flow Control Dashboard
1 2 3 4 5
The new approachIT considerations
Modular design following GSBPM Oracle RDBMS and development tools Automated procedure for xQ and SDMX
uploading with structural consistency E-Questionnaire online data collection Single set of metadata Single interactive consistency
procedure regardless of data collection means
«False positives» handling thru allowance issuing
Full screen data editor Dynamic content dissemination website Data workflow management module
8
Data Workflow
(country + user)
Qtable
(country + indic + survey)
LABORSTA data
Content &Documents
Current website services deliveryIndependent user interfaces
Colaboration
WCMS 10g
SAS
Plone
LABORSTA
backoffice
Oracle WebCenter Portal
Content &Documents(Dynamic)
ILOSTAT data Backoffice
Applications
WCMS 11g
WCMS 10g
Oracle DBMS
OBI EE
WebCenter Spaces
APEX
SAS
Oracle DBMS
Colaboration &
Social Network
New website services deliveryUnified user interface
Conclusions Increased coverage Improved opportunity Improved quality Reduced overburden Standards based General purpose Reduced TCO
Questions?
E-mail:[email protected]
Skype:egreising
Twitter: egreising
LinkedIn: http://www.linkedin.com/in/egreising
Thank you!
E-mail:[email protected]
Skype:egreising
Twitter: egreising
LinkedIn: http://www.linkedin.com/in/egreising