meeting today’s dissemination challenges – implementing international standards in oecd.stat...
TRANSCRIPT
![Page 1: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/1.jpg)
MEETING TODAY’S DISSEMINATION CHALLENGES:Implementing international standards in .Stat
Prepared by Jonathan Challener, OECDFor MSIS, April 2014 - Dublin, Ireland
![Page 2: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/2.jpg)
Doesn’t non-standard power supplies make things difficult?
![Page 3: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/3.jpg)
What happens when standards are not applied well?
![Page 4: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/4.jpg)
Picture: ‘The day Sweden changed from left-hand drive to right’
Confusion entails
![Page 5: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/5.jpg)
This all adds up…
![Page 6: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/6.jpg)
…high costs…
![Page 7: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/7.jpg)
and inefficiencies!
![Page 8: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/8.jpg)
and inefficiencies!“A little like the grade 8 student who doesn’t pay attention in class all year”.
![Page 9: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/9.jpg)
WHAT IS .STAT?
![Page 10: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/10.jpg)
What is .Stat?
.Stat is the central repository ("warehouse") of validated statistics and related metadata
.Stat is the central hub connecting data production, sharing & dissemination
processes
It is the corporate source of data for data sharing and dissemination purposes
![Page 11: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/11.jpg)
What is .Stat?
.Stat is the central repository ("warehouse") of validated statistics and related metadata
.Stat is the central hub connecting data production, sharing & dissemination
processes
It is the corporate source of data for data sharing and dissemination purposes
“.Stat is now being used and shared with 10 organisations including the OECD, as part of the Statistical Information System Collaboration Community (SIS-CC)”.
![Page 12: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/12.jpg)
.Stat Positioning in Statistical Information System
DATA DELIVERY
INTERNAL DATA SHARING
DATA DISSEMINATION
DATA PRODUCTION
.STAT
![Page 13: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/13.jpg)
.Stat Positioning in Statistical Information System
DATA DELIVERY
INTERNAL DATA SHARING
DATA DISSEMINATION
DATA PRODUCTION
.STAT
“The diagram illustrates the .Stat contribution to the SIS processes. .Stat’s core value-added lies in “Data Delivery”, a set of functions that enable dissemination and data sharing, and “Data Upload”, a set of functions interfacing data production processes into a single upload mechanism to feed dissemination channels”.
![Page 14: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/14.jpg)
.Stat Functional Representation
.STAT DATA DELIVERY ENGINE
DATA PRODUCTION
DATA SHARING DATA DISSEMINATION
SEARCH ENGINES
DATA ANALYSIS TOOLS
PC
WEBSITES, APPSPUBLICATIONS.STAT BROWSER
.STAT DATA UPLOAD ENGINE
FILEUPLOAD
SDMX IMPORT
DATA PRODUCTION TOOLS
TABLE & CHART EXTRACTION SERVICES
RELEASE MGTSERVICES
.STAT BROWSERCONFIGURATION
DATA EXTRACTION SERVICES
SDMX INPUT
E
P
BATCH UPLOAD
SDMX GLOBAL
REGISTRY
PUBLISHINGBACK
OFFICE
DATA MAPPING
SDMX OUTPUT
X
X
.Stat Component
Process
Human userData ProducerData EditorData Consumer
API orWebservice
OtherSDMX hubs
![Page 15: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/15.jpg)
.Stat Functional Representation
.STAT DATA DELIVERY ENGINE
DATA PRODUCTION
DATA SHARING DATA DISSEMINATION
SEARCH ENGINES
DATA ANALYSIS TOOLS
PC
WEBSITES, APPSPUBLICATIONS.STAT BROWSER
.STAT DATA UPLOAD ENGINE
FILEUPLOAD
SDMX IMPORT
DATA PRODUCTION TOOLS
TABLE & CHART EXTRACTION SERVICES
RELEASE MGTSERVICES
.STAT BROWSERCONFIGURATION
DATA EXTRACTION SERVICES
SDMX INPUT
E
P
BATCH UPLOAD
SDMX GLOBAL
REGISTRY
PUBLISHINGBACK
OFFICE
DATA MAPPING
SDMX OUTPUT
X
X
.Stat Component
Process
Human userData ProducerData EditorData Consumer
API orWebservice
OtherSDMX hubs
“The grey shaded boxes in the figure below show a visual representation of how .Stat fits within a broader Data Dissemination Information System of organisations; the boxes with dotted lines represent other components of the Data Dissemination Information System that are not supported by .Stat but are enabled by it”.
![Page 16: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/16.jpg)
.Stat Functional Representation
In particular, .Stat provides the following 3 key functional areas…
![Page 17: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/17.jpg)
.Stat Functional Representation.Stat Data Upload Engine
![Page 18: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/18.jpg)
.Stat Functional Representation.Stat Data Delivery Engine
![Page 19: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/19.jpg)
.Stat Functional Representation.Stat Data Browser
![Page 20: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/20.jpg)
.Stat Positioning in GSBPM Reference Model
.Stat contributes to Planned additions
Archive incorporated into the over-arching process of data and metadata management
![Page 21: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/21.jpg)
.Stat Positioning in GSBPM Reference Model
.Stat contributes to Planned additions
Archive incorporated into the over-arching process of data and metadata management
“.Stat can be mapped today to the Generic Statistical Business Process Model (GSBPM) under “Disseminate” and “Build”. In the future it will also incorporate archive functions as part of the over-arching process for data and metadata management”.
![Page 22: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/22.jpg)
Multipurpose SDMX within .Stat…
![Page 23: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/23.jpg)
For dissemination and data eXchange
SDMXWS and RESTful API• SDMX 2.0 compliant• SOAP + REST• Pull• SDMX-ML• SDMX Structural
metadata created on the fly
![Page 24: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/24.jpg)
For ‘Open Data’ dissemination
SDMX-JSON (beta)SDMX-TWG agreed in mid 2013 on proposal for data and their structural metadata (inc. flat & sliced layouts) and referential metadata (dataset, series, obs) as annotations.
Further enhancements to come: Complete data structures and referential metadata
![Page 25: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/25.jpg)
For data reporting
SDMX-Reference Infrastructure (RI)*• SDMX 2.0 and 2.1
compliant• SOAP + REST• SDMX Common APIs
(SdmxSource.NET)
• Pull + Push• SDMX-ML, GESMES ,
CSV• Structural metadata
stored in mini registry• One web service - several
mapped database instances
Mapping Store DB
XXX.StatData
warehouse
SDMX-RI Web Service
DisseminationMapping Assistant
SDMX-RI
* The integration of SDMX-RI in .Stat is based on collaboration with Eurostat, provider of the SDMX-RI component with ISTAT taking the lead on behalf of the OECD’s Statistical Information System Collaboration Community.
![Page 26: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/26.jpg)
For internal data sharing
DirectAccess• Restful SDMX query• Flat data, flags, units• Referential metadata
Excel-add-in• DirectAccess (Rest
SDMX)• Native Excel pivot table• Wizard to select data
![Page 27: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/27.jpg)
For a decentralised publishing environment
DataHub*• One interface to the
publishing tools• Centralised reporting and
auditing• SDMX based structural
metadata, and referential metadata management
• Flexible load tool that promotes ‘self publish’ for data custodians
• In-built checks and safeguards to minimise errors
• Manages security and access rights
• Can be extended to manage other outputs and not limited to .Stat
* DataHub has been developed and integrated with .Stat by Statistics NZ, with an additional connection to the Fusion Registry for managing structural metadata through the definition of DSDs.
![Page 28: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/28.jpg)
Future outlook…
![Page 29: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/29.jpg)
Further SDMX artifact support
![Page 30: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/30.jpg)
SDMX ingest (Import)
![Page 31: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/31.jpg)
SDMX global registry API
![Page 32: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/32.jpg)
SDMX-RDF data cube vocabulary pilot
![Page 33: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/33.jpg)
SDMX-RDF data cube vocabulary pilot
“Explore further semantic web/linked data opportunities (SDMX-RDF data cube vocabulary). To be taken forward by ISTAT and ABS under the SIS-CC umbrella”.
![Page 34: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/34.jpg)
• Lower technology adoption costs• Increased development consistency,
simplicity and predictability• Improved code reuse• Reduced cost, time and effort to transition
between different solutions
We all know the…
• Reduced focus on infrastructure• Ability to create composite interfaces that are tailored to the
needs of specific task• Improved application portability• Enable faster time to market because it is easier to use off the
shelf components and applications that can integrate and provide features for the solution
![Page 35: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/35.jpg)
References
1. Operationalising .Stat in a decentralised publishing environment (DataHub) by Tony Breen SNZ : https://community.oecd.org/docs/DOC-68362
2. Building a scalable architecture (.Stat) by Jens Dosse OECD: https://community.oecd.org/docs/DOC-68363
3. SDMX-RI and .Stat integration by Francesco Rizzo Istat: https://community.oecd.org/docs/DOC-68696
4. SDMX-JSON API: http://stats.oecd.org/opendataapi/Index.htm
![Page 36: Meeting today’s dissemination challenges – Implementing International Standards in OECD.Stat MSIS Dublin 14-16 April 2014](https://reader035.vdocument.in/reader035/viewer/2022070604/54639f6faf7959624d8b5031/html5/thumbnails/36.jpg)
Jonathan Challener, OECD [email protected]@Challener
MSIS - Dublin, 14-16 April 2014Meeting today’s dissemination challenges: Implementing international standards in .Stat
Thank you