escwa sdmx workshop session: role in the statistical lifecycle and relationship with ddi (data...
TRANSCRIPT
ESCWA SDMX Workshop
Session: Role in the Statistical Lifecycle and Relationship with DDI
(Data Documentation Initiative)
© Metadata Technology
General Statistical Business Process Model and SDMX
© Metadata Technology
General Statistical Process Business Model and SDMX
DDI DDISDMX
© Metadata Technology
General Statistical Process Model and SDMX
DDI DDISDMXSDMX
© Metadata Technology
DDI 3.0 and the Survey Life Cycle
• A survey is not a static process: It dynamically evolved across time and involves many agencies/individuals
• DDI 2.x is about archiving, DDI 3.0 across the entire “life cycle”• 3.0 focus on metadata reuse (minimizes redundancies/discrepancies,
support comparison)• Also supports multilingual, grouping, geography, and others• 3.0 is extensible
© Metadata Technology
DDI 3.0 Modules
• Conceptual Components (concepts, universes)• Data Collection (survey instruments and collection processing)• Logical Products (variables, categories, code lists)• Physical data product (descriptions of file structures)• Physical Instance (instances of data files)• Archiving (information about holding, storage, and organizations)• Comparative (mapping schemes)• Grouping (for comparison, and longitudinal studies, panels, and
series)• Instance (the wrapper)• DDI Profile (describes which DDI 3 elements are used)• Study Unit (describes a single study)
© Metadata Technology
Technical Specifications - Maintainable Schemes (that’s with an ‘e’ not an ‘a’)
• Category Scheme• Code Scheme• Concept Scheme• Control Construct Scheme• GeographicStructureScheme• GeographicLocationScheme• InterviewerInstructionScheme• Question Scheme• NCubeScheme• Organization Scheme• Physical Structure Scheme• Record Layout Scheme• Universe Scheme• Variable Scheme
Packages of reusable metadata maintained by a single agency
© Metadata Technology
DDI “Mind Map”DDI “Mind Map”
© Metadata Technology
DDI – “Mind Map”
© Metadata Technology
Technical Specifications – XML Schemas
• archive• comparative• conceptualcomponent• datacollection• dataset• dcelements• DDIprofile• ddi-xhtml11• ddi-xhtml11-model-1• ddi-xhtml11-modules-1• group• inline_ncube_recordlayout
• instance• logicalproduct• ncube_recordlayout• physicaldataproduct• physicalinstance• proprietary_record_layout (beta)• reusable• simpledc20021212• studyunit• tabular_ncube_recordlayout• xml• set of xml schemas to support
xhtml
© Metadata Technology
Use within data collection
<DDI 3.0>ConceptsUniverseMethodsPurposePeople/Orgs
<DDI 3.0>QuestionsInstrument
<DDI 3.0>Data CollectionData Processing
<DDI 3.0>Funding Revisions
SubmittedProposal
$€ £
Presentations
Archive/RepositoryPublication
+++
+
+
<DDI 3.0>VariablesPhysical Stores
PrincipalInvestigator
Collaborators
Research Staff
Data
© Metadata Technology
DDI and Generic Process Example
Survey/Register
Raw Data SetRaw Data Set
Anonymization, cleaning, Anonymization, cleaning, recoding, etc.recoding, etc.
Micro-Data Set/Micro-Data Set/Public Use FilesPublic Use Files
Tabulation, processing,
Tabulation, processing,
case selection, etc.
case selection, etc.
Aggregation,
Aggregation,
harmonizatio
n
harmonizatio
n
Aggregation, Aggregation, harmonizationharmonization
Aggregate Data SetAggregate Data Set(Lower level)(Lower level)
Aggregate Data SetAggregate Data Set(Higher Level)(Higher Level)
DDIDDI
SDMXSDMX
IndicatorsIndicators
© Metadata Technology
Concept Scheme
Code List, HCL,
SDMX RegistrySDMX DSD and data set
SDMX MSD and metadata set
SDMX
General Statistical Process Model and SDMX
SDMX
SDMX Process Model
© Metadata Technology
Process - Schematic
ProcessProcess
ComputationComputationTransitionTransitionProcess StepProcess Step
Process Artefact
Process Artefact
inputsoutputs
target
© Metadata Technology
Process - UML
1
The process step can reference any identifiable object as input or output.
MaintainableArtefact
Process
Computation
localId : StringsoftwarePackage : StringsoftwareLanguagesoftwareVersion : String
Transition
localId : String
InternationalString
+condition
1
ProcessStep
0..*+step
0..*
0..1
+computation
0..1
0..*+child 0..*+parent
1
+target
10..*+transition
0..*+source
ProcessArtefact
localID : String
0..*
+output
0..* 0..*
+input
0..*
IdentifiableArtefact
11
+artefact
AnnotableArtefact
InternationalString1..*
+description
1..*