MDG DATA AND METADATA MDG DATA AND METADATA EXCHANGE AMONG NATIONAL EXCHANGE AMONG NATIONAL

AGENCIES ANDAGENCIES ANDWITH INTERNATIONALWITH INTERNATIONAL

ORGANIZATIONSORGANIZATIONS

The Experience of MexicoThe Experience of Mexico

Enrique OrdazEnrique Ordaz19- 21 October, 201119- 21 October, 2011Manila Manila

Statistics and Geographic Information LawStatistics and Geographic Information Law

National Institute of Statistics National Institute of Statistics and Geography, INEGIand Geography, INEGI

AUTONOMY

NATIONAL COUNCILNATIONAL COUNCIL

INEGIINEGI

SUBSYSTEMSUBSYSTEM SUBSYSTEMSUBSYSTEM SUBSYSTEMSUBSYSTEM SUBSYSTEMSUBSYSTEM

Catalog of National Indicators

National Information Programs

Data dissemination

Norms Coordination

Socio DemographicInformation

SpecializedTechnicalCommittee

UE

Working groups UE

Goverment, PublicSecurity and JusticeInformation

Specialied TechnicalCommittees

Workinggropus

UE

UE

Geographic andEnviromentalInformation

SpecializedTechnicalCommittees

Workinggropus

UE

UE

EconomicInformation

SpecializedTechnicalCommittee

Workinggroups

UE

UE

Society

State

The new statistical The new statistical (and geographic information) system(and geographic information) system

MDG Indicators CommitteeMDG Indicators Committee

A Specialized Technical Committee was set up in February 2010 to

coordinate the integration of the Millennium Development Goals

Indicators.

Mexico’s MDG IndicatorsMexico’s MDG Indicators

Lista oficial de los indicadores de la ONUSistema de Información de los Objetivos de Desarrollo del Milenio,

México 2010

Objetivo Metas Indicadores Metas

Indicadores

Total Preexistente ReformuladosPropuestos por México

De Más allá de las Metas del

Milenio Total 22 70 22 80 40 8 12 20

1. Erradicar la pobreza extrema y el hambre

3 9 5 14 8 1 0 5

2. Lograr la enseñanza primaria universal

1 5 3 14 5 0 1 8

3. Promover la igualdad de género y el empodera-miento de la mujer

1 7 1 7 5 1 1 0

4. Reducir la mortalidad de los niños menores de 5 años

1 3 1 6 3 0 0 3

5. Mejorar la salud materna 2 6 2 6 4 2 0 0

6. Combatir el VIH/SIDA, el paludismo y otras enfermedades

3 13 4 15 4 1 7 3

7. Garantizar la sostenibilidad del medio ambiente

4 12 4 13 8 3 1 1

8. Fomentar una alianza mundial para el desarrollo

6 16 1 3 3 0 0 0

General schedule of the work programGeneral schedule of the work program

Activities

2010 2011Enero Febrero Marzo Abril Mayo Junio Julio Agosto Septiem-

breOctubre Noviem-

breDiciem-

breEnero Febrero

1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4

•Committee Meetings                                                                                                               

•Data integration by working groups

                                                                                                               

MDG System                                                                                                               

1st Phase                                                                                                               

2nd Phase                                                                                                               

3rd Phase                                                                                                               

•Drafting of the 2010 country report

1.Poverty, income and nutrition

Working groups

2. Poverty, employment

3. Education and gender equality

4. Maternal and child mortality, rep. health

5. VIH/AIDS, malaria, other diseases

6. Environment and natural resources

7. Improved water access, and sanitation

8. Global partnerships for development

Ministries responsible

for reviewing and updating

metadata; updating

basic data and indicators

INEGI:technical assistance; data integration and development and management of the website.

General Coordination: Chrístel Rosales (DGPAE)Agreements follow up: César Garcés (CONAPO)

CoordinationCoordination

Process for updating the MDG systemProcess for updating the MDG system

oceso de actualización Sistema ODMoceso de actualización Sistema ODM

ActivitiesActivities

1. Prepare manuals of procedure and rules.

2. Create an FTP site for data exchange.

3. Backing up time series and metadata from the FTP site.

4. Classify, analyze and assess the data and metadata to identify possible inconsistencies.

5. Ask for clarifications from each responsible agency when problems are found in the data.

6. Update the MDG system.

Activities for the operation of the project by INEGI

Form to incorporate data and indicators in Form to incorporate data and indicators in the MDG systemthe MDG system

Metadata formMetadata form

Concept

Name of the indicator

Definition

Algorithm

Meaning of acronyms

Source of primary data

Geographic coverage

Frequency

Updating date

Name of agency responsible for the indicator

Importance and usefulness of the indicator

International reference

Remarks

Procedure to compile and disseminate Procedure to compile and disseminate the information the information

1. The ftp://200.23.8.226/ site contains the following information: Formal the creation of the MDG Technical Committee.

MDG 2006 Manual; draft of the 2009 manual; Report from de Secretary General.

Mexico’s reports from 2005, 2006. Relationship of responsible agencies, indicators and data

sources. Terms of reference of the MDG Technical Committee. Dossiers for each one of the working groups. Each dossier

contains: Indicators’ metadata and the rules for updating it on line. Raw statistics produced or integrated by INEGI for updating

the indicators. A document with the method of calculation for each

indicator

Procedure to compile and disseminate Procedure to compile and disseminate the information the information

2. Within each working group a person is designated to calculate the indicator.

3. INEGI provides access to FTP://200.23.8.226, with different rights.

Procedure to compile and disseminate Procedure to compile and disseminate the information the information

4. Once the work has been finished and authorized, the designated person will up load on the FTP site all the statistical series of raw data to calculate the indicator, as well as the series of the indicator itself, with the corresponding metadata.

5. INEGI reviews and assesses the data and the indicators, and asks for clarifications.

6. Once the data are cleared they are put on an internal server for final review by the Technical Committee.

7. Data are published on the website.

Weekly reportWeekly report

ObservationsObservations

1. Differences in name of indicators: system vs. working groups.

2. Differences or missing metadata.

3. Lack of URL links in the metadata.

4. Acronyms not explained.

5. Inconsistencies in the raw data

6. Differences between the data produced by INEGI and the data published by the government agencies.

7. Unexplained breaks in time series of indicators or the raw data.

Typology of comments made by INEGI on the data produced bay other agencies

Moving MDG to SDMXMoving MDG to SDMX

• We used a preliminary version of the DSD for MDG and we made the following changes and we finally have a NEW DSD for MDG

• It was very useful, but we had to make the following changes in order to adapt it to our information and tools. So we have a NEW DSD for MDG:

– We changed the encoding XML ISO-8859-1 per utf-8 because our tools use encoding = "utf-8“

– The XML namespaces are generated directly from our Web Service for each type of message. So it can be seen that some namespaces (i.e. the required for data messages as "compact", "generic", etc.) do not appear in the DSD proposed by INEGI, but on the other side, it is a new one that is the xmlns: registry that is required in a DSD.

• 34 MDG indicators.• We have adjusted some code lists: 9 were

used, of which 6 are consistent with the UNSD and 3 were included by the INEGI.

• The codes are alphabetically sorted in Spanish.– Two age groups were added because of the

particularities of the indicators presented by Mexico.

– One class added: ND Not determined

Moving MDG to SDMXMoving MDG to SDMX

Progress in the conversion of MDGs to Progress in the conversion of MDGs to

SDMXSDMX

The names of the catalogs were changed, removing _SDMX or _ODM, because by recommendation the identification should be as generic as possible so they can be used by other flows. ANTES AHORA

CL_UNIT_MULT_SDMX CL_UNIT_MULT

CL_NATURE_MDG CL_NATURE

CL_UNIT_MDG CL_UNIT_MEASURE

CL_SOURCE_TYPE_MDG CL_SOURCE_TYPE

CL_LOCATION_MDG CL_LOCATION

CL_AGE_GROUP_MDG CL_AGE_GROUP

CL_SEX_MDG CL_SEX

CL_SERIES_MDG CL_SERIES

CL_REF_AREA_MDG CL_REF_AREA

CL_FREQ_MDG CL_FREQ

CL_UNIT_MULT_SDMX CL_UNIT_MULT

CL_NATURE_MDG CL_NATURE

CL_UNIT_MDG CL_UNIT_MEASURE

CL_SOURCE_TYPE_MDG CL_SOURCE_TYPE

Progress in the conversion of MDG to SDMXProgress in the conversion of MDG to SDMX

• Added ConceptScheme CS_MDG– This component (is required for SDMX 2.0), it contains a list of concepts

applicable to the DSD• Key Family MDG was changed to DSD_MDG because it is recommended

that the nomenclature of the id is based on the first initials of the type of device (DSD DataStructure, Codelist CL, Scheme Concept CS, etc) followed by the identifier.

• The following 3 attributes changed from observation to sets level, because for Mexico, in the retrieval systems and original databases, these features are specified for each series. This change produces smaller SDMX files .

– UNIT_MULT– TIME_DETAIL– FOOTNOTES

• The next three code lists were updated because the INEGI information could not be classified with the original code lists:

– CL_FREQUENCY,– CL_AGE_GROUP  – CL_UNIT_MEASURE

UnitedUnited Nations Statistics Division DSD for Nations Statistics Division DSD for

the MDGthe MDG

• DSD STRUCTURE CODELIST (CL) APPLICATION

DIMENSIONS

1. Time period Yes

2. Frequency CL_FREQ_MDG Yes- CL Supplemented3. Series CL_SERIES_MDG Yes

4. Location CL_LOCATION_MDG Yes

5. Sex CL_SEX_MDG Yes

6. Age group CL_AGE_GROUP_MDG Yes- CL Supplemented

7. Reference area CL_REF_AREA_MDG Yes

8. Units of measurement CL_UNIT_MDG Yes- CL Supplemented9. Source Type CL_SOURCE_TYPE_MDG Yes

ATTRIBUTES

1. Unit multiplier CL_UNIT_MULT_SDMX Yes

2. Time period details Yes

3. Nature of data points CL_NATURE Yes

4. Source Detail   Yes

5. Footnotes Yes

CODE LIST FOR UNITS OF MEASUREMENTCODE LIST FOR UNITS OF MEASUREMENTCODE DESCRIPTION SOURCE APPLICATION

1 Not applicable ONU Not used2 Deaths of children under five per thousand live births INEGI-New Used3 Deaths of children under one year per thousand live births INEGI-New Used4 USD in end-2006 net present value terms ONU Not used5 USD ONU Not used6 Children per thousand women INEGI-New Used7 Number ONU Used8 Kilograms per person INEGI-New Used9 Kg oil equivalent per USD1,000 constant 2005 PPP GDP ONU Not used

10 Square kilometers ONU Not used11 Local currency ONU Not used12 Local currency per USD (PPP) ONU Not used13 Women ONU Not used14 Women for men INEGI-New Used15 Live Births ONU Not used16 Population ONU Not used17 Per hundred thousand INEGI-New Used18 Percent ONU Used19 Per 1 USD GDP (PPP) ONU Not used20 Metric tons ONU Used

Nota: For the Mexico’s DSD of 34 MDG indicators,  9 classes were used (in purple), of which 3 are consistent with the Catalogue of the UN and 6 were included by the INEGI. About the codes, were used consecutive numbers, arranged sorted alphabetically terms in Spanish

CODE LIST FOR AGE GROUPSCODE DESCRIPTION SOURCE APPLICATION

Z Not applicable ONU Yes

1 Under 1 year olds ONU Yes

2 Under 5 year olds ONU Yes

3 6-11 year olds INEGI Yes

4 10-14 year olds ONU Not used

5 14 and more year olds INEGI Yes

6 15-19 year olds ONU Yes

7 15-24 year olds ONU Yes

8 15-49 year olds ONU Yes

9 All age ranges ONU Yes

Note: Two age groups were added because of the particularities of the indicators presented by Mexico.

CODE LIST FOR FREQUENCY

CODE DESCRIPTION SOURCE APPLICATION

A Annual ONU Yes

2A Two-year average ONU Yes

3A Three-year average ONU Not used

S Half-yearly, semester ONU Not used

Q Quarterly ONU Not used

M Monthly ONU Not used

ND Not determined INEGI – NewYes

Note: Added one class to characterize the statistics that have no clearly defined periodicity: ND: Not determined

SDMX – MDGConsulta del DSD desde el Web

Service de INEGIDataFlows de INEGI (Para saber qué flujos disponibles hay y sus correspondientes DSD se

hace primero una consulta dinámica de todos los DataFlows publicados por INEGI)http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Dataflow/ALL/ALL/ALL

(o si ya se sabe el nombre del DataFlow se puede acotar por ejemplo a DF_MDG)http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Dataflow/ALL/DF_MDG/ALL

El nombre del DSD de MDG se puede ver en el resultado de la consulta anterior. Por ejemplo, para el caso de los MDG el nombre es DSD_MDG. Para consultarlo se construye la URL como sigue:

 DataStructurehttp://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/DataStructure/ALL/DSD_MDG/ALL

DataStructure con artefactos de referencias (codelist, consept scheme)http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/DataStructure/ALL/DSD_MDG/ALL?references=shallow

CODELISTS (Ejemplo para consultar dinámicamente el código CL_FREQ, los nombres de los códigos los sabemos en el DSD de la consulta anterior)

http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Codelist/IAEG/CL_FREQ/ALL

SDMX – MDGConsulta de Datos desde el Web Service de INEGI en diferentes

formatosDATOS del Flujo ODM INEGIDatos xml (Se pueden visualizar en el Explorer o en cualquier visualizador como

XML-Marker)http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?

Key=ALL&format=compact

Datos xml filtrados, Ejemplo (Trae exclusivamente el indicador de MEXICO cuya SERIE es SL_EMP_TOTL)

KEY =  [FREQ].[REF_AREA].[SERIES].[SEX].[AGE_GROUP].[LOCATION].[SOURCE_TYPE].[UNIT]

http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?Key=.MEX.SL_EMP_TOTL.....&format=compact

Datos csv (Archivo separado por comas, reduciendo mas del 50% de su tamaño xml, y se puede visualizar en varias herramientas como Excel, ó llenar directamente una tabla de BD)

http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?Key=ALL&alt=csv

Datos chart (Trae una imagen, este es el mismo ejemplo anterior de datos filtrados, que trae sólo el indicador de MEXICO cuya SERIE es SL_EMP_TOTL, para Total, Hombres y Mujeres, en una imagen)

http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?Key=.MEX.SL_EMP_TOTL.....&format=compact&alt=chart

Datos json y jsonp (Para utilizar los datos del flujo en aplicaciones WEB de terceros como Facebook, Twitter, etc. -OPEN DATA-)

http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?Key=ALL&format=compact&alt=jsonhttp://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?

Key=ALL&format=compact&alt=json&callback=jsonp

SDMX – MDGCAMBIOS EN EL DSD

• Dynamic chart sourced by a SDMX flow

Proposed Timeline for Conversion to SDMX the MDG indicators

Actividades A2

A3

A4

A5

S1

S2

S3

S4

O1

O2

O3

O4

Area responsible

1. Analysis of the DSD - MDG provided by the UN and its applicability to integrated indicators in the Mexican project

* * DGAI – DGAII 

2.     Meeting with conceptual and database staffs to present the structure of DSD (dimensions and attributes) and catalogs, in order to analyze the concepts used in the DSD and to define from where will be taken

* DGAI – DGAII

3.  Establishing the equivalence between the dimensions, attributes and catalogs of DSD vs Project database

* * DGAII 

4. Analysis of the database for the transformation process

* * DGAI – DGAII

5. Amendment proposed to the DSD * DGAI

Proposed Timeline for Conversion to SDMX the MDG indicators

Actividades A2

A3

A4

A5

S1

S2

S3

S4

O1

O2

O3

O4

Area responsible

6. Generating  of a SQLServer scheme database and migrate data from Access, or agreement the access if there is a copy

* * DGAI-DGAII

7. Generating a SQL query to the database to obtain the specific information for the required flow

* DGAI

8. Loading the modified DSD and the query to the Mapping  Assistant and make appropriate correlations, saving it with its stream identifier

* * DGAI

9. Run the flow and validate the data against the original data file, and make corrections where necessary

* * * * DGAI - DGAII

10. Plenary meeting with conceptual staff in order to present them  the results

* * * DGAI

11.  Developing a plenary meeting with conceptual staff in order to present the results.

* DGAI - DGAII 

12. Sending the flow to the UN for validation * DGIAI

MDG DATA EXCHANGE AMONG MDG DATA EXCHANGE AMONG NATIONAL AGENCIES ANDNATIONAL AGENCIES AND

WITH INTERNATIONALWITH INTERNATIONALORGANIZATIONSORGANIZATIONS

The Experience of MexicoThe Experience of Mexico

Enrique OrdazEnrique Ordaz19- 21 October, 201119- 21 October, 2011Manila Manila