cdi implementation
TRANSCRIPT
Pan-European infrastructure for Ocean & Marine Data management
An EU Integrated research Infrastructure Initiative (I3)
How to implement CDI ?M. Fichaut, V. Tosello
Training course - June 2007, Oostende, Belgium
15/08/13
How to proceed with the fi les for trainingIn the shared directory :With 2 f i les CDI_implementation.ppt : this presentation training2_programme.doc : programme sent by email
And 5 directories :
CDI documentation Detailed documentation on CDI and mapping
Example database3 directories for ACCESS, MySQL and EXCEL database
Exercises Mikado
Mikado software and configuration files
SolutionsSolution of the exercises, empty now, will be filled during the training after each exercises session
Now : Copy the common directory on your computer
15/08/13
Example database (1)
10 tables describing : the stations of datasets collected during cruises 5 cruises on 2 ships, 10 datasets, 1399 stations the datasets are either CTD or Bottle stations (mikado_dataset.database_id) Measured parameters are linked to the datasets (mikado_dataset_parameter) Mapping with common vocabularies is made through mapping tables
(mikado_map_…)
Document : \example_database\example_database_description.rtf
15/08/13
Example database (1)
One database in several environmentsOn Microsoft ACCESS \access\cdi_training_course_2.mdb
On MySQL Name of the database : mikado, user : mikado, password :
mikadoOn EXCEL \excel\ cdi_training_course_2.xls
15/08/13
First set of Exercises
Manipulate the example database in different environments
Exercise 1 : MySQL database
Exercise 2 : ACCESS database
Exercise 3 : Excel file
15/08/13
CDI implementation progress
USER LOCAL
DATABASEACCESS
MYSQL
EXCEL
EXCEL FILES
STATION METADATA(CTDs, Bottles, Current meters, …)
15/08/13
Generating the CDI - First STEP
MAP YOUR DATABASE to THE CDI FIELDS
15/08/13
Database mapping to XML schema (1)
Using the documents : \CDI documentation\ CDI_XML_V0_documentation_3_00_June2007.doc:
Metadata Format and full description of XML schema version 2.04 – as prepared in Sea-Search
\CDI documentation\ CDI_V1_04_mapping.xls : Excel file describing the mapping between the CDI fields, ISO-19115 and the local database
\example_database\example_database_description.rtf :Description of the tables of the example database
15/08/13
On the right hand side : local database reference Table_name.field_name
Database mapping to XML schema (2)
Level 2 Level 3 Level 4 …WHERE Latitude1 (var1) <dataIdInfo> (15) <geoBox> (42) <southBL> (346)WHERE Longitude1 (var2) <dataIdInfo> (15) <geoBox> (42) <westBL> (344)WHERE Latitude2 (var3) <dataIdInfo> (15) <geoBox> (42) <northBL> (347)WHERE Longitude2 (var4) <dataIdInfo> (15) <geoBox> (42) <eastBL> (345) WHERE Measuring area type (var5) <spatRepInfo> (12) <VectSpatRep> (176) <geometObjs> (178) WHERE Data Coordinate system (var6) <refSysInfo> (13) <MdCoRefSys> (189) <datum> (192)
… WHEN Start date (var12) <dataIdInfo> (15) <dataExt> (45) <tempEle> (337) WHEN Start time (UT) var(12) see Start date WHAT Parameters measured (var50) <dataIdInfo> (15) <descKeys> (33) <keyword> (53)
<KeyTyp> (54)
HOW Instrument or gear type (var42) <dataIdInfo> (15) <descKeys> (33) <keyword> (53)<KeyTyp> (54)
HOW Type of platform (var43) <dataIdInfo> (15) <descKeys> (33) <keyword> (53)<KeyTyp> (54)
WHO Originator (var16) <dataIdInfo> (15) <idPoc> (29) <rpOrgNameId> (not ISO)<role> (379)
PART CDI FIELD XML TAGS LOCAL DATABASE FIELD(TABLE_NAME.FIELD_NAME)
On the left hand side : CDI reference For each : XML tree (XML tags, with ISO-19115 number)
15/08/13
Database mapping to XML schema (3)
WHERE Latitude1 (var1) MIKADO_STATION.STATION_LATWHERE Longitude1 (var2) MIKADO_STATION.STATION_LONWHERE Latitude2 (var3) not relevantWHERE Longitude2 (var4) not relevantWHERE Measuring area type (var5)WHERE Data Coordinate system (var6)
… WHEN Start date (var12)WHEN Start time (UT) var(12)WHAT Parameters measured (var50)HOW Instrument or gear type (var42)
HOW Type of platform (var43)WHO Originator (var16)
PART CDI FIELD LOCAL DATABASE FIELD(TABLE_NAME.FIELD_NAME)
Mandatory CDI f ields in bold characters
Example of mapping
15/08/13
Second set of Exercises
Map the local database to the CDI f ields
Exercise 4 : Find all CDI fields mapping with MIKADO_CRUISE table
Exercise 5 : Find the mapping for the CDI field Data Coordinate system and Sampling Interval
Exercise 6 : Full mapping Look at the final mapping file. Questions?
15/08/13
CDI implementation progress
ISO-19115
XML SCHEMA
defined for the CDI
by SDN – TTT
USER LOCAL
DATABASEORACLE
MSSERVER
MYSQL
…
EXCEL FILES
STATION METADATA(CTDs, Bottles, Current meters, …)
MAPPING
Common vocabularies
MA
PP
ING
15/08/13
Principles of the CDI
ISO-19115
XML SCHEMA
defined for the CDI
by SDN – TTT
USER LOCAL
DATABASEORACLE
MSSERVER
MYSQL
…
EXCEL FILES
STATION METADATA(CTDs, Bottles, Current meters, …)
MAPPING
Central CDI
Common vocabularies
MA
PP
ING
TOOLSMikadoOthers
Configuration Files
Compliant to
XML CDI fi les
15/08/13
Configure Mikado - Second STEP
MAP YOUR DATABASE to THE CDI FIELDS
MIKADO CONFIGURATION
15/08/13
MIKADO : Reminder
MIKADO has been developed by IFREMER in the framework of the SEA-SEARCH project.
MIKADO can be used to generate EDMED, CSR and CDI XML files.
MIKADO can be used into 2 different ways : One manual way, to input manually information for EDMED,
CSR and CDI fields in order to generate XML files. One automatic way, to generate these descriptions automatically
if information is catalogued in a relational database.
15/08/13
MIKADO principle
MIKADOJava code
JDBCJava DataBase Connectivity
JVM – Java Virtual Machine
Native DriversMYSQL
ORACLEPOSTGRESSQLServer
Bridge Driversusing Microsoft ODBC
(ACCESS, EXCEL, SQL SERVER)
DATABASEConfigurat ion
Fi les(*.properties)
Download driversfrom web sites
if not provided with Mikado
15/08/13
List of drivers provided with Mikado
ODBC Bridge (from Java JDK 1.5) MS Access 97, 2000, 2003 MS Excel
JTDS 1.1 (Open Source licence) Sybase 10, 11, 12, 15 SQL Server
MySQL 3.1.10 (GPL licence) MySQL 4.1, 5.x
Oracle 14 (Oracle Technology Network Development and Distribution License) Oracle 8.1.7, 9i, 10i (with restrictions)
PostgreSQL 8.0.312 (BSD licence) PostgreSQL
MS SQL server 2005 driver (Microsoft download) SQL Server 2000 with Service Pack 1 or higher
Please note that Mikado source level is JDK 1.5
15/08/13
How to find the appropriate JDBC driver?
Native drivers must be downloaded from the appropriate vendors websites(i.e Oracle, Sybase, MySQL, PostGres, …) It must f i t with the database and the JDBC versions.
This l ink may help you to find out the native drivers you need : http:/ /developers.sun.com/product/ jdbc/drivers
For Oracle RDBMS , a native driver may be found on http:/ /www.oracle.com/technology/software/tech/ java/sqlj_jdbc/ index.html
For Microsoft SQL server, a native driver may be found on Microsoft MSDN http://msdn2.microsoft .com/en-us/data/default.aspx
Bridge drivers (using Microsoft-ODBC Open DataBase Connectivity) are bundled with the Java 2 SDK.
http:// java.sun.com/j2se/1.3/docs/guide/jdbc/getstart/bridge.doc.htmlDrivers have then to be installed following the given recommendations The corresponding .jar files must be copied in the \mikado\dist directory
15/08/13
MIKADO configuration f i les
See Mikado user manual \mikado\sdn_Mikado_Manual_V0.1.doc
Mikado needs 2 configuration fi les in the \mikado\conf directory files.properties file which contains
all the information about the name and the location of the files used by the software
Cdi.properties file which contains:Database connection parametersQueries to retrieve data from the database Mapping between the queries results (SQL variables) and the CDI fields
15/08/13
Mikado fi les.properties f i le All the information about the name and the location of
the f i les used by the softwareName of the files containing the list of
values for CDI [and EDMED and CSR] : Parameters codes, platforms codes, …
############### CDI###############cdiPartnerCodescdiPartnerCodes.f i le = ./conf/cdi_partner_codes.csv#cdiMeasurementAreaTypeCodescdiMeasurementAreaTypeCodes.f i le = ./conf/cdi_measurement_codes.csv
Name of XML schemas used for CDI [and EDMED and CSR]
#xsdxsd.cdi = . /conf/CDI_V1_04.xsdxsd.cdi_mult irecords = . /conf/CDI_V2_01_mult irecords.xsd
Name of the files used for CDI XML automatic generation [and EDMED and CSR]
#mapping.cdi = . /conf/cdi_training2_access.propert ies#mapping.cdi = . /conf/cdi_training2_mysql.propert iesmapping.cdi = . /conf/cdi_training2_excel.properties
15/08/13
Mikado cdi.properties f i le
3 parts in the f i le Database connection parameters
Depending on the DATABASE : ACCESS, MySQL, ORACLE …
Queries to extract data from the database To be written from the mapping
Mapping between the queries results (SQL variables) and the CDI fields
Must not be modified!!!
15/08/13
cdi.properties fi le : database connection (1)
DATABASE ACCESS JDBC (Java Data Base Connectivity) connection parameters
###############################JDBC parameters for querying database###############################bdd.query.timeout = database connection time outbdd.className= driver Java Class Name (the *.jar file which
implement this Class Name must exist in the mikado\conf\dist directory)
bdd.connectURL = connect string to the databasebdd.user.name = user namebdd.user.password = user password
15/08/13
cdi.properties fi le : database connection (2)
Examples of ODBC bridge drivers ACCESS database
bdd.className = sun.jdbc.odbc.JdbcOdbcDriverbdd.connectURL =jdbc:odbc:Driver={Microsoft Access Driver
(*.mdb)};DBQ=N://example_database/access/cdi_training_course_2.mdbbdd.user.name =bdd.user.password =
EXCEL filebdd.className = sun.jdbc.odbc.JdbcOdbcDriverbdd.connectURL =jdbc:odbc:Driver={Microsoft Excel Driver
(*.xls)};DBQ=X:/example_database/excel/cdi_training_course_2.xlsbdd.user.name =bdd.user.password =
MS SERVEUR Not recommended, use nat ive driver
15/08/13
cdi.properties fi le : database connection (3)
Examples of Native driversMySQL database
bdd.className = com.mysql. jdbc.Driverbdd.connectURL = jdbc:mysql: / / localhost/mikadobdd.user.name = mikadobdd.user.password = mikado
ORACLEbdd.className = oracle.jdbc.OracleDriver bdd.connectURL = jdbc:oracle:thin:@server:port:instance User and password must be providedExample : jdbc:oracle:thin:@josephine. ifremer.fr:1521:btest920
• MS SERVERbdd.className = com.microsoft. jdbc.sqlserver. SQLServerDriver bdd.connectURL =
jdbc:sqlserver://serverName;instanceName:portNumber;property=value[;property=value]
User and password must be providedExample : jdbc:sqlserveur:/ /www.bkrclass.org:1065;databaseName=io_mikado;
integratedSecurity=false;
15/08/13
cdi.properties fi le : mapping
Mapping of the query results to the CDI f ieldsMapping table : SQL variables = CDI fields
var1 = southLatitudevar2 = westLongitudevar3 = northLati tude…..var42 = instrumentOrGearTypevar43 = typeOfPlatformvar50 = parametersMeasured.value
WHERE Latitude1 (var1) MIKADO_STATION.STATION_LATWHERE Longitude1 (var2) MIKADO_STATION.STATION_LONWHERE Latitude2 (var3) not relevantWHERE Longitude2 (var4) not relevant
PART CDI FIELD LOCAL DATABASE FIELD(TABLE_NAME.FIELD_NAME)
Must not bechanged
15/08/13
cdi.properties fi le : SQL queries (1)
Main query
It is the query which identifies all CDI entries that must be exported. This request must return only one column which is the identifier of each entry and that will be used for sub-queries.
SQL syntax must be adapted to your local database############################# Main query (station identifier list)############################query.=select mikado_station.station_id from mikado_station
15/08/13
cdi.properties fi le : SQL queries (2)
Sub-queries For each row of the main query, it is possible to define single or
multi sub-queries.Single sub-queries : return one row for each row of the main query (ex: dataset_id, cruise_name)Multi sub-queries : return several rows for each row of the main query (ex: parameters).
The links between the fields of the database and the variables of the mapping (var1 to varN) table have to be defined in theses sub-queries as synonyms of the selected columns
15/08/13
cdi.properties fi le : SQL queries (3)
Single Sub-queries : return one row
The number of single sub-queries must be defined
queries.single.number=5
The keyword for single sub-queries is :queries.single.i= where i varies from 1 to queries.single.number
Example : queries.single.1 = select tabA.colX var1 from tabA where
tableA.colY= :$var1 = synonym for the column name from the mapping table:$ = the identifier of the row returned by the main query
15/08/13
cdi.properties fi le : SQL queries (4)
Mult iple Sub-queries : return several rows
All the names of the multi sub-queries are defined and must not be changed
For CDI one multi sub-query is defined
The keyword is :queries.multi.parametersMeasured =
Example : queries.multi.parametersMeasured =select tabA.colX var50 from
tabA, tabB where tabB.colX = :$ and tabB.colY = tabA.colZ
var50 = synonym for the column name from the mapping table:$ = the identifier of the row returned by the main query
15/08/13
Third set of Exercises
Write the cdi.properties f i les for ACCESS or MY SQL or EXCEL fi les In the exercises only the stations of the Cruise « CITHER2 LEG 1 » will
selected to have a smaller number of stations
Exercise 7 : Look at the main query. Questions?
Exercise 8 : Fulfil the missing information in query.single.1.
Query on MIKADO_DATASET table
Exercise 9 : Fulfil the missing information in query.single.2.
Query on MIKADO_STATION table
Exercise 10 : Write the query.single.4 to f ind the datum coordinate system
Exercise 11 : Have a look at the final property f i le, questions?
15/08/13
ISO date format
Date format must have the following syntax YYYY-MM-DD YYYY-MM-DDTHH24:MI YYYY-MM-DDTHH24:MI:SS
15/08/13
CDI implementation progress
ISO-19115
XML SCHEMA
defined for the CDI
by SDN – TTT
USER LOCAL
DATABASEORACLE
MSSERVER
MYSQL
…
EXCEL FILES
STATION METADATA(CTDs, Bottles, Current meters, …)
MAPPING
Common vocabularies
MA
PP
ING
TOOLSMikadoOthers
Configuration Files
15/08/13
XML fi les generation - Third STEP
MAP YOUR DATABASE to THE CDI FIELDS
MIKADO CONFIGURATION
Generate the XML fi les
15/08/13
XML CDI f i les generation
Use the Mikado software to create the XML CDI f i les Make sure that you have the right « properties » files in the
\mikado\conf directory files before running the software Change the connection to the database in the
CDI.properties file, with the path to your local database Run Mikado Import the common vocabularies lists for CDI using Mikado
: Menu configuration, library, CDI Create and output directory for your XML files
15/08/13
XML fi les check - Fourth STEP
MAP YOUR DATABASE to THE CDI FIELDS
MIKADO CONFIGURATION
Generate the XML fi les
Check the XML fi les
15/08/13
CDI implementation progress
ISO-19115
XML SCHEMA
defined for the CDI
by SDN – TTT
USER LOCAL
DATABASEORACLE
MSSERVER
MYSQL
…
EXCEL FILES
STATION METADATA(CTDs, Bottles, Current meters, …)
MAPPING
Common vocabularies
MA
PP
ING
TOOLSMikadoOthers
Configuration Files
Compliant to
XML CDI fi les
15/08/13
Send XML fi les - Fifth STEP
MAP YOUR DATABASE to THE CDI FIELDS
MIKADO CONFIGURATION
Generate the XML fi les
Check the XML fi les
Send the XML fi lesto central CDI
15/08/13
Fourth set of Exercises
Generate the XML CDI f i les and check them
Exercise 12 : Generate the XML files with Mikado
Exercise 13 : Check the XML files