Dataset registration process

Sergey Sukhonosov, Dr. Sergey BelovNational Oceanographic Data Centre, Russia

Training course on establishment of the ODP regional node and data network for SNDM-Argentina,7 - 11 October 2013, Buenos Aires, Argentina

2

Registration stepsDesignRegistrationMaintenance

3

DesignData source storage type

DBMS – all types of data;Structured data files – forecasts, climate, small volumes of

real-time data;Objective files – images, documents, shape-files, …;Web-application – WMS;

Data source structure (data files, tables/views)

4

Data source storage type - DBMSSupported databases: MySQL, PostgreSQL, Oracle, MS

SQL Server (native JDBC driver)Table structure requirement: one parameter – one

column.Possibility to get the data from one or several tablesFilter the dataBenefits:Metadata is updating automaticallyCaching modeFast processing of large amounts of data.Flexible adjustment of the data.

5

Data source storage type – Structured data files

Supported formats: text files with separator or text files with fix column positions.

Structure must be ‘plain’, non-hierarchical

Options to link data files:Specify URL to HTTP/FTP server,Upload data file to Data Provider server using web

interface,Copy data file to local Data Provider file system

6

Data source storage type – Structured data files

Benefits:• Metadata is updating automatically• Caching mode• In case of using HTTP/FTP data source it’s enough to

update data files, Data Provider will synchronize data, update data cache (instances) and update dynamic metadata (temporal extent, geographic bounding box,…)

• Connect set of data files to one metadata (specifying all file names or using filename mask (e.g. “*.txt”))

7

Objective data filesSupported formats: any data format – Data Provider

don’t analyze the content and provide data “as is”. In case if several object data files should be linked to one

metadata, they can be zipped to archive. E.g. GIS data (shp, shx, dbf).

The ways to link objective data files:• specify URL to HTTP/FTP server,• upload data files to Data Provider server using web

interface

8

Objective data filesAt least one object data file must be linked to discovery

metadata. Linked data files call ‘instances’ and have their own discovery metadata.

Metadata for instance based on resource metadata.Features:• Once data file is uploaded it can’t be updated

automatically: only manually.• You should add new data file manually via web-interface• Dynamic metadata is not updating automatically by

scheduler

9

Design

Data granularity issue: a) Split data to several metadata records by geographic

box, time period, measured parameter;

10

Designb) Split data to data subsets (instances) by “key”

parameter;

11

Data granularity – point data

12

Data granularity – point data

2001-08-06T00:00:00;-7.778;119.603;V2AP6.00;0;27.3

2001-08-06T00:00:00;-7.778;119.603;V2AP6.00;68;27.2

2001-08-06T00:00:00;-7.778;119.603;V2AP6.00;92;26.4

2001-08-06T00:00:00;-7.778;119.603;V2AP6.00;100;25.6

2001-08-06T00:00:00;-30.515;-155.227;DCFH2.00;2;17.9

2001-08-06T00:00:00;-30.515;-155.227;DCFH2.00;4;17.9

2001-08-06T00:00:00;-30.515;-155.227;DCFH2.00;6;17.9

2001-08-06T00:00:00;-30.515;-155.227;DCFH2.00;8;17.9

13

Data granularity – profile data

14

Data granularity – profile dataSpecifying record header

15

Data granularity – profile dataSpecifying list of parameters in the data table

16

Data granularity – profile data

2001-08-06T00:00:00;-7.778;119.603;V2AP6.000;27.368;27.292;26.4100;25.6

2001-08-06T00:00:00; -30.515;-155.227;DCFH2.002;17.94;17.96;17.98;17.9

17

System parameters analysisMake cross-mapping between local data columns and

system parametersContact global node (MINCYT) to add new parameters if

required

18

RegistrationUse web-interface to register metadata and connect the

data (Notice: Chrome browser is not supported yet)Use metadata templatesSpecify maximum details to fit metadata quality

requirements

19

MaintenanceUpdate data in-timeUpdate metadata on data frequency basis – create

schedulers for automatic metadata actualizationAnalyze daily reports from Integration Server

20

Questions?