1 nodc, russia gisc & dcpc developers meeting langen, 29 – 31 march e2edm technology...
Post on 18-Dec-2015
224 views
TRANSCRIPT
1NODC, Russia
GISC & DCPC developers meeting
Langen, 29 – 31 March
E2EDM technology implementation for WIS GISC
development
S. Sukhonosov, S. Belov
2NODC, Russia
History
• E2EDM was originated as one of the JCOMM/ETDMP pilot projects in 2003
MetOffice, UK
IFREMER, France
VLIZ, Belgium
NODC, Russia
3NODC, Russia
Pilot project data sources
The following data sources are involved in the E2EDM prototype• Historical marine meteorological data• Historical ocean cruise data• Real-time GTS ocean (BATHY and TESAC) data• Real-time GTS ocean (TESAC/ARGO data)• Real-time GTS (SHIP) data• Monthly climatic fields of ocean parameters (imageries)• Ocean SST satellite data
Geographic area – North Atlantic (test-area)
4
Scope
NODC, Russia
E2EDM provides following functionality:
• Wrapping local data by means of E2EDM components• Mapping mechanisms (codes, structures)• Data search and retrieving from distributed data sources on
user requests (real-time and scheduled)• Data obtaining and further processing, visualization and
exchange
E2EDM is integrating operational and delay-mode time scale data; across oceanographic and marine meteorological disciplines; from multiple data source formats and multiple data providers in different geographic regions.
6NODC, Russia
E2EDM concept model
NODC, Russia
Providing formalization, identification, description and interpretation for model objects. Model is based on the following semantic objects:
• Element• Class (set of elements)• Record (set of classes)In terms of the model each technological object is described by
means of listed semantic objects. E.g. object called “Resource” is record, which contains features – classes. Each resource record (or description) contains granularity structures for the source data, e.g. first level of granularity – platform, second level – date and time, third – measurements (sea surface temperature, water salinity, etc.).
7NODC, RussiaNODC, Russia
E2EDM namespace
NODC, Russia
Defines metadata and data structures. Within E2EDM consists of two sub-namespaces: global and conceptual (thematic).
• Global namespace – structures for the model objects• Conceptual (thematic domain) namespace – list of thematic
elements dealing with some discipline, e.g. meteorology, and available for discovery and exchange.
Both based on XML Schema.1. Global XML Schema (ISO 19115 + WMO Core Metadata)2. Concept XML Schema (DiGIR)
8NODC, RussiaNODC, RussiaNODC, Russia
E2EDM data exchange protocol
NODC, Russia
Data exchange protocol is responsible for searching and data retrieving.
Request message
(XML,DiGIR-protocol)
Response message
(XML,DiGIR-protocol)
Transport data file(NetCDF)
9NODC, Russia
<request xmlns="http://digir.net/schema/protocol/2003/1.0" xmlns:e2edm="http://data.meteo.ru/digir/schema/conceptual/cruise" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://digir.net/schema/protocol/2003/1.0 http://digir.net/schema/protocol/2003/1.0/digir.xsd
http://data.meteo.ru/digir/schema/conceptual/cruise http://data.meteo.ru/digir/schema/conceptual/cruise/pilot2cruise.xsd"> <header> <version>0.95</version> <sendTime>2005-08-03T14:47:52+03:00</sendTime> <source>127.0.0.1</source> <destination resource="RU_RIHMI_01">http://data.meteo.ru:80/digir/DiGIR.php</destination> <type>search</type> </header> <search> <filter> <and> <lessThanOrEquals> <e2edm:Latitude>78.54</e2edm:Latitude> </lessThanOrEquals> </and> </filter> <records limit="100" start="0"> <structure> <xsd:element name="record" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:complexType> <xsd:sequence> <xsd:element ref="e2edm:dateTime"/> <xsd:element ref="e2edm:DEPH"/> <xsd:element ref="e2edm:TEMP"/> </xsd:sequence> </xsd:complexType> </xsd:element> </structure> </records> <count>false</count> </search></request>
Request message example
10NODC, Russia
Response message example<responseWrapper>
<response xmlns="http://digir.net/schema/protocol/2003/1.0"><header>
<sendTime>2005-09-29T15:35:17+04:00</sendTime><source resource="RU_RIHMI_03"/><type>search</type>
</header><content>
<record><E2ETransportSpecification>
<transportFileURL>http://data.meteo.ru:8090/dpms/ncResults/1127993716743-result.nc</transportFileURL>
</E2ETransportSpecification><count>7</count>
</record></content>
</response></responseWrapper>
11NODC, Russia
E2EDM prototype components
1. E2EDM prototype specifications -- structures, metadata/data formats, codes and other means of data unification from the local data systems
2. E2EDM Prototype software – E2EDM Data Provider system and E2EDM Integration Server
12NODC, Russia
Functionality:Integration server• Data discovery (through resource
catalog)• Monitoring of distributed data
sources• Request/reply mechanism• Data distribution from the multiple
data sources• Metadata synchronizationData Provider• Request handling• Access to the local data source
(DBMS, files)• Data extraction based on search
criteria• Transport data file generation
Namespace
Global XML Schema, conceptual XML
Schema USER
Integration Server Management Service
Metadata service
Data access service
Interface service
Integration Server
Transport service
Request message to data
provider
Response message from data provider
Data Provider
Data Provider Management Service
Database Access Service (DiGIR-based)
Local database
Local database
SQL query
Result
Extended transform
API
Configuration metadata
Data format service
File Access Service (OPeNDAP based)
Local data system(files)
Transport data files
Navigation service
Request message to data
provider N
Response message from data provider N
Response from data provider
Diagram of prototype
13NODC, Russia
General scheme
User
Integration Server
Interface layer (HTTP, Web-services)request
Data ProviderData Provider
Data Provider
Request message
reply
Transport data file
Transport data file
Response message
14NODC, Russia
Integration Server user web-interface
Resource catalogue
Parameters
Additional search options
16NODC, Russia
• E2EDM Web site http://data.meteo.ru/e2edm/
• E2EDM Integration Server - IODE Project Office (VLIZ)
http://e2edm.vliz.be - RIHMI-WDC (mirror) http://data.meteo.ru:8080/iserv/
Integration Server user web-interface (cont.)
Point data grid dataprofile data
17NODC, Russia
Current status
• Released technical specification document• Builded software framework using SOA• Extended community of national Data Providers• Solved performance problems (e.g. response time)• Implemented caching mechanism• Defined user policy
18NODC, Russia
Future…
• It is important to develop administrative services for component management and maintenance• To build up predefined queries, push/pull and notification mechanisms• System metadata mapping into WMO Core Metadata Profile• To develop web-services in order to improve compatibility with other systems