deutscher wetterdienst dwd gisc implementation gisc development team

Deutscher Wetterdienst

DWD GISC ImplementationGISC Development Team

GISC Audit 17.09.2010 Page: 2

Agenda Welcome

Architecture– High level Architecture

– Hardware

Performance Tests

International Collaborations and Co-operations

Contributions to WIS

Q&A


Design Principles of DWD GISC

Reuse of existing well established software and implementation methodology

Use of established Interfaces – in particular to „critical components“ to allow later exchange if necessary

– Metadata Providing

– Authentication and Authorization

Modular concept to address performance and scalability requirements

Clear separation of responsibilities

Keep things simple

Describe external interfaces


DWD GISC - Development In house development team and support of business units

Professional Services – Fraunhofer – research facility

– Con terra – experts on Metadata

Co-operation with companies– ask

• Contributes to the development with 2-3 persons

• Supports WMO events (EC Demo)

– IBL

• Long term partner

• Supports WMO events (EC Demo)

• Face lift of DWD GISC

• Offers professional services (integration and maintenance) for DWD GISC


High Level Architecture: Functional Blocks

Security Layer

Monitoring and Reporting Layer

MetadataDatabase

SubscriptionManagement

Web Interfaces– Customer– Operator– Administrator

Data & ProductImport Management

&GISC Synchroniser

DAR Components– Metadata Harvesting– Metadata Providing

Data & Product AccessManagement

24 hCache

WIS Global Ex.

File pools,Archives,Databases

DAR Generator/Importer

SRUSearch and Retrieval by URL


Timeline

Project Start

Concept Phase

11/08 01/09 03/09 05/09 07/09 09/09 11/09 01/10 03/10 05/10

VGISC Workshop

G2G com. Workshop

Impl. Start:Web InterfaceSubscription

DAR MetadataGenerator

DAR CataloguePerf. Study

G2G com. Tests

G2G com. Workshop

MoU CMA

VGISC Backup WS

OAI Infrastructure

SRU

OAI & WIS Workshop

Data & ProductManagement

Test

Test

06/10

EC Demo

08/10

OAI Tests

09/10

MetadataStudy



Security Layer


MetadataDatabase



&GISC Synchroniser



24 hCache

WIS Global Ex.






High Level Architecture: Web Interfaces

GISC– Redesign of WebWerdis

– Navigation structure

– Search functions

• Simple Search

• Extended Search (e.g. Full Text Search)

• Expert Search

– Internationalization (I18N)

– Integrate logging framework (necessary for monitoring)

– Code clean-up

– Entry points for External Application (e.g. Online Resource)

AUST– Redesign of AUST



Security Layer


MetadataDatabase



&GISC Synchroniser


24 hCache

WIS Global Ex.







High Level Architecture: DAR Generator, Importer and Components DAR Metadata Provider

– jOAI (also used by NCAR/UCAR – reference site)

DAR Metadata Harvester– OAIHarvester2

– Modified Source Code with CMA together

– In house development of a scheduler and workflow component (done by ask)

DAR Metadata Generator– In house development by DWD shared with CMA, Hong Kong, WMO, Brazil, Norway, Australia, South Africa, …

DAR Metadata Importer– In house development by DWD



Security Layer


MetadataDatabase







&GISC Synchroniser


24 hCache

WIS Global Ex.



High Level Architecture: Delivery Components

Subscription Management– Controller: time and event triggered management

AFD– Workhorse in the background

– GISC-to-GISC communication

– File distribution

– White List

Message Switching System– REST Web-Service to manage and control Moving Weather

– First proposal for generic Interface



Security Layer


MetadataDatabase



&GISC Synchroniser



24 hCache

WIS Global Ex.






High Level Architecture: SRU

SRU– WMO SRU2JDBC implementation

Cooperation between WMO and DWD available as Open Source

– In house development by DWD: Mapping of CQL (Contextual Query Language) to SQL (done by ask)

– External configuration file for Mapping

• It is easy to add new indices

• Database specific feature (e.g. functions, special operators) are kept outside of the code

– All Queries and Responses are encodedaccording to the SRU-Standard allowingfor both interactive as well as automatedusage – e.g. by catalogue systems


Hardware Architecture –Infrastructure

DMZ

INTRANET

Web Cache OAI Infrastructure

Application Server

Adapter &Scheduler

Operations

Application Server

Adapter &Scheduler

OAI Infrastructure

Integration System


Performance 05/2009 Performance Study of DAR Metadata Catalogues – Both Systems terraCatalog 2.3

and Geonetworks 2.2 do not fulfill all Performance Requirements

09/2009 CMA and DWD did performance study of GISC-to-GISC communication solutions – JMA Blog solution and AFD

03/2010 CMA and DWD did performance tests of OAI Provider solution

04/2010 INMET and DWD did performance tests of GISC-to-GISC communication solutions – AFD and Moving Weather (IBL)

07/2010 Internal performance tests of DAR Metadata Catalogue and SRU – Lessons learned– Database is hot spot, especially when we talk about Full Text Search

– Oracle standard reports

– Limit the result set – configuration parameter

– Optimize SQL Statements

• Explain plan

• Rewrite SQL Statements

• Indices

– Improved responses for most critical queries by an order of magnitude


International collaboration: CMA (GISC) GISC-to-GISC communication

– Synchronization of 24 hour cache

• Extensively tests of JMAs Blog solution

• Extensively tests of AFD

GISC-to-GISC communication– Description for GISC-to-GISC backup based on XML Schema


International collaboration: CMA (GISC) DAR Metadata

– DAR Metadata Generator

– Metadata Template

– Request for Clarification paper 11/2009 (CMA, JMA and DWD)

Metadata Providing / Harvesting– Functional tests of standard OAI provider

– Common development of OAI Harvester

– Extensively performance tests for DAR Metadata synchronization

SRU– Mapping indices to DAR Metadata elements

Workshops:– GISC-to-GISC communication workshop 04/2009 at DWD

– GISC-to-GISC communication workshop 09/2009 at CMA

– OAI workshop 03/2010 at CMA


International co-operation: JMA (GISC) DAR Metadata

– Design of DAR Metadata

– Request for Clarification paper 11/2009 (CMA, JMA and DWD)

Metadata Providing / Harvesting– Synchronization tests based on OAI

GISC-to-GISC communication– Knowledge exchange JMA Blog solution

– JMA will take XML Schema (based on a proposal from CMA and DWD)as working draft for GISC-to-GISC backup

Workshop– WIS Workshop at JMA 03/2010


International co-operation: Brazil (GISC) DAR Metadata



GISC-to-GISC communication– Moving Weather

– AFD


International collaboration: EUMETSAT (DCPC) DAR Metadata

– Conceptual design of DAR Metadata

– Knowledge exchange

DAR catalogue– Extensively knowledge exchange

VGISC ITT– Attended as expert

EUMETSAT is connected as DCPC since 23th of August 2010– Test Phase 06/07 2010


International co-operation: Hong Kong (DCPC) DAR Metadata



International co-operation: NCAR/UCAR (DCPC)

DAR Metadata– Knowledge exchange



International co-operation: VGISC Since 2002 VGISC – Météo France, UK MetOffice, DWD, met.no, ECWMF and EUMETSAT

SIMDAT

VGISC ITT

Several workshops regarding Backup and Recovery


Contributions to WIS DWD participates several ETs, ICG, …

Released Metadata catalogue performance study to community Geonetwork improvements

Shared results of DAR Metadata analysis with WIS community

DAR Metadata Generator

Released SRU2JDBC in co-operation with WMO as Open Source http://sru2jdbc.svn.sourceforge.net/

Active member of jOAI community

Contribute experts for WIS Jump Start Offer


Q&A


Live Demo

http://gisc.dwd.de/

deutscher wetterdienst dwd gisc implementation gisc development team

Documents