Slide: 1

WGISS-37, April 2014Yonsook Enloe <yonsook.k.enloe@nasa.gov>Martin Yapur <martin.yapur@noaa.gov>

CWIC Report

2

What Is CWIC?

• The CEOS WGISS Integrated Catalog (CWIC) is being developed by WGISS to provide a framework that allows for easier search and access of Earth Observation data via partnering of CEOS agency data systems. For these partnering agencies, CWIC provides a single point of search and access.

• CWIC is a virtual clearinghouse of spatial, temporal, and science keyword metadata that brokers access to Earth Observation data by interoperating with data archives of other agencies and countries

• CWIC is envisioned to be a community catalog for satellite data that will implement the CEOS data systems harmonization results.

3

CWIC Information Architecture

The CWIC architecture consists of 3 main items:

CWIC middleware - provides an access point for an inventory-level search at CEOS agencies who are CWIC data providers

CWIC clients/portals - user interfaces to access cross-discipline data from CWIC data providers

IDN – International Directory Network (IDN) provides clients/portals directory-level search of CEOS agencies who have registered their collections in the IDN

IDNCWIC

Portal/Client

CWIC Architecture

CWIC Mediator

OpenSearch Service Interface

Mediator Core

IDN

Legend

CWIC – Wrapper

CWIC – Mediator

OGC CSW Protocol

Java API

Native Protocol

OpenSearch API

CWIC Clients

OpenSearch/CSW

Interfaces

USGS Landsat Connector

USGS Landsat Connector

OpenSearch/CSW

Interfaces

NASA ECHO Connector

NASA ECHO Connector

OpenSearch/CSW

Interfaces

Brazil INPE ConnectorBrazil INPE Connector

OpenSearch/CSW

Interfaces

CCMEO Connector

CCMEO Connector

USGS LSI Brazil INPE

NASA ECHO

CanadaCCMEO

OpenSearch/CSW

Interfaces

NOAA GHRSST Connector

NOAA GHRSST Connector

NOAA GHRSST

CSW Service Interface

5

CWIC Basics

• Provides an access point for an inventory (granule) search at multiple CEOS agencies.

• CWIC provides translation from OGC CSW 2.0.2 (ISO) or “CEOS OpenSearch” to the agency inventory system protocol.

• CWIC uses “Connectors” to provide translation from CSW/OpenSearch to the agency system.

• CWIC does NOT contain any copies of data or metadata• Search responses are returned using ISO 19115 or Dublin core

metadata for CSW and Atom encoding for OpenSearch.• Search responses contain urls to download data or urls to

agency data ordering system or email orders.• All CWIC data is searchable and accessible.• CWIC clients expected to support directory/collection search at

the IDN using CSW.

CWIC Team

Who is on the CWIC team?• CWIC Design team• CWIC Server Implementation team• CWIC Data Partners• CWIC Client partners • International Directory Network (IDN)

Work together to solve common challenges for search and access – approaches and solutions that work for CEOS agencies’ operational EO systems.

• Need to consider end to end implications for assumptions and approaches, diversity of providers and clients, and scalability and maintainability of solutions

7

CWIC Support from USG

At the October 2012 CEOS Plenary, Mike Freilich (NASA) announced that the US government agencies - NASA, NOAA, and USGS – will provide long term funding and resources to the CWIC activity starting in May 2013.

The IDN existed before the CWIC initiative and provides other services to CEOS and GEO beside collection level discovery for CWIC. For this reason, at the present time the IDN funding will remain a contribution from NASA.

8

ISRO

CWIC Data Partners

CCMEO

NASA

AOE

USGS

INPE

NOAA/GHRSST

NASA - National Aeronautics and Space Administration NOAA - National Oceanic and Atmospheric Administration CCMEO-Canadian Center for Mapping and Earth ObservationGHRSST – Global High Resolution Sea Surface Temperature

ISRO - Indian Space Research Organisation

INPE – National Institute for Space Research USGS – United State Geological Survey AOE - Academy of Opto-Electronics, Chinese Academy of Sciences (CAS) -- (Beijing-1, NSMC/FengYun, CRESDA)

CWIC Data Partners & Data

CWIC provides a major source of satellite data inventory search and data access in GEO.

Current data partners include NASA, USGS (LSI), INPE, NODC/GHRSST, CCMEO, (dev partners: AOE, ISRO, NOAA CLASS, ROSCOSMOS,…EUMETSAT?)• NOAA will add access to most of the data in CLASS when the new API is

operational.• NASA working to make all EO unrestricted online satellite data accessible from

CWIC, including near real time data (from LANCE).• GHRSST (SST VC) is working to make all their data accessible from CWIC at

NOAA NODC. NODC will also add data to this CWIC connection.• CCMEO provides search and email data ordering to access to Radarsat-1 &

Radarsat-2• AOE working to add CRESDA (HJ1A, HJ1B, CBERS01, CBERS02, CBERS2B),

Beijing-1 (BJ-1), and NSMC/Feng-Yun (FY3A, FY2D, FY2E).• ISRO is making progress on their CWIC access (Nitant Dube).• ROSCOSMOS to start work on their connections soon.• EUMETSAT has scheduled an intro telecon in early May for data partner info

CWIC Data Partners API Consistency

CWIC Data Partners API Consistency – making all the CWIC partners look “the same” – (makes for happier clients) for CSW interface and also for the “CEOS OpenSearch” interface

• Support same search criteria• Support pagination• Support chronological (most recent first) sort order for returned

results• Support online data access/order url or email order• Support exceptions code handling (to convey to the user, the data

partner specific info or other status)• If spatial criteria is mandatory for provider, what the default spatial

criteria should be (e.g. global).• If temporal range is required, then send exception condition No

“natural” default for temporal range.Details of each data partner API is on the CWIC webpage in the CWIC Connector API Report

CWIC Data Partner Responsibilities

• CWIC Data Partner responsibilities:• Register all data collections (with CWIC accessible inventories) in the IDN• Offer search access to the agency inventory system• Offer data access to the granules/files to the CEOS/GEO users –

o Direct data downloado Direct link to data ordering system with specific data granule informationo Email order with specific data granule information

• Provide technical POC (programmer type) who will assist the CWIC Server team in implementing a CWIC Connector that translates the CWIC supported protocol to their native protocol.

• Test the Connector and test access to collections of inventories• Work with the broader CWIC team to come to agreement on resolutions to

common issues.

• CWIC Data Partner do *NOT* need to• Change native protocols • Change metadata format

CWIC Client Partners

CWIC-Start Prototype– user interface client for scientists (supports CSW)

• Provides collection search by IDN keywords (mission/instrument, science,..) via the IDN CSW

• Provides search and access to all CWIC accessible inventories via the CWIC Server

CWIC-Smart Prototype – user interface client for scientists (supports OpenSearch) – accesses IDN & CWIC

LSI Portal• Provides search and access to the LSI data of interest

COVE• CCMEO Radarsat2 through the CWIC connection

GEO Partners : includes GEO DAB, GENESI, GEO Portal

CWIC Support for New Partners

Welcoming new partners! • Bi-lateral introductory telecons – new partner can ask initial

questions in “private” and not in full view of the BIG TEAM.

CWIC Data Partners• CWIC DocBuilder for CWIC DIFs – plus IDN staff assistance

with questions.• CWIC Server team builds “Connector” to translate CWIC

searches to Data Partner system with assistance from new data partner.

• CWIC Server team assists with any testing issues,

CWIC Client Partners• CWIC Server team assists with any testing issues

CWIC Servers : (DEV, TEST, PRODUCTION, STATIC)

CWIC test clients

CWIC Road to OpenSearch

• Broad review of the OGC Geospatial and Temporal Extensions & OGC Extensions for EO documents in summer 2013 with ESIP OpenSearch implementers; Comments given to the CEOS OpenSearch team who submitted it to ESA (who funded the spec author).

• Most comments incorporated into the revised OGC spec released in late fall/winter 2013.

• Fall: Series of bi-weekly telecons with the CWIC, IDN, and CWIC-Smart teams, discussing an overall design for the OpenSearch implementation, identifying design/implementation issues, coming to resolution on these issues, etc.

• January 2014 CWIC team meeting, focused on making decisions on OpenSearch implementation issues and firming up the CWIC OpenSearch Best Practices.

CWIC Road to OpenSearch

• Feb – early April : OpenSearch implementation according to the CWIC OpenSearch Best Practices document at IDN, CWIC, and CWIC-Smart

• End March/early April – Early prototype implementations made accessible for team testing

• Future plans to compare the CWIC & IDN OpenSearch implementation interfaces with the FedEO OpenSearch interfaces;

• Major parts of the CWIC OpenSearch Best Practices will be put into the CEOS OpenSearch Best Practices wherever possible.

• Future plans to come to agreement within the CEOS OpenSearch team on the CEOS OpenSearch Best Practices and to revise the software, if necessary;

CWIC Status

• CWIC Servers (DEV, TEST, PRODUCTION, STATIC) working well on a new hardware platform at USGS/Sioux Falls since summer 2013. The CWIC servers at GMU will be shut down in early summer.• Cwic.wgiss.ceos.org• Cwictest.wgiss.ceos.org• Cwicdev.wgiss.ceos.org• Cwicstatic.wgiss.ceos.org

• Solidifying Test plans and processes to support new CWIC server enhancements, new data partner addition, IDN/CWIC synchronization, data accessibility testing (new data, confirmation of existing data); Automating testing wherever possible.

• Enhanced metrics collection and display

CWIC Status (cont)

• CWIC offers support for OpenSearch!• CCMEO became an operational data partners

(Radarsat-1 & Radarsat-2)• The CWIC server team implemented email ordering

support in the CCRS CWIC Connector – which populates an email with detailed information about the granule the user wants to order

• COVE (CEOS Visualization Environment) tool starting work to access Radarsat-2 metadata through CWIC

• CWIC-Start (new client prototype that implements OpenSearch)

CWIC Status (cont)

• IDN/CWIC synchronization plan simplified with the development of the IDN “tagging”/mapping tables. The metadata about metadata is kept separately as “tags” or mapping table. Example: Project=CWIC is now a CWIC tag.

• Late 2013, ROSCOSMOS announced plans to become a CWIC data partner

• April 2014, Eumetsat asked for more information about becoming a CWIC data partner using the OPenSearch API; Info telecon scheduled for early May

CWIC Future Work

• Work with the CEOS OpenSearch team to come to agreement on the CEOS OpenSearch Best Practices• Align IDN/CWIC OpenSearch implementations with the final version

of the CEOS OpenSearch Best Practices• Interoperability testing with the FedEO OpenSearch implementation• Study the feasibility of FedEO Connector to CWIC

• Produce new documents to update the CWIC Data Partner Guide, CWIC Client Partner Guide for OpenSearch.

• Support the developing CWIC Data Partners to become operational

• Strategy/plans for supporting VCs, WGD, SDMS, etc. for data access issues

• Continue to support GCI integration• Strategy for “Service Level Agreements” – how to get

agreements on improving the reliability of the CWIC service, including agreements with data partners

GCI Integration

• Both the IDN and CWIC integrated with GEO, and accessed by the GEO components (Geo Web Portal, DAB, GENESI).

• IDN provides collection metadata for over 24,700+ data collections available to GEOSS (April 2014), with 9600+ of these data collections tagged as “geossdatacore”.

• Registering a data collection in the IDN in effect registers the data collection in GEOSS.

• A data collection must be registered in the IDN to be CWIC accessible.

• CWIC provides access to over 1900+ (April 2014 figure) data collections with inventories (over 70 million granules, adding granules daily from live missions)

• Providing technical support for GEO component teams accessing IDN and CWIC.

• GEO Director, Barb Ryan, noted during the GEO session at the Nov 2013 CEOS Plenary, that of the 65 million granules accessible in GEO (Oct 2013), CWIC provided access to over 50 million granules.

GCI Picture

22

http://wgiss.ceos.org/cwic

CWIC Project

CWIC Info and Resources

http: wgiss.ceos.org/cwic• Link to youtube CWIC demo• How-to documents : CWIC Data Provider, CWIC Client Provider,

CWIC DIF Guide• CWIC Connector API Report, CWIC Exceptions Handling Guide,

…• Link to CWIC DIF Builder tool• Links to CWIC enabled clients• Links to open source CSW software (for Partners)• Link to Test Environment

cwic@wgiss.ceos.org email list• Monthly CWIC Team telecons

Farewell to Dr. Yuanzheng Shao

Yuanzheng Shao, the development lead for the CWIC Server is leaving the team to return to China.

Shao will be an associate professor at the University of Wuhan.

THANKS very much for your insights and hard work!

Our very best wishes to you and to your family in your new life in China!