Summary of the EOSDIS Reference Architecture

ESIP Federation MeetingESDSWG SPGJuly 22, 2010

Outline

• Background• EOSDIS Architecture Framework

– Science Data System Functions– Cross-cutting Functions

• Scenarios– Satellite Data– Field Campaign– Airborne

• Issues

3

EOSDIS Context

Tracking & Data Relay Satellite (TDRS)

Instrument Teamsand Science

Investigator-ledProcessing System

(SIPS)

EOSSpacecraft

Direct Broadcast/Direct Readout

Stations

DirectBroadcast

(DB)

EOS Data and Operations System

(EDOS) Data Processing

White SandsComplex

(WSC)

EOS PolarGround Stations

Internet

(Search, order,distribution)

Research

Education

Value-AddedProviders

InteragencyData Centers

InternationalPartners

EarthSystem Models

Decision Support Systems

Distribution and Data

Access

Data Acquisition Flight Operations,Data Capture,

Initial Processing,Backup Archive

Data Transport

to Data Centers/

SIPSs

Science Data Processing,Data Mgmt, Interoperable

Data Archive, & Distribution

NASA Integrated

Services Network

(NISN) Mission Services

EOS Operations Center (EOC) Mission

Control

ESMO ESDIS

EOSDIS DataCenters

EOSDIS Facilities

4

Data centers, collocated with centers of science discipline expertise, archive and distribute standard data products produced by Science

Investigator-led Processing Systems (SIPSs)

ASF DAACSAR ProductsSea Ice, Polar

Processes

SEDACHuman Interactions

in Global Change

LP DAACLand Processes

& Features

PO.DAACOcean Circulation

Air-Sea Interactions

ASDCRadiation Budget,Clouds, Aerosols,Tropo Chemistry

ORNL DAACBiogeochemical

Dynamics, EOS Land Validation

GES DISCAtmos Composition &

Dynamics, GlobalModeling, Hydrology,

Radiance

LAADS/MODAPS

Atmosphere

OBPGOcean Biology &Biogeochemistry

GHRCHydrological Cycle &

Severe Weather

CDDISCrustal Dynamics

Solid Earth

NCAR, U of Col.HIRDLS, MOPITT,

SORCE

GSFCGLAS, MODIS,

OMI, OBPG

LaRCCERES, SAGE III

GHRCAMSR-E, LIS

JPLMLS, TES

San DiegoACRIM

NSIDC DAACCryosphere, Polar

Processes

SIPSs

Key

DataCenter

Purpose

• Purpose of the EOSDIS Reference Architecture– Documents the way EOSDIS does business, in all of its

variations– Enables discussion with other communities and agencies

on earth science data system development– It provides a framework for the insertion of community

developed systems into the core system– Informs new missions of the current processes– Serves as a starting point for a future-looking Earth Science

Data System Architecture

• Discipline and mission specific data systems

• Only community-specific standards

• Only directory functions to point and link users to various data systems

• Multi-site access and data inter-use cumbersome

• Improved access to data with 90s technology

• Cross-system search and order access via interoperability data model

• Site-specific interfaces at DAACs/SIPs

• Common distribution format (HDF); other formats also supported

<1990 mid-1990s

Evolution of EOSDIS Data System Features

Technology,Funding,Lessons

Technology,Funding,Lessons

6

• Support for high data volumes, ambitious performance requirements, high availability/reliability

• Integrated core infrastructure plus loosely coupled elements from community

• Common data model– Automated metadata

creation and ingest– No need for cross-site

metadata translation– FGDC standards

compliance• Expanded set of software

tools and services• Flexible options for

supporting or interoperating with external data sources

• Coexistence of heterogeneous, distributed data providers/ information partners

• Optimal set of core standards

• Support for community-specific standards

• Reusable software components

• Exploitation of Service Oriented Architecture

• On-line archives and cross-system service invocation

• Ease of innovation and technology infusion

• More services to meet community demands

Late 90s to present Near-Future

Evolution of EOSDIS Data System Features

Technology,Funding,Lessons

Technology,Funding,Lessons

Technology,Funding,Lessons

7

Core and Community Capabilities - Characteristics

CORE COMMUNITY

Projects Subject to Programmatic Review Projects Competitively Selected

Substantive NASA Oversight ‘Light Touch’ Oversight w/Significant Community Involvement

Tight Integration of Data System Tools, Services and Functions

Community-based Tools and Services Loosely-Coupled

Employ Well Established Information Technologies

Employ ‘Edgy’ or Emerging Technologies

8

EOSDIS Architecture Framework

Satellite Data Scenario #1Function EOSDIS Approach Description

1) Level Zero Processing EDOS Level 0 data handling system

Data captured from satellite-based instruments enters EOSDIS via the EOS Data and Operations System (EDOS) Level 0 data handling system

2) Science Processing Science Investigator-led Processing Systems (SIPS)

The Science Investigator-led Processing Systems (SIPSs) collect Level 0 data and process these products into higher level products.

3) Archive ECS, Data Center specific systems

The EOSDIS data centers, many of which are also Distributed Active Archive Centers (DAACs), perform the functions of ingest, storage, data management and access.

4) Discovery ECHO/WIST/Data Center specific systems

The EOSDIS provides a metadata handling function by which the data system keeps track of its vast data collection and data center unique search and order systems.

5) Networks NISN, Internet2Effective access to EOSDIS depends on the end-to-end network connectivity between users and geographically distributed EOSDIS Data Centers.

6) Data System Standard Interfaces

Interface Definition, Verification and Validation

EOSDIS fosters the development of standards used within science data processing systems. The Hierarchical Data Format (HDF), was selected as the data format to be used for archiving and distributing data products for EOS instruments.

7) Performance Monitoring

ESDIS Metrics System, Annual User Surveys

The EMS supports the ESDIS project management by collecting and organizing a variety of metrics from the distributed data centers and other data providers (such as the GCMD and ESDIS-sponsored websites).

8)Systems and Sustaining Engineering

ESDIS Systems Management Process

EOSDIS is managed by the ESDIS Project which oversees the planning and funding for the EOSDIS system delegating local authority to individual data centers to implement the system needed to satisfy their discipline community.

AMSR-E Data Flow

TLSCF ScienceSoftware I&T SIPS-GHCC

PO.DAAC

JAXA-EOCL1A Product Generation

SIPS-RSSL2A Product Generation

L2B and L3 Product Generation(science, metadata, qa, ph, subsets, browse, DAPs)

L2 and L3DAPs

L2A algorithm

NSIDC DAAC Data Archive and Distribution

Level-1

Level-2 and Level-3

Level-1

Level-1

NASAEDOS

RBD & PDS

GroundStations

L2B and L3 Algorithms

Level-0

Product Delivery Record Server (PDRS)

Algorithm Teams

Ocean Sea Ice

Rain

Land

Snow Data

NASAFDS GBAD

Descriptions of EOSDIS data holdings can be found at the archiving data center.

See AMSR-E at the National Snow and Ice Data Center (NSIDC).

Select the Order Data option.

http://nsidc.org/data/amsre/

http://nsidc.org/data/amsre/order_data.html

Users have a variety of ways to access EOSDIS data.

For example select the WIST option.

Warehouse Inventory Search Tool (WIST)

• The EOS ClearingHouse (ECHO) is a metadata catalog of NASA’s EOS data and a registry for related data services (e.g. reformatting, pattern recognition). ECHO’s catalog contains more than 2800 data sets held at 12 EOSDIS data centers.

• WIST, the Warehouse Inventory Search Tool, is the primary web-based client for discovering and ordering cross-discipline data from all of ECHO’s metadata holdings. WIST allows users, including those without specific knowledge of the data, to search science data holdings, retrieve high-level descriptions of data sets and detailed descriptions of the data inventory, view browse images, and submit orders via ECHO to the appropriate data providers.

• The WIST search-and-order tool has the following features: • Search capability—Allows a user to explore the list of data sets or granules returned by a search by viewing their temporal

coverage, spatial coverage, attributes (metadata), related documents (guide search), and browse images.• Order function—Allows the user to select data for ordering, choose packaging information, enter ordering information (such

as shipping address), place an order, and view order status.

Reference:http://wist.echo.nasa.gov/~wist/api/imswelcome/

Other Scenarios

• Field Campaign (In situ) scenario• Ground and Ocean-based scenario• Airborne scenarios• Ancillary scenarios• Satellite scenarios

– US– International

Examples of EOSDIS Data Center Implementations

EOSDISData Centers

Level Zero Processing

ScienceProcessing Archive Discovery

ASF DAAC GAP GAP GAP ECHO/GCMD/URSA

CDDIS NA IGS/IGN CDDIS ECHO/GCMD/Website

GHRC EDOS SIPS GHRC ECHO/GCMD/HyDRO

GES DISC EDOS SIPS/S4PM S4PA ECHO/GCMD/

MIRADOR/OPeNDAPLP DAAC EDOS SIPS ECS ECHO/GCMD/GloVis

ASDC EDOS SIPS ECS/ANGe ECHO/GCMD/ANGe

MODAPS/LAADS EDOS SIPS LAADS ECHO/GCMD/ Website

NSIDC EDOS SIPS ECS ECHO/GCMD

OBPG EDOS/OBPG OCDPS OBPG

OBPG/GCMD/SEADAS/SEABASS

OPENDaP, others

ORNL NA NA ORNL ECHO/GCMD/ Mercury

PO.DAAC PO.DAAC SIPS DMAS ECHO/GCMD/POET/OPeNDAP/ FTP

SEDAC NA NA NA ECHO/GCMD/SEDAC Information Gateway

EOSDIS Reference Architecture• Plan both a document & website• EOSDIS Reference Architecture Draft

– Under discussion and drafting components• DAACs• Committee of NASA data system planners

– Scenarios Developed• Satellite Data• Field Campaign• Airborne

• Basic Website in place• However, the reference architecture only documents EOSDIS

functions in a very broad approach– Does not take the place of official NPR documentation– Each new Mission requires direct communication and distinct

documentation

Example Science Interfaces & Documentation

Networks ESDIS Metrics System

ESDIS Project

Users

Flight Project

ECHO

DMRADURD

SIPSs

MRT Mission Readiness Test Plan

IPA

ICD W/A

ICD

ICD ICDIRDICDOA

ATBDICD

ICD ICDICD

A subset of interfaces are shown for mission operations

ESMO Project

ESDIS Project

ICD

Science Team

Data CentersEDOS

http://esdis.eosdis.nasa.gov/dsdocs

Interoperability on Many Levels• Process Methodology

– Configuration control– Business processes– Policy processes

• Software Reuse – S4PA, subsetting• Standards

– Metadata model– Data formats– Interface controls

• EOSDIS ClearingHouse• EOSDIS Metrics components

EOSDIS Business Model• Budget preparation and tracking of actual

expenses is performed across the Project.• Components of EOSDIS submit yearly budgets

and work plans • Budgets and actuals are tracked monthly against

phasing plans and schedules• Metrics are collected for both technical and

fiscal management• Coordination across the architecture allows for

cost savings

Specific Areas of cost tracking

Category Definition

Mission and Science Support

Quality assurance and verification, metadata development, data architecture and definition, Science guide documentation and science team interaction, metadata catalog maintenance

User Services Web support (content development and maintenance) and outreach (NASA booth), tutorial development, Answering phones and emails

Operations Code execution and job scheduling, processing, distribution, Ingest Management, metrics operations & troubleshooting

Engineering Web and tool development, evolution of existing systems, System development, sustainment, database development

Infrastructure Configuration mgmt, systems admin, DBA

IT Security Applications and network security, contingency planning

Administration Project and resource managementAdministrative assistance

Conclusion

• The EOSDIS Reference Architecture is still under draft by many staff members– Thanks to all of you!!!

• Any questions? Comments?• Feel free to contact me:

– Jeanne Behnke– jeanne.behnke@nasa.gov

AMSR-E – Advanced Microwave Scanning Radiometer - Earth Observing System JAXA – National Space Development Agency (Japan)TLSCF – Team Leader Science Computing Facility (AMSR-E)RSS – Remote Sensing Systems (company)GHCC – Global Hydrology and Climate Center (MSFC, Huntsville, Ala)DAAC – Distributed Active Archive CenterPO.DAAC – Physical Oceanography DAACNSIDC – National Snow and Ice Data Center (NSIDC)DAP – Delivery Algorithm PackageGBAD – Ground based attitude determinationFDS – Flight Dynamics SystemRBD – Rate buffered dataPDS – Production data set