Past and Future Operations Concepts of NASA’s Earth Science Data and Information SystemJohn F Moses1, Francis Lindsay1, Jeanne Behnke1

1 NASA Goddard Space Flight Center, Greenbelt, MD, United States.

www.nasa.gov

IN51F-0695Fall AGU Meeting December 9-13, 2019San Francisco, CA

Earth Observing System Context• Flight mission operations separated from Science Data Systems for EOS platform command and

control, maneuvers, instrument planning & operations, spacecraft housekeeping, attitude/orbit determination, science data capture• Independent mission/instrument operations enabled by standardized flight products managed by

interface control documents (ICD) and sent to science data systems via secure network• Coordination and planning for all instruments/platforms managed through unique interfaces with

each mission science team. Instruments operate full duty cycle, ASTER is only instrument supporting prioritized science research acquisition requests.

EOS Data and Information System separated elements• Archive and distribution systems developed and operated by DAACs

• EOSDIS Core System (ECS) designed by NASA for selected complex/high volume instruments; implementation limited to three DAACs.

• Each DAAC built and operates custom software systems and data services for discipline-unique collections, ancillary data, campaigns and missions, other agency and international satellite data

• Each DAAC and host institution selected system hardware, provided network and security• Science Investigator-led processing systems (SIPS): separated from ECS, evolved from PGS.

• Custom designed to efficiently integrate science algorithms and process specific instrument data.• Each SIPS and host institution selects system hardware, provides network and security• Interfaces managed by ICD and use an EOSDIS-wide standardized scripted protocol

• Enterprise user registration, search, visualization, metrics and toolkit support • ECS Toolkit: Basic tools for integration of science algorithms into production systems, including

a common product format (HDF) and physical parameter and earth location metadata• Widely adopted prior to launch and continued use by all EOS science algorithm developers• Enabled cross discipline search and interoperability among diverse instrument measurements

• Earthdata Search and Common Metadata Repository (CMR): Evolved from Research Search Tool and IMS. Based on unified EOSDIS metadata model custom built and operated by NASA to achieve microsecond search response for internet users.

• Worldview and Global Image Browse Service (GIBS) based on creating standard browse imagery from science data and custom build to achieve rapid display of high resolution imagery

• Earthdata Login: custom configured using industry standard libraries for user email registration prior to data download. Evolved from SMC.

• EOSDIS Metrics System: Custom configured commercial tools for collecting formatted DAAC system logs per ICD and for periodic analysis and annual reporting. Evolved from SMC.

EOSDIS Data Collections in-the-cloud Target Architecture

EOSDIS Architecture and Operations Concept 1990’s

• NASA General Application Platform (NGAP): provides a cloud-based platform with access and security monitoring for DAAC applications. • Deployed with applications code to operate on a commercial cloud environment.• Enterprise operations manage cloud accounts, resource utilization and access monitoring, security,

ingress/egress, networks for application owners.• Cumulus: an application framework for data ingest and distribution utilizing object-based storage

archives. Configured for each data collection to perform acquisition, ingest, validation, backup, processing, metadata harvesting & creation, publication to CMR, Image creation for GIBS data distribution and metrics reporting. DAAC Operators use a dashboard to monitor multiple Cumulus automated workflows, discipline specific data services and cloud utilization and workflow status metrics.• EOSDIS Product Delivery Record/Product Availability Notice (PDR/PAN) legacy

interface protocol used for acquisition of science data from on-premise facilities• EOSDIS Cloud Notification Message utilized for Cloud-to-Cloud interface for Mission

Science Data Systems or Product Generation Systems. Application subscribes with vender service to receive data availability notices, and on receipt pulls data from object storage archive using cloud service.

• Application Programming Interfaces (APIs) are utilized by Cumulus Ingest and Distribution workflows to interface with Enterprise services Earthdata Login, CMR, GIBS, Cloud Metrics and science data backup systems.

• DAACs develop data collection ingest workflows utilizing Cumulus and data service applications in development ‘sandbox’ accounts. Developers integrate and test custom changes to Cumulus through System Integration and Test accounts, and compete end-user testing through User Acceptance Test accounts. DAACs release Cumulus and Data Services in production accounts. Data Services are separate from Cumulus to maintain minimum resources for ingest workflows. Egress throttle responds to ESDIS established limits and DAAC configured controls to manage cloud network utilization.

• Mission Science Data Systems perform instrument calibration and science product generation, scale cloud resources and retrieve raw mission data from the DAAC as needed for reprocessing campaigns.

• Enterprise Operations expand to add cloud vendor provisioning and resource monitoring, security access controls and monitoring through NGAP, cloud egress controls through Network Operations and backup of irreplaceable mission science data.

EOSDIS As-Built

EOSDIS Core System defined 3 Segments• Flight Operations: manages and controls EOS platforms and

instruments.• Science Data Processing: science data processing, archival and

distribution elements and a data information management system (IMS). Distributed Active Archive Centers (DAACs) host the SDPS elements a distributed portion of the IMS. Toolkits & Application Programming Interfaces (APIs) for DAAC-unique extensions and integration into Science Computing Facilities.

• Communication and System Management: overall management and operation of ground system resources. Includes the System Management Center (SMC) and the EOSDIS Science Network (ESN) at GSFC. DAACs host a distributed portion of SMC and ESN.

Science Data Processing Segment• Data Archive and Distribution System: receives and stores

science data, ancillary data. Updates metadata with inventory and quality assurance. Receives orders from IMS, updates inventory, generates accounting and request status.

• Product Generation System: routine processing and reprocessing to standard products.

• Information Management System: provides user information on EOS data and documentation, accesses databases throughout ECS, processing status, science data products, and mission descriptions and schedules. Tools for the user to enter a data acquisition requests (DAR); the IMS sends the DAR to the EOC for scheduling and tracks status.

• Research Search Tool: accesses portion of ECS resources for new search, filtering, subsetting, special product generation. User interfaces with ECS via APIs.

DAAC operations• Each DAAC performs data management and processing for a

specific set of Earth science data; data product generation, on-demand processing, algorithm integration, data archiving and distribution. Standard products, browse, subsets, summaries are created during routine and on-demand processing in response to user requests.

• DAACs control data quality, integration and testing of algorithms, storage, software support and provision of user access to data and related information. DAACs provide science consulting support and software tools to EOS users to allow them to perform algorithm updates, quality assessment and test product reviews. DAAC operations are 24 hours x 7 days a week.

• Product generation, processing & reprocessing, data ingest and distribution are scheduled, monitored, controlled, reviewed, managed by the DAAC operations staff using ECS. The staff performs product distribution, archive maintenance, systems support and data backup and recovery. High level direction for science data processing provided by the NASA Science Operations Manager.

• Technical Assistance Group assists in providing science community access to and use of ECS. TAG assists users in registration, general inquiry, product requests, logistics and user accounting, product acquisition and tracking, and aids them in use and development of metadata, subsets of data, numerical methods and tools, vector and parallel processor techniques, visualization and graphic tools, analysis tools expert systems, data formats and computing techniques. TAG also support the scientist in testing and integrating their product generation algorithms into the PGS.

• System Maintenance and Operations management was to be established at each of the ECS sites to plan, budget, configure and control resources, finance, personnel, logistics, property, security, and training. M&O will maintain hardware, software, firmware, make system upgrades and perform sustaining engineering for system improvements.

Communication and System ManagementEach ECS element manages configuration, billing and accounting, performance measurement and security, SMC interfaces with all elements for monitoring, coordination, guidelines, directives and control. SMC authorizes user and operator access. End-to-end fault, configuration, performance, and security data are shared between all segments. DAACs host a distributed portion.

EOSDIS Metrics SystemDAAC metricsLog files

Eart

hdat

aW

eb

Earthdata LoginUser authentication cookie

Use

r Re

gist

ratio

nSession ID

DAAC Metadata Common Metadata Repository

EarthdataSearch

Inte

rnet

U

ser

API

DAAC Imagery Global Image Browse Service WorldviewAPI

Raw MISR, MOPITT, TES, CERES

NOAA

MLS SIPSat JPL

ASDC DAAC at LARC

GES DAAC at GSFC

EOS

Miss

ion

Ope

ratio

ns S

yste

ms a

t GSF

C Vi

a EB

NET

TES SIPSat JPL

OMI SIPSat GSFC

MISRprocess

AIRSprocess

MOPITT SIPSat UCAR CERES

process

GMAOat GSFC

ECS

Regi

ster

ed U

ser I

nter

net D

ownl

oad

Raw AIRS, MLS, OMI

HDF HDFRaw

HDFRaw HDFRaw model

model

HDF

HDF

File archive

File archive

ECS

LP DAAC at USGS EROS

MODIS Radiance & Atmosphere DAAC at GSFC

MODISProcessing

ASTER processing

AMSR SIPSUAH

Raw MODIS

Raw ASTER

Raw AMSR, ICESAT

Snow & Ice HDF

Land HDF

HDFRaw

NSIDC DAAC at U of CO

ECS File archive

File archive

File archive

HDF

HDF

HDF

ASF, GHRC, ORNL, SEDAC DAACsFile archiveHDF

Cam

paig

ns,

Inte

rnat

iona

l Miss

ions Products

via Internet

Ingest workflows &

distribution API

NG

AP

Dashboard

Discipline data

services

Object Store archive

DAAC operations

User Operations

cloudUser Application

met

adat

aim

ager

y

Use

r Aut

hent

icat

ion

met

rics

stre

am

Network Operations

secu

rity

log

stre

am

Egress Throttle

cloud

subscribe

data for reprocess

Raw & NetCDF

Cumulus

NetCDF products

Science Data System

Data Provider

on premise

Product Generation

Mission Science Data Systems

EarthdataSearchSearch API

Metadatarepository

CMR operations

upload

MetadataManagement

Tool

Use

r Dat

a Se

arch

DA

AC

Ope

ratio

ns

Tile Server

Imageryrepository

GIBS operations

upload

Worldview

Use

r Dat

a Vi

sual

izatio

n

cloud

NG

AP

cloud

NGAP

Use

r Ope

ratio

ns

dow

nloa

d

Use

r App

licat

ion

Earthdata Login Use

r Re

gist

ratio

nEnterprise operations

User Authentication

NGAPcloud

Dat

a ba

ckup

cloud

cloudcloudNGAP NGAP

NGAPmetadata

App

code

, con

trols

Analysis

Metricsrepository

Metrics operations

replicate

CommercialAnalysis Tools

DAAC

Ope

ratio

ns

NGAP

cloud cloudNGAP

metrics stream

cloud

imagery

NGAP NGAPcloud

Science data backup-restore

Raw + NetCDF

on premise

DAAC

Dev

elop

men

t

security and resource monitoring

system deployment

NGAP Log streamsApp Code, NGAP Configuration

Access control

on premiseon premise

Enterprise Cloud Provisioningand Security Operations

Cloudtamer

cloud

cloud provisioningaccount resource utilization

clou

d pr

ovis

ioni

ngAc

coun

t util

izat

ion

tape

ESMO Mission OperationsEarth Science Mission Operations

ESDIS Science OperationsEarth Science Data and Information System

DataAcquisition

Flight Operations,Data Capture,

Initial Processing,Backup Archive

DataTransport

Science Data Processing,Data Management,Interoperable Data

Archive, and Distribution

Discovery, Data

Access, and Distribution

Network Research

Education

Value-AddedProviders

EarthSystem Models

InternationalPartners

Decision SupportSystems

Internet

Search,Order,

Distribute

Tracking and DataRelay Satellite(TDRS)

Instrument Teamsand Science

Investigator-ledProcessing Systems

(SIPSs)

White SandsComplex (WSC)

EOS PolarGround Stations

DirectBroadcast(DB)

Direct Broadcast/Direct Readout

Stations

EOSDIS Distributed

Active ArchiveCenters (DAACs)

EOS Data Operations

System (EDOS) Raw

Data Processing

EOS OperationsCenter (EOC)

Mission Control

Earth Science Spacecraft

U.S. AgencyPartners

https://ntrs.nasa.gov/search.jsp?R=20190033965 2020-07-07T08:20:16+00:00Z