Data Management Subsystem

Jeff Valenti (STScI)

DMS Context

PRDS - Project Reference Database

PPS - Proposal and Planning

OSS - Operations Scripts

FOS - Flight Operations

WSS - WFS&C Software

DMS - Data Management

Science & Operations Center

Ground Segment

JWST

Element

Segment

Mission

Subsystem

Science Data Flow - The Big Picture

FocalPlaneArray

ApplicationSpecific I ntegratedCircuit

FocalPlaneElectronics

FocalPlaneArrayProcessor

SolidStateRecorder

DeepSpaceNetwork

NASA I ntegratedServicesNetwork

FlightOperationsSubsystem

Data Management Subsystem

HighGainAntenna

ISIM

Spacecraft

CommonSystems

Science &OperationsCenter

ObservatoryGround

Segment

Element

What is the DMS?

“The DMS provides the data processing, archive, catalog, calibration, distribution and analysis functions required to support the science program and maintenance of Observatory performance.”

Ground Segment Requirements Document(JWST-RQMT-1056, Level 2)

Mission Operations Concept Document (JWST-OPS-002018)

General

MO-599 Archive science data from JWST. Distribute data to users.

MO-601 Backup data onto removable media. Store offsite.

Engineering Data

MO-602 Do not processes engineering data (handled by FOS).

MO-603 Obtain processed engineering data from FOS.

Engineering Images

MO-609 Archive guide star acquisition images.

MO-1086 Calibrate wavefront sensing images within 3 hours.

MO-1089 Send calibrated wavefront sensing images to WSS.

Mission Operations Concept Document (JWST-OPS-002018)

Science Data

MO-600 Process and archive science data. Notify users when processed science data are available.

MO-602 Defines “data processing levels” for science data.

MO-604 Receive science data from FOS, process, and archive. Upon request retrieve, process, and distribute science data.

MO-605 Remove instrumental signatures and observation artifacts.

MO-606 Make science data available within 5 days of observation. Archive raw data (229 Gbits/day) plus derived data. Archive users must register. Protect proprietary science data.

MO-607 Process science data upon receipt. Verify data integrity. Record pipeline parameters and reference file names. Archive metadata (preview, object lists, NIRSpec image map). Upon request, reprocess data “on-the-fly”. Deliver FITS files. Protect proprietary data using archive accounts. Distribute public data to mirror archives.

Mission Operations Concept Document (JWST-OPS-002018)

Science Data (cont.)

MO-608 Recorded science data identifies exposure and observation. Use timestamps to merge engineering data with science data.

MO-610 Decompress, calibrate, format, and archive science data. Assess dataset completeness. Report incomplete processing. Produce calibrated science data suitable for analysis. Notify users when processed science data are archived. Distribute data via network, unless physical media requested. Store data online to allow “quick” data access.

MO-611 Base initial pipeline reference files on ground tests. Calibration scientists use flight data to update reference files. Use best reference files in pipeline and provide to community.

Data Processing Levels

Level 0

Recorded Science Data - Downlinked files containing packets of compressed pixel data, as stored by ISIM on the SSR. Data from a particular integration may be spread across multiple files.

Level 1

Raw Exposure Data – FITS file containing up-the-ramp image data for one detector and one exposure, which may consist of multiple integrations. Relevant packets are extracted from recorded science data files, uncompressed, and assembled into one image sequence per integration. Engineering telemetry and calibration keywords are added to the FITS header.

Level 2

Count Rate Images – Up-the-ramp image sequences are fitted to create a count rate image, an uncertainty image, and a data quality image for each integration. The slope fitting procedure removes cosmic rays, bias drifts, and dark current.

Level 3

Processed Science Exposures – Count rate images are processed to remove instrumental signatures, extract spectra, and convert to physical units. Measurements from contemporaneous calibration exposures (e.g., science target locations in the reference image) are used when creating processed science exposures. Data from multiple integrations are combined.

Level 4

Combined Science Exposures – Associated science exposures (e.g., dithers) are combined.

DMS Data Flow

Recorded Science Data Unpacking

Science Header Creation

Recorded Science Data Collection

FOS

Calibration Metadata Creation

Up-The-Ramp Processing

Science Exposure Processing

Science Exposure Combination

PPS / DSFOS / EDW

Science Data Archive

Science Header Service

Science Data Pipeline

Pipeline Executive (and OTFR)

Backup Manager

Calibration Metadata Update

Calibration Reference Database

Calibration Data Processing

Science Header Update

Science Header Update

Calibration Metadata Update

Calibration Metadata Update

Level 3 Processed Science Exposures

Level 4 Combined Science Exposures

Level 2 Count Rate Images

Level 1 Raw Exposure Data

Level 0 Recorded Science Data

Science Header Update

OTFR 1

OTFR 2

OTFR 3

0

Science Archive Interface

Reference File Creation Reference

File Server

Science Data Ingest

Jeff’s Rule for DMS Design

Reusing the Hubble DMS design and tools will usually be easier and faster (and hence cheaper) than creating new tools.

Reasons to Modify HST DMS

JWST requirements not met by HST DMS designNew modes, e.g. automatically extract multi-object spectra

New data models, e.g. engineering data warehouse

More data, e.g. can we routinely distribute raw exposure data?

Component of HST DMS will become obsoleteRewrite synphot in Python because SPP is obsolete [in progress]

Reduce overall cost of operating JWSTAutomatic calibration data processing (e.g., dark processing)

Update science headers and calibration metadata via server

Improve value of data products and servicesSource lists, object classification, etc. [e.g., Hubble Legacy Archive]