cos pipeline cdr jim rose july 23, 2001opus science data processing space telescope science...

COS PIPELINE CDRJim Rose

July 23, 2001 OPUS Science Data Processing

Space Telescope Science Institute

1 of 12

Science Data Processing

www.stsci.edu/software/OPUS/

www.dpt.stsci.edu/




2 of 12

OPUS Science Data Processing

PM D B

PD B

KeywordD atabase

D ataV alida tion

G enericC onversion

D ata C o llector

S upport S chedu le

W orld C oord ina teS ystem

KeywordExceptions

T able

KeywordR ules

ED T b inary dataset

Ÿ extract proposal and schedulingin form ation from PM D B

ED T b inary dataset + dgx

ED T b inary dataset, dgx + dvx

ED T b inary dataset, dgx, dvx + dcx

Ÿ dredge m nem onic va lues from te lem etry and apply convers ions if necessaryŸ com pare actua l vs. p lanned observation param eters

Ÿ convert po in ting in form ationinto s tandard F IT S W C S keywords

Ÿ convert data in to F IT S form atŸ properly orient im ages or spectraŸ populate keyword va lues

B est R ef/ S w itch

uncalibrated F IT S dataset

Ÿ populate ca libration reference file and sw itch keywords (a lso used byS tarV iew /web)


Ÿ hold ind ividual exposures for associa tion processing


ASSIST

C D BS

D ataP artition ing

Ÿ check for bad segm entsŸ insert fill data if te lem etry dropouts

pod file




3 of 12

OPUS Processes

Data PartitioningData Partitioning– segments the telemetry stream into standard EDT datasetsegments the telemetry stream into standard EDT dataset– appends multiple dumps into a single exposure fileappends multiple dumps into a single exposure file

> uses an index file to avoid sorting packetsuses an index file to avoid sorting packets> flags packet errors caught by PACORflags packet errors caught by PACOR

Support ScheduleSupport Schedule– gathers proposal information from PMDBgathers proposal information from PMDB– test proposals required for developmenttest proposals required for development

> test version of PMDB must be populated by TRANStest version of PMDB must be populated by TRANS> Thermal Vac support schedule required for full processingThermal Vac support schedule required for full processing




4 of 12

OPUS Processes (cont.) Data ValidationData Validation

– decodes the exposure and engineering parameters in the telemetry and decodes the exposure and engineering parameters in the telemetry and compares them to the planned valuescompares them to the planned values

– internal header specification (from Ball)internal header specification (from Ball)> PDB (EUDL.DAT, TDFD.DAT) must be fully populated and defined in PDB (EUDL.DAT, TDFD.DAT) must be fully populated and defined in

DM-06 [DONE]DM-06 [DONE]– Planned value comparisons (“flags and indicators”) need to be determined Planned value comparisons (“flags and indicators”) need to be determined

by the Instrument Scientistsby the Instrument Scientists World Coordinate SystemWorld Coordinate System

– implements a translation from telescope coordinates through the implements a translation from telescope coordinates through the instrument light-path to an astronomically valid pointinginstrument light-path to an astronomically valid pointing

– aperture positions must be definedaperture positions must be defined




5 of 12

OPUS Processes (cont.) Generic ConversionGeneric Conversion

– Generic Conversion outputs uncalibrated dataGeneric Conversion outputs uncalibrated data> rawtag (Time-Tag), rawimage, pha, asn, spt, trlrawtag (Time-Tag), rawimage, pha, asn, spt, trl

– data will be output in standard FITS format with image or table extensionsdata will be output in standard FITS format with image or table extensions– primary header will contain keywords inherited by all extensions and a primary header will contain keywords inherited by all extensions and a

null data arraynull data array– ExtensionsExtensions

> Time-Tag mode will be FITS binary tables (events + good times intervals)Time-Tag mode will be FITS binary tables (events + good times intervals)> ACCUM mode will be images grouped by imsets consisting of science ACCUM mode will be images grouped by imsets consisting of science

array, data quality array, and null error array.array, data quality array, and null error array.> FUV data will contain separate files for each detector (a and b)FUV data will contain separate files for each detector (a and b)> data quality array will be null if no telemetry dropoutsdata quality array will be null if no telemetry dropouts




6 of 12

OPUS Processes (cont.)

Generic Conversion (cont.)Generic Conversion (cont.)– Required for developmentRequired for development

> DM-06 to develop algorithms for data formatting DM-06 to develop algorithms for data formatting [DONE][DONE]

> keyword definitions (ICD-19) are currently being worked keyword definitions (ICD-19) are currently being worked by the Instrument Scientistsby the Instrument Scientists

• world coordinate definitionsworld coordinate definitions

• exposure time calculationsexposure time calculations

• calibration switches and selection criteriacalibration switches and selection criteria

• calibration file name keywordscalibration file name keywords




7 of 12

OPUS Processes (cont.) Data CollectorData Collector

– OPUS will ensure all component exposures are present before OPUS will ensure all component exposures are present before processing furtherprocessing further

> rules for processing incomplete associations have been defined rules for processing incomplete associations have been defined – association table contains information about the product association table contains information about the product

datasetdataset> dataset self-documentingdataset self-documenting

– only associations required for data processing will be only associations required for data processing will be constructed in the OPUS pipelineconstructed in the OPUS pipeline

> FP splitsFP splits> Repeat obsRepeat obs




8 of 12

Calibration

OPUS will use STSDAS calibration softwareOPUS will use STSDAS calibration software run on ALPHA / Tru64 UNIX platform in run on ALPHA / Tru64 UNIX platform in

operations operations expands size of datasetexpands size of dataset

– converts integer raw data to realconverts integer raw data to real– possible expansion of data quality array for ACCUM possible expansion of data quality array for ACCUM

modemode– expansion of the error array.expansion of the error array.

Need calibration reference files for testing (at least Need calibration reference files for testing (at least dummies)dummies)




9 of 12

Other Science Data Modes

requirements for data content of each of these other requirements for data content of each of these other science data modes must be defined by Instrument science data modes must be defined by Instrument ScientistsScientists– target acquisitionstarget acquisitions– microprocessor memory dumpmicroprocessor memory dump– engineering diagnostic dataengineering diagnostic data– cumulative detector imagecumulative detector image

> MAMA cumulative image like STIS for NUVMAMA cumulative image like STIS for NUV> cumulative image for FUV detectorcumulative image for FUV detector




10 of 12

CODE REUSE Core OPUS system (OPUS 12.1)Core OPUS system (OPUS 12.1)

– ~236,000 lines of code~236,000 lines of code– 100% reuse100% reuse

COS specific processesCOS specific processes– Based on FUSE study (Rose et al. 1998, “OPUS: The FUSE Data Based on FUSE study (Rose et al. 1998, “OPUS: The FUSE Data

Pipeline”, Pipeline”, www.www.stscistsci..eduedu/software/OPUS/kona2.html/software/OPUS/kona2.html))> 5076 lines of code5076 lines of code> 71% reuse of existing OPUS modules71% reuse of existing OPUS modules

Expect > 99% reuse of existing data processing software for Expect > 99% reuse of existing data processing software for COS, based on lines of code.COS, based on lines of code.

– All SI complexity contained in relatively few lines of code.All SI complexity contained in relatively few lines of code. Efficient use of existing system!Efficient use of existing system!




11 of 12

Major Science Data Processing Requirements Summary Internal header specificationInternal header specification

– DM-06DM-06 to document content and format of science internal to document content and format of science internal headerheader

– PDB (EUDL.DAT, TDFD.DAT) defined and fully populatedPDB (EUDL.DAT, TDFD.DAT) defined and fully populated

Keyword definitions Keyword definitions Planned values “flags and indicators” for Data Planned values “flags and indicators” for Data

ValidationValidation Aperture definitionsAperture definitions




12 of 12

Test Data Requirements

Test data from detectors on ops bench expected in November Test data from detectors on ops bench expected in November 2001 and from integrated instrument in March 20022001 and from integrated instrument in March 2002

Test data to be provided by IDT/Instrument Scientists and Test data to be provided by IDT/Instrument Scientists and Engineers should include all science and engineering modesEngineers should include all science and engineering modes

Test data must includeTest data must include– PMDB population and PDB definitionPMDB population and PDB definition– spectra that enable OPUS to determine proper orientationspectra that enable OPUS to determine proper orientation– list of possible error conditions to simulate list of possible error conditions to simulate – data that simulate error conditionsdata that simulate error conditions– enough data for throughput testenough data for throughput test

cos pipeline cdr jim rose july 23, 2001opus science data processing space telescope science...

Documents