stsci pipeline thermal-vacuum data handling i. neill reid for wfc3 instrument calibration team
DESCRIPTION
ICAL. STScI Pipeline Thermal-vacuum data handling I. Neill Reid for WFC3 Instrument CALibration team. The WFC3 thermal-vacuum campaign will incorporate several innovations, including: Fully-planned science calibration campaign All images obtained during T/V will be archived in the HDA - PowerPoint PPT PresentationTRANSCRIPT
1
STScI PipelineThermal-vacuum data handling
I. Neill Reid for WFC3 Instrument CALibration team
ICAL
2
Introduction
The WFC3 thermal-vacuum campaign will incorporate several innovations, including:
• Fully-planned science calibration campaign
• All images obtained during T/V will be archived in the HDA
- science: calibration & performance verification
- technical: optical alignment
• Data characterising the configuration of the illuminating optical stimulus (OS) will be incorporated automatically in image headers
3
Background
• Thermal-vacuum tests scheduled for Feb-April, 2003 at GSFC
• Science calibration mid-April 2003 (17 days)
• Expect up to 10,000 images; ~2,000 science
• Data-handling system design has been considered by the WFC3 ICAL
Full system concept has been developed and impact on pipeline assessed
Details of other sub-systems still under consideration
4
Functional Requirements for T/V data system
• Data acquisition - Acquire SDI and SHP files from SITS and OS snapshots from the Stimulus; merge and convert into a uniquely-named FITS file.
• Data logging - Automatically log all calibration activities -records what SMSs are being run and what pictures are being generated
• OS monitoring - Store all OS configuration and telemetry data in ancillary files.• Quick-look and real-time access to local database - Images taken during T/V shall be
available for immediate display and real-time science assessment on site. • Data archiving – Data will be transferred to STScI for permanent storage and retrieval
in the HDA.• Exposure traceability - FITS files will be traceable back to the SMS scripts. • Archive retrieval - DADS will allow use of keywords to retrieve images from HDA• Automation - Provide automation to minimise data-handling errors and optimise use of
manpower.• Reuse - Legacy software will be used to minimize development costs• Compatibility – For efficiency, the same software should be compatible with both
UVIS and IR channels
5
System Concepts
Two main architectures considered, both utilising heritage software:
• ACS-like system– Local operating system and database designed specifically for T/V – Flexible, rapid access to data via custom-built cataloguing facilities– Limited capabilities for easily-accessible long-term archive
• OPUS-like system– STScI-based system linked to Hubble Data Archive– Designed for pre-specified programmatic data acquisition– Well-defined, easily-accessible long-term archive
Proposed system combines strengths of alternatives in a hybrid system
6
Recommended Data Handling System
CalibrationPlan TestProcedure
Quick LookReal TimeImage Displayand Analysis
Data LoggingAcquisitionSoftware
OpticalStimulus
SITS ExcelSpreadsheetSMS Tool
Real TimeManual CCLCommands
SMS Scripts
WFC3
Retrieval /Traceability
STSDASIRAF Tools
Real TimeLocal Database
Local ShortTermArchive
Science IPT /SOC RemoteImage DisplayAnalysis WorkStation
HST RemoteLong TermArchive
StimulusComputer
DataArchivingDistributionSystem(DADS)
OSConfigurationScripts
Science IPTWorkstation
OPUSPipelineWorkstation
SHP Files
SDI Files
OS
Snap
sho
t C
onfig
uration F
iles
Local Real Time Quick Look System - GSFC
OS Ancillary Files
Data Ingest / Retrieval
TelemetryCommands
Proposed WFC3 Thermal Vacuum Data Flow
OPUS System - STScI
OPUSStagingDisk
Time StampFiles
LocalStagingDisk
SMS
Unique Merged FITSFiles with OS Key Words
7
System characteristics
• Utilise existing capabilities and resources
• Modular design
- Science Calibration plan
- SITS
- Optical stimulus
- File transfer & time stamp
- Quick-look ACS-style Analysis System
- OPUS/pipeline system
This allows concurrent development and a phased integration schedule
• Minimise development cost and schedule impacts
• All calibration files will be generated from HDA images
8
Schedule
Detailed schedule for full system currently under review - outline:
• Science Calibration Plan – final version, Nov 30 2001
• Prototype systems for SITS/OS/Quick-look/Time-stamp modules will be developed for testing with Optical Stimulus, late-May to July 2002
• Remaining issues can be addressed post-prototype testing, but before December 2002
• Full system test in December 2002
9
Pipeline deliverables
1. File names 15 Oct 2001
unique identifier to associate SDI, OSC & log files and permit cross-referencing with quick-look data:
ipppssoot_csiiyyddhhmmss.extension2. Definition of unique T/V-specific keywords 31 Oct 20013. Incorporation of auxiliary information 31 May 2001 - OS configuration files - comment/log files4. Identify instrument set-up parameters for incorporation in science
header 31 May 2001 - supplement/replace parameters normally in PMDB5. T/V specific archive retrieval tools 1 April 2001