overall architectureoverall architecture use of associationsuse of associations outline of...

•Overall ArchitectureOverall Architecture

•Use of AssociationsUse of Associations

•Outline of ProcessingOutline of Processing

•Reuse of CALACS and CALNIC CodeReuse of CALACS and CALNIC Code

•Processing Steps and KeywordsProcessing Steps and Keywords

•Reference FilesReference Files

WFC3 Pipeline CalibrationWFC3 Pipeline Calibration

February 13, 2001 CALWF3 2

WFC3 PIPELINE PDR

H. BushouseCALWF3 ArchitectureCALWF3 Architecture

• Code written in ANSI CCode written in ANSI C

• Use CVOS and HSTIO interfaces to IRAF/TABLES librariesUse CVOS and HSTIO interfaces to IRAF/TABLES libraries

• All run-time data files in FITS format with extensionsAll run-time data files in FITS format with extensions

• Exposures consist of multiple-extension IMSETsExposures consist of multiple-extension IMSETs

• Use associations to process and combine related exposuresUse associations to process and combine related exposures


WFC3 PIPELINE PDR

H. BushouseCALWF3 OutlineCALWF3 Outline

• Maximal reuse of ACS and NICMOS calibration softwareMaximal reuse of ACS and NICMOS calibration software

• CALWF3 based on CALACS flow modelCALWF3 based on CALACS flow model

• Includes single and multiple image operations within one shellIncludes single and multiple image operations within one shell

• UVIS and IR images treated as similarly as possibleUVIS and IR images treated as similarly as possible– UVIS identical to ACS WFCUVIS identical to ACS WFC

– IR similar to NICMOSIR similar to NICMOS

• Grism/Prism images receive 2-D processing only; spectral Grism/Prism images receive 2-D processing only; spectral extraction tools to be supplied by ST-ECF (as is the case for extraction tools to be supplied by ST-ECF (as is the case for NICMOS & ACS)NICMOS & ACS)


WFC3 PIPELINE PDR

H. BushouseBasic FunctionsBasic Functions

• Apply instrumental calibrations to individual imagesApply instrumental calibrations to individual images– Includes handling of binned and subarray imagesIncludes handling of binned and subarray images

• Perform various types of combining for associated imagesPerform various types of combining for associated images– Combine UVIS CR-Splits and IR MultiAccumsCombine UVIS CR-Splits and IR MultiAccums

– Sum Repeat-Obs for UVIS and IR imagesSum Repeat-Obs for UVIS and IR images

– Drizzle Dithered UVIS and IR imagesDrizzle Dithered UVIS and IR images

• Interface to Drizzle module currently in development by ACS Interface to Drizzle module currently in development by ACS team for CALACS; adoption of CALACS backbone will allow team for CALACS; adoption of CALACS backbone will allow for easy CALWF3 integrationfor easy CALWF3 integration

• Drizzle module can be used to perform geometric correctionsDrizzle module can be used to perform geometric corrections


WFC3 PIPELINE PDR

H. BushouseUse of AssociationsUse of Associations

• Associations will be used for:Associations will be used for:– CR-SplitsCR-Splits

– Repeat-ObsRepeat-Obs

– Dither patternsDither patterns

– All possible combinations of the aboveAll possible combinations of the above

• Associations will Associations will notnot be used for: be used for:– Mosaic patterns (logically-related exposures specified across >1 Mosaic patterns (logically-related exposures specified across >1

exposure logsheet line)exposure logsheet line)

• CALWF3 will use list of exposures in association table to CALWF3 will use list of exposures in association table to identify files that need to be processed as groups & subgroupsidentify files that need to be processed as groups & subgroups


WFC3 PIPELINE PDR

H. Bushouse

ACS2DDQI Nonlin Dark Flat

Phot

CALNICACALNICADQI Nonlin Dark Flat

Phot

CALACS Flow ModelCALACS Flow Model

Another Set of CR-SPLITS?Another Set of CR-SPLITS?

DetectorCCDCCD

MAMAMAMA

Another ImageAnother Imagein Association?in Association?

CRCORR

ACSREJCR Rejection

YesYes

ACS2DDark Flat Shad Phot

ACSCCDAtoD DQI Blev Bias

Another ImageAnother Imagein CR-SPLIT?in CR-SPLIT?

RPTCORR

ACSSUM

YesYes

ACSDTH(Drizzle)

YesYesDTHCORR

IRIR


WFC3 PIPELINE PDR

H. BushouseCALWF3 Flow DiagramCALWF3 Flow Diagram

Detector UVISUVIS

IRIR

WF3IRZoff DQI Blev Bias Nlin Dark Flat Phot

Another ImageAnother Imagein Association?in Association?

WF3CCDAtoD DQI Blev Bias

Another ImageAnother Imagein CR-SPLIT?in CR-SPLIT?

CRCORR

WF3REJCR Rejection

YesYes

WF3CCDDark Flat Shad Phot

Another Set of CR-SPLITS?Another Set of CR-SPLITS?

YesYesDTHCORR

RPTCORR

WF3SUM

YesYes

WF3DTH(Drizzle)

??


WFC3 PIPELINE PDR

H. BushouseCode ReuseCode Reuse

• CALACS backbone used for overall flow controlCALACS backbone used for overall flow control• CALACS module reuse:CALACS module reuse:

– acsccd: CALWF3 UVIS processingacsccd: CALWF3 UVIS processing– acsrej: CALWF3 UVIS CR-Split combinationacsrej: CALWF3 UVIS CR-Split combination– acssum: CALWF3 UVIS and IR Repeat-Obs combinationacssum: CALWF3 UVIS and IR Repeat-Obs combination– acsdth: CALWF3 UVIS and IR Dither combinationacsdth: CALWF3 UVIS and IR Dither combination

• CALNICA reuse:CALNICA reuse:– All applicable MultiAccum-mode calibration modulesAll applicable MultiAccum-mode calibration modules

• No CALNICB reuse:No CALNICB reuse:– All image combination done via CALACS and Drizzle methodsAll image combination done via CALACS and Drizzle methods

• Backwards compatibility with CALACS and CALNIC allows Backwards compatibility with CALACS and CALNIC allows for easy implementation of future developments that are of for easy implementation of future developments that are of value to WFC3 (e.g. NICMOS post-SAA persistence and value to WFC3 (e.g. NICMOS post-SAA persistence and pedestal corrections, ACS Drizzle development, etc.)pedestal corrections, ACS Drizzle development, etc.)


WFC3 PIPELINE PDR

H. BushouseRequired UpgradesRequired Upgrades

• UVIS binned imagesUVIS binned images– STIS: calstis rebins reference data on-the-fly to match science imagesSTIS: calstis rebins reference data on-the-fly to match science images

– ACS: no binning capabilities; reference data read line-by-lineACS: no binning capabilities; reference data read line-by-line

– WFC3: difficult to merge STIS on-the-fly rebinning with ACS line-by-WFC3: difficult to merge STIS on-the-fly rebinning with ACS line-by-line I/O; will be handled instead via binned reference dataline I/O; will be handled instead via binned reference data

• IR subarray imagesIR subarray images– necessary for short exposures of bright standardsnecessary for short exposures of bright standards

– not supported in CALNIC; will use algorithms from CALACS WFCnot supported in CALNIC; will use algorithms from CALACS WFC

• IR overscan subtractionIR overscan subtraction– used to track bias levels in each non-destructive readoutused to track bias levels in each non-destructive readout

– not supported in CALNIC; will use algorithms from CALACS WFCnot supported in CALNIC; will use algorithms from CALACS WFC

• Interface to DrizzleInterface to Drizzle– currently under development for CALACScurrently under development for CALACS


WFC3 PIPELINE PDR

H. BushouseCALWF3 UVIS Cal StepsCALWF3 UVIS Cal Steps

Input FilesInput Files Processing StepProcessing Step Keyword SwitchKeyword Switch Output FileOutput File

_asn or _raw_asn or _raw

CCDTABCCDTAB

Compute Statistical ErrorsCompute Statistical Errors(initialize ERR array)(initialize ERR array)

N/AN/A

BPIXTABBPIXTAB Assign Static DQ FlagsAssign Static DQ Flags(initialize DQ array)(initialize DQ array)

DQICORRDQICORR

ATODTABATODTAB A-to-D CorrectionA-to-D Correction ATODCORRATODCORR

OSCNTABOSCNTAB Overscan Bias Level SubtractionOverscan Bias Level Subtraction BLEVCORRBLEVCORR

BIASFILEBIASFILE Bias Image SubtractionBias Image Subtraction BIASCORRBIASCORR _blv_tmp_blv_tmp

_blv_tmp_blv_tmp

CRREJTABCRREJTAB

Cosmic-Ray RejectionCosmic-Ray Rejection CRCORRCRCORR _crj_tmp_crj_tmp


WFC3 PIPELINE PDR

H. BushouseCALWF3 UVIS Steps (cont.)CALWF3 UVIS Steps (cont.)


_crj_tmp_crj_tmp

DARKFILEDARKFILE

Dark Current SubtractionDark Current Subtraction DARKCORRDARKCORR

PFLTFILEPFLTFILE

LFLTFILELFLTFILE

Flatfield CorrectionFlatfield Correction FLATCORRFLATCORR

SHADFILESHADFILE Shutter Shading CorrectionShutter Shading Correction SHADCORRSHADCORR

PHOTTABPHOTTAB Compute Photometric ParametersCompute Photometric Parameters(populate PHOTxxxx keywords)(populate PHOTxxxx keywords)

PHOTCORRPHOTCORR

Compute Image StatisticsCompute Image Statistics STATFLAGSTATFLAG _flt or _crj_flt or _crj


WFC3 PIPELINE PDR

H. BushouseCALWF3 IR Cal StepsCALWF3 IR Cal Steps


_asn or _raw_asn or _raw Zeroth-read Signal CorrectionZeroth-read Signal Correction ZSIGCORRZSIGCORR

Zeroth-read SubtractionZeroth-read Subtraction ZOFFCORRZOFFCORR

BPIXTABBPIXTAB Assign Static DQ FlagsAssign Static DQ Flags(initialize DQ arrays)(initialize DQ arrays)

DQICORRDQICORR

OSCNTABOSCNTAB Overscan Bias Level SubtractionOverscan Bias Level Subtraction BLEVCORRBLEVCORR

Compute Statistical ErrorsCompute Statistical Errors(initialize ERR arrays)(initialize ERR arrays)

N/AN/A


WFC3 PIPELINE PDR

H. BushouseCALWF3 IR Cal Steps (cont.)CALWF3 IR Cal Steps (cont.)


DARKFILEDARKFILE Dark Current SubtractionDark Current Subtraction DARKCORRDARKCORR

NLINFILENLINFILE Non-Linearity CorrectionNon-Linearity Correction NLINCORRNLINCORR

PHOTTABPHOTTAB Compute Photometric ParametersCompute Photometric Parameters(populate PHOTxxxx keywords)(populate PHOTxxxx keywords)

PHOTCORRPHOTCORR

CRREJTABCRREJTAB Cosmic-Ray RejectionCosmic-Ray Rejection(also combines readouts)(also combines readouts)

CRCORRCRCORR

PFLTFILEPFLTFILELFLTFILELFLTFILE

Flatfield CorrectionFlatfield Correction FLATCORRFLATCORR

Compute Image StatisticsCompute Image Statistics STATFLAGSTATFLAG _ima and _crj_ima and _crj


WFC3 PIPELINE PDR

H. BushouseCALWF3 Image CombiningCALWF3 Image Combining


_flt or _crj_flt or _crj Sum Repeat-Obs ImagesSum Repeat-Obs Images RPTCORRRPTCORR _sfl_sfl

_flt or _crj or _sfl_flt or _crj or _sflIDCTABIDCTABDITHTABDITHTAB

Combine Dithered ImagesCombine Dithered Images11 DITHCORRDITHCORR _dth_dth

11DITHCORR currently a stub in CALACS; drizzle algorithm development is in DITHCORR currently a stub in CALACS; drizzle algorithm development is in progress by ACS & SSG teamsprogress by ACS & SSG teams


WFC3 PIPELINE PDR

H. BushouseCALWF3 Reference ImagesCALWF3 Reference Images

Image NameImage Name SuffixSuffix DescriptionDescription UVISUVIS IRIR Selection CriteriaSelection Criteria

BIASFILEBIASFILE biabia BiasBias ** DETECTOR GAINDETECTOR GAINBINAXISBINAXIS11

DARKFILEDARKFILE drkdrk Dark CurrentDark Current ** ** DETECTOR GAINDETECTOR GAINBINAXISBINAXIS11

SAMP_SEQSAMP_SEQ

NLINFILENLINFILE linlin Non-Linearity CoefficientsNon-Linearity Coefficients ** DETECTORDETECTOR

PFLTFILEPFLTFILE pflpfl Pixel-to-Pixel Flat FieldPixel-to-Pixel Flat Field ** ** DETECTOR FILTERDETECTOR FILTERBINAXISBINAXIS11

LFLTFILELFLTFILE lfllfl Low-order Flat FieldLow-order Flat Field ** ** DETECTOR FILTERDETECTOR FILTERBINAXISBINAXIS11

SHADFILESHADFILE shdshd Shutter Shading CorrectionShutter Shading Correction ** DETECTORDETECTORBINAXISBINAXIS11

11Separate reference images will be maintained in CDBS for each binning mode.Separate reference images will be maintained in CDBS for each binning mode.


WFC3 PIPELINE PDR

H. BushouseCALWF3 Reference TablesCALWF3 Reference Tables

Table NameTable Name SuffixSuffix DescriptionDescription UVISUVIS IRIR Selection Selection KeywordsKeywords

Selection Selection ColumnsColumns

Parameter ColumnsParameter Columns

ATODTABATODTAB a2da2d Analog-to-Digital Analog-to-Digital CorrectionCorrection

** AMPAMP

GAINGAINREF_KEYREF_KEY

REF_KEY_VALUEREF_KEY_VALUE

NELEMNELEM

ATODATOD

BPIXTABBPIXTAB bpxbpx Bad Pixel TableBad Pixel Table ** ** DETECTORDETECTOR PIX1PIX1

PIX2PIX2

LENGTHLENGTH

AXISAXIS

VALUEVALUE

CCDTABCCDTAB ccdccd CCD ParametersCCD Parameters ** ** DETECTORDETECTOR AMPAMP

CHIPCHIP

GAINGAIN

OFSTOFST

BINAXIS1BINAXIS1

BINAXIS2BINAXIS2

ATODGAINATODGAIN

BIASBIAS

READNSEREADNSE

SATURATESATURATE

AMPXAMPX

AMPYAMPY


WFC3 PIPELINE PDR

H. BushouseCALWF3 Ref Tables (cont.)CALWF3 Ref Tables (cont.)




CRREJTABCRREJTAB crrcrr Cosmic-Ray Cosmic-Ray RejectionRejection

ParametersParameters

** ** DETECTORDETECTOR CRSPLITCRSPLIT

MEANEXPMEANEXPSCALENSESCALENSE

INITGUESSINITGUESS

SKYSUBSKYSUB

CRSIGMASCRSIGMAS

CRRADIUSCRRADIUS

CRTHRESHCRTHRESH

BADINPDQBADINPDQ

CRMASKCRMASK

IDCTABIDCTAB idcidc Image DistortionImage Distortion

CoefficientsCoefficients

** ** DETECTORDETECTOR FUNCTYPEFUNCTYPE

NCOEFF1NCOEFF1

COEFF1[ncoeff1]COEFF1[ncoeff1]

NCOEFF2NCOEFF2

COEFF2[ncoeff2]COEFF2[ncoeff2]


WFC3 PIPELINE PDR

H. BushouseCALWF3 Ref Tables (cont.)CALWF3 Ref Tables (cont.)




OSCNTABOSCNTAB oscosc Overscan ParametersOverscan Parameters ** ** DETECTORDETECTOR AMPAMP

CHIPCHIP

BINXBINX

BINYBINY

NX NYNX NY

TRIMX[1,2]TRIMX[1,2]

TRIMY[1,2]TRIMY[1,2]

VX[1,2]VX[1,2]

VY[1,2]VY[1,2]

BIASSECTA[1,2]BIASSECTA[1,2]

BIASSECTB[1,2]BIASSECTB[1,2]

PHOTTABPHOTTAB phtpht Photometry Photometry ParametersParameters

** ** DETECTORDETECTOR FILTERFILTER NELEMNELEM

WAVELENGTHWAVELENGTH

THROUGHPUTTHROUGHPUT


WFC3 PIPELINE PDR

H. BushouseTest PlanTest Plan

• A comprehensive test plan will be created for UVIS and IR A comprehensive test plan will be created for UVIS and IR instrumental calibrations, as well as all image combination instrumental calibrations, as well as all image combination methods, to be executed and evaluated by WFC3 IPTmethods, to be executed and evaluated by WFC3 IPT

• Tests will include:Tests will include:– Correct selection and sequence of calibration steps depending on Correct selection and sequence of calibration steps depending on

science data inputscience data input

– Proper trapping and handling of common failure modes (e.g. Proper trapping and handling of common failure modes (e.g. missing, inappropriate reference data files, missing assocation missing, inappropriate reference data files, missing assocation members, missing or invalid keyword values, etc.)members, missing or invalid keyword values, etc.)

– Verification of intended scientific results from each operationVerification of intended scientific results from each operation

– Correct generation of trailer filesCorrect generation of trailer files

overall architectureoverall architecture use of associationsuse of associations outline of...

Documents