russ laher (1) s13.0 test readiness review, august 29, 2005 downlink segment test status (s13.0)...

Russ Laher (1)S13.0 Test Readiness Review, August 29, 2005

Downlink Segment Test Status (S13.0)

Russ Laher

August 29, 2005


Summary of S13 D/L work performed

D/L-group software changes implemented for S13.0 99 pipeline-related CRs/AIs (105 for S12.0) 38 database-related CRs/AIs (67 for S12.0) 22 ARs (31 for S12.0)

D/L segment-testing to check specific CR/AR/AI changes Small cluster of 16 pipeline-drones (mini “ops-like” environment) SODB (fresh S12 database via EdJackson) Product archiver run on pipeline results for end-to-end verification

ISTs doing science verification of pipeline products and testing of SPICE QA team verification of QA sub-system upgrades Test plan documents written by CogEs to help I&T Code walk-throughs and training sessions conducted on a weekly basis over

the last four months to improve software quality and help I&T Weekly downlink meetings with developers and IST Cog. E.s Many “Advanced Data Products” development team meetings with developers

and customers Weekly “Archive Working Group” meetings with developers, IRSA group, SSC

deputy manager (L. Storrie-Lombardi), and archive scientist (M. Lacy)


What’s New for S13.0 IRS

New IRS pipeline to produce background-subtracted images and spectra ((plScriptId 2023) New IRS pipelines to generate mosaics and point-source extractions for peakup images (plScriptIds 2340

and 2348 for FIF and MOPEX pipelines) Upgrades to SPICE (various improvements; new spectral rectification

IRAC New IRAC imask product (SOSDL-SIS-PD-3030) IRAC delay-time calculation extended across AOR boundary Superboresight creation, archival and _usage_ in pointing transfer to ensure all IRAC channels get best

refined pointing in future campaign reprocessings (based on optimal reprocessing of channel 1 and 2 data) New database table (QA_ptg_refine), schema, store procedures to store IRAC pointing-refinement

information on a BCD basis MIPS:Si

All P-masked pixels will be explicitly NaN'd in final MIPS-24 products All legitimate 0-valued pixels in the difference plane of MIPS-24 SUR-mode DCEs will be set to NaN and

propagated to the final BCD For all MIPS channels/modes: use calibrated CSM_PRED versus Y_spot position to correct for pointing

due to unpredictable scan-mirror wobble All MIPS BCD products will have background model estimate keywords: ZODY_EST, ISM_EST, CIB_EST Significant enhancements to MIPS-24 offline pipelines. e.g., user-friendly tutorials and the inclusion of

many utility scripts For the very first time, a document outlining Frequently Asked Questions on MIPS-24 processing. To be

used by ISTs, I&T, CogEs and the PO. See: http://spider.ipac.caltech.edu/staff/fmasci/home/sirtfwork/MIPS24_FAQs.pdf


What’s New for S13.0 – cont.

MIPS:Ge Upgrades to GeRT (offline Germanium Reprocessing Tool) Fixed source bug preventing clean compile on Linux and Solaris with GCC Modification to treat Infinity as NaN in calibration files Modification so that NaN's get written to calibration files, not Infinity Modification to allow namelist to interchange order of Column and Hipass filters WARMFLAG included in as filtering criterion in caltrans calls Separate WARMFLAG namelist blocks added

Infrastructure Changes for “Warm” Spitzer Observation Campaigns New warmFlag field in campaigns, MipsGe Fallback/Metadata database tables and supporting

software Upgrade to C-API function and database-stored function to read warmFlag from the database Modification to job.c to write WARMFLAG to fos16.fits FITS header

Post-BCD Software New offline script to create MIPS SED mosaics Bandmerge upgrade Porting of MOPEX and bandmerge modules to Linux and Mac OsX MOPEX GUI development for Solaris Improved user documentation for mosaicker Various post-BCD-pipeline bug fixes (e.g., increased CRVAL precision by one digit)

Generic Software Robust scheme in pointing-transfer to guard against outliers in input Boresight Pointing History

File (BPHF; all instruments) New adaptive-smoothing tool


What’s New for S13.0 – cont.

On-the-fly reprocessing (automated pipeline running and product archival) Database schema, stored procedures, triggers, and perl API functions Webform for SSC-internal job submission “Lights out” perl script

IRSA Connectivity Populating new database tables for storing IRSA metadata for BCD and post-BCD

products Creation of JPEG image previews for all primary BCD and post-BCD products and

compressed/lossy FITS files for mosaics (all imaging instruments), and GIF spectral-plot previews for IRS, in order to support the user quick-look interface in Leopard

QA sub-system upgrades SDAT upgraded to run under proxy apache server New showQaPtgXfer.cgi CGI script to query QA_PTG_XFER database table and

display results Populating new QA_IRS_REFFRM and QA_IRS_RHFIND database tables New fields being populated in QA_PTG_XFER database table Improvements in SDAT image display Miscellaneous minor improvements


Tests Under S13.0

IRS All Sci (BCD&BQD), DarkCal, DarkEns, Flatcal and peak-up-imaging pipelines; used fallback

cal; all data run through archiver Details in IRS S13 Test Plan Document

MIPS-Ge Sci (scanmap, photometry, sed, tpm), darks, flats, warm AOR, fine photometry; used fallback

cal; all data run through archiver Details in MIPS-Ge S13 Test Plan Document

MIPS‑24 SUR-mode sci scanmap (slow, med, fast cases); SUR- and RAW-mode photometry; SUR- and

RAW-mode TPM; flat-field cal w/ pointing transfer, non-linearity cal w/ pointing transfer; SUR & RAW dark cal w/ pointing transfer ; used fallback cal; all data run through archiver

Details in MIPS-24 S13 Test Plan Document IRAC

sub-, full-array science and cal pipelines for all exposure modes, frame times, and combinations of internal and external repeats; product records and file system contents checked for data run thru each pipeline; used metadata and fallback cal; all data run through archiver

Details in IRAC S13 Test Plan Document Post-BCD

FIFcreation; pointing refinement;single_frame_pnt_src_ext;multi-frame pnt-src_ext; mosaic pnt-src_ext; overlap correction; image_interp, outlier; image_reinterp; mosaic_coadd for IRAC, MIPS24, 70 &160; all data run through archiver

Details in PostBCD S13 Test Plan Document QA, SDAT

Loader run for all tested pipelines, SDAT tool run, qa pages and statistics pages brought up Details in QA/Infrastructure S13 Test Plan Document


Tests Under S13.0

Testing of the “lights out” script for on-the-fly reprocessing for many reqKeys Testing of SDM’s new channelized product archiving many reqKeys


S13.0 Functions/Test Data Flows


S13.0 Capabilities

Sub-system Capability

Ingest Supports DCEs, Pntng, ECSV, Rpt files, Mem Dmps. Sped up, SigEvents, FEIdelete, ReRun, Archiver naming, HK compression

Pipeline Exec real-time data reduction; reprocessing

Instrument pipelines All three instruments science and calibration

Calibration Transfer Fallback rules (incl NIT) for all three instruments, three nearest in time rules for all three instruments, blessed status flag, asymmetric discriminant selection, interpolation rule

Instrument Performance Monitoring Create/Populate HK tables in the HK DB; web-based viewing and data retrieval; updated menus

Quality Assessment Creation and populating of QA tables; SDAT web-based QA inspection; analysis and display functions; support archive release

CDF Populating of version controlled CDF files on the pipeline drones; retrieve CDF in pipelines, version tracking in the SODB

Imaging post-BCD pipeline Point source extraction, pointing refinement (w/ & w/out 2MASS, USNO, UCAC, FIRST), tiling, overlap correction, outlier identification and mosaic coadding, bandmerging


S13.0 Success Criteria

Run, re-run requests that failed on OPS Interact with the SODB via perl & c-APIs Deposit raw data into the archive file system Create pipeline manifests & ensembles Run manifests through APES Run multiple jobbers on 16 pipeline drones Use CDF to get configuration files Use CalTransfer to get calibration data (NIT and/or fallbacks) Execute post-BCD pipelines including pointing refinemnet Put QA statistics into QA DB tables Put processing results in sandbox and set file pointers in DB Use SDAT to access and view results on segment test Get “better” BCDs & post-BCD prods IST sci verification of seg test data & buy off (incremental from IOC) Pass the running of the (channelized) archiver


Calibration SOSDL-SIS-CL-3000 : darkcal output FITS file, dark image SOSDL-SIS-CL-3001 : lincal output FITS file, linearity model cube SOSDL-SIS-CL-3002 : lincal output table file, pixel cal data SOSDL-SIS-CL-3003 : lincal output cmask image SOSDL-SIS-CL-3004 : darkcal output cmask image SOSDL-SIS-CL-3005 : pmask definitions SOSDL-SIS-CL-3006 : Electronic Nonlinearity Calibration File SOSDL-SIS-CL-3007 : MIPS-Ge stim_latent SOSDL-SIS-CL-3008 : function loadcalkeywords SOSDL-SIS-CL-3009 : IRAC-loadcalkeywords version-1 SODB Table SOSDL-SIS-CL-3010 : darkest cmask SOSDL-SIS-CL-3011 : CalTrans Rule List SOSDL-SIS-CL-3012 : IRS Umask SOSDL-SIS-CL-3013 : IRS Lmask SOSDL-SIS-CL-3014 : IRAC flatcal data cube SOSDL-SIS-CL-3015 : IRAC flatcal cmask SOSDL-SIS-CL-3016 : IRS flatcal cmask SOSDL-SIS-CL-3017 : IRS Omask SOSDL-SIS-CL-3018 : IRS Flux Conversion Table SOSDL-SIS-CL-3019 : IRS FOV PSF Table SOSDL-SIS-CL-3020 : IRS COADD2D AMASK SOSDL-SIS-CL-3021 : IRS COADD2D FMASK SOSDL-SIS-CL-3022 : IRS WAVSAMP TABLE SOSDL-SIS-CL-3023 : MIPS-Ge cmask

Data Management SOSDL-SIS-DM-3000 :

Data Tools SOSDL-SIS-DT-3000 : SOSDL-SIS-DT-3004 : profile.tbl (IRS BQD) SOSDL-SIS-DT-3005 : ridge.tbl (IRS BQD) SOSDL-SIS-DT-3006 : extract.tbl (IRS BQD) SOSDL-SIS-DT-3007 : irs_tune.tbl (IRS BQD)

Executive SOSDL-SIS-EX-3000 : plinexyz.cdf SOSDL-SIS-EX-3001 : run time environment variables

Ingest SOSDL-SIS-IN-3000 :

Instrument Performance Monitoring SOSDL-SIS-IP-3000 :

Applicable SISs

Products SOSDL-SIS-PD-3000 : cvti2r4 output FITS file, 32-bit real data SOSDL-SIS-PD-3001 : cvti2r4 output FITS file, dmask SOSDL-SIS-PD-3002 : muxbleedcorr output DCE FITS file SOSDL-SIS-PD-3003 : imfliprot output FITS file SOSDL-SIS-PD-3004 : MIPS Ge bmask SOSDL-SIS-PD-3005 : imagest output bmask SOSDL-SIS-PD-3006 : tranhead keyword-translate-only list SOSDL-SIS-PD-3007 : Tile/BCD association files SOSDL-SIS-PD-3008 : Mosaic-geometry interface SOSDL-SIS-PD-3009 : BCD delta file SOSDL-SIS-PD-3010 : PREDICTSAT s-mask SOSDL-SIS-PD-3011 : dldeltafile library SOSDL-SIS-PD-3012 : PRF/PSF map SOSDL-SIS-PD-3013 : Point source extraction table SOSDL-SIS-PD-3014 : Detection/Tile/BCD Association File SOSDL-SIS-PD-3015 : imagest output slope image SOSDL-SIS-PD-3016 : latimflag l-mask SOSDL-SIS-PD-3017 : fiducial frame definition parameters SOSDL-SIS-PD-3018 : mosaic_interpolation table SOSDL-SIS-PD-3019 : matched-point-source refined position table SOSDL-SIS-PD-3020 : fiducial frame input parameters SOSDL-SIS-PD-3021 : bandmerged point source output file SOSDL-SIS-PD-3022 : bandmerger pointer dump file SOSDL-SIS-PD-3023 : band-pair registration uncertainties SOSDL-SIS-PD-3024 : bandmerge statistics SOSDL-SIS-PD-3025 : MIPS24 bmask SOSDL-SIS-PD-3026 : FPGen Command Line Input SOSDL-SIS-PD-3027 : FPGen SODB Entries file (input) SOSDL-SIS-PD-3028 : FPGen Template Input file SOSDL-SIS-PD-3029 : Bandmerge (QA) spectral combination counts SOSDL-SIS-PD-3030 : IRAC imask

Pointing Transfer SOSDL-SIS-PT-3000 : pixel-scale table file SOSDL-SIS-PT-3001 : Ptg Refinement Absolute Source List Input Fmt SOSDL-SIS-PT-3002 : MIPS Scan Mirror Parameters Table SOSDL-SIS-PT-3003 : Instrument FOV Table SOSDL-SIS-PT-3004 : Super Boresight Pointing History File SOSDL-SIS-PT-3005 : Refined DCE Position Table File SOSDL-SIS-PT-3006 : Aperture-ID-to-name translation CDF table

Quality Assessment SOSDL-SIS-QA-3000 : QA file (*.qa) format

New

Updated

Updated

New


Pipeline-Related CRs/AIs (99)

6576 downlink Hesselroth_Ted In libtblio build, skip soft link step... 6578 downlink Hesselroth_Ted In libmathc90 build, skip soft link step... 6845 downlink Laher_Russ ParchiverForDnl bug fix 6844 downlink Laher_Russ Update to D/L segment-test build script 6545 downlink Hesselroth_Ted Tblio null filepointer bug 6595 downlink Fowler_John Have cvti2r4 return an exit code of "1" for MIPS Ge stims 6565 downlink Hesselroth_Ted Move shared library suffix definitions to platform-dependent makefiles 6608 downlink-analysis Pesenson_Meyer Release adaptive smoothing tool 6254 downlink-irac Lee_Wen-Piao Creation of imask, final product mask file 6246 downlink-irac Lee_Wen-Piao Skydark Delaytimes within the Header 6604 downlink-irac Khan_Iffat Change needed in CRVAL precision 6697 downlink-irs Narron_Bob Update BCD extraction wrapper: w_irs_bqd_process2.pl 6403 downlink-irs Makovoz_David New IRS Post-BCD Peakup Imaging pipeline 6625 downlink-irs Narron_Bob Coadd2d to pass bits from BMASK to AMASK 6655 downlink-irs Narron_Bob Do not perform a simple average of RA_SLT, DEC_SLT and PA_SLT 6533 downlink-irs Narron_Bob IRS background subtracted images and spectra 6698 downlink-irs Narron_Bob Update w_irs_bqd_process.pl to include call to "LoadIrsaPtgMetaData". 6529 downlink-irs Narron_Bob Add header card for cal_set ID 6715 downlink-mips-24 Masci_Frank Ensure that all p-masked pixels are NaN'd in final mips24 BCD products. 6839 downlink-mips-24 Makovoz_David Do source extraction for MIPS-24 on mosaic, not individual BCDs 6416 downlink-mips-24 Masci_Frank Make 0-valued pixels in raw difference frames NaN's for bcd pipeline 6675 downlink-mips-ge Henderson_David Add warmflag keyword to caltrans calls 6594 downlink-mips-ge Henderson_David Allow changing the order of MIPS-70 filtering 6848 downlink-mips-ge Henderson_David Slopecal hipass filter does not always produce reliable results 6674 downlink-mips-ge Laher_Russ Write new WARMFLAG FITS-keyword to fos16.fits 6822 downlink-pl-cal Wheelock_Sherry Upgrade MIPS calibration-observation database 6693 downlink-pl-cal Henderson_David Add "WARMFLAG" to calkeywords input-keyword-list for MIPS:Ge 6692 downlink-pl-cal Lee_Wen-Piao Mod. to calkeywords library for new "warmFlag" field 6805 downlink-pl-cal Lee_Wen-Piao Caltrans.h modification


Pipeline-Related CRs/AIs (99) – cont.

6571 downlink-pl-cal Fang_Fan Wavsamp module output signed offset files 6572 downlink-pl-cal Fang_Fan Wavsamp module pixel sizes need to be changeable 6765 downlink-pl-exec Laher_Russ APES static-file changes for new IRS "reffr" and "mopex" threads 6642 downlink-pl-exec Makovoz_David Rename fif.tbl to mosaic_fif.tbl in mosaic.pl, mosaic_mask.pl, ... 6789 downlink-pl-exec Laher_Russ APES static-table changes for new IRS plScriptId=2023 6680 downlink-pl-exec Laher_Russ ReqMode=111 not recognized by createEnsembles.pl 6682 downlink-pl-exec Makovoz_David New offline script to create MIPS SED mosaics 6723 downlink-pl-exec Makovoz_David Update w_bqd_run_overlap.pl not to create empty offsets tables 6725 downlink-pl-exec Makovoz_David Modify w_bqd_run_fiducial_frame.pl to accomodate IRS peak-up imaging 6726 downlink-pl-exec Makovoz_David Modify createFifTable.pl to accomodate IRS peak-up imaging 6727 downlink-pl-exec Makovoz_David Modify getFifTableFilenameAndStatus in WrapperUtils.pl to include IRS 6756 downlink-pl-exec Laher_Russ "Lights out" script for on-the-fly reprocessing 6764 downlink-pl-exec Laher_Russ APES static-file changes for new Onthefly job-manifests 6599 downlink-pl-ing Masci_Frank Add "refinedPointing" fileType to subroutine "getFileTypeAbbreviation" 6741 downlink-pl-prod Masci_Frank Code walk-through fixes/updates to "LoadIrsaPtgMetaData" perl subroutine. 6670 downlink-pl-prod Laher_Russ New productTypes table entries for preview images for IRSA connectivity 6777 downlink-pl-prod Masci_Frank Update w_bqd_run_mopex.pl for IRSA previews and metadata-generation 6640 downlink-pl-prod Fowler_John Improve imagest data-reduction logic and chi-square computation 2597 downlink-pl-prod Henderson_David Add photometric nonlinearity module to MIPS:Ge pls 1891 downlink-pl-prod Lee_Wen-Piao Module needed to flag "banding" effect in IRAC channels 3 & 4 6621 downlink-pl-prod Makovoz_David Add call to new subroutine "addQA_ptg_refine" in w_bqd_pointrefine.pl 6525 downlink-pl-prod Khan_Iffat New module to replace data in a FITS file by a given value 6632 downlink-pl-prod Masci_Frank Add subroutines "LoadIrsaPtgMetaDataTable()", ... 6549 downlink-pl-prod Hesselroth_Ted String literal treated as character array in nl_and_params.c 6550 downlink-pl-prod Khan_Iffat New module to combine coverate weighted images 6553 downlink-pl-prod Hesselroth_Ted Multiply defined global variables in irs modules 6630 downlink-pl-prod Masci_Frank New libraries/modules to create new downlink products to support IRSA 6555 downlink-pl-prod Masci_Frank Register some pointing-refinement (IRAC) keywords into the DB...



6620 downlink-pl-prod Laher_Russ Please add new subroutine "addQA_ptg_refine" to WrapperUtils.pl 6584 downlink-pl-prod Narron_Bob Libim should optionally not to convert all non-finite numbers to NaNs 6840 downlink-pl-prod Khan_Iffat New mode in mosaic_outlier to create minimum value mosaic 6593 downlink-pl-prod Masci_Frank Write background model estimate keywords: ZODY_EST, ISM_EST, CIB_EST 6681 downlink-pl-prod Khan_Iffat New module to compute FIF for MIPS70 SED mosaic 6384 downlink-pl-prod Lee_Wen-Piao Remaining code walk-through action items for latimflag module... 6679 downlink-pl-prod Masci_Frank Update w_pointinghistory.pl to add calls to "LoadIrsaPtgMetaData" 6427 downlink-pl-prod Hesselroth_Ted Tblio/dlimage duplicate symbol errors 6694 downlink-pl-prod Makovoz_David Update w_bqd_run_mopex.pl to include call to LoadIrsaPtgMetaData. 6728 downlink-pl-prod Hesselroth_Ted Port of mopex to OS X 6612 downlink-pl-prod Khan_Iffat New module to compute the covariance matrix in a set of images 6438 downlink-pl-prod Masci_Frank Use calibrated CSM_PRED to correct for MIPS pointing 6733 downlink-pl-prod Masci_Frank Update cvtR4i2 module to report MIN, MAX pixel values to stdout. 6468 downlink-pl-prod Fowler_John Upgrade/bug fixes to bandmerge 6734 downlink-pl-prod Masci_Frank New library and modules to create of lossy FITS images for U/L. 6735 downlink-pl-prod Masci_Frank New module to robustly compute image noise... 6737 downlink-pl-prod Fowler_John Linux and OS X ports of bandmerge 6740 downlink-pl-prod Khan_Iffat Bug in tblio related to delete_rows processing 6818 downlink-pl-ptg Masci_Frank Update pointing transfer thread to write ORIGINAL pointing keywords 6607 downlink-pl-ptg Masci_Frank Update getPH_online for "refinedPointing" fileType (SBPHF) queries. 6437 downlink-pl-ptg Masci_Frank Come up with scheme to archive and register superboresight PH files.. 6554 downlink-pl-ptg Masci_Frank Implement robust scheme in pointing transfer to guard against outliers 6710 downlink-pl-ptg Masci_Frank Update GenSBPHFsForCampaign.pl following comments from code walk-thru.. 6600 downlink-pl-ptg Masci_Frank Update GenSBPHFsForCampaign.pl to optionally ingest SBPH files 6100 downlink-pl-ptg Masci_Frank Update pointing transfer to optionally use new SBPH files 6517 downlink-pl-qa Hesselroth_Ted Allow "Frame" window to be able to close in SDAT 6752 downlink-pl-qa Hesselroth_Ted Improvements for SDAT image display 6781 downlink-pl-qa Jacobson_Jeff Add qa_irs_rhfind table for qaloader 5706 downlink-pl-qa Laher_Russ Write SDAT.log to $SOS_LOGS/cgiout directory



6564 downlink-pl-qa Laher_Russ Dqa_deploy.csh: check to see if SOS_LOGS is defined 6774 downlink-pl-qa Narron_Bob Qaloader change for the new QA_irs_reffrm database table 6769 downlink-pl-qa Jacobson_Jeff New qa_irs_reffrm database table 6799 downlink-pl-qa Masci_Frank Make association of refined ptg tables with BPHFs more robust 6751 downlink-pl-qa Wheelock_Sherry Allow SDAT to be displayed in the proxy server environemnt 6783 downlink-pl-qa Narron_Bob Qaloader change for the new QA_irs_rhfind database table 6808 downlink-pl-qa Laher_Russ New SODB_Utils.pl function SODB_Get_QaPtgXfer 6736 downlink-pl-qa Laher_Russ QA scripts to dynamically read PERL_PATH from environment 6479 downlink-pl-qa Wheelock_Sherry Display some fields in table qa_ptg_xfer 6775 downlink-pl-qa Jacobson_Jeff New database stored procedure addQA_irs_reffrm 6129 sos Laher_Russ Change QA primary proxy URL 1577 sys-engring-i+t Bennett_Lee Super-Boresight Test Plan and operational instructions for S13.0. 1732 sys-engring-i+t Yu_Felix MIPS-24 S13.0 Test Plan..


Database-Related CRs/AIs (38)

6660 archive-database Jacobson_Jeff Update SPs to support the archive by channel 6701 sdm Jacobson_Jeff Add softwareVersion column in processingVersion table 6339 sdm-archive Jacobson_Jeff Be able to archive on an channel basis 6763 sdm-database Jacobson_Jeff New plScripts DB records for IRS "reffr" and "mopex" threads 6696 sdm-database Jacobson_Jeff Add warmFlag to C-API-function sdm_getDceRecForDP (sdm_sodb.ec) 6677 sdm-database Jacobson_Jeff New "warmFlag" boolean field in DB campaigns table 6782 sdm-database Jacobson_Jeff New database stored procedure addQA_irs_rhfind 6784 sdm-database Jacobson_Jeff Mod. to database procedure deleteNonArchivedProducts 6785 sdm-database Jacobson_Jeff New IRS background-subtraction pipeline (plScriptId=2023) 6787 sdm-database Jacobson_Jeff Database SP to compute new cross-AOR delaytimes from r_sclk 6795 sdm-database Jacobson_Jeff Perl API function for createPvIdsForOnTheFlyReprocessing 6798 sdm-database Jacobson_Jeff New database stored function getQaInfoForRequest 6402 sdm-database Laher_Russ New DB stored function for calculating delay time 6800 sdm-database Jacobson_Jeff Perl API for computeIracDelayTimeBetweenSuccessiveRequests 6434 sdm-database Laher_Russ New perl API-function computeIracDelayTime 6802 sdm-database Jacobson_Jeff New and modified stored procedures to support OTF request archiving 6807 sdm-database Jacobson_Jeff New DB stored function getQaPtgXferDataForRequest() 6809 sdm-database Laher_Russ New perl API function getQaPtgXferDataForRequest() 6813 sdm-database Jacobson_Jeff Need schema and perl API update for new columns in QA_ptg_xfer table. 6816 sdm-database Jacobson_Jeff Need stored function to query QA_ptg_xfer table for input dceId. 6678 sdm-database Jacobson_Jeff New "warmFlag" boolean field in MIPS-Ge cal. DB tables 6631 sdm-database Jacobson_Jeff Update campaign stored procedures for new "warm" MIPS campaign. 6743 sdm-database Laher_Russ Perl APIs for on-the-fly reprocessing 6615 sdm-database Jacobson_Jeff New "QA_ptg_refine" table requested by IRAC IST.. 6712 sdm-database Jacobson_Jeff New records to be deleted in DB SP deleteNonBestProducts 6747 sdm-database Jacobson_Jeff New database SP createPvIdsForOnTheFlyReprocessing() 6749 sdm-database Jacobson_Jeff New database tables for on-the-fly reprocessing 6753 sdm-database Jacobson_Jeff New database fields for on-the-fly reprocessing


Database-Related CRs/AIs (38) – cont.

6754 sdm-database Jacobson_Jeff New database stored procedures for on-the-fly reprocessing 6695 sdm-database Jacobson_Jeff Return warmFlag in DB stored function getDceProfile 6627 sdm-database Jacobson_Jeff Schemas, sql loading stored procedures and perl-API for IRSA metadata 1719 downlink Laher_Russ New productTypes entries for RSA-related products 1720 downlink Bennett_Lee Change exposure type for several MIPS fine scale flat observations 1695 sdm Jackson_Ed New ensemble-creation rules for new IRS "reffr" and "mopex" threads 1716 sdm Jackson_Ed New IRS ensemble-creation ruleId=8 for plScriptId=2023 1667 sdm Bennett_Lee Data-content change in ProductTypes database table 1664 sdm Bennett_Lee New data content for ProductTypes database table 1727 sos Laher_Russ Update release level in product types table for 7157


ARs (22)

6575 downlink Hesselroth_Ted In libim build, skip soft link step ... 6583 downlink Fowler_John Pick does not display Infinity values ... 6577 downlink Hesselroth_Ted In libintegrate build, skip soft link step (conditional) 6773 downlink-irac Lee_Wen-Piao Darkdrift module change 6748 downlink-irac Lee_Wen-Piao INSBPOSDOM writes no processing history to header 6833 downlink-irs Narron_Bob RIDGE not reading FOV file 6713 downlink-irs Narron_Bob EXTRACT module: improper dependency of table output on input header 6758 downlink-mips-ge Henderson_David Slopecal mistreats calibration files with NaN values 6122 downlink-mips-ge Henderson_David Slopecal module generates fitsio errors under certain conditions 6431 downlink-mips-ge Henderson_David Ge_STIMEXTRAP bmask bit 6705 downlink-mips-ge Henderson_David Typo in exptype to cdf block variant lookup for exptype="dfl" 6503 downlink-mips-ge Henderson_David Hardcode interp module to use geb_STIMEXTRAP 6704 downlink-mips-ge Pesenson_Meyer Mipsflat needs command line option for coverage map file name 6501 downlink-mips-ge Henderson_David Mips_ge_mask.h 6820 downlink-mips-ge Henderson_David MIPS-160 exptype=FINE doesn't exist 6827 downlink-mips-ge Henderson_David Downlink/include/nanvalue.h does not produce linkable code ... 6582 downlink-mips-ge Henderson_David NaN's in a data file turn into "Infinity" when read in by fitsio 6563 downlink-pl-prod Fowler_John The pick utility has a display error for unsigned integer... 6838 downlink-pl-prod Makovoz_David Apex_1frame.pl can not handle images > 10000 pixels in one dim 6750 downlink-pl-qa Hesselroth_Ted SDAT can not display MIPS channel 3 DCE 6770 sdm-database Jacobson_Jeff More robust getCalProducts... database stored procedures 6755 sdm-pr-archiver Jacobson_Jeff GetCalProductsForRequest need to return int8 type for file size


Major issues to be tackled in S14

Recent bug found in post-BCD position-refinement pipeline: the QA_IRAC_REFINE database table is not being populated!

Bandmerge pipeline wrapper upgrade so it can be executed in routine operations

Upgrades to MOPEX GUI Porting MOPEX GUI to Linux and Mac OsX Bandmerge GUI development Porting bandmerge GUI to Linux and Mac OsX More user-friendly e-mail notification from on-the-fly reprocessing Improved user documentation for point-source extractor More work needs to be done to make before/after pointing-refinement

performance data more accessible (need to store it in the database) Least-square stim interpolation for MIPS:Ge TPM MIPS:Ge pipeline that uses self-generated darks Multi-threaded GeRT GeRT port to Mac OsX


Sample of new on-the-fly reprocessing webform


Sample output from new on-the-fly reprocessing script

sscpipe14% $SIRTF_SCRIPTS/otfReprocessing.pl -o 20050812T1825.out

otfReprocessing.pl, Version 1.0

By Russ Laher ([email protected]), Ron Beck ([email protected]), and Jeff Jacobson ([email protected]).Copyright (C) 2005Spitzer-Space-Telescope Science Center (SSC),California Institute of Technology, Pasadena, CA

Script for "lights out" on-the-fly reprocessing.

Database, role = sodb_dnl5, pipelineswrole

Date/time = Fri Aug 12 18:22:31 2005

Open database connection...

List of unique reqKeys:

Checking (reqKey, chanNum) pairs for current status...

CampaignId Instrument ReqKey ChanNum ReprocessFlag 662 MIPS 7120384 1 1 708 IRSX 4980736 1 0 708 IRSX 4980736 2 0 708 IRSX 4980736 3 1

List of pending onTheFlyReprocessing records:

ProcId UserId E-mail Instrument ReqKey ChanNum 54 6 [email protected] IRSX 4980736 1 54 6 [email protected] IRSX 4980736 2 54 6 [email protected] IRSX 4980736 3 55 6 [email protected] MIPS 7120384 1


Sample output from new on-the-fly reprocessing script – cont.

Deleting records from WIP tables...Loading WIP tables...Making new pvIds...Aug 12 18:22:32: Querying WIP tables...

CampaignId Instrument ReqKey ChanNum ReprocessFlag pvId 662 MIPS 7120384 1 1 1055 708 IRSX 4980736 1 0 0 708 IRSX 4980736 2 0 0 708 IRSX 4980736 3 1 1056

ProcId UserId E-mail Instrument ReqKey ChanNum 54 6 [email protected] IRSX 4980736 1 54 6 [email protected] IRSX 4980736 2 54 6 [email protected] IRSX 4980736 3 55 6 [email protected] MIPS 7120384 1

Aug 12 18:22:32: setup manifests ...Aug 12 18:22:32: create campaign_reproc thread entries ... dnlAccess -> setPlHopper 54 dnlAccess -> setPlHopper 55 Aug 12 18:22:34: monitor pipeline processing ... checking progress of reqkeys ... Onthefly-IRSX-3 4980736 has not started yet ... Onthefly-MIPS-1 7120384 has not started yet ... Aug 12 18:22:34: sleeping 300 ... checking progress of reqkeys ... start: Aug 12 18:27:13 Onthefly-IRSX-3 4980736 2017 stop: Aug 12 18:27:14 Onthefly-MIPS-1 7120384 1030Aug 12 18:27:35: sleeping 300 ... dnlAccess -> setPlDone 54 dnlAccess -> setPlDone 55 Aug 12 18:32:35: End pipeline processing...


Sample output from new on-the-fly reprocessing script – cont.

Aug 12 18:32:35: Begin product archiving... Setting archiveCampaign flag for pvId = 1055 Setting archiveCampaign flag for pvId = 1056

Aug 12 18:32:35: Polling to see if product archiving has completed... Certified flag for pvId = 1055 is equal to 0 Certified flag for pvId = 1056 is equal to 0 Certified flag for pvId = 1055 is equal to 0 Certified flag for pvId = 1056 is equal to 0 Certified flag for pvId = 1055 is equal to 0 Certified flag for pvId = 1056 is equal to 0 Certified flag for pvId = 1055 is equal to 1 Certified flag for pvId = 1056 is equal to 1

Aug 12 18:35:35: Terminating OTF-reprocessing episode...Aug 12 18:35:35: The following e-mail notifications were sent:[email protected] -> New Spitzer products are available for reqKeys: 4980736 7120384 Aug 12 18:35:35: Close database connection...

Date: Fri, 12 Aug 2005 18:35:35 -0700 (PDT)From: Rick Ebert <[email protected]>To: [email protected]: SIG EVENT #305740, 050812181451F1 INFORMATION Received 2005-08-12T18:35:35 from terminateOnTheFlyReprocessingEpisode( PID N/A) on host DB Procedure, message # 1, New Spitzer products are available for reqKeys: 4980736, 7120384


Sample of new ptg xfer Q/A statistics


Sample of new ptg xfer Q/A statistics – cont.

russ laher (1) s13.0 test readiness review, august 29, 2005 downlink segment test status (s13.0)...

Documents

mips bcd products

final mips

russ laher

summary of s13

test readiness review

irs new irs pipeline

whats new

mips channelsmodes