18 april 2007 second generation vlt instruments 1 vircam & cpl: lessons learned jim lewis and...
TRANSCRIPT
18 April 200718 April 2007 Second Generation VLT InstrumentsSecond Generation VLT Instruments 11
VIRCAM & CPL: Lessons VIRCAM & CPL: Lessons LearnedLearned
Jim Lewis and Peter BunclarkJim Lewis and Peter Bunclark
Cambridge Astronomy Survey UnitCambridge Astronomy Survey Unit
18 April 2007 Second Generation VLT Instruments 2
Introduction to CASUIntroduction to CASU
► Small group within the Institute of Astronomy, specialising Small group within the Institute of Astronomy, specialising in survey astronomy.in survey astronomy.
► Pipeline reduction of imaging dataPipeline reduction of imaging data APM (Schmidt Plates 1 x 40k x 40k)APM (Schmidt Plates 1 x 40k x 40k) INT Wide Field Camera (4 x 2k x 4k)INT Wide Field Camera (4 x 2k x 4k) ESO WFI on 2.2m at La Silla (8 x 2k x 4k)ESO WFI on 2.2m at La Silla (8 x 2k x 4k) MOSAIC-1 on KPNO 4m (8 x 2k x 4k)MOSAIC-1 on KPNO 4m (8 x 2k x 4k) MOSAIC-2 on Blanco 4m at CTIO (8 x 2k x 4k)MOSAIC-2 on Blanco 4m at CTIO (8 x 2k x 4k) AAO WFI on AAT (8 x 2k x 4k)AAO WFI on AAT (8 x 2k x 4k) CIRSI on INT (4 x 1k x 1k)CIRSI on INT (4 x 1k x 1k) INGRID on WHT (1k x 1k)INGRID on WHT (1k x 1k) UFTI on UKIRT (1k x 1k)UFTI on UKIRT (1k x 1k) WFCAM on UKIRT (4 x 2k x 2k)WFCAM on UKIRT (4 x 2k x 2k)
18 April 2007 Second Generation VLT Instruments 3
Introduction to VISTA and Introduction to VISTA and VDFSVDFS
► VISTA: 4m survey telescope on ParanalVISTA: 4m survey telescope on Paranal► VIRCAM: The IR camera for VISTAVIRCAM: The IR camera for VISTA
16 Non-buttable 2k x 2k detectors16 Non-buttable 2k x 2k detectors► VDFS: PPARC funded facility to provide an end-to-VDFS: PPARC funded facility to provide an end-to-
end data-flow system for VISTA and WFCAM.end data-flow system for VISTA and WFCAM.► Quality control and calibration pipelinesQuality control and calibration pipelines
Paranal and GarchingParanal and Garching► Science pipeline for full calibration of science data.Science pipeline for full calibration of science data.
CambridgeCambridge► Science archive acts as the point of access of the Science archive acts as the point of access of the
reduced data. Plus some further processing.reduced data. Plus some further processing. WFAU, EdinburghWFAU, Edinburgh
18 April 2007 Second Generation VLT Instruments 4
Data FlowData Flow
►Raw telescope data is assessed by the Raw telescope data is assessed by the summit pipeline (QC1)summit pipeline (QC1)
►Shipped to Garching (discs)Shipped to Garching (discs)►Shipped to CASU for science reduction Shipped to CASU for science reduction
and calibration. (discs)and calibration. (discs)►Calibrated data shipped to Edinburgh Calibrated data shipped to Edinburgh
for archiving (ftp)for archiving (ftp)
18 April 2007 Second Generation VLT Instruments 5
IR Data Reduction WorriesIR Data Reduction Worries
► IR detectors are currently inherently more IR detectors are currently inherently more unstable than optical CCDs. unstable than optical CCDs. Some odd electronic effectsSome odd electronic effects Much poorer cosmetic quality than CCDsMuch poorer cosmetic quality than CCDs
► Sky emission > 100x brighter than most Sky emission > 100x brighter than most objectsobjects And it’s variable both spatially and temporally!And it’s variable both spatially and temporally!
► Exposure times are short, so data rates are Exposure times are short, so data rates are very high.very high. 200-500 Gb/night expected for VISTA public surveys200-500 Gb/night expected for VISTA public surveys Rice tile compression can save factors of 3-4 in 32 bit Rice tile compression can save factors of 3-4 in 32 bit
integer datainteger data
18 April 2007 Second Generation VLT Instruments 6
VDFS Pipeline RecipesVDFS Pipeline Recipes
►Create master calibration frames (dark, Create master calibration frames (dark, twilight flats, confidence maps, etc)twilight flats, confidence maps, etc)
►Linearity analysisLinearity analysis►Detector noise & dark current propertiesDetector noise & dark current properties►Persistence and crosstalk analysisPersistence and crosstalk analysis► Illumination correction analysisIllumination correction analysis►Full reduction recipes for standard star Full reduction recipes for standard star
and programme fieldsand programme fields
18 April 2007 Second Generation VLT Instruments 7
Summit & Garching PipelinesSummit & Garching Pipelines
►QC1 parametersQC1 parameters e.g. photometric zeropoints, astrometric fit e.g. photometric zeropoints, astrometric fit
qualityquality
►Written using ESO qfits/CPL infrastructureWritten using ESO qfits/CPL infrastructure Both use the same software modulesBoth use the same software modules
►Amount of processing can be scaled downAmount of processing can be scaled down►Calibration images (flats etc) from a Calibration images (flats etc) from a
master library.master library.
18 April 2007 Second Generation VLT Instruments 8
Cambridge PipelineCambridge Pipeline
►Full reductionFull reduction e.g. better sky correction e.g. better sky correction
►Results shipped to WFAUResults shipped to WFAU►Some of the same CPL based modules Some of the same CPL based modules
reused. Some additional modules reused. Some additional modules required.required.
18 April 2007 Second Generation VLT Instruments 9
Current Status of VISTA Current Status of VISTA PipelinesPipelines
►Version 0.5 released to ESO in MarchVersion 0.5 released to ESO in March►Full compliment of recipes (almost!)Full compliment of recipes (almost!)►PAEPAE
18 April 2007 Second Generation VLT Instruments 10
The Decision To Use CPL (1)The Decision To Use CPL (1)
►We were given the choice.We were given the choice.►We already had advanced pipeline We already had advanced pipeline
software in our own environment for software in our own environment for WFCAM and other IR instruments.WFCAM and other IR instruments.
►First view of CPL and qfits was not First view of CPL and qfits was not favourable.favourable.
18 April 2007 Second Generation VLT Instruments 11
The Decision To Use CPL (2)The Decision To Use CPL (2)
►CPL v2.0 was a great improvementCPL v2.0 was a great improvement►CASU has an ‘ESO workstation’ to CASU has an ‘ESO workstation’ to
generate ‘data’ using the VIRCAM generate ‘data’ using the VIRCAM acquisition softwareacquisition software
► Integration into Paranal and Garching Integration into Paranal and Garching systems greatly simplified by using systems greatly simplified by using CPLCPL
18 April 200718 April 2007 Second Generation VLT InstrumentsSecond Generation VLT Instruments 1212
Things I wish I had Things I wish I had known…known…
18 April 2007 Second Generation VLT Instruments 13
My First Attempt At A RecipeMy First Attempt At A Recipe
► INPUT:INPUT: FITS image of WFCAM dataFITS image of WFCAM data Binary FITS table of astronomical objectsBinary FITS table of astronomical objects Binary FITS tables containing the 2MASS catalogueBinary FITS tables containing the 2MASS catalogue
► TASK:TASK: Work out RA, Dec coverage of image using header Work out RA, Dec coverage of image using header
FITS WCS keysFITS WCS keys Extract 2MASS standardsExtract 2MASS standards Cross match standards to object catalogueCross match standards to object catalogue Fit a WCSFit a WCS Write resulting WCS to FITS imageWrite resulting WCS to FITS image
18 April 2007 Second Generation VLT Instruments 14
World Coordinate SystemsWorld Coordinate Systems
►CPL has no WCS facilityCPL has no WCS facility But it soon willBut it soon will
18 April 2007 Second Generation VLT Instruments 15
Input File AccessInput File Access
►CPL doesn’t give write access to input CPL doesn’t give write access to input files.files. cpl_frame, cpl_image cpl_frame, cpl_image Data abstraction is not perfectData abstraction is not perfect cpl recipes can’t be written to update a cpl recipes can’t be written to update a
filefile
18 April 2007 Second Generation VLT Instruments 16
CPL vs FITS tablesCPL vs FITS tables
► CPL does not allow for direct manipulation of CPL does not allow for direct manipulation of FITS tablesFITS tables cpl_table structures can be created from a FITS table. cpl_table structures can be created from a FITS table. But one must read the WHOLE table even though you But one must read the WHOLE table even though you
might only be interested in a few rows or columns.might only be interested in a few rows or columns.► Tables with many rows take too long to readTables with many rows take too long to read
Break them upBreak them up► Tables with many character type entries cause Tables with many character type entries cause
the system to run out of memory pointersthe system to run out of memory pointers Remove unnecessary character columnsRemove unnecessary character columns
18 April 2007 Second Generation VLT Instruments 17
User DocumentationUser Documentation
►Reference manual, users guide, Reference manual, users guide, software developers kitsoftware developers kit
►User’s guide and SDK are helpful but User’s guide and SDK are helpful but limited in scopelimited in scope Very few examplesVery few examples Get a copy of a working CPL pipelineGet a copy of a working CPL pipeline
18 April 2007 Second Generation VLT Instruments 18
QCQC
►Garching and Summit pipelines generate QC Garching and Summit pipelines generate QC parametersparameters Provides data for QC trending to be done both on Provides data for QC trending to be done both on
site and in Garchingsite and in Garching The ‘ESO System’ is really two systemsThe ‘ESO System’ is really two systems
►QC passed through FITS headers (Garching)QC passed through FITS headers (Garching)►QC passed through PAF files (Chile)QC passed through PAF files (Chile)►Neither system appears to be documented.Neither system appears to be documented.►PAF files contents are not documented in CPL users or PAF files contents are not documented in CPL users or
reference manuals.reference manuals.►CPL has no standard class for passing QC parametersCPL has no standard class for passing QC parameters
18 April 2007 Second Generation VLT Instruments 19
File TagsFile Tags
►DPR TYPEDPR TYPE OBSTYPEOBSTYPE
► PRO CATGPRO CATG Product typeProduct type No two recipes can produce the same valueNo two recipes can produce the same value
►DO CATGDO CATG File type for files used as input to recipesFile type for files used as input to recipes Raw frame DO CATG values must be unique to Raw frame DO CATG values must be unique to
each recipe.each recipe. There are some rules on the values here. There are some rules on the values here.
18 April 2007 Second Generation VLT Instruments 20
Output File NamesOutput File Names
►Recipe creates and names output files.Recipe creates and names output files. File names must be hard coded!File names must be hard coded! Several files of the same PRO CATG must Several files of the same PRO CATG must
be indexed in a predictable way.be indexed in a predictable way.
►esorex can change the file nameesorex can change the file name