KIDS requirements

Dave Wilman for the KIDS team

• The KIDS operational plan and the role of AstroWise.

• List focussed on open questions: discussion required!

• By no means an exhaustive list…

1. Monitoring Tool

Requirements:

We would like a tool to: – Define the observations

And to monitor and manage:– Progress of observations– Progress of reduction– Delivery to ESO

Current Implementation:

• The VISTA consortium is developing the Survey Area Definition Tool (SADT) which will interface with P2PP to automatically generate OBs and guide stars. Availability: early 2007.

• P2PP will be upgraded to allow timelink, concatonation and grouping of OBs so that survey strategy can be optimized.

• VISTA consortium developing a tool to monitor survey progress: The EST will look at this and see if a version of this tool can be made available to ESO and other survey teams.

1. Monitoring Tool

1. Monitoring Tool

Action Items:

• Check the existence, capability and availability of the ESO monitoring tool.

2. Strategic Organisation and Distribution of Data Reduction

• Management Tool (item 1)• Uniform plan for data reduction:

– Consistent scripts / use of web-interface• Distribution of data for actual reduction.• Distribution of responsibilities for recipes and

testing.

3. Field Layout and meshing with VIKING

Observing Strategy and Dithering of KIDS:– see KIDS webpage

Meshing with VISTA:– Probable solution: Matching KIDS tiles in Dec,

but not in RA.

4. Dealing with Tiling and overlaps

• Split into 1 square degree chunks (imaging and catalogue)?

• Provide catalogue only once the full depth is available in all bands for that chunk?

• Deal separately with the overlap regions and the resulting cross-calibration?

• How will these divisions work technically? (i.e. can SWARP apply its own boundaries to a set of images, which due to dithering and tiling overlaps will cover a larger area than the chunk?)

5. The size of KIDS

There has been discussion with ESO about extending the area of KIDS. Technically, what is the maximum data-rate which we could handle for reduction?

– Manpower (reduction, testing, re-reduction)– Computer power (rate of processing)

6. Absolute Photometric Calibration

Calibration of the absolute photometric zeropoint:

North: Calibrate using SDSS sources: – SDSS accuracy not good enough for our relative photometry

(~0.02mags) but OK for the absolute photometry?– Well understood calibration in worse seeing SDSS data– SDSS photometry already exists.

South: Calibrate using our own photometric pass:– Easier to calibrate, with reduced systematics– No reliance on external source of data– Possible synergy with ATLAS– May help improve absolute astrometry– Alternatives (e.g. SSS) rely on data which does not yet exist,

and with unknown systematics.

7. Quality Control

• Local / Short-term: – during data processing; – Simple checks of data products: plots and QC

parameters (see next slide);• Global / Long-term:

– Photometric accuracy using multi-band catalogues;

– Colour-colour diagrams (and photo-z);– Number counts;– Delegated responsibility by expertise?

(e.g. Munich has expertise in fine-tuning photometry using a multi-band catalogue)

8. QC Parameters and Plots

• QC parameters to be computed from each node when processing data?

• Code required to make certain tests: build on existing inspect methods (plots) and QC parameters (values)

• Store QC parameters in database and process to management tool?

• Require a full list of such tests, frequency and responsible persons.

8. QC Parameters and PlotsExample QC Plots and Parameters:

– Astrometry: • Local: plots of residuals (dx,dy,dra,ddec) vs (x,y,ra,dec)• Global: cross-checks (and a long-term check might be to look for

systematic and random offsets between bands)• Values: rms, Npairs, coefficients and errors on linear fits of

residuals?....– Photometry:

• Images: – Background level variation in subwindows?– PSF(time, chip, position, colour, mag)– Manual(?) check for fringe-patterns, ghosts, satellite tracks, cosmic

rays etc• Catalogues:

– STD: plots: raw ZP (time, chip, position, colour, mag)– STD: values: raw ZP rms, Nobj etc– SCI: in overlap regions: ZP residuals (time, chip, position, colour, mag)

etc etc…..

QC Params: KIDS page 1Test when By whom

PSF: plots: width, PA, ellip as fn of(position, mag, time, chip, colour)Vector plot

New Reduced, Regridded, Coadded frames- Becomes less frequent when more stable

Creator

PSF: pars: width, ellip, PA: mean/median and dispersion, (radial gradients?); Later: quantify comparison with the expectation

New Reduced, Regridded, Coadded frames

Creator

Astrometry: Local: Plots: residuals (dx,dy,dra,ddec) vs (x,y,ra,dec)Vector plotVs mag

New Astrometric Solution- At beginning, and when solution is poor

Creator

QC Params: KIDS page 2Test when By whom

Astrometry: Local: pars:Rms, Npairs, rms(dra, ddec, dX, dY), NrejectedPairs, Fit params, subwindow stats (where rejected stars are clustered)

New Astrometric Solution

Creator

Gastrometry: plots using all overlaps – residuals vs position, chip; stars used for solution

New GAstrometric Solution- When fit is poor

Creator

Gastrometry pars: rms, Npairs, rms(dra, ddec), NrejectedPairs, Fit params, subwindow stats (where rejected stars are clustered)Contrast with expectation (trend analysis on solution)

New GAstrometric Solution

Creator

QC Params: KIDS page 3Test when By whom

Astrometric Solution in different filters: compare positions in multi-band catalogue

Multi-band catalogue: NOT aperture matched

Local

Overlaps between adjacent fields (not dithers): similar to the Gastrometric tests

Single-band catalogues

PMs / Astrometry team

Photometric: plots – STD : raw ZP (chip, position, time, filter, colour, mag, airmass, PSF width)

New Photometric Parameters- When required

Creator / Local (time/filter dependance)

Photometric Pars: - STD : ZP, rms(ZP), Npairs, NRejectedPairs, fit params for above plots

New Photometric Parameters

Creator / Local (time/filter dependance)

QC Params: KIDS page 4Test when By whom

Photometry of Science Observations – overlaps:ZP consistency (dithers, pointing overlaps): dmag between different exposures – 1 number per overlap computed using high S/N stars

Before coaddition (on regidded frames) using SourceLists from regridded frames

Local

Stellar Colour-Colour plots & photo-z (via spec-z)- implementation?

Multi-band catalogues

Taskforce (Munich?)

Number Counts – includes depth check (also in overlap regions with greater depth) - implementation?

Single-band catalogues

Taskforce (Paris?)

Cross-Survey Comparisons (SDSS, CFHT-LS, SSS…)

Multi-band catalogues with matched magnitude types

Designated

QC Params: KIDS page 5Test when By whom

Visual Inspection of background:Fringes, flat-fielding, bad pixels, artifacts, actual sky background

New ReducedScienceFrames

Creator

9. Photometric Accuracy

Require accuracy of 0.01mag required for photo-z.

(Potential) Astrowise tools to help reach this accuracy:• Galctic Extinction Map Ingestion• Sextractor Double-Image mode (measure the flux at

the same position in all bands, even where in some bands we only get upper limits)

• Shapelets: The GAAP (PSF-matching) method (Kuijken et al): can we reach the required accuracy with this method?

• Convolution (brute-force): Johannes has plans to implement within AstroWise, and to test the computational requirements.

10. Ingestion of NIR data

How will NIR data be ingested into AstroWise?• At what stage of reduction will it be ingested?

Pre-regridding?• What kind of Weight-Maps will/can come with the

data?• Is Ingestion of reduced images already tested?• UKIDSS / VISTA / 2MASS??

11. Data Storage

Deletion of files !!! Is it necessary to consider deletion of:• Redundant files?• Intermediate files?

Our tutorial in Munich, plus Johannes’ WFI data is already filling our diskspace…….

Please consider how we can safely delete data within a Project context!!!

12. Visual Inspection of Data

• PanoraPix: Will it be ready in time for KIDS?

• What is the status of the cut-out server?

• It would be nice to view each ‘chunk’ of KIDS data separately: if PanoraPix is not ready, will this be possible?

13. Tracking of artifacts

We will want to know automatically which sources in the catalogue are real.

How do we treat:• Ghosts?• Bright-star artifacts?

14. Node Project Managers

Who are they?• Munich: David Wilman <dwilman@mpe.mpg.de>, Hans Vaith

<hav@mpe.mpg.de>,• Groningen: Edwin A. Valentijn <valentyn@astro.rug.nl>,• Paris: Herve Aussel <herve.aussel@cea.fr>,• Leiden: Merijn Smit <msmit@strw.leidenuniv.nl>,• Bonn: Oliver Cordes <ocordes@astro.uni-bonn.de>,• Napoli: Mario Radovich radovich@na.astro.it

Each manager should be able to manage/organise the people at their node.

A good start would be to get started with test cases ASAP!

15. KIDS Daily Operations

Contact between nodes / people.Frequency / method:

• A forum / mailing list?• Regular telecons?• On a longer timescale, regular meetings?

16. Timescale for ESO delivery

What we told ESO in the Management Plan RIx reply:“A good multi-band catalogue is essential to qualify the photometric

(and astrometric) calibration of our data products, and to our science (in particular to get good photo-z's). We will reevaluate seeing requirements for u, g & i (but not r) bands each semester, with the aim to keep the bands in step. However, particularly at the beginning of the survey, observing practicalities will mean that there is no way that we have identical coverage in all bands.

Photometric calibration will require the photometric pass which will precede the main survey pass and establish zero-points across the KIDS area. Also we require re-observation with OmegaCAM of our secondary standard star fields during the first year, as outlined in the OmegaCAM Calibration Plan. Multi-band photometry is also required for vital fine-tuning of photometric accuracy (colour-colour and photo-z checks). For all these reasons, we request a full year (after delivery of the final band for a given tile) for initial catalogue delivery. Once the photometric calibration is better established, we are happy to catch-up to a 6 month timescale. The frequency of delivery can be as ESO prefers. This can be almost as continuous as desired.”

17. WFI test programThere is a need for a ‘federated’ test program which

‘simulates’ the reduction of KIDS data (i.e. managing the distribution of reduction and testing amongst ourselves).

Pre-requirements:• Federation of AstroWise - required• Nodes ready with the AstroWise system installed working

on the clusters – required• Stable and relatively bug-free installation of AstroWise –

required• Consistent plan for reduction on an appropriate dataset –

required• All software elements in place (e.g. aperture-matched

photometry etc) - preferable

18. VLT followup

Is there any action required regarding the ESO VLT followup of KIDS targets?