raw data recipes SGDP

Science Grade Data ProductsWolfram Freudling

DPD / SDP Group

Science grade data products (SGDPs) are data products which can be used as-is to extract scientific conclusions, or to carry out quantitative measurements.

Instrument signature removed physical units signal-to-noise ratio close to the optimum error estimates all aspect of data taking and processing fully documented

Data Product Survey Nov 2008

AMBER: 4

CRIRES: 11

FORS1: 3

FORS2: 3

FORS12: 16

GIRAFFE: 9HAWK-I: 3ISAAC: 12

MIDI: 6

NACO: 2

SINFONI: 4

UVES: 6

VIMOS: 5

VISIR: 10

About 50% overlap with DFS tickets

Proposed Projects

Science Ready Pipelinesas reported by Survey

• CRIRES LS Spectroscopy • GIRAFFE/MEDUSA• Hawk-I• UVES• VISIR

address issues which apply to several instruments

• Multi-OB image combination • Telluric lines tool• illumination correction • distortions • background subtraction• detector “monitoring”

Austrian contribution

Austrian contribution

Looking for science-ready instrument to develop Reflex

workflow• Hawk-I:– Major problem with background subtraction

=> fixed– Distortions:

• not enough calibration data• needed recipe to analyze distortions• distortions not stable

=> Austrian in-kind

Looking for science-ready instrument to develop Reflex

workflow

• UVES:– error-bars– Combination of calibrations– CR-like artefacts

=> most issues fixed draft of report by Daniel Bramich

• Extrapolation: Every instrument needs substantial work before data products can be considered science-grade

• Many easy to spot problems• Problems often involve both calibration data

and pipeline• Bottle-neck is the resources available for

validation and verification• Only 2.5 astronomers work in SDP group => need IOT involvement

Conclusions so far

New process to set priorities

http://eso.org/observing/dfo/sdp/

Why?• Tickets driven development not good way to set

priorities• Rush to submit something to pipeline priority

meeting• PSD resources are spread too thin:

– do fewer things with more resources

• Projects are too software-centric– full support of projects by IOT/SDP

• Goals of projects are ill-defined– start with good specifications, test plan

“Proposals should be articulated, comprehensive, and set some specific measurable goals to improve data quality. “

Proposals will be assessed by DData PProducts BBoard.

Instrument upgrades should use the same route.

SDP Proposals

The Board

• The Data Product Board is chaired by the head of SDP

• meets twice per year, monthly progress meetings

• Members: IOT Coordinator, head of LPO/PSO, the head of DMO/DPD, head of SDD/PSD, representative from INS/IPD

• First meeting: Feb 2011

but...

• IOT/IS does not have the time, resources, motivation ... to write a SDP proposals

• It is de-motivating if IOT proposal needs DPB approval

• We will not be able to process some data if operational critical tickets are not addressed right away

• I prefer to talk directly to software developer

We should work only on projects which we can complete.

Approved projects get lots of resources.

We need to use our resources efficiently.

This is understood and accepted.

What should be in a proposal / data products requirements

• Specific requirements for data quality• Proposal is not compilation of tickets• Separate requirements from

implementation suggestions• Prototypes for algorithms are great, but...• Detail resources

Effect on Pipeline Development

• All pipeline development frozen be default• Tickets should be submitted but will not be

attended immediately.• Exceptions have to be approved on a case-

by-case basis• “new” pipelines under development not

affected.

Pipelines not frozen• VIMOS: support for upgrade• XSHOOTER: still in development for several

modes• VISIR: basic support for the new detector

only; additional development must go through an enhancement proposal

• OMEGACAM: support for commissioning• VIRCAM: pipeline still in development for

several modes• PRIMA: still in development

Questions?

Comments?