NEON School

NEON Archive Observing School

Alberto Micol

ESA

Space Telescope Operations Division15 July 04

ESO & HST ArchivesESO & HST Archives

NEON School

NEON School 15 July 2004 NEON School 15 July 2004 LayoutLayout

What’s the role of the Data Provider

Three levels of Archive complexity– Raw Data (ESO Archive)– Reduced Products (The importance of Service

Observing)– Enhanced Products (HST/WFPC2 Associations)

Enhanced Interfaces (Querator & VO) Publications and Observing Programs

NEON School

Operations Technical Support

The ESO Data Flow Model and its Supporting Organizational Structure

Data Archive & Supporting Databasesstorage and re-distributionPI packages distribution

Phase Iproposal submissionprogram selectiontime allocation

Phase IIproposal submissionobserving blocksmedium term schedule

Observationshort term schedulearchival of data

Quality ControlMaster Cal files prod.Instrument trendingPI products preparation

User S

uppo

rt Gro

up

Par

anal

/La

Sil

la O

bser

vato

ries

Data Flow Operations Group

Data Flow, what you were toldData Flow, what you were told

NEON School

Data Flow, the joyless TruthData Flow, the joyless Truth

NEON School

Space-based observatories:•The archive is a core activity from the start.

Ground-based observatories:•Priority given to Observers, not to CommunityWith the difference that Observers can go observer themselves.

ESO has changed such paradigm, by introducing Service Mode.But it takes time for the community to accept it,

and effort within the Observatory to implement it

The Archive: last in the chain?The Archive: last in the chain?

NEON School

http://www.eso.org/org/dmd/usg/philosched.htmlService ModeService Mode

NEON School

Real Archive vs Virtual ObservatoryReal Archive vs Virtual Observatory

Real ArchivesThree types of Archives

– Level 0: Data Repository– Level 1: Ability to offer reduced products– Level 2: Ability to offer enhanced products

1 Instrumental signature-free data

2 Enhanced Products (e.g., mosaics)

0 Data Repository

“Real” Archives

BoringHandy

NEON School

Level 0: Data RepositoryLevel 0: Data Repository

Data Repository:1. Acquisition2. Data Management3. Metadata4. User Interface

NEON School

Level 0: Data Repository (1)Level 0: Data Repository (1)

Acquisition:· Observing run · The Control S/W acquires the Data· Stored in the on-line data archive· Transmitted to Science Archive

NEON School

Level 0: Data Repository (2a)Level 0: Data Repository (2a)

Data management:Integrity checks

· Did I get all I should have gotten?· Check for corrupted data· Take action if corrupted

NEON School

Level 0: Data Repository (2b)Level 0: Data Repository (2b)

Data management:Storing on medium of choice Optical disks, CDs, DVDs, Magnetic Disks

Bookkeeping Which file was stored where?When? In which format?Which compression?Store all this info in a Database (RDBMS)

NEON School

Level 0: Data RepositoryLevel 0: Data Repository

Data Repository:1. Acquisition2. Data Management3. Metadata4. User Interface

NEON School

Level 0: Data Repository ( 3 )Level 0: Data Repository ( 3 )

Metadata:· Where was the telescope pointing? (RA, Dec)· Which telescope/instrument was used? (telescope, instrument)· When did the observation occur? (start time, end time, exptime)· In which band? (filter or dispersive element used)· By whom? (Principal Investigator [PI])· For which reason? (Observing Program Identifier)· Plus many other details about the observing mode, etc.

· All this information is usually extracted from the Header of the FITS files containing the Data.· Metadata are typically stored in a Database

NEON School

Level 0: Data RepositoryLevel 0: Data Repository

Data Repository:1. Acquisition2. Data Management3. Metadata4. User Interface

NEON School

Level 0: Data Repository ( 4a )Level 0: Data Repository ( 4a )

User Interface

Finally, an astronomer wants to seewhether some information about photons regarding a certain celestial object are already stored in our archive.

Metadata (Database) -> Internet (Web)

NEON School

Level 0: Data Repository (4 Archive)Level 0: Data Repository (4 Archive)

User Interface

NEON School

Level 0: Data Repository (4 Form)Level 0: Data Repository (4 Form)

User Interface

Finally, an astronomer wants to seewhether some information about photons regarding a certain celestial object are already stored in our archive.

Metadata (Database) -> Internet (Web)

NEON School

Level 0: Data Repository ( 4b )Level 0: Data Repository ( 4b )

User Interface

•Name Resolvers: SIMBAD (CDS), NED (IPAC)•Search by Box/Cone (RA, Dec, size/radius)

•Search by Night (Observing run, calibration obs.)•Observing Blocks, Data Product Types: very ESO specific •Exptime, Filter, Grism, Grating, Slit: very instrument specific

•Release Date: Are the data public?Proprietary period implements Observatory policy

NEON School

Level 0: Data Repository ( 4 Query )Level 0: Data Repository ( 4 Query )

User Interface

Finally, an astronomer wants to seewhether some information about photons regarding a certain celestial object are already stored in our archive.

Metadata (Database) -> Internet (Web)

NEON School

Level 0: Data Repository ( 4 Results )Level 0: Data Repository ( 4 Results )

User Interface

Finally, an astronomer wants to seewhether some information about photons regarding a certain celestial object are already stored in our archive.

Metadata (Database) -> Internet (Web)

NEON School

Level 0: Data Repository ( 4 Submission )Level 0: Data Repository ( 4 Submission )

User Interface

Finally, an astronomer wants to seewhether some information about photons regarding a certain celestial object are already stored in our archive.

Metadata (Database) -> Internet (Web)

NEON School

Level 0: Data Repository ( 5 )Level 0: Data Repository ( 5 )

Request HandlerA system which takes good care of the submitted archive

requests. The RH consists of a set of procedures, interfaces, C

programs and libraries, and relational tables; It allows to: • submit archive requests, • keep track of their handling, • inform (via email or via a web interface) the operators of

any intervening problem,• notify the archive users about the status of their requests • up to successful closure.

NEON School

Level 0: Users’ point of viewLevel 0: Users’ point of view

A Level 0 archive only offers RAW Data.Removing the instrumental signature from the

RAW files is left to the astronomer:

• Identify suitable BIAS, DARK and FLAT frames• Produce Master frames (e.g., mean averaging, etc.)

• Run reduction software (e.g., bias&dark, flat fielding)

Couldn’t the Observatory provide that service?

NEON School

Level 1: Ability to offer reduced products Level 1: Ability to offer reduced products (1)(1)

Data Repository & Processing:1. Actual Storage2. Data Management3. Metadata4. User Interface5. Calibration DB + Rules6. Reduction Pipeline

NEON School

Level 1: Ability to offer reduced products (2) Level 1: Ability to offer reduced products (2)

Calibration DB + Rules· Instrument Scientists (ESO: Quality Control group [QC])

build the “Master Calibration Files” (or “Reference Files” in HST terminology)

· Reference files are stored in the Archive

· They associate to each observation a set of reference files via a set of rules.

· Rules are applied and results stored in the Calibration Database

NEON School

Level 1: Ability to offer reduced products Level 1: Ability to offer reduced products (3)(3)

Calibration DB + Rules

Simple-minded example· Rules and recipe to build a master DARK

file:

· Retrieve 5 DARK images · the ones closer to the date of the observation, · With the same read-out mode and exposure time; · De-bias each DARK image; · Average them while removing cosmic rays;· Normalise to “dark current per second”; · Flag dubious pixels (Data Quality flag)

NEON School

Level 1: Ability to offer reduced products Level 1: Ability to offer reduced products (4)(4)Reduction Software

· Instrument Scientists and Software Group must implement the Reduction Software:

a collection of processes necessary to remove the instrumental signature from a given observation.

If such software can run unattended, we call it Reduction Pipeline.

NEON School

Level 1: Ability to offer reduced products Level 1: Ability to offer reduced products (5)(5)

Reduction Products· Once the Calibration DB and the

Reduction Pipeline are in place,

the Science Archive Request Handler can make use of it

to offer Reduced Products to the relieved community.

NEON School

Level 1: HST reduced products (form)Level 1: HST reduced products (form)

HST

NEON School

Level 1: HST reduced products (results)Level 1: HST reduced products (results)

HST

NEON School

PreviewPreview

PREVIEW

•A highly compressed view of an observation, to provide a quick-look product for quality assessment.

It is generated via a specialised pipeline takes in input the reduced (or otherwise raw) observation and processes it (resizing, cropping, reducing dynamical range, applying wavelets, etc.).

Look at it, do not use it for your science!

NEON School

Level 1: HST reduced products (preview)Level 1: HST reduced products (preview)

HST

NEON School

Level 1: HST reduced products Level 1: HST reduced products (submission)(submission)

HST

NEON School

Level 2: Enhanced Products & InterfacesLevel 2: Enhanced Products & Interfaces

How to improve the archive beyond the on-the-fly recalibration?

1. Deeper and cleaner products2. Better User Interfaces

NEON School

Level 2: Deeper and Cleaner: AssociationsLevel 2: Deeper and Cleaner: Associations

HST/WFPC2 (Wide Field Planetary Camera 2)

An observation in space is affected by Cosmic RaysNo detector is perfect: bad columns, hot pixels, etc.

Special observing strategy to cope with these facts:

CR-SPLIT: Splitting one single long exposure into many short ones;

POS-TARG: Dithering is the technique of displacing the telescope between observations either on integral pixel scales (to assist in removing chip blemishes such as hot pixels) or on sub-pixel scales (to improve sampling and thus produce a higher-quality final image).

NEON School

Level 2: Deeper and Cleaner: AssociationsLevel 2: Deeper and Cleaner: Associations

HST/WFPC2 (Wide Field Planetary Camera 2)

CR-SPLIT and POS-TARG have to do with the Data Flow System of the HST observatory, not with the scientific exploitation of the data.

The associations, i.e., grouping together those observations which cover the same sky region, provide a way to isolate the archive userfrom those technical details.

NEON School

Level 2: WFPC2 Associations (form)Level 2: WFPC2 Associations (form)

NEON School

Level 2: WFPC2 Associations (asn)Level 2: WFPC2 Associations (asn)

NEON School

Level 2: WFPC2 Associations (members)Level 2: WFPC2 Associations (members)

NEON School

Level 2: WFPC2 Associations (comparison)Level 2: WFPC2 Associations (comparison)

NEON School

Level 2: Advanced Interfaces: QueratorLevel 2: Advanced Interfaces: Querator

Astronomers requires Scientific interfacesThey don’t want to know Observatory detailsHence, not only instrumental signature but also Data Flow signature must be removed

Querator goes in this direction.Querator in two words:

A multi-archive search engine specialised on multi-colour imaging data.

NEON School

Querator (form)Querator (form)

NEON School

Querator (results)Querator (results)

NEON School

Querator: (Filters)Querator: (Filters)

Filters

NEON School

Last but not list: PublicationsLast but not list: Publications

NEON School

Last but not list: PublicationsLast but not list: Publications

NEON School

Last but not list: PublicationsLast but not list: Publications

NEON School

Last but not list: PublicationsLast but not list: Publications

NEON School

Astrophysical Virtual ObservatoryAstrophysical Virtual Observatory

NEON School

RecapRecapYou have seen the exciting day to day life ofYou have seen the exciting day to day life ofa Data Provider.a Data Provider.

URLs:URLs:

Service mode:Service mode:http://www.eso.org/org/dmd/usg/philosched.html

Archive Home Page:Archive Home Page:http://archive.eso.org/http://archive.eso.org/

Querator:Querator:http://archive.eso.org/querator/http://archive.eso.org/querator/