astro-wise- federation omegacen

AstroWise((G)A)VO Meeting 6 May 2004

ASTRO-WISE- federation OmegaCEN

AstroWise a Virtual Survey System

OmegaCAM – Lofar – AstroGrid –((G)A) VO

Edwin A. Valentijn


((G)A)VO meeting• Goal: Connect raw data to end user in VO

environment– Exchange views on design level– 3 years after– Design – bound / unbound– Review the layers

• Can we build a working near-VO system? – OmegaCAM VISTA Lofar SExtractor VOT GAVO AVO


Programme – 2 rounds• AstroWise

– Vision - Edwin– Implementation – Object maker– Danny– Prototype – demo - Ewout– Object viewer - Michiel

• AstroGrid• AVO• Sextractor - AVO• GAVO


VST


Paranal

Mid 2005

+ 3 year


Lofar

IBM- Blue Gene/L


Large Data Volume• Wide-field imaging instruments, vast amounts of data

– VST sees equivalent of Southern sky 30 min exp, 300 nights/y in 3 years. 100 Tbyte of image data and Tbytes of source list data

• Handling of the data is non-trivial– Pipeline data reduction– Calibration and re-calibration– Image comparisons and combinations– Working with source lists– Visualization

• Science can only be archive based

ESOcompliant}


ASTRO-WISE Mission Virtual Survey System

• Environment that provides systematic and controlled– Access to all raw and calibration data– Execution and modification image/calibration pipelines– Execution of source extraction algorithms- catalogues– Archiving or regenerate dynamically– federated to link different data centers

• Dynamical archive continuously grows, can be used for – small or large science projects– generating and checking calibration data– exchanging methods, scripts and configuration

raw pixel data pipelines/cal files catalogues


Keys -Solution

• Procedurizing– Data taking at telescope for both science and

calibration data• Observing Modes: —Stare —Jitter —Dither —SSO• Observing Strategies: —Stan —Deep —Freq —Mosaic

– Full integration with data reduction– Design- ADD – Data model (classes) defined for data reduction and

calibration– View pipeline as an administrative problem


Monitoring Photometric Calibration


Calibration procedures

Sanity checks

Quality controlCalibration procedures

Image pipeline

Source pipeline


AstroWise Pipelines

Photometric pipeline

Bias pipeline

Flatfield pipelineImage pipeline

Source pipeline

SourceList

RA0, Dec0

t0

V filterSource

ListV filter

t2

RA0?, Dec0?

SourceList

V filterRA0?, Dec0?

t1

SS

C

I

TES

A

A

O

B filterRA0, Dec0

SourceList

SourceList

V filterRA0, Dec0

SS

C

I

TES

A

A

OS

S

C

I

TES

A

A

O

SourceList

B filterRA0, Dec0

SourceList

B filterRA1, Dec1

SourceList

V filterRA0, Dec0

SourceList

B filterRA1, Dec1

SourceList

B filterRA0, Dec0

SourceListB - VRA0, Dec0

SourceList

V filterRA0, Dec0

t0

SourceList

V filterRA0?, Dec0?

t1

SourceList

V filter

t2

RA0?, Dec0?

SourceList

V filterRA0?, Dec0?

tn

Proper motion

Red quasars

Supernovae

Tbyte source lists brains make the associations

Link -lists

as fast as possible


DB – engine

Oracle contract:– reference – licenses - consultancy

Oracle 9i -> 10g:- federation through “Advanced replication” to evolve into cross

site links (=references= pointers)- Python I/F SQL – OCI- Oracle db- Full oo support + Python to db persistency- Terabyte scalability through “partitioned tables”- Administration tools– Interoperability: STREAMS connects to others: SyBase, MySQL


Share the loadAstroWise Survey System • Processing

– Hardware • Beowulf processors – 32 (most cases)• Multi Terabyte disks (10 – 100)

– Data reduction• Derive calibration• Run image pipeline (1 Mpx/s)

• Archiving– Storage

• Images (100’s Tbyte), Calibration files (10 Tbyte)• Source parameters (1-10 Tbyte)

– Federate (network speed)• 5 Mb/s (24 hours/day) full replication (MegaCAM 10 Mb/s)• 200 Mb/s no replication, on-the-fly retrieval


Contents of federation• Raw data

– Observed images– Ancillary information

• Calibration results– Calibration files time stamped

• Reduced images– Single observation– Coadded images

• Software– Methods (pipelines) for processing calibration– Configuration files

• Source lists – catalogues– Extracted source information– Associated among different data objects


Concepts of federation• Federation maintained by a single database- Oracle9i• Full history tracking

– of all input that went into result – providing on-the fly reprocessing

• Dynamical archive - Context as object attributes– Project: Calibration, Science, Survey, Personal– Owner: Pipeline, Developer, User– Strategy: —Standard —Deep —Freq — Mosaïc– Mode: —Stare —Jitter — Dither —SSO– Time: Time stamping VO interface

• Software standards– Classes/data model/procedures– 00 – inheritance/ persistency– Python scripts/ c-libraries USER Python


How to use the system?UI: Ingest raw data

UI: Run standard pipelines – GUI

UI: Modify pipeline/modules

UI: Make/associate source lists

5LS: Query – whatever


VST - Virtual Survey Telescope


What ? science ?Collaborate - EU-wide- Projects

- user defined qualifications

- Surveys = projects

- Archive studies = Surveys =projects


New components

• Python binding to Oracle – 5LS ۷• Fileserver ۷• Parallelisation code ۷• Catalogue class/ associating HTM ۷• Federating with Oracle Streams• User interfaces

– Tell me everything tool<- >on the fly reprocessing


Schedule• Hardware

– OmegaCEN: 6Tbyte + servers + 8 node cluster operational ۷

– OAC , USM , Terapix + CVS federation ۷• Software

– Design – review Q2 2002- ۷– Design – formalized July 2003 ۷– WFI +INT Prototype operational Q2 2003 ۷– Basic operations – Q4 2003۷– Evaluate and prepare for mass production 2004– Qualify and populate 2005– Deliver survey system – satellites

astro-wise- federation omegacen

Documents

data reductiondesign

tbyte of image data

data model classes

calibration dataexecution

vo system

calibrationview pipeline

replication megacam

advanced replication