vstcen data management divisionnapolita/vstmeeting/vstcen-dmd_vst... · 2009. 8. 24. · quality...

VSTceN-DMD Napoli 20-07-2009

VSTceN

Data Management

Division

A. Grado

INAF


VSTceN-DMDGoals

� Data processing hardware infrastructure� Data processing and data storage

� Data processing software infrastructure � Pipelines and data archive

� Coordination of data reduction � Users support

INAF


VSTceN-DMD

WF related projects

Wide field HW/SW related projects since the VST start:

4 projects for SW development

4 projects for HW/GRID R&D/procurement

3 projects for SW development

INAF


VSTceN-DMD

WF related projects INAF

1998 - OACDF (OAC � pilot project at [email protected] )1999 - WFIX (OAC-Terapix � tools used in Terapix and Bonn pipeline)2000 - BEOWFI (OAC-ESO � photometric calib pipeline of [email protected]

running on a beowulf cluster)2001 - COFIN Capaccioli (PartenoBEO � Beowulf procurement )2001 - AstroWise (OmegaCen NL, Terapix, ESO, USM, OAC � QC on

pre-reduction, variability tool)2001 - FIRB-GRID Wp10 (OAP, OAT, OAC � WF pre-reduction on grid)2002 - COFIN-GRID (OAT, OAP, OAC � tests on Beowulf in grid)2004 - COFIN Capaccioli (Data Storage procurement)2005 - AstromC (OAC � WFIX porting to C++, used in LBC pipeline)2005 - VSTarchive (VSTceN � archive SW for VST images)2006 - Efits (VSTceN-Sternberg Institute � C library for fits files) 2006 - VST-Tube (OAC-VSTceN � flexible pipeline for WF images )


VSTceN-DMD

VST data flow/volumeVSTceN GoT: ~ 60 nights/yearKids: ~ 40 nights/year/node

Disk shipping trough Diplo-bag each 4 working days

Data transferred to Naples via network.

Each night: 50 ÷ 170 GB (16 bits not compressed, 2 bands/nights)

INAF


VSTceN-DMD

Available HW

Image processing: 80 cores (2 concurrent users)

Data analysis and numerical simulations: 32 CPU

Data storage: 80 TB (very high quality) (enough for 2 years of VST operation)

New cluster (80 cores) and data storage extension (65 TB) installed on May 2009 with Regione Campania funds


VSTceN HW infrastructuresEMC NS704 schematic view

The system is fully redundant (no single point of failure)

The system is remotely controlled by the EMC, intervention in 4 hours in case of failure.

Presently populated with 77 TB disks (raw capacity)

13 CPU just to control the disks!!


VSTceN-DMD SW

VST data handling

needs implemented

Archive raw data and pipeline products AW, VSTarchive

Process images up to science grade level 2 pipelines (AW, VST-Tube)

Provide Quality Control AW, QualityFits

INAF


VSTceN-DMD AstroWise

An FP5 RTD EC funded Project

AW goal: develop and maintain a computational environment to process wide-field imaging data

AW is: a pipeline embedded in a data base using an object oriented like approach

AW official release: November 2006 (see www.astro-wise.org)



The AstroWise idea

AstroWise has been designed to operate in a multi-node environment and to trace the reduction history of large and distributed projects (e.g. KIDS)

� Raw pixel data� Pipelines/cal files All integrated in one information � Catalogues system

This implies a complex DB administration and SW structure



The AstroWise features

� distributed services� multiple instrument support� processing GRID / Storage DISTRIBUTED� driven by DB query of user� publish directly in EURO-VO� easy to go from source in catalog back to pixel



The federate AstroWise environment

Official pipeline for KIDS

Napoli-node has contributed:

quality control pre-reduction

data analysis (variability tool)Scamp in AW (recently)

Planned an Astro-Wise course for the national community


VSTceN-DMD SWVST-tube

Complementary pipeline to Astro-Wisedeveloped completely in Naples.

Why?

� Pipeline working without DB� Easy to adapt to peculiar data reduction

model� Easy for new algorithms testing � Data reduction cross check

Unique characteristics: e.g. true noise map propagation



It is a pipeline but also a collection of tools

� aperture and PSF (S. Zaggia) photometry � Illumination correction surface tool� MAgliM (limiting magnitude vs magnitude and surface brightness � Mask (automatic halo/spike mask)� VSTSim (images simulator)



Automatic spikes and halos detection (ZhuoyiHuang)


VSTceN-DMD Processing capability3 tough examples - versatility

VST-Tube adapted for the reduction of CFHT12k images (VIRMOS collaboration)

4 deg2 distributed over 51 nights

(Lamareille et al. 2009 A&A 495,53)

Photometric comparison with SDSS as function of celestial coordinates Typical σ < 0.1 mag

∆mag

(sds

s–

us)


VSTceN-DMD Processing capability 3 tough examples – astrometry

Globular cluster NGC2808 4 deg2 observed with [email protected] in V and I band

V-I

V

PSF photometry on coaddedmosaics for a total of 16 pointings:

Completely automatic reduction and catalog extraction in collaboration with S. Zaggia (INAF-OAPd).

S. Zaggia, A. Grado et al. , Tidal Tails in Globular Clusters.The case of NGC2808(A&A, in preparation)

color turn-off stars σ < ± 0.02 mag


VSTceN-DMD Processing capability 3 tough examples - photometric accuracy

NGC3115 observed with [email protected]

V-I

V

Surface photometry of NGC3115

With plates (Capaccioli et al. 1987 ) exp.time = 34 Ks

With WFI + VST-Tube (Grado et al. in preparation)

exp.time = 4.2 Ks

1 2 3 4 5 6 7

18

20

22

24

26

28

30

32

(distance from center)1/4

Sur

face

brig

htne

ss (

mag

arc

sec−

2 )

(distance from center)1/4

Sur

face

brig

htne

ss (

mag

arcs

ec-2

)


VST-DMDSW

Other SW products

- VSTceN data Archive

- AstromC : sw for astrometric calibration (used also for LBC pipeline)

- EFits : new Fits library with noise propagation map A. Grado, M. Pavlov and L. Limatola, 2007, "Efits: A New Efficient and Flexible FITS Library ", The 2007 ESO Instrument Calibration Workshop. Ed. Springer, p. 217

- Fitshdr :Fits header manipulation- Astrobench : hw benchmark for astronomical applications


Example of Efits weight map

NGC4697

mosaic obtained with [email protected] data

Section of the weight map


VSTceN-DMD Cost breakdown

Costs since VSTceN start activities

HW Total cost 222.725 €

Cofin Capaccioli

Regione Campania

INAF

AW

PremioFondazioneCorteseBanco Napoli

SW total cost164.450 €

Cofin Capaccioli

Regione Campania

INAF

AW

Premio FondazioneCorteseBanco Napoli


VSTceN-DMD

Conclusions

• The hardware is in place and working

• AW is available (federated DB under set-up)

• The VST-archive is ready

• VST-Tube is currently used for science projects

• VST-Tube 2.0 with Efits under test

INAF


VSTceN-DMDInvolved people Activities

2 - VST-Tube

6 - Efits

1 - Astro-Wise

4 - Hardware

* Sternberg Institute Moscow

** now at OAT

3 - DB admin

Bold-underlined-blue are not staff

8 - VSTarchive

A. Grado1,2,4,6,8

F. Getman1,3.4

L. Limatola2,6,8

M. Radovich1,5,7

M. Pavlov*6

E. Puddu1

R. Silvotti**1

J. Alcala’1

S. Zaggia2

5 - AstromC

7 - Photcal

vstcen data management divisionnapolita/vstmeeting/vstcen-dmd_vst... · 2009. 8. 24. · quality...

Documents