astrowise((g)a)vo meeting 6 may 2004 astro-wise- federation omegacen astrowise a virtual survey...
TRANSCRIPT
AstroWise((G)A)VO Meeting 6 May 2004
ASTRO-WISE- federation OmegaCEN
ASTRO-WISE- federation OmegaCEN
AstroWise a Virtual Survey System
OmegaCAM – Lofar – AstroGrid –((G)A) VO
AstroWise a Virtual Survey System
OmegaCAM – Lofar – AstroGrid –((G)A) VO
Edwin A. ValentijnEdwin A. Valentijn
AstroWise((G)A)VO Meeting 6 May 2004
((G)A)VO meeting((G)A)VO meeting
• Goal: Connect raw data to end user in VO environment– Exchange views on design level– 3 years after– Design – bound / unbound– Review the layers
• Can we build a working near-VO system? – OmegaCAM VISTA Lofar SExtractor VOT GAVO AVO
• Goal: Connect raw data to end user in VO environment– Exchange views on design level– 3 years after– Design – bound / unbound– Review the layers
• Can we build a working near-VO system? – OmegaCAM VISTA Lofar SExtractor VOT GAVO AVO
AstroWise((G)A)VO Meeting 6 May 2004
Programme – 2 roundsProgramme – 2 rounds
• AstroWise– Vision - Edwin– Implementation – Object maker– Danny– Prototype – demo - Ewout– Object viewer - Michiel
• AstroGrid• AVO• Sextractor - AVO
• GAVO
• AstroWise– Vision - Edwin– Implementation – Object maker– Danny– Prototype – demo - Ewout– Object viewer - Michiel
• AstroGrid• AVO• Sextractor - AVO
• GAVO
AstroWise((G)A)VO Meeting 6 May 2004
VSTVST
AstroWise((G)A)VO Meeting 6 May 2004
Paranal Paranal
Mid 2005Mid 2005
+ 3 year+ 3 year
AstroWise((G)A)VO Meeting 6 May 2004
LofarLofar
IBM- Blue Gene/LIBM- Blue Gene/L
AstroWise((G)A)VO Meeting 6 May 2004
Large Data VolumeLarge Data Volume
• Wide-field imaging instruments, vast amounts of data– VST sees equivalent of Southern sky 30 min exp, 300 nights/y in 3 years. 100 Tbyte of image data and Tbytes of source list data
• Wide-field imaging instruments, vast amounts of data– VST sees equivalent of Southern sky 30 min exp, 300 nights/y in 3 years. 100 Tbyte of image data and Tbytes of source list data
• Handling of the data is non-trivial– Pipeline data reduction– Calibration and re-calibration– Image comparisons and combinations– Working with source lists– Visualization
• Science can only be archive based
• Handling of the data is non-trivial– Pipeline data reduction– Calibration and re-calibration– Image comparisons and combinations– Working with source lists– Visualization
• Science can only be archive based
ESOcompliantESOcompliant}}
AstroWise((G)A)VO Meeting 6 May 2004
ASTRO-WISE Mission Virtual Survey SystemASTRO-WISE Mission Virtual Survey System
• Environment that provides systematic and controlled– Access to all raw and calibration data– Execution and modification image/calibration pipelines– Execution of source extraction algorithms- catalogues– Archiving or regenerate dynamically– federated to link different data centers
• Environment that provides systematic and controlled– Access to all raw and calibration data– Execution and modification image/calibration pipelines– Execution of source extraction algorithms- catalogues– Archiving or regenerate dynamically– federated to link different data centers
• Dynamical archive continuously grows, can be used for – small or large science projects– generating and checking calibration data– exchanging methods, scripts and configuration
• Dynamical archive continuously grows, can be used for – small or large science projects– generating and checking calibration data– exchanging methods, scripts and configuration
raw pixel data pipelines/cal files catalogues raw pixel data pipelines/cal files catalogues
AstroWise((G)A)VO Meeting 6 May 2004
Keys -SolutionKeys -Solution
• Procedurizing– Data taking at telescope for both science and
calibration data• Observing Modes: —Stare —Jitter —Dither —SSO
• Observing Strategies: —Stan —Deep —Freq —Mosaic
– Full integration with data reduction– Design- ADD – Data model (classes) defined for data reduction and
calibration– View pipeline as an administrative problem
• Procedurizing– Data taking at telescope for both science and
calibration data• Observing Modes: —Stare —Jitter —Dither —SSO
• Observing Strategies: —Stan —Deep —Freq —Mosaic
– Full integration with data reduction– Design- ADD – Data model (classes) defined for data reduction and
calibration– View pipeline as an administrative problem
AstroWise((G)A)VO Meeting 6 May 2004
Monitoring Photometric CalibrationMonitoring Photometric Calibration
AstroWise((G)A)VO Meeting 6 May 2004
Calibration proceduresCalibration procedures
Sanity checksSanity checks
Quality controlQuality controlCalibration proceduresCalibration procedures
Image pipelineImage pipeline
Source pipelineSource pipeline
AstroWise((G)A)VO Meeting 6 May 2004
AstroWise PipelinesAstroWise Pipelines
Photometric pipelinePhotometric pipeline
Bias pipeline
Flatfield pipeline
Image pipeline
Source pipeline
SourceList
RA0, Dec0
t0
V filterSource
ListV filter
t2
RA0?, Dec0?
SourceList
V filterRA0?, Dec0?
t1
SS
C
I
TES
A
A
O
B filterRA0, Dec0
SourceList
SourceList
V filterRA0, Dec0
SS
C
I
TES
A
A
OS
S
C
I
TES
A
A
O
SourceList
B filterRA0, Dec0
SourceList
B filterRA1, Dec1
SourceList
V filterRA0, Dec0
SourceList
B filterRA1, Dec1
SourceList
B filterRA0, Dec0
SourceListB - VRA0, Dec0
SourceList
V filterRA0, Dec0
t0
SourceList
V filterRA0?, Dec0?
t1
SourceList
V filter
t2
RA0?, Dec0?
SourceList
V filterRA0?, Dec0?
tn
Proper motion
Red quasars
Supernovae
Tbyte source lists brains make the associations
Tbyte source lists brains make the associations
Link -lists
as fast as possible
AstroWise((G)A)VO Meeting 6 May 2004
DB – engineDB – engine
Oracle contract:– reference – licenses - consultancy
Oracle 9i -> 10g:- federation through “Advanced replication” to evolve into cross
site links (=references= pointers)- Python I/F SQL – OCI- Oracle db- Full oo support + Python to db persistency- Terabyte scalability through “partitioned tables”- Administration tools– Interoperability: STREAMS connects to others: SyBase, MySQL
Oracle contract:– reference – licenses - consultancy
Oracle 9i -> 10g:- federation through “Advanced replication” to evolve into cross
site links (=references= pointers)- Python I/F SQL – OCI- Oracle db- Full oo support + Python to db persistency- Terabyte scalability through “partitioned tables”- Administration tools– Interoperability: STREAMS connects to others: SyBase, MySQL
AstroWise((G)A)VO Meeting 6 May 2004
Share the loadAstroWise Survey System
Share the loadAstroWise Survey System • Processing
– Hardware • Beowulf processors – 32 (most cases)• Multi Terabyte disks (10 – 100)
– Data reduction• Derive calibration• Run image pipeline (1 Mpx/s)
• Processing– Hardware
• Beowulf processors – 32 (most cases)• Multi Terabyte disks (10 – 100)
– Data reduction• Derive calibration• Run image pipeline (1 Mpx/s)
• Archiving– Storage
• Images (100’s Tbyte), Calibration files (10 Tbyte)• Source parameters (1-10 Tbyte)
– Federate (network speed)• 5 Mb/s (24 hours/day) full replication (MegaCAM 10 Mb/s)• 200 Mb/s no replication, on-the-fly retrieval
AstroWise((G)A)VO Meeting 6 May 2004
Contents of federationContents of federation
• Raw data– Observed images– Ancillary information
• Calibration results– Calibration files time stamped
• Reduced images– Single observation– Coadded images
• Software– Methods (pipelines) for processing calibration– Configuration files
• Source lists – catalogues– Extracted source information– Associated among different data objects
• Raw data– Observed images– Ancillary information
• Calibration results– Calibration files time stamped
• Reduced images– Single observation– Coadded images
• Software– Methods (pipelines) for processing calibration– Configuration files
• Source lists – catalogues– Extracted source information– Associated among different data objects
AstroWise((G)A)VO Meeting 6 May 2004
Concepts of federationConcepts of federation
• Federation maintained by a single database- Oracle9i• Full history tracking
– of all input that went into result – providing on-the fly reprocessing
• Dynamical archive - Context as object attributes– Project: Calibration, Science, Survey, Personal– Owner: Pipeline, Developer, User– Strategy: —Standard —Deep —Freq — Mosaïc– Mode: —Stare —Jitter — Dither —SSO
– Time: Time stamping VO interface• Software standards
– Classes/data model/procedures– 00 – inheritance/ persistency
– Python scripts/ c-libraries USER Python
• Federation maintained by a single database- Oracle9i• Full history tracking
– of all input that went into result – providing on-the fly reprocessing
• Dynamical archive - Context as object attributes– Project: Calibration, Science, Survey, Personal– Owner: Pipeline, Developer, User– Strategy: —Standard —Deep —Freq — Mosaïc– Mode: —Stare —Jitter — Dither —SSO
– Time: Time stamping VO interface• Software standards
– Classes/data model/procedures– 00 – inheritance/ persistency
– Python scripts/ c-libraries USER Python
AstroWise((G)A)VO Meeting 6 May 2004
How to use the system?How to use the system?
UI: Ingest raw data
UI: Run standard pipelines – GUI
UI: Modify pipeline/modules
UI: Make/associate source lists
5LS: Query – whatever
UI: Ingest raw data
UI: Run standard pipelines – GUI
UI: Modify pipeline/modules
UI: Make/associate source lists
5LS: Query – whatever
AstroWise((G)A)VO Meeting 6 May 2004
VST - Virtual Survey TelescopeVST - Virtual Survey Telescope
AstroWise((G)A)VO Meeting 6 May 2004
What ? science ?What ? science ?
Collaborate - EU-wide- Projects
- user defined qualifications
- Surveys = projects
- Archive studies = Surveys =projects
Collaborate - EU-wide- Projects
- user defined qualifications
- Surveys = projects
- Archive studies = Surveys =projects
AstroWise((G)A)VO Meeting 6 May 2004
New componentsNew components
• Python binding to Oracle – 5LS ۷• Fileserver ۷• Parallelisation code ۷• Catalogue class/ associating HTM ۷• Federating with Oracle Streams• User interfaces
– Tell me everything tool<- >on the fly reprocessing
• Python binding to Oracle – 5LS ۷• Fileserver ۷• Parallelisation code ۷• Catalogue class/ associating HTM ۷• Federating with Oracle Streams• User interfaces
– Tell me everything tool<- >on the fly reprocessing
AstroWise((G)A)VO Meeting 6 May 2004
ScheduleSchedule
• Hardware– OmegaCEN: 6Tbyte + servers + 8 node cluster
operational ۷– OAC , USM , Terapix + CVS federation ۷
• Software– Design – review Q2 2002- ۷– Design – formalized July 2003 ۷– WFI +INT Prototype operational Q2 2003 ۷– Basic operations – Q4 2003۷– Evaluate and prepare for mass production 2004– Qualify and populate 2005– Deliver survey system – satellites
• Hardware– OmegaCEN: 6Tbyte + servers + 8 node cluster
operational ۷– OAC , USM , Terapix + CVS federation ۷
• Software– Design – review Q2 2002- ۷– Design – formalized July 2003 ۷– WFI +INT Prototype operational Q2 2003 ۷– Basic operations – Q4 2003۷– Evaluate and prepare for mass production 2004– Qualify and populate 2005– Deliver survey system – satellites