astro-wise- federation omegacen
Post on 28-Jan-2016
45 Views
Preview:
DESCRIPTION
TRANSCRIPT
Virtual Survey System sept 04
ASTRO-WISE- federation OmegaCEN
ASTRO-WISE- federation OmegaCEN
AstroWise a Virtual Survey System
OmegaCAM – Lofar – AstroGrid –((G)A) VO
AstroWise a Virtual Survey System
OmegaCAM – Lofar – AstroGrid –((G)A) VO
Edwin A. ValentijnEdwin A. Valentijn
Virtual Survey System sept 04
VSTVST
Virtual Survey System sept 04
Paranal Paranal
Mid 2005Mid 2005
20072007
Virtual Survey System sept 04
Virtual Survey System sept 04
Large Data VolumeLarge Data Volume
• Wide-field imaging = vast amounts of data– VST sees equivalent of Southern sky with 0.2” pixels in 3 years. 100 Tbyte of image data and Tbytes of source list data A project like KIDS (1000 Sq Deg) has >10^6 8Mpix raw data
images
• Wide-field imaging = vast amounts of data– VST sees equivalent of Southern sky with 0.2” pixels in 3 years. 100 Tbyte of image data and Tbytes of source list data A project like KIDS (1000 Sq Deg) has >10^6 8Mpix raw data
images
• Handling of the data is non-trivial– Pipeline data reduction– Calibration with very limited resources– Things change in time:
–Physical changes (atmosphere, various gains)–Code (new methods, bugs)–Human insight in changes
– Working with source lists
Science can only be archive based
• Handling of the data is non-trivial– Pipeline data reduction– Calibration with very limited resources– Things change in time:
–Physical changes (atmosphere, various gains)–Code (new methods, bugs)–Human insight in changes
– Working with source lists
Science can only be archive based
Virtual Survey System sept 04
ASTRO-WISE Mission Virtual Survey SystemASTRO-WISE Mission Virtual Survey System
• Environment that provides systematic and controlled– Access to all raw and calibration data– Execution and modification image/calibration pipelines– Execution of source extraction algorithms- catalogues– Archiving or regenerate on the fly dynamically – Paradigm: no static data releases but dynamic on request data– federated to link different data centers
• Environment that provides systematic and controlled– Access to all raw and calibration data– Execution and modification image/calibration pipelines– Execution of source extraction algorithms- catalogues– Archiving or regenerate on the fly dynamically – Paradigm: no static data releases but dynamic on request data– federated to link different data centers
• Dynamical archive continuously grows, can be used for – small or large science projects– generating and checking calibration data– exchanging methods, scripts and configuration
• Dynamical archive continuously grows, can be used for – small or large science projects– generating and checking calibration data– exchanging methods, scripts and configuration
raw pixel data pipelines/cal files catalogues raw pixel data pipelines/cal files catalogues
Virtual Survey System sept 04
Keys -SolutionKeys -Solution
• Procedurizing– Data taking at telescope for both science and
calibration data• Observing Modes: —Stare —Jitter —Dither —SSO
• Observing Strategies: —Stan —Deep —Freq —Mosaic
– Full integration with data reduction– Design- ADD – Data model (classes) defined for data reduction and
calibration– View pipeline as an administrative problem
• Procedurizing– Data taking at telescope for both science and
calibration data• Observing Modes: —Stare —Jitter —Dither —SSO
• Observing Strategies: —Stan —Deep —Freq —Mosaic
– Full integration with data reduction– Design- ADD – Data model (classes) defined for data reduction and
calibration– View pipeline as an administrative problem
Virtual Survey System sept 04
Calibration proceduresCalibration procedures
Sanity checksSanity checks
Quality controlQuality controlCalibration proceduresCalibration procedures
Image pipelineImage pipeline
Source pipelineSource pipeline
Virtual Survey System sept 04
AstroWise PipelinesAstroWise Pipelines
Photometric pipelinePhotometric pipeline
Bias pipeline
Flatfield pipeline
Image pipeline
Source pipeline
Virtual Survey System sept 04
DB – engineDB – engine
Long search: Objectivity, CERN HEP July 2002 Oracle contract
– reference – licenses - consultancy
Oracle 9i -> 10g:- Full oo support + Python to db persistency- Terabyte scalability through “partitioned tables”- Administration tools- federation through “Advanced replication” to evolve into cross
site links (=references= pointers)– Interoperability: STREAMS connects to others: SyBase, MySQL– Python I/F SQL – OCI- Oracle db (Python binding)
Long search: Objectivity, CERN HEP July 2002 Oracle contract
– reference – licenses - consultancy
Oracle 9i -> 10g:- Full oo support + Python to db persistency- Terabyte scalability through “partitioned tables”- Administration tools- federation through “Advanced replication” to evolve into cross
site links (=references= pointers)– Interoperability: STREAMS connects to others: SyBase, MySQL– Python I/F SQL – OCI- Oracle db (Python binding)
Virtual Survey System sept 04
Concepts of federationConcepts of federation
• Federation maintained by a single database- Oracle9i• Full history tracking by linking (joints, references)
– of all input that went into result – providing on-the fly reprocessing
• Dynamical archive - Context as object attributes– Project: Calibration, Science, Survey, Personal– Owner: Pipeline, Developer, User– Strategy: —Standard —Deep —Freq — Mosaïc– Mode: —Stare —Jitter — Dither —SSO
– Time: Time stamping VO I/F Publish• Software standards
– Classes/data model/procedures– 00 – inheritance/ persistency
– Python scripts/ c-libraries USER Python
• Federation maintained by a single database- Oracle9i• Full history tracking by linking (joints, references)
– of all input that went into result – providing on-the fly reprocessing
• Dynamical archive - Context as object attributes– Project: Calibration, Science, Survey, Personal– Owner: Pipeline, Developer, User– Strategy: —Standard —Deep —Freq — Mosaïc– Mode: —Stare —Jitter — Dither —SSO
– Time: Time stamping VO I/F Publish• Software standards
– Classes/data model/procedures– 00 – inheritance/ persistency
– Python scripts/ c-libraries USER Python
Virtual Survey System sept 04
Contents of federationContents of federation
• Raw data– Observed images– Ancillary information
• Calibration results– Calibration files time stamped
• Reduced images– Single observation– Coadded images
• Software– Methods (pipelines) for processing calibration– Configuration files
• Source lists – catalogues– Extracted source information– Associated among different data objects
• Raw data– Observed images– Ancillary information
• Calibration results– Calibration files time stamped
• Reduced images– Single observation– Coadded images
• Software– Methods (pipelines) for processing calibration– Configuration files
• Source lists – catalogues– Extracted source information– Associated among different data objects
SourceList
RA0, Dec0
t0
V filterSource
ListV filter
t2
RA0?, Dec0?
SourceList
V filterRA0?, Dec0?
t1
SS
C
I
TES
A
A
O
B filterRA0, Dec0
SourceList
SourceList
V filterRA0, Dec0
SS
C
I
TES
A
A
OS
S
C
I
TES
A
A
O
SourceList
B filterRA0, Dec0
SourceList
B filterRA1, Dec1
SourceList
V filterRA0, Dec0
SourceList
B filterRA1, Dec1
SourceList
B filterRA0, Dec0
SourceListB - VRA0, Dec0
SourceList
V filterRA0, Dec0
t0
SourceList
V filterRA0?, Dec0?
t1
SourceList
V filter
t2
RA0?, Dec0?
SourceList
V filterRA0?, Dec0?
tn
Proper motion
Red quasars
Supernovae
Tbyte source lists brains make the associations
Tbyte source lists brains make the associations
Link -lists
as fast as possible
Virtual Survey System sept 04
StatusStatus
• From abstract design to working prototype FIRST VIRTUAL LIGHT
17 April 2003
– WFI@2.2– INT
• Federated – USM– OAC
• Summer 2004
– First massive ingest
• From abstract design to working prototype FIRST VIRTUAL LIGHT
17 April 2003
– WFI@2.2– INT
• Federated – USM– OAC
• Summer 2004
– First massive ingest
Virtual Survey System sept 04
VST - Virtual Survey TelescopeVST - Virtual Survey Telescope
Virtual Survey System sept 04
New componentsNew components• Python binding to Oracle – 5LS ۷• Scripting paradigm-۷
– Wrappers– Make metaphor– Persistency, links as attributes
• Fileserver ۷• Parallelisation code ۷• Catalogue class/ associating HTM ۷• Federating with Oracle Streams• User interfaces
– Tell me everything tool<- >on the fly reprocessing
• Python binding to Oracle – 5LS ۷• Scripting paradigm-۷
– Wrappers– Make metaphor– Persistency, links as attributes
• Fileserver ۷• Parallelisation code ۷• Catalogue class/ associating HTM ۷• Federating with Oracle Streams• User interfaces
– Tell me everything tool<- >on the fly reprocessing
Virtual Survey System sept 04
New technologies-paradigmNew technologies-paradigm
• raw data centralized – on the fly reprocessing = VSS
• sociology- project management• ++ • Python at top layer(s) = VSS • parallel – IP - seti@home• user tunable pipelines = VSS• ++ db• 5 LS – 5 lines script = VSS• Distribution – VO = VSS
• raw data centralized – on the fly reprocessing = VSS
• sociology- project management• ++ • Python at top layer(s) = VSS • parallel – IP - seti@home• user tunable pipelines = VSS• ++ db• 5 LS – 5 lines script = VSS• Distribution – VO = VSS
Virtual Survey System sept 04
development issuesdevelopment issues
• Indexing Terabytes on the sky• Associating entries • World wide linking entries/distribution
– (referencing) – registries– centralized – replication – p2p– Other p2p’s?
• Visualisation - imaging • Visualisation -multidimensional data• VO world - Euro-VO
• Indexing Terabytes on the sky• Associating entries • World wide linking entries/distribution
– (referencing) – registries– centralized – replication – p2p– Other p2p’s?
• Visualisation - imaging • Visualisation -multidimensional data• VO world - Euro-VO
Virtual Survey System sept 04
top related