theory in the virtual observatory (tvo)

Theory in the VO, Garching, 7.4.2008

Theory in the Virtual Theory in the Virtual Observatory (TVO)Observatory (TVO)

Goals of Euro-VO DCA WP4Gerard Lemson, GAVOARI-ZAH, HeidelbergMPE, Garching

OverviewOverview

• Recap VO• Why “Theory in the VO”?• Theory in the IVOA

– Simple Numerical Access Protocol

• Intro to this workshop.

Recap VO Recap VO

• Reminder, what is VO about?– “Universe on your desktop”– All astronomical resources online available– Behind friendly interfaces– Interoperable

• What is an “astronomical resource”?– data (all stored results of astronomical

experiments)– software packages (IRAF,AIPS)– (web) services (Simbad, NED)– publications (LANL, ADS) – people (you)

Web helps to access Web helps to access resourcesresources

• Interesting astronomical resources may be– unavailable– unknown – not here– large (the farther away, the larger!)– complex

• Web technologies help:– Discovery: search engines, Google-like or structured– Documentation: HTML – Retrieval: relatively easy access– Filtering: server-side reduction of data streams– Web applications: services as resources

• Main issue, understanding each other...

EsperantoEsperanto

• Standardisation– Discovery (registries)– Data description (“meta-data”) – Data formats (FITS, VOTable)– Protocols– (Web) Application Interfaces– Query language

• Organised in IVOA

VO’s EsperantoVO’s Esperanto

Observations in the VOObservations in the VO

• Most IVOA standardisation efforts concentrate on observational data sets – image archives– source catalogues– spectra

• Standards observationally biased– Sky-based query protocols: cone search, SIAP,

SSAP– Source catalogue combination: ADQL, XMatch– Data models: spectra, STC, characterisation

(sky/time/energy/flux)

Theory in the VO: issuesTheory in the VO: issues

• Good reasons for emphasis on observations– simple observables: photons detected at a certain time

from a certain area on the sky in a certain wavelength interval

– pre-existing (meta-)data format standards (FITS, “csv”)– long history of archiving– valuable over long time (digitising 80yr old plates)

• Simulations not so simple– more varied “observables”: anything that can be

modelled is explicitly there– no standardisation (not even HDF5)– archiving ad hoc, for local use– Moore’s law makes useful lifetime relatively short: few

years later can do better

““Moore’s law” for N-body Moore’s law” for N-body simulationssimulations

Courtesy Simon White

InteroperabilityInteroperability

• Current IVOA standards not always relevant• Distributed resources hard to join

– no common sky– no common objects– no common observables– data models tailored to observations

• Complex data structures, not supported by messaging format standards – AMR, trees, graphs, Voronoi tesselations

• Individual data products often VERY LARGE and not obviously reduced without explicit user interaction.

So why bother?So why bother?

• Simulations are interesting:– For many cases only way to see processes in action– Others can think of science cases you may not have

thought of– Complex observations require sophisticated models for

interpretation

• Bridging gap in specialisations: not everyone has required expertise or resources to create simulations, though they can analyse them.

• Many use cases do not require the latest/greatest – exposure time calculator– survey design

Di Matteo, Springel and Hernquist, 2005

John Hibbard http://www.cv.nrao.edu/~jhibbard/n4038/n4038.html

NASA/CXC/SAO/G. Fabbiano et al.

Toomre & Toomre, 1972

Courtesy Volker Springel

IVOA: Theory Interest IVOA: Theory Interest GroupGroup

• http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/IvoaTheory

• “Provide a forum for discussing theory specific issues in a VO context. “

• Use cases for working groups.• Projects

– Semantics– Micro-simulations– Simple Numerical Access Protocol (SNAP)

SNAPSNAP

• Goal:– create a VO protocol for discovering, querying

and retrieving simulation data – Similar to other S*AP protocols

• Restricted to 3+1D simulations: – At least some common elements– Challenging

• large• complex• diverse• no support in IVOA (compare theory spectra)

Data access protocolsData access protocols

1. Find standard services in registry (say SIAP or SSAP) • Filter on type of service, sky-footprint, wavelength.

2. Query these services using protocol syntax, in general based on location on the sky.• Spectra in a circle on the sky, images overlapping

a certain rectangle• Results in VOTable, providing some metadata per

image/spectrum.3. Retrieve desired results in standardised

format– FITS for images or spectra– VOTable or other XML representation for spectra

or source lists.

SNAP 1: registrySNAP 1: registry

• Different motivations for querying a simulation registry.

– no “interesting patch in the sky”– no object about which more information is

desired– no standard set of variables

• How do we classify simulation archives?

• Need new features for describing SNAP services.

SNAP 2: query protocolSNAP 2: query protocol

• Is it possible to conceive of queries that makes sense for all simulation access services?– No common-sky based simple query to

send to lots of simulation archives

• Need new model to describe simulations and base queries on.

• Less is known, more abstract model.

SNAP Data ModelSNAP Data Model

• Goal: assist in describing and retrieval.• Meta-data model.

– We only know that part of space is evolved in time.– Properties, objects, dimensions, coordinate systemss, units all

flexible.– Compare to (RA/DEC, JD, λ, Flux)

• Should answer common questions about simulations, such as– What type of object is being simulated?– What physics is included?– What “observables” are available?– What are the typical dimensions? – How are the objects represented?– What numerical algorithms were used?

• Support:– “Locate simulations that contain a galaxy cluster of about

1014 Msun, used SPH type hydrodynamics”– etc

PosterBourges et al

SNAP RegistrySNAP Registry

• Difficult to separate steps 1 and 2.• Registries not fine-grained.• Individual institutes may lack expertise to deal

with complex data model.– Metadata describing simulations not easy to fit in “flat”

table. – S*AP-like HTTP GET queries not flexible

• SNAP Registry– Few centers acting as registries for fine grained

simulation data– Registration and browsing interfaces– Evt ADQL query interface based on SNAP data model

SNAP 3: data retrievalSNAP 3: data retrieval

• Often very large datasets.• Need server-side filtering to reduce size of

transferred byte streams:– cut-out (how to decide which part of box?), projection,

gridding, cluster finder, visualisation (full virtual telescopes?)

• What data formats?– FITS, binary VOTable, HDF5?– how about more complex data structures?

• Server side analysis– 2pt correlations, power spectra, density profiles, ...

• For now concentrate on discovery and links to web services.

Theory in the Euro-VO DCATheory in the Euro-VO DCA

• Work package 4: theory in the VO.• Deliverable

– this workshop– whitepaper

A Framework for the inclusion of theory data in the VO

• Theory Experts Group (this workshop’s SAC)

This workshop: goalsThis workshop: goals

• Use cases– Science with TVO-like aspects.– (How) might TVO facilitate work?

• Early implementations• Presentations on VO-like facilities• Discussions• Questionnaire• Whitepaper

This workshop: This workshop: sessionssessions

• 3+1D simulations• micro-simulations• theory-theory interoperability• theory-observational interface• computational infrastructure

Simulation typesSimulation types

• 3+1D simulations– Subject of SNAP

• Overview (V. Springel)• Projects (H. Wozniak, J. Schaye)• VO efforts (R. Wagner, P. Hennebelle)

• Micro-simulations– individually small– different use cases from SNAP-like simulations

• parameter space sampling• MANY parameters• MANY observables• on-line simulations feasible

– Large variety (all speakers)

InteroperabilityInteroperability

• Theory-theory– Not as straightforward as for observations.– Examples

• MODEST (P. Teuben)• Code comparisons (I.Iliev):

– Santa Barbara Cluster Comparison Project– Aspen-Amsterdam Void Finder Comparison Project

• Data reuse (S.Charlot)• Theory-observational

– Assist observers to use theoretical resources (vice versa?).– Use cases.

• survey planning, exposure time calculator• analysis of detailed observations, using detailed models

– Where do theory and observations meet? (Qi Guo)• virtual telescopes (E.Bertin, S. Borgani)• analyse observations as far as possible and compare physical

properties (G.Kauffmann)

Detailed observationsDetailed observations

electron density

gas pressuregas temperature

Courtesy Alexis Finoguenov, Ulrich Briel, Peter Schuecker, (MPE)

Detailed predictionsDetailed predictions

Courtesy Volker Springel

Computational infrastructureComputational infrastructure

• New technologies can assist, or may be required to implement these ideas.

• Real life examples:– Grid (M. Steinmetz, M. Spaans)– Algorithms (L.M.Sarro)– Relational databases (J. Blaizot)– VO aware visualisation tools (G.

Caniglia)

DiscussionsDiscussions

• Address “typical” VO issues– what is it good for, why should I

participate, doesn’t it lead to bad science...?

• Possibly formalised in questionnaire to be sent around to participants after workshop

• Feedback for WP4 whitepaper.

Thank you.Thank you.

Questions IQuestions I

• Which resources are important?– raw simulation results, post-processed, analysis, virtual

observations– services to produce these

• What considerations should we apply to decide what types of resources should be available?– reproducibility, (re-)usability (by ...), ...

• What should accompany resources published on line?– documentation, software readable metadata, help desk– ...

• What scientific content is of most interest?– large vs small

Questions IIQuestions II

• What questions do you want to ask of a registry?– content, methods, physics,

characterisations• What dangers do you see in

publishing resources online?– quality control, bad science

• What reasons do you have to publish resources online?– what reasons to not do this?

theory in the virtual observatory (tvo)

Documents

virtual observatory water_final

ledas, astrogrid and the virtual observatory

the virtual solar-terrestrial observatory the virtual...

russian virtual observatory

virtual observatory --architecture and specifications

virtual observatory single sign-on u.s. national virtual...

sxds database and japanese virtual observatory

the astronomical virtual observatory

canadian virtual observatory project

the astrophysical virtual observatory

the virtual solar-terrestrial observatory*

theory standards for the virtual observatory

japanese virtual observatory (jvo)

the virtual observatory

discovering data in the virtual observatory

the australian virtual observatory

theory in the vo, garching, 7.4.2008 theory in the virtual...

current status of japanese virtual observatory

the virtual observatory europe and the vo: the astrophysical...

data mining and virtual observatory