NVO Summer School, Aspen Sept 14 2004 1

Data Access Layer Working GroupImage and Spectral Access

Doug TodyNational Radio Astronomy Observatory

National VIRTUAL OBSERVATORY

NVO Summer School, Aspen Sept 14 2004 2

DAL Services

Dataset

Time Series

Table Source Catalog

Event List

Visibility Data

Image NDImage

1D Spectrum

SED

NVO Summer School, Aspen Sept 14 2004 3

Simple Image Access (SIA)

• Provides access to "image" data– (instead of spectrum, catalog, etc.)– regularly sampled (pixelated) data– generally an image of the sky, with a WCS

• Service-oriented data discovery– query service to discover data

• Access to image metadata– can get image metadata without retrieving the actual

image– uniform description based on standard data models

• Access to image datasets– data may be virtual or computed on demand– uniform interface to any type of image data

NVO Summer School, Aspen Sept 14 2004 4

SIA - Basic Usage

• Simplest possible query– HTTP GET, e.g., http://myvo.nvo.org/SIAPServlet?POS=12.0,10.0&SIZE=0.2

– similar to cone search but ROI is a rectangle defining the ideal image coverage on the sky, not merely a search region

• Query response– VOTable describing images, one image per row– for each image there is an access reference, a URL

• Get Data– simply fetch the image at the given URL

• SIA is deceptively simple! It can do a lot more than is apparent, but simple usage should be kept simple.

NVO Summer School, Aspen Sept 14 2004 5

SIA - Interface Concepts

• Service protocol independent– URL (REST), WS, ADQL all implement the same interface

model– document-oriented, pass through

• Relational query model– relational: flat table, relationships inferred from metadata– generally it is up to the client to refine the query

• Uniform access to data– atlas, cutout, etc., images all treated the same– mediation to a standard data model (partial currently)

• Image service types– cutout, mosaic, atlas, pointed

NVO Summer School, Aspen Sept 14 2004 6

SIA - Interface Concepts

• Virtual data– most VO data analysis uses virtual data– virtual data is generated on the fly by the service– may involve subsetting, filtering, transformation,

analysis

• Data Model-based data access– addresses problem of heterogeneous data– allows disparate software to share the same data– same physical data can be viewed via different

models, e.g., image or spectrum

NVO Summer School, Aspen Sept 14 2004 7

SIA - Interface Summary

• Query– Simple positional query

• POS, SIZE– Image FORMAT

• FITS, graphic, HTML, metadata

• Image generation parameters– fully specify projection on the sky but simplify FITS WCS– naxis, cframe, equinox, crpix, crval, cdelt, rotang, proj– defaults are derived from POS, SIZE

• Others– Intersect (covers, enclosed, center, overlaps)– service defined, e.g., filter or bandpass name

NVO Summer School, Aspen Sept 14 2004 8

SIA - Interface Summary

• Query response– Response is a VOTable– One candidate image per table row– Includes standard metadata, including WCS

• title, date, pos, naxes, naxis, scale, format, etc.

– FITS WCS parameters• subset but includes CD matrix

– spectral bandpass metadata– service processing metadata

• did service interpolate pixels?

NVO Summer School, Aspen Sept 14 2004 9

SIA - Interface Summary

• Get Data– The image "access reference", a URL, is

used to fetch the dataset– URL often points to a service which

generates data on-the-fly (OTF)• e.g., image cutout or mosaic

– A separate get is required for each image– Note the query and get may be performed

by different clients• multiple get operations may proceed concurrently• Use of URL permits caching of images

NVO Summer School, Aspen Sept 14 2004 10

SIA - Interface Summary

• Staging Data– Included in SIA interface design, but not yet implemented

– Asynchronous staging of data is required for large computations

• e.g., a large image mosaic, or generation of 10000 cutouts

– Interface• same as for synchronous image access (same query, getData)• adds accessImage method, messaging, polling, multiple

clients• third party delivery possible, including MySpace

NVO Summer School, Aspen Sept 14 2004 11

SIA - Future Work

• Advanced queries– query on additional image metadata, e.g., collection,

bandpass, time– syntactical queries (ADQL), queries on virtual tables

• Extended data model– metadata standardization (UCD normalization)– dataset characterization, identification, provenance– image subtypes, e.g., image cube, synoptic imagery

• Query response– intelligent ranking of query response (like Google)– logical grouping of related images, e.g., multi-band survey

data– metadata extension mechanism, e.g., as for AVO demo

• Data Access– Staging of data, authentication

NVO Summer School, Aspen Sept 14 2004 12

Simple Spectral Access

• Provides access to "spectral" data– similar to SIA but deals with tabular spectrophotometric

data• Service-oriented data discovery

– query service to discover data• Access to dataset metadata

– can get dataset metadata without retrieving actual dataset– uniform interface based on standard data models

• Access to actual dataset– data may be virtual, i.e., computed on demand– uniform interface to any type of spectral data– hides details of how data is stored or represented

externally

NVO Summer School, Aspen Sept 14 2004 13

SSA - Basic Usage

• Simplest possible query– HTTP GET, e.g., http://myvo.nvo.org/SSAPServlet?

POS=12.0,10.0&SIZE=0.2– other query types, e.g., WS, or ADQL in the future, also possible

• Query response– VOTable describing spectral datasets, one per row– for each dataset there is an *access reference*, a URL

• Get Data – simply fetch the dataset at the given URL– returned data adheres (normally) to a standard data model and data

format• Data Format

– A returned 1D spectrum may be a simple VOTable (or text file, or FITS binary table, etc.) with some general metadata followed by a simple spectrum table with wavelength, flux, and uncertainty columns.

• Once again, although basic usage is simple, the interface can do more than is immediately apparent.

NVO Summer School, Aspen Sept 14 2004 14

SSA - Interface Scope

• SSA deals with several types of data– Spectral Energy Distributions (SEDs) – 1D spectra– time series

• Why this grouping?– common spectrophotometric data model– all are sampled, spectrophotometric, tabular

data

NVO Summer School, Aspen Sept 14 2004 15

SSA Data Model

• Sampled spectrophotometric sequence– projected at constant time results in 1D spectrum– projected at constant spectral value results in time

series– projecting both results in a photometry point

• A SED is:– a collection of these three types of objects– at a constant point on the sky (usually!)– typically spanning a wide range of spectral values

• Summary– a SED attempts to describe the full spectral energy

distribution of an object, encompassing as much of the emitted energy as possible

NVO Summer School, Aspen Sept 14 2004 16

SED Composition

spectrum segment

Photometry point

NVO Summer School, Aspen Sept 14 2004 17

NVO Summer School, Aspen Sept 14 2004 18

OTF Generation of Spectral Data

• Spectral Archives– spectral data resembles catalog data as much as

image data– most spectral data access will probably be to pre-

computed data

• Virtual Data examples– spectrum from an image cube– SED from multi-band image data (plus catalog

data etc.)– time series from synoptic imagery or catalog data– spectrum or time series from radio Pulsar data

NVO Summer School, Aspen Sept 14 2004 19

SSA Data Formats

• Concepts– science data model (SDM)

• semantic model for the data - what IS this data– export data format (EDF)

• expresses the SDM in a specific data representation• identically the same SDM regardless of representation

• Formats– native XML– VOTable– FITS binary table– text table, e.g., CSV– plus graphics, HTML– spectral data can also be viewed as an image, with

restrictions

NVO Summer School, Aspen Sept 14 2004 20

SSA Service Interface

• Each class of data gets a separate interface– SED, spectrum, time series– similar but separate access interfaces preferred

• Similar to SIA– query, query response, getData– additional query parameters

NVO Summer School, Aspen Sept 14 2004 21

SSA Query

• Required parameters– POS, SIZE, FORMAT– region is circular, as for cone search, unlike SIA– FORMAT provides more options than just FITS for

the EDF, including XML (native and VOTable), and text

• Optional parameters– time, bandpass, collection, ID, rank– aperture, verbosity

NVO Summer School, Aspen Sept 14 2004 22

SSA Query Response

• VOTable– one table row per candidate dataset– access reference (URL) used to fetch data– component data models included directly as

objects– uses GROUP, UTYPE from VOTable 1.1