2008 NVO Summer School 1

Data Access Layer Services

Doug Tody (NRAO)

THE US NATIONAL VIRTUAL OBSERVATORY

Data Access Layer (DAL) Servers

• Scope– This talk is about serving data to the VO

• Publishing catalogs, image and spectral data collections

– Client side data analysis merely uses the services• End user does not see any of this• Those developing data analysis s/w will however• Audience: data service and client s/w developers

• Topics– Part 1: Data services (presentation)– Part 2: The DALServer framework (hands-on)

Why a Service Framework?

• Not required– DAL protocols are open

• separation of interface from implementation– Building from scratch is always permitted

• Sometimes one needs to do this

• However...– This often results in minimal or noncompliant services– Increasingly impractical as VO protocols mature and

become more complex– Use of a service framework can make service construction

easier while providing more complete, robust service implementations.

Service Frameworks

• Examples– Apache Web server, Tomcat, etc. are non-astro examples

• Basic Concept– Framework provides all the generic functionality– You provide the data and metadata - the content

• Other– Also provides a reference implementation for a standard– Service verification is also an important issue

DAL Data Service Classes

• Generic Dataset– Table

• SCS, TAP

– Image• SIAV1, SIAV2

– Spectrum• observational (SSAP)• synthetic spectrum (TSAP)

– SED• based upon SSA

• Generic Dataset– TimeSeries

• based upon SSA

– Spectral line list• SLAP

– Theory data• model data

– Instrumental data• any observatory

instrument

• Aggregations– Complex data

DAL Services Share Common Elements

• Common Service Elements– Query parameters (POS, SIZE, BAND, TIME, etc.)– Generic dataset metadata (ident, curation, char, etc.)– Service methods (queryData, getData, getCapabilities,

etc.)– Form of interface (operations, parameters, errors, etc.)– Usage of VOTable, UTYPE, etc. for query response

• Other Common Elements– Output serializations (VOTable, CSV, text, FITS, etc.)– Region specification, STC-based coordinates– Grid capabilities (async, auth, vospace, etc.)

Service Profile

• Operations– REQUEST, VERSION parameters– queryData, getData, getCapabilities, getAvailability

• Params– Semantics, e.g., no repeated params– Syntax, e.g., range list syntax

• Errors– use of VOTable for errors

Data Models

• Query response– each service type defines a query response data model– standard metadata for a class of data

• Dataset (image, spectrum, etc.)– dataset also defines a data model

• part of this is common with the query response– Spectrum is a good example

• UTYPE usage– tag elements of a data model– "flattens" a hierarchical data model– separation of model from representation

DAL Service Functionality

• Service Architecture– Data query

• data discovery• metadata retrieval• iterative specification of virtual data

– Data access• data retrieval• virtual data generation• data staging (async, vospace)

• Advanced Capabilities– getCapabilities, getAvailabilty– authentication, asynchronous processing, vospace

The DALServer Framework

• Unifies all DAL services within a consistent framework– common mechanisms, framework elements, semantics– maximizes code sharing and reuse

• Provides a reference implementation of all DAL services– required as part of the standards development process– augments specification with a working implementation example

• Separates external concerns from service implementation– service code does not know how it is being called– service code can focus on data manipulation– framework deals with external interface (HTTP, servlets, etc.)

Framework Elements

• Service Implementations– base class for each service protocol (SIA, SSA, etc.)– often a real working service, e.g. echo/test or builtin data

• Input Parameters– standard input parameter set for each class of service– facilities for parameter management, parsing, and access

• Keyword Dictionary– standard keyword dictionary for each class of service– defines standard metadata including UCD, UTYPE, etc.

• Data Model– Service code writes to the data model– Does not deal directly with VOTable etc. (unless it needs to)– Framework responsible for serializing model in various formats

Data Model Usage

• ExampleRequestResponse r = new RequestResponse();SsapKeywordFactory ssap = new

SsapKeywordFactory(r);

r.addGroup(ssap.newGroup("Access“));r.addField(ssap.newField("AcRef“));r.addField(ssap.newField("Format“));

r.addRow();r.setValue("AcRef“, “http://...”);r.setValue("Format“, "image/fits“);

Adding a New Data Service

• Configurable Service– Service functionality provided by DALServer– Configure service to access local data

• cone search example illustrates this

• Custom Service– Subclass base class and provide custom methods

(queryData etc.)– Custom service functionality provided by the new class– Much of the generic framework is still inherited

• servlet code, paramset, keyword dictionary, data model, etc.

– More work, but more adaptable

SCS support

• Provided– Working service using built-in copy of Messier table– Service user-configurable to access any DBMS table

• New Services– Configure generic service– Subclass dalserver.ScsService to make custom service

SIA support

• Provided– Echo/test service implementing protocol including

metadata– Service user-configurable to access a local image collection

• DBMS table required to provide SIAP metadata for collection• getData operation provided to access image data• Image data accessed by local URL (file or network-based)

• New Services– Configure generic service, provide SIAP metadata in a table– Subclass dalserver.SiapService to make custom service

SIA Image Index

• Table Metadata (for each image)– ra RA in degrees (ICRS)– decl DEC in degrees (ICRS)– title Short descriptive title– instr Instrument description– dateobs Date of observation (MJD)– naxes Number of image axes– naxis1, naxis2 Size of each axis (pixels)– scale1, scale2 Scale of each axis (deg)– format Image format (MIME type)– fpath File path relative to dataDirURL

– cframe Spatial coord frame as in FITS – equinox Equinox of spatial coords, if used – proj Celestial projection– crpix Coordinates of reference pixel– crval WCS coords of reference pixel– cd CD matrix as in FITS WCS

SSA support

• Provided– Echo/test service implementing protocol including all

metadata– Demonstration proxy service for JHU/SDSS spectra

• working SSA service with access to ~1 million SDSS DR6 spectra

• example of a complex custom subclassed service

• New Services– Subclass dalserver.SsapService to make custom service