Sept. 16 2004NVO Summer School 1

The NVO DataScope: Internals

Tom McGlynnNASA/GSFC

THE US NATIONAL VIRTUAL OBSERVATORY

Sept. 16 2004NVO Summer School 2

Summary

• DataScope as an example VO application

• Issues in building DataScope

Sept. 16 2004NVO Summer School 3

Overall Organization

CacheManagerDataScope

VO Registry

VO ResourcesUser

Sends request to DataScope

Is this request cached?

Yes: Just send results back to user

No: find resources to query

RequestManager

Queue the requests and start them off

Start telling the user about requests that have returned Cache the results

as they come in

Sept. 16 2004NVO Summer School 4

Perl VO Objects: Services

Service

URLAggregater

ScannerCone

SIAP

PositionalService

TarrerOASIS

AladinHeasarcResolver

Resolver

JHURegistryRegistry

HeasarcScanner ServicesNot easily usable in other

applications

Portable object

Sept. 16 2004NVO Summer School 5

Perl VO Library: Display

Display

Criterion

State

VOTableRendererServiceFilter

Sept. 16 2004NVO Summer School 6

Perl VO Library: Control

CacheManager RequestManager CacheableRequest

Sept. 16 2004NVO Summer School 7

Simple Galaxy Checker

• Look for all resources with subject of galaxy (or galaxies) and see if there are any matches at a given position.

• Run service at: http://skyview.gsfc.nasa.gov/tam/gTest.html

Fix bug in SOAP…This is where VO library was placed

Going to use these

Convert user position to standard decimal J2000 RA/Dec

Errors and defaults

Create objects

Query registry

Loop over matchesMake a cone search service and indicate where to search

Display the results. Note how we use eval which allows us to recover from a parsing error if the VOTable is malformed.

Query the cone search and get the VOTable

We only look for galaxies at the beginning of the subject. We might use “%Galax%” to match anywhere.

Sept. 16 2004NVO Summer School 9

How does DataScope use these objects?

• Main DataScope page is basically a HeasarcScanner which gets the data and a ServiceFilter which shows the results.

• Cone Search and SIAP results are shown using a VOTableRenderer– A few custom filters which have knowledge not in the

VO registries are used to make links to ADS, MAST and HEASARC archives

• The URLAggregater handles most of the details of getting information to OASIS and Aladin

• Single file services (VOPlot, VOStat) currently use custom code at CGI script level.

Sept. 16 2004NVO Summer School 10

DataScope: Registry Issues

• Registry is the most rapidly evolving element of the VO• SOAP interface to registry

– SOAP::Lite used in Perl• One local fix to library to retrieve multiple string fields properly.

– Setup may be non-trivial• How do we select resources of interest?

– Restrictions on ServiceType• Cone or SIAP

– Restrictions on ContentLevel– Should we use positional information about service?

• Not currently. The data is too ill-formed.

• Do we need to cache registry queries?– Takes several seconds, so not needed for DataScope

• Organizing resources– Use ServiceType, Subject and Content fields

• DataScope organization is just one of many possible ideas– Redundant and superseded resources

• Currently minimal information on these– Priority of resources

• Currently no information– More work needed if we are going to handle 103 to 104 resources

• No information on analysis services in registry– Hardwired into top level DataScope routines

Sept. 16 2004NVO Summer School 11

DataScope: Cone and SIAP issues

• Search radius:– SIAP implicitly handles overlapping regions but Cone search does not. May need to

increase size of region by ServiceSize to get all relevant rows from cone search services. • Failed queries:

– Should we retry?• Overloading provider sites

– Don’t want to spit hundreds of queries to the same site simultaneously• DataScope sorts resources by destination URL and only sends one query to a given host at a time.

• Non-compliant services– Cone searches returning multiple tables– Cone searches without position columns– SIAP services with improper UCD columns– But these are useful sites! (ADS, VizieR, XMM-Newton)

• Need to cache SIAP results before they expire• How do we handle deferred SIAP results? Don’t have any now.• SIAP can have both quicklook and FITS files but the relationship between them is

not defined.• A registry entry may describe only some of the results returned by a SIAP

request (using ID_MAIN column == ServiceName convention)– Look for multiple registry entries with identical ServiceURL’s

• When will these services be available as Web services?

Sept. 16 2004NVO Summer School 12

DataScope: Analysis software issues

• Currently two types of `analysis’ tools, some handle only one file at a time (VOPlot, TopCat, DS9,…) others take a set of URLs (Aladin, OASIS).

• URL Aggregating services may run over GET limits or limits on number of URLs that can be used.

• Single file tools may have more capabilities in non-applet modes

• Tools are not described in registry so knowledge of them is built into DataScope itself.

Sept. 16 2004NVO Summer School 13

DataScope caching issues

• Not necessarily a VO issue• Limits loads on data providers• Enhances response time• Data may go out of date.

– Failed queries may now be available

• SIAP results may expire if left on provider• Some services (e.g., Aladin) cannot currently

handle FTP URLs (though this should change with latest versions of Java). SIAP (and SSAP) URLs may need to be cached to be available to such services.

Sept. 16 2004NVO Summer School 14

Miscellaneous

• SkyNode’s not accessible through DataScope.– Cone search probably provides appropriate

functionality for DataScope.– Are all positional SkyNode tables available

through cone search?

Sept. 16 2004NVO Summer School 15

HEASARC Perl library distribution

• Very preliminary distribution with limited documentation and testing for the non-VOTable modules.

• Available at:– http://skyview.gsfc.nasa.gov/vo/volib.0.1.tar or volib.0.1.zi

p

– Installation documentation at http://skyview.gsfc.nasa.gov/vo/volibinstall.doc

Simple command line script to invoke an SIAP service and download all of the referenced files.