nvo summer school, september 20061 desktop integration with vo voclient, dalclient nvo summer...

NVO Summer School, September 2006 1

Desktop Integration with VOVOClient, DALClient

NVO Summer School, Aspen Sept 2006

Doug Tody (NRAO), Mike Fitzpatrick (NOAO)

US National Virtual Observatory

NVO Summer School, September 2006 2

Application Development in the VO

• Types of VO-Aware Applications– Webapps (e.g., datascope)

• interface is a web browser– Application services (wesix, wcsfixer, etc.)

• interface is a web service– Application environments (IRAF, IDL, Python, PHP, etc.)

• interface is a full user applications environment– Grid workflows (e.g., astrogrid)

• interface is a user-generated workflow script (batch job)

• At the back-end, all of these can share the same client-side interface to the VO.

NVO Summer School, September 2006 3

VO Client-Side Development: Requirements

• Support for multiple languages/environments– Every language or environment has its application domain– Let the user decide which is best for their application

• High level functionality– Applications generally prefer a high level environment– Integrate VO with existing powerful data analysis tools

• Isolate application from details of VO infrastructure– VO interface is relatively low level– Functionality can be complex

• multiple versions of interfaces• multiple protocols, grid and web service technologies• storage managers (VOSpace), asychronous data transfers, etc.

– Java is the standard language for the VO infrastructure• but is not favored for most science applications development

NVO Summer School, September 2006 4

Option 1: Custom Integration for each Language

• Approach– Thin layer implemented natively in each target environment

• Advantages– Simplest approach so long the as middleware is thin– Easier to customize interface for each target language– Avoids need for multi-language integration

• Issues– Doesn't scale well as middleware becomes more complex

• have to reimplement functionality for every target environment

• ok, until it gets beyond a certain size– Interface and capability matrix will tend to be different for

each target environment.

NVO Summer School, September 2006 5

Option 2: Common Facility with Multiple Bindings

• Approach– A common client-side middleware facility with a thin binding for each

target language or environment– MySQL (and many other such facilities) use this architecture

• Advantages– Portable– Uniform interface and capability matrix for all target environments– Scales well

• Only one implementation required for all target languages• Also scales well in the sense of parallel computing• New capabilities appear immediately in all target languages

– Can use Java• Most VO and web/grid system-level software is in Java

• Issues– More complex as architecture is multi-language and distributed– Distributed computing performance can be a concern if not careful

NVO Summer School, September 2006 6

The VOClient Middleware

• Objectives– Provide a high level interface to the VO for user development– Support all major languages and environments– Provide reference code within NVO for the DAL services

• Origin– Subsystem in the Opticon/NVO scalable framework architecture

• stands on its own due to open systems approach

– History• began in 2005 with the dalclient Java library• expanded this year to add multi-language support

NVO Summer School, September 2006 7

Architecture

• Applications layer– Python, Java, IRAF, IDL, PHP, CASA, etc.

• Message bus– RPC and async progress events

• VOClient daemon (voclientd)– dalclient Java library– Registry interface– Sesame name resolver– VOSpace, security, async operations– Data access objects

• table, image, spectrum, etc.

PythonIRAF

IDL etc. Java

voclientd

dalclientregistry

NVO Summer School, September 2006 8

VOClient Functionality

• Capabilities– Sesame name resolver

• convert astronomical object names to coordinates– Registry query

• required for data collection and service discovery– Client side of DAL interfaces

• cone search (catalogs), SIAP (images)• soon: SSAP (spectra) and SLAP (line lists)

• Basic interface– Multi-threaded, multiple clients and connections– Stateful client-server connection

• connection-oriented, like a DBMS or other object interface• state is maintained where the data is managed (voclientd)• concurrent access to a shared object is possible

– DAL service interface is RPC to dalclient running in voclientd

NVO Summer School, September 2006 9

Sesame Name Resolver

• Start voclientd if not already running

• Open a connection voc_initVOClient (config_opts)

• Name resolver API sr = voc_nameResolver (target) pos_str = voc_resolverPos (sr) radeg = voc_resolverRA (sr)decdeg = voc_resolverDEC (sr)

• Close connection voc_closeVOClient (shutdown_flag)

NVO Summer School, September 2006 10

DALClient Interface

• DAL class hierarchy– Generic dataset (any dataset)

• Cone catalogs• SIA images• SSA spectra (soon)• SLAP spectral line lists (soon)• SNAP theoretical models (future)

(etc.)

• Basic API– Similar for each class of data

• Form query, execute query, walk result set(Similar to a JDBC database interface)

– Multi-level outputs• stream, votable, data model

NVO Summer School, September 2006 11

DAL Query (Base class)

dal = new DALConnection () dal = new DALConnection (service-url) dal.addService (service-url) dal.getServiceCount () dal.getServiceURL (index)

query = dal.getDALQuery () query.addParameter (name, value) query.getQueryString (serviceIndex) query.executeCSV () query.executeCSV (fname) vot = query.executeVOTable () is = query.executeRAW ()

NVO Summer School, September 2006 12

CONE Query (extends DAL Query)

cone = new ConeConnection ()cone = new ConeConnection (service-url)

query = cone.getConeQuery ()query = cone.getConeQuery (ra, dec, sr) query.addParameter (name, value)

qr = query.execute () query.executeCSV () query.executeCSV (fname)vot = query.executeVOTable () is = query.executeRAW ()

NVO Summer School, September 2006 13

SIAP Query (extends DAL Query)

siap = new SiapConnection () cone = new SiapConnection (service-url)

query = siap.getSiapQuery () query = siap.getSiapQuery (ra, dec, size) query = siap.getSiapQuery (ra, dec, size, format) query = siap.getSiapQuery (ra, dec, ra_size, dec_size) query = siap.getSiapQuery (ra, dec, ra_size, dec_size,

format) query.addParameter (name, value)

qr = query.execute () query.executeCSV () query.executeCSV (fname)vot = query.executeVOTable () is = query.executeRAW ()

NVO Summer School, September 2006 14

Query Response (data model view)

• Format– Result set: a sequence of records

• Interface qr.getRecordCount () rec = qr.getRecord (i)

rec.setAttribute ("<attrname>", "<value>") v = rec.getAttribute ("<attrname>")

v.intValue () v.floatValue () (etc.)

rec.getDataset ("<path>") path = rec.getDataset ()

NVO Summer School, September 2006 15

Registry Interface

• Approach– Very similar to DAL query– Form query, execute query, step through result set

• Interface res = voc_regSearch (sql, keywords, orValues)res =voc_regSearchByService (svc, term, orValues)

query = voc_regQuery (term, orValues) voc_regAddSearchTerm (query, term, orValue) res = voc_regExecute (query)

count = voc_resGetCount (res) str = voc_resGetStr (res, attribute, index) dval = voc_resGetFloat (res, attribute, index)

NVO Summer School, September 2006 16

Current status

• Language Support– C/C++, Java, IRAF, FORTRAN (so far)– Partial support for Python, PHP, etc.

• Messaging– Message bus currently uses a simple brute-force approach

• VOClient Daemon– Connection context currently limited to single service

• Usage– Java and C interfaces are usable– IRAF interface is a good example of where we are headed

with this

NVO Summer School, September 2006 17