japanese virtual observatory and workflowjvo.nao.ac.jp/documents/2008/ogf22-astrorg-tanaka.pdffeb...

Feb 26, 2008 Astro-RG, OGF22 1

Japanese Virtual Observatory and Workflow

Masahiro TanakaNational Astronomical Observatory of Japan

Feb 26, 2008 Astro-RG, OGF22 2

This talk

• Overview of JVO• JVO Workflow system• Investigation of GUI Workflow builder• Services called from Workflow

Feb 26, 2008 Astro-RG, OGF22 3

SkyNode1.0

SDSS QSO

… 2MASS

Portal

P. Reg.

S. Reg.

SkyNode 0.7.4

SIAP/SSAP

解析解析

解析AnalysisServers

MDS

P. Reg.

Other VOs

Invoke

JVO Portal

Subaru

Grid System

Registry

Data Services

JVO system : Portal and Services

UserStorage

User

Feb 26, 2008 Astro-RG, OGF22 4

JVO Portal - top page

4http://jvo.nao.ac.jp/portal/

January 24, 2008 ADC International Review

Feb 26, 2008 Astro-RG, OGF22 5

JVO SkyNode Architecture

JVOQLADQL

SIAPSSAP

HTTP/SOAPAXIS

HTTP/ParameterTomcat

JDBC forSkyNode

SelectSQLTranslator

Query ExecuterFormatter

Internet JVO SkyNode

JVO Portal

• Uses Apache Axis and Tomcat.

• Accepts Four kinds of query languages.

• Query is executed through JDBC-like interface.

• Query result is formatted as VOTable or CSV.

JVO SkyNode

DBMSResultSetgetMetadata()

getString()getFits()getJpeg()

Feb 26, 2008 Astro-RG, OGF22 6

Finding Services – RegistrySearchable Registry

Publishing Registry

Publishing Registry

Data Service

Analysis Service

Virtual Observatory Client

Query Services

Feb 26, 2008 Astro-RG, OGF22 7

Grid service for Subaru image reduction

• Automated job assignment with Monitor and Discovery Service (MDS)

Client

MDS ServerJVO Portal

Service Request

Available Server Info

Confirm Job Completion

Job Completion Info

Analysis Servers# of JobsCPU loadJob status

Analysis ServerManagement Data

Job submission

Results

Parallel Executions on Multiple Server

Execute

Feb 26, 2008 Astro-RG, OGF22 8

JVO Workflow system

Feb 26, 2008 Astro-RG, OGF22 9

Example of workflow

Analysis Services

SkyNode

SExtractorWebServices

HyperZWebServices

JVOworkflow control SIA Service

QSO Catalog

SourceCatalog

Subaruimage

Query

Feb 26, 2008 Astro-RG, OGF22 10

JVO Workflow LanguageXML Diagram of JVO Workflow Description Language

<<rootnode>>Process

<<abstract>>Activity

<<abstract>>LoopControl

<<abstract>>Control

<<abstract>>ConditionControl

<<abstract>>BasicActivity

<<abstract>>SequenceActivity

Sequence Flow

Command

OutputInput

Switchif

invokeScript

Then Else Case Otherwise

Parfor

For

WhileVariables

Variable

<<abstract>>ActivityContainer

1

1..*

1..*1

1

1

0..1

10..*

1 1

0..1 0..1

1 0..1 1..* 0..1

11

1

1

awk

Feb 26, 2008 Astro-RG, OGF22 11

Current workflow builder for JVO

• Editing XML directly– Difficult to write XML– Use Templates

• No GUI workflow builder for JVO

WF Editor

Execution monitor

Feb 26, 2008 Astro-RG, OGF22 12

Investigation of GUI Workflow builder for JVO

Feb 26, 2008 Astro-RG, OGF22 13

JFLOW

• Developed at CDS• Good User Interface• Does not support flow

controls:– Condition– Loop

Feb 26, 2008 Astro-RG, OGF22 14

Workflow editor for NAREGI

• Good Graphical WF builder

• Developed for Grid workflow for NAREGI

• Applicable to VO Services??

Feb 26, 2008 Astro-RG, OGF22 15

NAREGI

• Building Grid environment using NAREGI Middleware β2.0.1

• Trying Interoperability with KEK (Institute for high energy physics)

• Problems:– Difficult to install NAREGI middleware– No support for VO Protocols

Feb 26, 2008 Astro-RG, OGF22 16

Taverna .• Developed for Biology but usable for

general purposes• Evaluated by AstroGrid• JVO team has just started evaluation• Client-side application

– JVO has server-side workflow

Feb 26, 2008 Astro-RG, OGF22 17

Trying Taverna• Workflow calling a single Web Service

Feb 26, 2008 Astro-RG, OGF22 18

Executing…

Feb 26, 2008 Astro-RG, OGF22 19

Result

Feb 26, 2008 Astro-RG, OGF22 20

Taverna good things

• Good User Interface• Easy to call Web Services• Able to save workflow and results as XML• Good workflow engine

– Job monitoring and control– Logging (intermediate results and status)

Feb 26, 2008 Astro-RG, OGF22 21

Taverna issues…

• Still need knowledge on– Web Services– Input/Output data types– Programming

• Need Communication with :– Client-side software– VO Registry

Feb 26, 2008 Astro-RG, OGF22 22

Scripting languages

• Define new Workflow Language easier to write than XML?– Not difficult to define it from XML definition– We have already workflow engine.– Need language design– Learning cost for users

• Use Existing language?– Perl, Python, Ruby, …– No learning cost if user knows

Feb 26, 2008 Astro-RG, OGF22 23

SOAP call example usingScripting Language

• Ruby script :require 'soap/wsdlDriver'wsdl = 'http://ion.mtk.nao.ac.jp:8080/axis2/services/DetectLine?wsdl'

url = 'http://jvo.nao.ac.jp/skynode/sdss/spectrumRequest.do?'+

'db=sdss&table=spectrum&id=51630-0266-001&format=spectrum/fits'

driver = SOAP::WSDLDriverFactory.new(wsdl).create_rpc_driver

result = driver.invoke(:url=>url).return

• Easy to call Web Services• Script code can be a workflow

Feb 26, 2008 Astro-RG, OGF22 24

We have choices

• We just started evaluation.• Taverna is a promising tool.• Need more experience.

Feb 26, 2008 Astro-RG, OGF22 25

Services called from Workflow

Feb 26, 2008 Astro-RG, OGF22 26

Services in VO

• VO framework– Data archives

• Registry, SIAP, SSAP, ConeSearch, SkyNode, TAP,– Data storage

• VO Space

• Need more– Analysis services– Visualization services– Interoperability with client tools

• SAOImage, VOPlot, Aladin, …

Feb 26, 2008 Astro-RG, OGF22 27

Use Cases of Workflow

• Subaru data reduction– Service: Parallel execution with cluster

• Study on AGN environment– Service: SExtractor and HyperZ

• Search for Metal-poor stars– Service: Line detection service

Feb 26, 2008 Astro-RG, OGF22 28

Automatic Spectral Line Detection Service

• Detect every scale of line width• Wavelet-like algorithm

Feb 26, 2008 Astro-RG, OGF22 29

Grid Challenge

• Public Contest of HPC– Held in mid 2008 by Grid Scientists– JVO provides Subaru data and scientific scenario :

• Supernova search– Good experience to build Grid services for

Astronomy

Feb 26, 2008 Astro-RG, OGF22 30

Useful Services

• Re-usable, general-purpose services• Services which require computer

resources– CPU– Storage

• Services with clearly-defined interface

Feb 26, 2008 Astro-RG, OGF22 31

Issues

• Too few available services– Publicly available services are needed.

• WSDL can be written freely :– Variety of interface can be confusing.– Granularity of Services?– Need Methodology / Guideline / Standard ?

Feb 26, 2008 Astro-RG, OGF22 32

Conclusion

• JVO system and workflow are reviewed.• GUI WF builder for JVO is evaluated :

– Taverna is a promising tool.

• Useful services called from Workflow are needed.