virtual observatories, astronomical telescopes and instrumentation (august 2002)1 an architecture...

Virtual Observatories, Astronomical Telescopes and Instrumentation (August 2002)

1

An Architecture for Access To A Compute Intensive Image Mosaic

Service in the NVO

G. B. Berriman, D. Curkendall, J. Good, J. Jacob,D.S. Katz, M. Kong, S. Monkewitz, R. Moore, T.Prince,

R. Williams


2San Diego Supercomputer Center

Compute Resources Catalogs Data Archives

InformationDiscovery

Metadatadelivery

Data Discovery

Data Delivery

Catalog Mediator Data mediator

1. Portals and Workbenches

Bulk DataAnalysis

CatalogAnalysis

MetadataView

DataView

4.GridSecurityCachingReplicationBackupScheduling

2.Knowledge & ResourceManagement

Standard Metadata format, Data model, Wire format

Catalog/Image Specific Access

Standard APIs and Protocols Concept space

3.

5.

6.

7. Derived Collections

National Virtual ObservatoryData Grid


3

Montage - Custom Image Mosaics

http://montage.ipac.caltech.edu

• User specified size, WCS projection, coordinates, spatial sampling, rotation

• Rectification of backgrounds in images

• Supports drizzle algorithm

Delivery:• Semi-annual deliveries from Feb 2003 • Final Delivery Jan 2005• Available for download

Science Drivers:• Science Grade Images• Analyze diverse images as if part of same “multi-wavelength image”


4

SI MPLE = T /BI TPI X = - 64 /NAXI S = 2 /NAXI S1 = 3000 /NAXI S2 = 3000 /CDELT1 = - 3. 333333E- 4 /CDELT2 = 3. 333333E- 4 /CRPI X1 = 1500. 5 /CRPI X2 = 1500. 5 /CTYPE1 = ' RA- - - TAN' /CTYPE2 = ' DEC- - TAN' /CRVAL1 = 265. 91334 /CRVAL2 = - 29. 35778 /CROTA2 = 0. /

END

A rb itraryInp utIm age

F IT S head er d efines o utp ut p ro jec tio n

R ep ro jec tedIm age

In p u t p ixelsp r o jected o nceles tia l s p h er e

O u tp u t p ixelsp r o jected o nceles tia l s p h er e

To ta l F lu x

S k y A re a C o v e ra g e(s te ra d ia n s )

M o n ta g e R ep ro jec tio n M o d u le

C en tra l to th e a lg o rith m is a ccu ra te ca lcu la tio no f th e a rea o f sp h erica l p o lyg o n in tersectio nb etw een tw o p ixels (a ssu m es g rea t circle seg m en tsa re a d eq u a te b etw een p ixel v ertices)


5

C o o rdina te d E xe c utio n A c ro s sM ultiple G R ID P ro c e s s o rs

Inp utIm ages

S imp le c o a d d f o r f in a l mo sa ic(sk y a re a s u se d a s w e ig h ts )

M o n ta g e Pa ra lle liza tio n

SI MPLE = T /BI TPI X = - 64 /NAXI S = 2 /NAXI S1 = 3000 /NAXI S2 = 3000 /CDELT1 = - 3. 333333E- 4 /CDELT2 = 3. 333333E- 4 /CRPI X1 = 1500. 5 /CRPI X2 = 1500. 5 /CTYPE1 = ' RA- - - TAN' /CTYPE2 = ' DEC- - TAN' /CRVAL1 = 265. 91334 /CRVAL2 = - 29. 35778 /CROTA2 = 0. /

END

F IT S head er

R ep ro jec tedIm ages

m P ro je c t

m P ro je c t

m P ro je c t

m A dd


6

M o n ta g e B a c k g ro u n d C o rre c tio n Pro c e d u re

A co rre ctio n is ca lcu la te d fo r e a ch im a g e b a se d o n a ll th e d iffe re n ce s b e tw e e n it a n d itsn e ig h b o rs (a n a p p ro x im a tio n to a le a s t sq u a re s fi t to th e d if fe re n ce d a ta w ith b r ig h tn e s s o u tlie rp ix e ls e x clu d e d ). Th e co r re ctio n is cu r re n tly a p la n e b u t co u ld b e a h ig h e r o rd e r su r fa ce .

Th is is d o n e fo r a ll im a g e s , th e n h a lf th e co r re ctio n d e te rm in e d is a p p lie d (to a p a ra m e te rd a ta b a se ; e q u iv a le n t n u m e r ica lly to a p p ly in g it to th e im a g e s ).

Th e p ro ce s s is i te ra te d u n til s te p d iffe re n ce s fo r a ll im a g e s b e co m e s sm a ll.

D iffe r e n ce s in o v e r la p a r e a s

E xam p le: T h ree overla p p in grep rojected 2 M AS S im a g es


7

M o n ta g e B a c k g ro u n d C o rre c tio n R e su lts

R ep rojectedB a ckg rou n d C orrectedIm a g es


8

Deployment of Montage

Performance Goal:Sustain a throughput of at least 30 square degrees per min on 1024 x 400 Mhz R12K O3000 or equivalent

Deploy operationally on the Teragrid by January 2005

Teragrid will be used by NVO for compute intensive services


9

Managing Requests For Compute Intensive Services

•Access compute intensive service or bulk data delivery service from existing clients • Apache has no memory of requests, and so little control of multiple time intensive requests:

• restart automatically after failed request• stop large requests as needed• monitoring of requests• load balancing

Solution:Request Object Management Environment


10

• Application of Enterprise Java Bean e-business technology• Used everyday by e.g. banks to manage secure financial transactions• Mature technology based on specialized Java Virtual Machine, the EJB server• EJB’s manage and persist transactions, perform load balancing, handle security

• Middleware that sits between client and processor whose role is straightforward

•Accepts requests from clients through standard interfaces•Registers them in database•Sends them for processing•Capable of managing very large numbers of requests

What Is ROME?


11

MONTAGE ProcessingModules

RequestDB

ROME

GRIDClient

Application

Image Cacheand

Catalog

Requests

CustomClients

BrowserForms

Clients

ImageSurveyData

ImageReprojection

ImageReprojection

ImageReprojection

ImageReprojection

ImageReprojection

ImageReprojection

ImageReprojection

BackgroundModelling

Web Services

SurveyMetadata

RequestManagement

ROMEMONTAGEInteraction


12

RE Q UE S T M A NA G E R (A P P L ICA T IO N S E RV E R)

JV M

RE Q UE S T P RO CE S S O R

JV M


JV M


JV M

W orker ThreadW orker Thread

Reques tUpdate

W orker Threads

ApplicationApplicationApplication

s tdin s tdout

o p ti o n a l m e ssa g e so cke t

Status

Update

Reques t

In terruptEx ec

Reques t Mes s ages

Us erRegis tration

Us er

GetReques t

CurrentReques t

A pplic ationMes s age

Mes s age

Ma in

Proc es s orRegis tration

P ro ce sso r

R O M E R eq u est O b ject M a n a g em en t E n viron m en t

Key:

OtherJAVA

Objects

DatabaseTable

Servelet

EJB

C lien t

(Brow s er , OA SIS, e tc .)

GetStatus

UpdateReques t

Reques tSubmis s ion

D BMS

Us er

Reques t

Proc es s or

Mes s age

U pdateApplication

Job ID

virtual observatories, astronomical telescopes and instrumentation (august 2002)1 an architecture...

Documents

large requests

memory of requests

request manager

diverse images

functions of load balancing

small transactions

worker threads

bulk data delivery service