jsaga2 overview job desc. glite plug-ins globus plug-ins jsaga hidemiddlewareheterogeneity (e.g....
TRANSCRIPT
JSAGA 2
OverviewOverview
jobdesc.
gLiteplug-ins
Globusplug-ins
JSAGAJSAGA
hid
eh
ide
mid
dle
war
em
idd
lew
are
het
ero
gen
eity
het
ero
gen
eity
(e.g
. gLi
te, G
lobu
s, U
nico
re)
(e.g
. gLi
te, G
lobu
s, U
nico
re)
JDL RSL
JSAGA 3
WMSWMS
inputdata
SRMSRM
GridFTPGridFTP
WS-GRAMWS-GRAMLCG-CELCG-CELCG-CELCG-CE WS-GRAMWS-GRAMfi
rew
all
OverviewOverview
jobdesc.
gLiteplug-ins
Globusplug-ins
JSAGAJSAGA
jobjob
staginggraph
dele
gate
sel
ectio
n
& fil
es s
tagi
ng
jobjob
OPlastEGEE
hid
e g
rid
hid
e g
rid
infr
astr
uct
ure
s
infr
astr
uct
ure
sh
eter
og
enei
ty
het
ero
gen
eity
(e.g
. EG
EE
, OS
G, D
EIS
A)
(e.g
. EG
EE
, OS
G, D
EIS
A)
hid
eh
ide
mid
dle
war
em
idd
lew
are
het
ero
gen
eity
het
ero
gen
eity
(e.g
. gLi
te, G
lobu
s, U
nico
re)
(e.g
. gLi
te, G
lobu
s, U
nico
re)
JDL RSL
JSAGA 4
What is JSAGA ?What is JSAGA ?h
ide
gri
dh
ide
gri
din
fras
tru
ctu
res
infr
astr
uct
ure
sh
eter
og
enei
ty
het
ero
gen
eity
(e.g
. EG
EE
, OS
G, D
EIS
A)
(e.g
. EG
EE
, OS
G, D
EIS
A)
hid
eh
ide
mid
dle
war
em
idd
lew
are
het
ero
gen
eity
het
ero
gen
eity
(e.g
. gLi
te, G
lobu
s, U
nico
re)
(e.g
. gLi
te, G
lobu
s, U
nico
re)
An API for submitting a collectionAn API for submitting a collectionof jobs to several grid infrastructuresof jobs to several grid infrastructureswith a single job descriptionwith a single job description
A Java implementation of theA Java implementation of theSAGA specificationSAGA specification
JSAGA is…JSAGA is…
JSAGA 5
Global architectureGlobal architecture
For end-users: ready-to-use software adapted to their needs
For application developers: hide infrastructure heterogeneity
For application developers: hide middleware heterogeneity
For plug-in developers: As many interfaces as ways to implement the functionalities
ApplicationsApplications
Plug-ins interfacesPlug-ins interfaces
Job CollectionJob Collection
SAGASAGA
JSAGA 6planned
construction
done
Provided plug-insProvided plug-ins
SecuritySecurity X5
09
SS
H
InM
em
Cre
d
Glo
bu
s
G. L
eg
acy
G. R
FC
82
0
MyP
roxy
VO
MS
Lo
gin
/ pw
d
DataData
Exec.Exec.(control)(control)
ExpressionExpression
cata
log
rns
lfn
srb / iro
ds
http
http
s
sftp
rbyte
io
file
zip
gsiftp
tar
ftp
ma
il
cach
e
srm
Job controlJob control
ga
teke
ep
er
gL
ite-W
MS
wsg
ram
un
icore
6
ssh
loca
l
crea
m
grid
vm
PB
S
rem
ote
Exec.Exec.(monitor)(monitor)
ga
teke
ep
er
gL
ite-L
B
wsg
ram
un
icore
6
ssh
loca
l
crea
m
…
ba
sic
de
fau
lt
JEP
Be
an
Sh
ell
LanguageLanguage
JS
DL
+e
xt.
SA
GA
JDL
RS
L-2
RS
L-4
JSAGA 7
Hide middleware heterogeneity– Uniform interface (SAGA)
Hide infrastructure heterogeneity– Selection of the right security
context
– Selection of the right client API configuration
– Transport of the job input/output data to/from worker nodes
Describe the capabilities of the infrastructures you want to use
requirements
WorldWorld
Grid Grid
EGEEEGEE
localhostlocalhost
OpenPlastOpenPlast
CC-IN2P3CC-IN2P3
Describe grid infrastructuresDescribe grid infrastructures
wsgramwsgram
WMSWMS
VOMSVOMS
VOMSVOMS
GlobusGlobus
gsiftp://
JSAGA 8
Describe grid infrastructuresDescribe grid infrastructures
Hide middleware heterogeneity– Job management technologies
• e.g. CREAM, WMS, SSH, GK
Hide infrastructure heterogeneity– Computing Elements (e.g. GK)
• Different grid or site policies– e.g. network filtering, shared FS
• Different environment variables– e.g. $VO_?_SW_DIR, /usr/local
• Different configuration attributes– e.g. monitor service URL, shell
path on cygwin, default SE URL
• Commands available on worker– e.g. globus-url-copy, srmcp,
Scp, wget, tar
WorldWorld
Grid Grid
EGEEEGEE
OpenPlastOpenPlast
localhostlocalhost
CC-IN2P3CC-IN2P3
http://
tar://
srb://
gsiftp://
srm://
lfn://
gatekeepergatekeeper
gatekeepergatekeeper
wsgramwsgram
WMSWMS
gatekeepergatekeeper
VOMSVOMS
VOMSVOMS
GlobusGlobus
examples
JSAGA 9
url://
Transfer path depends on…Transfer path depends on…
Using a single grid– all files can be transferred to the worker through a single node
Using several grids– need to dynamically build a transfer graph, according to…
WorldWorld
Grid Grid
EGEEEGEE
OpenPlastOpenPlast
localhostlocalhost
CC-IN2P3CC-IN2P3
http://
tar://
srb://
gsiftp://
srm://
lfn://
gatekeepergatekeeper
gatekeepergatekeeper
wsgramwsgram
WMSWMS
gatekeepergatekeeper
VOMSVOMS
VOMSVOMS
GlobusGlobus
jobjobdesc.desc.
plug-insplug-ins
JSAGAJSAGA
JSAGA 10
Transfer path depends on…Transfer path depends on…
grid or site– network filtering policy
– commands available on workers
– services available from workers (close Storage Element, shared FS)
– supported context instances
data to stage– shared by several jobs– installed on some worker nodes– file size– required data protection level
execution service– protocols supported for staging
transfer protocol– access mode (RO, WO, RW)– third-party transfer– supported data protection level
jobjobdesc.desc.
WorldWorld
Grid Grid
EGEEEGEE
OpenPlastOpenPlast
localhostlocalhost
CC-IN2P3CC-IN2P3
http://tar://
srb://
gsiftp://
srm://lfn://
gatekeepergatekeeper
gatekeepergatekeeper
wsgramwsgram
WMSWMS
gatekeepergatekeeper
VOMSVOMS
VOMSVOMS
GlobusGlobus
url://plug-insplug-ins
JSAGAJSAGA
JSAGA 11
OpenPlastOpenPlast
OpenPlastOpenPlast
WorldWorld
Grid Grid
EGEEEGEE
OpenPlastOpenPlast
localhostlocalhost
CC-IN2P3CC-IN2P3
http://tar://
srb://
gsiftp://
srm://lfn://
gatekeepergatekeeper
gatekeepergatekeeper
wsgramwsgram
WMSWMS
gatekeepergatekeeper
VOMSVOMS
VOMSVOMS
GlobusGlobus
Transfer path depends on…Transfer path depends on…
grid or site– network filtering policy
– commands available on workers
– services available from workers (close Storage Element, shared FS)
– supported context instances
data to stage– shared by several jobs– installed on some worker nodes– file size– required data protection level
execution service– protocols supported for staging
transfer protocol– access mode (RO, WO, RW)– third-party transfer– supported data protection level
OPlast
OPlast
OPlast
OPlastGSIFTPGSIFTP
OPlastEGEECA
SMTPSMTP GSIFTPGSIFTP GSIFTPGSIFTP HTTPHTTPOPlastSRBSRB
jobjob
jobjob
jobjob
CC
RR11
EE11
C'C'
EE11
RR11
CC C'C' common
result
std-error
JSAGA 12
EE11
RR11
CC C'C'C''C'' common
EE EEsrcsrc executable
DD11input data
result
std-error
OPlast
OPlast
OPlast
OPlast
OPlastEGEECA
SMTPSMTP GSIFTPGSIFTP GSIFTPGSIFTP HTTPHTTP
GSIFTPGSIFTP
OPlastSRBSRB
Transfer path depends on…Transfer path depends on…
grid or site– network filtering policy
– commands available on workers
– services available from workers (close Storage Element, shared FS)
– supported context instances
data to stage– shared by several jobs– installed on some worker nodes– file size– required data protection level
jobjob
jobjob
jobjob
C"C"EEsrcsrc DD11
EE
CC
RR11
EE11
C'C'
TARTARTARTAR
execution service– protocols supported for staging
transfer protocol– access mode (RO, WO, RW)– third-party transfer– supported data protection level
iGetiGet
JSAGA 13
Example of generated graphExample of generated graph
Data flowData flowData flowData flowEE11
RR11
CC C'C'C''C'' common
EE EEsrcsrc executable
DD11input data
result
std-error
OPlast
OPlast
several protocols used,but
only 3 jobs submittedon 1 grid…
JSAGA 14
ConclusionConclusion
Assets of JSAGA– does not sacrifice scalability and efficiency for abstraction
• thanks to design (in particular the definition of plug-in interfaces)
• thanks to caching mechanisms
– does not only hide heterogeneity of middleware, but also hides heterogeneity of grid infrastructures
• thanks to the description of used grid infrastructures (configuration file)
• thanks to the information provided by the plug-ins
• thanks to the information provided by the user in his job description
http://grid.in2p3.fr/jsaga/
JSAGA 15
Related projectsRelated projects
JSAGA is used by…– Elis@
• a web portal for submitting jobs to industrial and research grid infrastructures
– JJS (Java Job Submission)• a tool for submitting job collections to EGEE
• optimized for short-life jobs (resource selection based on QoS observed while submitting jobs)
– JUX (Java Universal eXplorer)• a multi-protocol file browser
ApplicationsApplications
//