status of tsukuba, kek, 21 september 2010 sylvain reynaud
TRANSCRIPT
JSAGA 2
The SAGA specificationThe SAGA specification
SAGA : the specification– a recommendation of Open
Grid Forum
– a Simple API
– for using Grid middleware• execution management
– job submission, monitoring...
• data management– transfer, list files, search…
• service discovery• …
– in user Applications• implementations (or wrappers)
exist for C++, Java and Python
…in a uniform way
S A
G
A
use middlewares in a uniform way(independent of middlewares)
use functionalities in a uniform way(consistent "look-and-feel")
use language bindings in a similar way
(language-independent specification)
JSAGA 3
SAGA vs. gLite User InterfaceSAGA vs. gLite User Interface
Source: S Jha, T Kielmann, A Merzky
user
gLite-UIgLite-UI
VOMS,VOMS,MyProxMyProx
yy
WMS,WMS,LCG-CE,LCG-CE,
CREAM-CECREAM-CE
SRM,SRM,gsiftpgsiftp
LFC,LFC,AMGAMG
AABDIIBDII
JSAGA 4
Simple API for Grid ApplicationsSimple API for Grid Applications
Java code example (data management)
Session session = SessionFactory.createSession(true);URL url = URLFactory.createURL("gsiftp://cclcgseli01.in2p3.fr/tmp/");
// create SAGA object with a factory
NSDirectory dir = NSFactory.createNSDirectory(session, url);
// use the created SAGA objectList<URL> result = dir.list();for (URL r : result)
System.out.println(r);
list content of directory
gsiftp://…/tmp/
JSAGA 5
Simple API for Grid ApplicationsSimple API for Grid Applications
Java code example (execution management)
Session session = SessionFactory.createSession(true);URL url = URLFactory.createURL("wms://lapp-wms02.in2p3.fr/…");
// create SAGA object with a factory
JobService svc = JobFactory.createJobService(session, url);
// use the created SAGA objectJob job = svc.runJob("/bin/date --utc");job.waitFor();System.out.println(job.getState());
submit a job and wait for
the end of its execution
JSAGA 6
The JSAGA implementationThe JSAGA implementation
a Java implementation of the SAGA specification
focuses on– uniform usage of middleware…. (beyond what they natively support)– ease of extension……………….– efficiency and scalability……….– control of the API behavior........ (configure binding rather than trying)– operating-system independency (tested on )
Under LGPL license
(thanks to the design of adaptor interfaces)
JSAGA 7
user applicationsuser applications
VOM
S
WM
S
gsiftp
Gatekeep.
Globus
JSAGAJSAGA
GridFTPGridFTP
Example – uniform usageExample – uniform usageSAGA
jobSAGA
job
out.txtout.txt
out.txtout.txt
JSAGA 8
user applicationsuser applications
VOM
S
WM
S
gsiftp
Example – what happens behindExample – what happens behind
EGEE proxy + attributes translate submit job poll status delegate transfer
DGrid proxy adapt + translate submit job receive status transfer
– to any protocolG
atekeep.
Globus
JSAGAJSAGA
CREAM-CECREAM-CELCG-CELCG-CE UnicoreUnicore
WMSWMS
Globus GKGlobus GK
RSL
VOMSVOMS
modified
jobwrapper
script
GridFTPGridFTP
SAGAjob
D-GridEGEE JDL
out.txtout.txt
out.txtout.txt
JSAGA 9
Layered software architectureLayered software architecture
JSAGAJSAGA
interfaceinterface
implementationimplementation
JSAGA 10
Layered software architectureLayered software architecture
Used by end usersUsed by end users– web portals, GUI, CLI…web portals, GUI, CLI…
Used by application developersUsed by application developers– a single uniform interfacea single uniform interface
– object oriented, high-levelobject oriented, high-level
Implemented by adaptorsImplemented by adaptors– each "way" to implement a each "way" to implement a
given feature has 1 interfacegiven feature has 1 interface
– service oriented, low-levelservice oriented, low-level
Implemented by middlewareImplemented by middleware– each technology has its own each technology has its own
interfacesinterfaces
user applicationsuser applications
middleware APIsmiddleware APIsmiddleware APIsmiddleware APIs
core enginecore engineJSAGAJSAGA
adaptorsadaptorsJSAGAJSAGA
adaptors interfacesadaptors interfacesJSAGAJSAGA
SAGASAGA Java bindingJava binding
interfaceinterface
implementationimplementation
JSAGA 11
listen status
changestransfer
output
Layered software architectureLayered software architecture
user applicationsuser applications
GridFTPGridFTP Globus GKGlobus GK
adaptor
getstatus
getoutput
uploaddata
JSAGA JSAGA core enginecore engine
JSAGA JSAGA adaptor interfacesadaptor interfaces
adaptor
Used by end usersUsed by end users– web portals, GUI, CLI…web portals, GUI, CLI…
Used by application developersUsed by application developers– a single uniform interfacea single uniform interface
– object oriented, high-levelobject oriented, high-level
Implemented by adaptorsImplemented by adaptors– each "way" to implement a each "way" to implement a
given feature has 1 interfacegiven feature has 1 interface
– service oriented, low-levelservice oriented, low-level
Implemented by middlewareImplemented by middleware– each technology has its own each technology has its own
interfacesinterfaces
JSAGA 12
Layered software architectureLayered software architecture
user applicationsuser applications
GridFTPGridFTP Globus GKGlobus GK
adaptoradaptor
getstatus
getoutput
uploaddata
JSAGA JSAGA core enginecore engine
JSAGA JSAGA adaptor interfacesadaptor interfaces
Used by end usersUsed by end users– web portals, GUI, CLI…web portals, GUI, CLI…
Used by application developersUsed by application developers– a single uniform interfacea single uniform interface
– object oriented, high-levelobject oriented, high-level
Implemented by adaptorsImplemented by adaptors– each "way" to implement a each "way" to implement a
given feature has 1 interfacegiven feature has 1 interface
– service oriented, low-levelservice oriented, low-level
Implemented by middlewareImplemented by middleware– each technology has its own each technology has its own
interfacesinterfaces
listen status
changestransfer
output
JSAGA 13
Layered software architectureLayered software architecture
user applicationsuser applications
GridFTPGridFTP Globus GKGlobus GK
adaptoradaptor
getstatus
getoutput
uploaddata
JSAGA JSAGA core enginecore engine
JSAGA JSAGA adaptor interfacesadaptor interfaces
listen status
changes
Middleware API can be used Middleware API can be used more efficientlymore efficiently– engine selects best interface for engine selects best interface for
each user requesteach user request
Developing new adaptors is Developing new adaptors is easiereasier– just wrap functionalities just wrap functionalities
supported by middleware APIsupported by middleware API
transfer
output
JSAGA 14
adaptorsadaptorsinterfacesinterfacesJSAGAJSAGA
As many interfaces as ways to implement each functionality
Example: ways to monitor jobs– API mode
• poll job status• listen to notifications about job status
changes
– API granularity• individual jobs• list of jobs• jobs filter (e.g. by user, by date, by
tag…)
planned
construction
done
Job monitoringJob monitoring
ga
teke
ep
er
gL
ite-W
MS
wsg
ram
un
icore
6
crea
m
na
reg
i
Design of adaptors interfacesDesign of adaptors interfaces
ssh
loca
l
SAGASAGA
querystatus forindividual
job
listenstatus forindividual
job
querystatus for
filteredjobs
getState
waitFor
JSAGA 15
adaptorsadaptorsinterfacesinterfacesJSAGAJSAGA
Design of adaptors interfacesDesign of adaptors interfaces
Optional interfaces for optimization
Example: ways to copy a file– data read/write methods
• either stream methods• or get/put methods
– data copy• e.g. enable third-party transfer
– delegated transfer
planned
construction
done
PhysicalPhysicalrbyte
io
gsiftp
srm
srb
irod
s
http
http
s
sftp
ftp
file
zip
cach
e
SAGASAGA
datareader
datawriter
datacopy
copy
JSAGA 16
LogicalLogical PhysicalPhysical
Supported technologiesSupported technologies
Minimal dependencies on external libraries external tools
– e.g. no gLite-UI operating system
– tested on
planned
construction
done
DataData
ExecutionExecution
SecuritySecurity
InM
em
Cre
d
Glo
bu
s
G. L
eg
acy
G. R
FC
82
0
MyP
roxy
VO
MS
JKS
SS
H
Lo
gin
/ pw
d
X5
09
ga
teke
ep
er
gL
ite-W
MS
wsg
ram
un
icore
6
crea
m
na
reg
i
ssh
loca
l
rns
lfn
srb
irod
s
cata
log
rbyte
io
gsiftp
srm
srb
irod
s
http
http
s
sftp
ftp
file
zip
cach
e
Grid
techno.
Grid
techno.
Grid
techno.
Others
Others
Others
JSAGA 17
Latest developmentsLatest developments
New JSAGA adaptors– for data management
• gLite-LFC, dCache (SRM)
– for execution management• OGSA-BES (Unicore, ARC…)
Improvements in some gLite adaptors and their libraries– features
– thread-safety
– memory leaks
– …
JPySAGA is now compliant with– reference Python binding for
SAGA (as user interface)– reference Java binding for
SAGA version 1.1 (as implementation interface)
JSAGA has a new configuration engine– enable dynamic configuration– support 1 configuration/session– detect conflicts at config.-time
JSAGA 18
Security context selectionSecurity context selection
Try all security contexts
attractive for beginners
can be inefficient– waste of time (while trying)
– waste of computing resource (risk of output staging failure)
can be unreliable– may create files with wrong
ownership
– may block an account with too many failed connection attempts
Bind contexts with URL patterns
predictable/controllable behavior
requires some knowledge about– used infrastructures– JSAGA configuration capabilities
JSAGA has a new configuration engine– enable dynamic configuration– support 1 configuration/session– detect conflicts at config.-time
JSAGA 19
Software qualitySoftware quality
Adaptors– validated by a middleware-
independent SAGA test suite
– maven 'archetype' to generate skeleton of new adaptor project
Automated build process– download and install
• build tools• external libraries
– generate source code– execute test-suites
• unitary tests• integration tests
– generate project web site• documentation• reports
– generate installer GUI• extract library dependencies
from maven project description
# SAGA protocols test-suite configurationgsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/gsiftp.context=OpenPlast_proxy
https.base=http://grid.in2p3.fr/html/Private/https.context=Web_X509
file.base=file:///c:/tmp/file.base2=file:///c:/
JSAGA 21
LicensesLicenses
LGPL license– for the core engine and for
most adaptors
Optional licenses– for adaptors having external
dependencies, which license is not compatible with LGPL
– then, end-user must…• either accept the terms of the
license agreement,• or go back to previous screen
and uncheck the adaptor(s)
JSAGA 22
Example applicationsExample applications
Pandora Gateway
– the grid platform of the
company
• a leading Grid/Cloud Computing
Business Services Provider
– used in several grid projects,
including
SimExplorer• software for simulation
experiment management• includes a workflow engine that
uses JSAGA to run simulations on grids
• used by experiences in– biology
– computer science
– physics
– food industry
JSAGA 23
Example applicationsExample applications
jCAE (Java CAE)• Computer Aided Engineering
with support for distributed computing facilities
Elis@• web portal for industrial
and academic production grids
• Elis@ is used in several grid projects, including
//
Calcul Intensif pour les LOgicielsde CAO Electronique et lesapplications embarquées
BioInformatique(protein sequence alignement...)
Plasturgy (simulation of injection…)
JSAGA 24
Other interoperability tools – JJSOther interoperability tools – JJS
Java Job Submission
tool to submit jobs on EGEE/EGI grid infrastructure– efficient and scalable
– resource selection based on the QoS observed while submitting jobs
http://cc.in2p3.fr/docenligne/269
uses JSAGA in order to hide middleware evolutions for– security
• globus proxy VOMS proxy
– data management• gridFTP SRM
– execution• LCG-CE CREAM-CE• WMS
used by D0 experiment to submit 15 000 jobs/day on– EGEE/EGI (mainly Europe)– Open Science Grid (USA)
JSAGA 25
Other interoperability tools – JUXOther interoperability tools – JUX
Java Universal eXplorer
multi-protocols file explorer
extensible– for protocols
• just put the JSAGA
plug-ins into the
directory "lib/".
– for viewers
http://cc.in2p3.fr/docenligne/821
JSAGA 26
Example applicationsExample applications
JSAGA provides command line interfaces for…– security
• jsaga-context-init• jsaga-context-info• jsaga-context-destroy
– execution management• jsaga-job-run• jsaga-job-status• jsaga-job-cancel• jsaga-job-output
Also useful as code examples to start with SAGA
– data management• jsaga-cat• jsaga-cp• jsaga-ls• jsaga-mkdir• jsaga-mv• jsaga-rm• jsaga-rmdir• jsaga-stat• jsaga-test• jsaga-logical
JSAGA 27
Interoperability of SAGA implementationsInteroperability of SAGA implementations
Java SAGAJSAGASAGA-C++
Java Bindings for SAGAC++ Bindings SAGA
several implementations of SAGA– different languages (not necessarily)– different characteristics
• features (e.g. automatic/configurable context selection, timeout…)
• optimizations (e.g. caching of connections, meta-data, states…)
• design choices (e.g. high-level / low-level adaptor interfaces)
– different supported API extensions (e.g. SD, Advert, Messaging)– different supported technologies (adaptors)
how to use all these API & technologies simultaneously ?
JSAGA 28
Interoperability of Interoperability of JavaJava implementations implementations
Java SAGAJSAGA
Java Bindings for SAGA
System.setProperty("saga.factory", IMPL_NAME);
File f = FileFactory.createFile(url); // old codeFile f = FileFactory.getInstance().createFile(url); // new codeFile f = FileFactory.getInstance(IMPL_NAME).createFile(url); // new (thread-safe)
SAGA-C++
C++ Bindings SAGA
Application
+ getInstance()
+ getInstance() + getInstance()
+ getInstance()Application
JSAGA 29
Interoperability of Interoperability of PythonPython implementations implementations
C PythonC Python JythonJython
JPySAGA
Python Bindings for SAGA (PySAGA)
Java SAGAJSAGA
Java Bindings for SAGA
SAGA-C++
C++ Bindings SAGA
JySAGA
Application Application
CppySAGA
CppSAGA (legacy)
JSAGA 30
Interoperability : all together…Interoperability : all together…
C PythonC Python JythonJython
JPySAGA
Python Bindings for SAGA (PySAGA)
Java SAGAJSAGA
Java Bindings for SAGA
SAGA-C++
C++ Bindings SAGA
JySAGA
Application
CppySAGA
CppSAGA (legacy)
+ getInstance()
+ getInstance() + getInstance()
+ getInstance()S A G A
JSAGA 31
PerspectivesPerspectives
Develop plug-ins for batch systems– DRMAA-based
• Grid Engine
– CLI-based, with ssh• torque
• Grid Engine
Develop API extensions– Grid-RPC (from SAGA core)– Service Discovery API (a
SAGA extension)• with plug-ins for
technologies– BDII
• with plug-ins for schemas– GLUE– CIM
JSAGA 32
PerspectivesPerspectives
Job submission– principle
• describe user's job• submit it to the grid
– middleware select the execution host
– OGF standards• BES, DRMAA, SAGA
– examples
Remote Procedure Call (RPC)– principle
• deploy user's service on grid• call a procedure
– middleware select the service instance
– OGF standards• Grid-RPC, SAGA
– examples
job Grid-RPC