![Page 1: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/1.jpg)
SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI-283481
gUSE Services
Remote API, DCI Bridge, Data Bridge, Robot Certificate
Zoltán Farkas, Péter Kacsuk, Gábor Herman, István Márton, Tibor Gottdank, Ákos Balaskó
MTA SZTAKI [email protected]
![Page 2: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/2.jpg)
Outline
• Remote API• DCI Bridge• Data Bridge• Robot Certificate
![Page 3: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/3.jpg)
REMOTE API
![Page 4: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/4.jpg)
Goals - Solutions
• Submitting a WS-PGRADE/gUSE workflow using an HTTP client without the web interface of WS-PGRADE
• Solution: The Remote API web service extension of gUSE will be used as application layer in order to communicate with the backend (gUSE) instead of WS-PGRADE portlets
• The workflows to be submit should be available at the caller– As WS-PGRADE workflow definitions contain own set of input files, reference
to input files must be changed, but the files of job executable, and the command line parameters can be changed as well
Solution: Manual change of files to be replaced and the creation of new association table (coded in the „portmapping.txt” file). The main workflow descriptor file <workflow>.xml should be modified only in the case when command line parameters belonging to the individual nodes (jobs) of the workflow must be modified
• On completion of a workflow submission session the common resources should be cleared
• Solution: The servers side data belonging to the workflow (submission) will be cleared upon the successful execution of the „Get output” command of the client
![Page 5: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/5.jpg)
Typical workflow development scenario in case of remote call
WS PGRADE SERVER 1
Portal
gUSE
DCI Bridge
<wf>.xml
<inputs>.zip
<file_i><file_i>
<file_i>
portmapping.txt
HTTP Client
Copy (modify)
Upd. user files
Copy and rename
Eventual update
Set of computational
resources
WS PGRADE SERVER 2
Portal
gUSE
Remote API Serv.
DCI Bridge
Browser
GRAPH Ed
Orig user files
1.Create and submit original workflow
<wf>.zip<wf>.xml
tree of <wf> files
2. Download tested workflow
3. Reengineering of wf definition
4. Submit the modified workflow
Client sideServer side
![Page 6: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/6.jpg)
Summary of workflow development and remote submission process
1. Download the original workflow from WS-PGRADE2. Do some reengineering:
• separate the structure description from the other parts• explore the user defined (not channel) input files connected to the
numbered ports of the named jobs and the files of executable of the named jobs
• Pack the needed files together and create a text file which describes the association of the files and the named jobs and ports
3. Create the needed script files needed to habitual submission process (call, observe, download (delete))
4. Execute the scripts on the client machine
![Page 7: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/7.jpg)
Basic assumptions of remote call setup
• There is a working gUSE set of services where the Remote API Servlet extension has been installed
• There is a client machine containing the description (structure, input files) of a workflow
• The client can reach the server by the HTTP protocol• There is a server-created password known by the client(s)• The client has the necessary proxy certificate file if the submissions
involved in the workflow are directed to such resources which need certificate bound authorization
*(see installation file and documentation on Sourceforge for version <V> :
Code: http://sourceforge.net/projects/guse/files/<V>/remote-3.4.tgz/download
Documentation: http://sourceforge.net/projects/guse/files/<V>/Documentation/RemoteAPI_Install_Manual.pdf/download
)
![Page 8: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/8.jpg)
DCI Bridge
![Page 9: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/9.jpg)
What is DCI Bridge - 1
A web service based application that provides standard access to various distributed computing infrastructures (DCIs), such as: grids, desktop grids, clusters, clouds and service based computational resources (connecting through its DCI plugins to the external DCI resources).
Supp
orte
d D
CIs
![Page 10: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/10.jpg)
When a user submits a workflow, its job components can be submitted transparently into the various DCI systems using the OGSA Basic Execution Service 1.0 (BES) interface. As a result, the access protocol and all the technical details of the various DCI systems are totally hidden behind the BES interface. The standardized job description language of BES is JSDL.
What is DCI Bridge - 2G
ener
ic V
iew
From gUSE or independently (from other workflow systems) can also use DCI Bridge for job
submission
From gUSE or independently (from other workflow systems) can also use DCI Bridge for job
submission
![Page 11: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/11.jpg)
Administration Interface – 1
A sample view of Middleware settings
page.
http:
//LO
CLAH
OST
:808
0/dc
i_br
idge
_ser
vice
/con
f
![Page 12: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/12.jpg)
Submenu name FunctionalityAdd new A new resource reference can be named and added to the groupEdit An existing resource can be selected and its access attributes can be modifiedMonitor Observation of jobs belonging to the given resource groupMiddleware settings Set of generic parameters common in the given middlewareLog entries Opens the history log where the main user activities of the Administrator can be
traced.
Submenu name FunctionalityManager The general flow of jobs can be enabled or disabled in the Manager submenu.Settings It contains properties for generic settings.Log entries Opens the history log where the main user activities of the Administrator can be
traced.
Base
Men
usM
iddl
ewar
e M
enus
Administration Interface – 2
![Page 13: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/13.jpg)
The
job
flow
in th
e D
CI B
ridge
and
bet
wee
n th
e co
mpo
nent
s.
WFI generates and submits the jobs' JSDL to the DCI-BRIDGE.
WFI generates and submits the jobs' JSDL to the DCI-BRIDGE.
pluginspluginsPluginsPlugins
Architecture
![Page 14: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/14.jpg)
DCI BridgeDCI Bridge
Admin Interface (JSP pages)
Semi-autogenerated EventhandlersBES
Middleware Brokering and Management Layer (Plugin Manager)
UnicoreUnicore gLitegLite GT2GT2 GT4GT4 CBCBPBSPBSLocalLocal ...GT5GT5
Middleware Class
<ns3:SDL_Type> <ns3:Constraints> <ns3:Middleware> <ns3:DCIName>local</ns3:DCIName> <ns3:MyProxy/> <ns3:ManagedResource>dci-bridge host(64bit)</ns3:ManagedResource> </ns3:Middleware> <ns3:Budget>0</ns3:Budget> </ns3:Constraints> </ns3:SDL_Type>
JSDL Processing
DCI Bridge accepts standardized JSDL job description documents. These documents are based on a well-defined XML scheme containing information about the job inputs, binaries, runtime settings and output locations.
DCI
Brid
geW
S-P
GR
AD
E
WF
I JSDL
![Page 15: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/15.jpg)
WFI
Job Processing
DCI BRIDGE
DCIWFS Storage
PLUGINWFI requests configuration
data for job submission from WFS. From response WFI
creates a JSDL
1
JSDL
WFI sends JSDL to DCI Bridge2
DCI Bridge gets inputs from
Storage
3
PLUGIN submits the job together with inputs to a DCI, monitors the job
status and gets the outputs.
4
DCI Bridge sends outputs to the Storage5
DCI Bridge sends back job status to WFI.6
![Page 16: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/16.jpg)
Steps of Plugin Creation - 1
1. Developing middleware specific partsImplementing of 4 methods:• submit (invoked on job submission; performs job submission• getStatus (invoked when job status need to be queried; queries
job status and sets job status accordingly• abort (invoked when the job need to be aborted; aborts the
execution of the job)• getOutputs(invoked when the outputs of the job need to be
downloaded; downloads the outputs to a local folder)
![Page 17: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/17.jpg)
Steps of Plugin Creation – 22.Developing configuration interface on DCI Bridge2.1 Adding new middleware name in mb_scheduling_description_language.xsd (other fields relevant for resource (VO, DCI) selection are generic)
….. <xsd:simpleType name="DCINameEnumeration"><xsd:restriction base="xsd:string">
<xsd:enumeration value="glite"/><xsd:enumeration value="arc"/><xsd:enumeration value="unicore"/><xsd:enumeration value="boinc"/><xsd:enumeration value="gemlca"/><xsd:enumeration value="pbs"/><xsd:enumeration value="lsf"/><xsd:enumeration value="gae"/><xsd:enumeration value="service"/><xsd:enumeration value="local"/>
<xsd:enumeration value=”newmid"/>
</xsd:restriction></xsd:simpleType>
![Page 18: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/18.jpg)
Steps of Plugin Creation - 3
<xsd:element name="unicore" minOccurs="0" maxOccurs="1"> <xsd:complexType> <xsd:sequence> <xsd:element name="keystore"> <xsd:simpleType> <xsd:restriction base="xsd:string"/> </xsd:simpleType> </xsd:element> <xsd:element name="keypass"> <xsd:simpleType> <xsd:restriction base="xsd:string"/> </xsd:simpleType> </xsd:element> <xsd:element name="keyalias"> <xsd:simpleType> <xsd:restriction base="xsd:string"/> </xsd:simpleType> </xsd:element> <xsd:element name="subjectdn"> <xsd:simpleType> <xsd:restriction base="xsd:string"/> </xsd:simpleType> </xsd:element> <xsd:element name="truststore"> <xsd:simpleType> <xsd:restriction base="xsd:string"/> </xsd:simpleType> </xsd:element> <xsd:element name="trustpass"> <xsd:simpleType> <xsd:restriction base="xsd:string"/> </xsd:simpleType> </xsd:element> </xsd:sequence> </xsd:complexType></xsd:element>
2.3 Extending existing classes that process tab/menu selection logic
2.4 Creating JSPs
2.2 Extending configuration schema with middleware-specific configuration 2.2 Extending configuration schema with middleware-specific configuration possibilities in possibilities in dci-bridge_configuration_schema_2012.xsddci-bridge_configuration_schema_2012.xsd
![Page 19: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/19.jpg)
Steps of Plugin Creation - 4
New middleware
WS-
PGRA
DECreating new class
(JobConfigUI_newmid)Implementing the getJsp and the getJobParameters methods.
Modification in WFI: Modify the JobConfig class: Adding the new middlewares name in mbsdlMiddleware() method (it maps the job's configuration to the middlewares configured in DCI-Bridge).
3.Developing WS-PGRADE and WFI specific parts
![Page 20: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/20.jpg)
DATA BRIDGE
![Page 21: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/21.jpg)
Outline
• Problem statement• Data Bridge as independent DCI service:
– Data Bridge concept– Use-cases– Data Bridge architecture
• WS-PGRADE integration– Data browsing portlet
• gUSE integration
![Page 22: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/22.jpg)
Problem statement
• Scientific applications:– Individual jobs or workflows– Access data from diverse sources– Science Gateways can hide the details, but…
• Data sources:– Diverse types: HTTP, FTP, GridFTP, SRM, iRODS, …– Thus, different APIs are needed to access these
• One possible solution is to use a service that can be used to access the sources through a unified interface
![Page 23: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/23.jpg)
Data Bridge
• Offers a simple service that provides a generic interface above different DCI's storage services to handle the data stored
• The service in different use cases offers a way to browse, upload and download data, and with the help of multiple server instances it enables inter-DCI data transfer as well
![Page 24: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/24.jpg)
Use cases
• Use case 1: Browse a single DCI data storage from WS-PGRADE, upload data
• Use case 2: Transfer data files between different DCIs
• Use case 3: Fetch input data on a DCI worker node from an other DCI
• Use case 4: Cloud storage usage
![Page 25: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/25.jpg)
Use case 1: Storage browsing and data upload
WS-PGRADE
Storage BrowsingPortlet
Data Bridge
Adaptor InterfaceStorage Adaptor
Storage
Browse and upload
![Page 26: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/26.jpg)
Use case 2: Data Transfer – Using multi-level Data Bridge
Data Bridge
Adaptor Interface
Storage1 Storage2
Data Bridge Adaptor
Client:•Storage Browsing Portlet•Custom application•…
Data Bridge
Adaptor InterfaceStorage Adaptor2
Storage Adaptor1
![Page 27: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/27.jpg)
Use case 3: Fetch data on a DCI’s worker node from a „foreign” DCI’s storage
Data Bridge
Adaptor InterfaceStorage Adaptor
Storage
DCI Worker node
Wrapper
Pre-process
Executable
Post-process
• Data bridge usage guidelines:– First try to fetch the data using native tools– Only if this fails, use the Data Bridge
![Page 28: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/28.jpg)
Use case 4: Cloud Storage access from WS-PGRADE/gUSE
• Currently, no S3 support in WS-PGRADE• An S3 Data Bridge adaptor would fix this
WS-PGRADE/gUSE
DCI
Worker node
Amazon S3 Data Bridge
Job
![Page 29: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/29.jpg)
Data Bridge Architecture
Public Interface
Adaptor Manager
Worker Pool
Thread1 Thread2 Threadn
Adaptor InterfaceDCI Adaptor1 DCI Adaptor2 DCI Adaptor3 DCI Adaptorm
jSAGA
Temporary URL queue
HTTP servlet
URI URI URI
![Page 30: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/30.jpg)
Data Bridge components
• Interfaces:– Public Interface– Adaptor Interface
• Adaptor Manager• Worker Threads• DCI Adaptors
![Page 31: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/31.jpg)
Data Bridge components- Interfaces
• Public Interface:– Provides the public interface for external
components (Portlets, gUSE, …)– Web Service interface
• Adaptor Interface:– A Java interface that hides the details of the
different adaptors
![Page 32: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/32.jpg)
Data Bridge Public Interface
• Operations:– List– Mkdir– Delete– Get– Put– Copy– Move
• Entities:– URI (either a path, an URL or some specific class)
• Error reports:– Common exceptions
![Page 33: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/33.jpg)
Data Bridge Public Interface - URI
• Represents an element with a given URI (a directory, a file, metadata attributes, …)
• Also needs to carry security credentials (if needed)• Attributes:
– Nothing special in the base class– For gLite, e.g:
• Path: the full path• Type: directory or file• Size: length of the entity (0 for directories)• Attributes: optional, contains information as returned by the
Adaptor Interface's Stat function
![Page 34: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/34.jpg)
Data Bridge Public Interface – Get and Put
• Two-phase up- and download with the temporary URL queue:• First, the web service interface is invoked to
register the transfer request• Next, a simple HTTP client may use HTTP GET or
POST/PUT to down- or upload the data
• This way, web service invocation („heavyweight” SOAP) is separated from data transfer („lightweight” HTTP)
Public Interface
Adaptor Manager
Worker Pool
Thread1 Thread2 Threadn
Adaptor InterfaceDCI Adaptor1 DCI Adaptor2 DCI Adaptor3 DCI Adaptorm
Temporary URL queue
HTTP servlet
URI URI URI
![Page 35: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/35.jpg)
Adaptor Manager and Worker threads
• Provided by JAX-WS web service API• Tasks:
– Manage incoming requests– Initialize worker threads to perform the requested
operation– With the help of different adaptors
![Page 36: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/36.jpg)
DCI Adaptors
• Implement: Adaptor Interface• Tasks:
– Perform operations requested by the Worker Threads, that is operations invoked through the web service
• Types:– gLite (using jSAGA)– GridFTP (using jSAGA)– FTP (using jSAGA)– …– Data Bridge: special adaptor to forward requests to other
Data Bridges
![Page 37: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/37.jpg)
Data Bridge clients
• Web Service clients:– Create your own based on the WSDL (or REST)
• Java API:– Provides a convenient tool to use Data Bridge
Public Interface functions– Data transfer functions should accept InputStream
and OutputStream objects as their arguments
![Page 38: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/38.jpg)
WS-PGRADE integration
• A Data Browsing portlet that eases storage management
![Page 39: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/39.jpg)
WS-PGRADE Workflow I/O configuration
• During a workflow node's IO configuration the user should be able to select files from storages
• The provided interface should be the same as the selected storage's Storage Browsing portlet (only with one panel)
![Page 40: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/40.jpg)
Current status, future work
• Core Data Bridge (available as a web service) ready, working with most major protocols (FTP, GridFTP, SRM)
• User Interface development has been started, first version will be available as part of WS-PGRADE/gUSE shortly
![Page 41: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/41.jpg)
ROBOT CERTIFICATE
![Page 42: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/42.jpg)
The concept of robot certificates
• The normal certificate is used to identify users• The robot certificate is used to identify applications• As a consequence the application should be trusted• When the CA provides the certificate for the
application, the certificate contains the identifier of the person or organization that validated the application and takes the responsibility for it
• It is the policy of the user community and the CA to decide whose name should be in the certificate
42
![Page 43: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/43.jpg)
EGI VO Portal Policy I.● The Portal, the VO to which the Portal is associated,
the Portal manager are all individually and collectively responsible and accountable for all interactions with the Grid, unless a credential of a Strongly Identified Web User is used to interact with the Grid
● The Portal must be capable of limiting the job submission rate
● The Portal must keep audit logs for all interactions with the Grid (https://documents.egi.eu/document/81)
![Page 44: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/44.jpg)
EGI VO Portal Policy II.● Portal classes (in fact, these are working mode classes, i.e.
the same portal can be in parameter mode from the point of view of a certain user and in the same time can be in job management mode from the point of view of another user):
SZTAKI gUSE
![Page 45: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/45.jpg)
EGI VO Portal Policy III.● Robot certificates can be used only for the first 3
working modes of the portal● Job management mode portals/applications must
not use robot certificates
![Page 46: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/46.jpg)
SCI-BUS portals and EGI• According to the EGI classification :
– WS-PGRADE/gUSE is a portal that can be used in 1,2,3,4 modes
– The community portals could also work in any of the 4 modes depending of the needs of the corresponding user community
– Robot certificates are needed only for the 1,2,3 modes
46
![Page 47: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/47.jpg)
Relationship between robot certificates and WF applications in WS-PGRADE/gUSE
• The WF applications can have different robot certificates• Even the jobs within a workflow can have different robot
certificates (this enables that different jobs of a WF can be executed in different DCIs requiring different robot certificates)
• This robot certificate contains the name of the community who set up the gateway
• Example:– Autodock gateway set up by SZTAKI and UoW– The robot certificate will contain the name: SCI-BUS
47
![Page 48: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/48.jpg)
WS-PGRADE/gUSE extensions to support robot certificates
• The robot certificates should be hidden for the end-users but manageable for the portal developer/administrator
• The WF applications with robot certificates will be stored in the internal repository of gUSE
• Consequences:– The internal repository should be extended to be able to store the
identification of the robot certificates for every node– The portal developer/administrator (with a new privileged role to be
introduced besides the power and end-user roles) should be able to assign the robot certificates for the WF nodes
48
![Page 49: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/49.jpg)
Suggested process to assign robot certificates to WF nodes
• When a WF is tested and ready to use in the community portal the next step is to assign robot certificates to the nodes of the WF
• This will happen in the following way:1. Portal developer imports the WF from the internal repository2. Use the WF configuration facility of WS-PGRADE (this will be
extended to enable the definition of robot certificates, see next slide)3. Test the workflow with the assigned robot certificates4. Export the WF with assigned robot certificates into the internal
repository• Notice that some nodes of the WF can work with robot certificates while
other nodes require user certificate. Therefore even during the execution of a WF the portal can change among working modes.
49
![Page 50: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/50.jpg)
UI extension to assign robot certificates to WF nodes
50
+ line (robot cert opt) + checkbox
+ auto ModWin
![Page 51: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/51.jpg)
Usable Certificate typesX509 and Myproxy:
A:
B: DN and HOST cert
Unicore:
A:
B: User’s own assertion
CloudBroker Platform:
A:
B: User’s own user/pass
PBS or LSF cluster:A: Generating new keyB: User’s own key
*
![Page 52: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/52.jpg)
Storing the robot certificate for a given job
• When the portal developer saves the WF that contains Job A with robot certificate the following happens:– The portal stores Job A with the required robot certificate
at a secure position. This stored job bundle will get a job bundle identifier
– In the configuration of Job A this identifier will be add the original configuration information
– When the workflow is saved to the repository this new extended configuration field (containing the Job A bundle identifier) will be saved
52
![Page 53: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/53.jpg)
Job Bundle structure
• Binary (executable)• Robot certificate parameters (depending on
the middleware type). This could be:– Login/password (e.g. PBS, cloud)– Certificate access information (e.g. gLite, GTx)
53
![Page 54: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/54.jpg)
Execution of a WF with robot certificates
• When the WFI interprets Job A with the robot certificate it will place into the JSDL the Job A Bundle identifier, too
• When the DCI Bridge receives a job with a bundle identifier, it:– Goes to the Bundle service and asks all the information
related to this bundle– If the security information is a certificate access, then goes
to the credential provider service (in WS-PGRADE) by passing the certificate access information as input and gets back a certificate proxy
54
![Page 55: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/55.jpg)
Further requirements for gUSE to provide
• Suitable log to identify the person who submitted a WF with robot certificate
• Restriction of the number of jobs submitted to a certain VO during a certain time period
55
![Page 56: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/56.jpg)
Suitable log to identify the person who submitted a WF
• Each logged event must contain:– A timestamp– The portal user’s ID– The portal user’s IP
• Job submit event should contain:– WF’s name– Job’s name– Job’s PID (in case of parametric jobs)– Job’s DCI Bridge ID– Credential used (proxy DN, username, …)– Input file list with sizes– Grid ID in case of successful submission– Error message in case of job submission failure
• These are already stored in DCI Bridge (see next slide)
![Page 57: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/57.jpg)
Suitable log to identify the person who submitted a WF
• Job status change events should contain:– The job’s DCI Bridge ID– The job’s new status– Optionally: the job’s old status
• Terminal job status events should contain:– Outcome (success, failure, with exit code)– List and size of output files transferred
![Page 58: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/58.jpg)
Log information in DCI Bridge
• After completing a job a zip file is created identified by the job identifier and containing all the information of the previous slide. This contains 3 files:– DCI Bridge log– JSDL– Log created in the DCI resource
• We will store these zip files in a temporary storage area but archiving them will be the responsibility of the portal admin
58
![Page 59: gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate](https://reader035.vdocument.in/reader035/viewer/2022062217/56813d54550346895da71163/html5/thumbnails/59.jpg)
Restriction of the number of jobs submitted to a certain VO
• This information will be stored in the DCI Bridge
• This is a parameter configurable by the portal admin for each DCI VO
• This information should be given when the robot certificate is assigned to a certain a job
59