jsdl/bes tutorial moreno marzolla, infn padova [email protected] omii-eu all hands meeting,...

JSDL/BES TutorialMoreno Marzolla, INFN Padova

[email protected] All Hands Meeting, Stockholm, jul 31—aug 2, 2007

EU project: RIO31844-OMII-EUROPE

Talk outline

• Introduction to job submission• The JSDL specification• The OGSA-BES specification• Conclusions


Aim of this presentation

• This presentation aims at providing the audience with an overview of the Job Submission Description Language (JSDL) and OGSA Basic Execution Service (BES) specifications

• Presentation will be implementation-agnostic– We focus here on technologies, not specific

implementations


Introduction to Job Submission

• Job Submission is one of the basic functionalities of any Grid system

• User submits a (computational) job to an appropriate Execution Service, and uses the same service to manage the job– Start/Stop execution– Pause/Resume execution– Query job status


So, what is the problem?

• Different Grid Systems provide different interfaces for job submission and management

• Those interfaces provides mostly a common set of functionalities, but with slightly different semantics and/or syntax– Different names of job management operations– Different syntax used to describe a job– Different state models– ...


The OGSA/GGF Solution

• Job Submission Description Language (JSDL)– OGF Specification– XML-based notation for describing computational jobs

and their resource requirements• OGSA Basic Execution Service (BES)

– OGSA Specification– WebService-based interface for job submission and

management– Uses JSDL to describe the jobs


JSDL

• Job Submission Description Language• Most recent specification is Version 1.0

– Version 1.1 being worked out• Specification (V1.0) available at

http://www.gridforum.org/documents/GFD.56.pdf• JSDL Working Group

https://forge.gridforum.org/projects/jsdl-wg/


JSDL: Motivations• Many organizations accommodate a variety of job

management systems– Each system has its own language for describing job

submission requirements. – This makes interoperability between these systems for job

management difficult• Grid environments may involve the interaction of a

number of different types of job management systems– The description of a job may be transformed by

intermediaries – This description may be passed between systems or

instantiated on resources matching the resource requirements for that job.


Job Submission Scenario

Job Manager Job Manager

Super-scheduler, or Broker, or …

Super-scheduler, or Broker, or …

A Grid Information

System

Local Information

System

WS intermediaryWS intermediary

DRM

Local Resource

(e.g., super-computer)

Another JSDL System

J

J

J

J

J

JJSDL document:

WS Client

WS Client

J

J

WS Client

Source: JSDL Specification, version 1.0


JSDL Scope

• What JSDL is– JSDL is a language for describing the submission

requirements of individual jobs. • What JSDL is not

– It does not attempt to address the entire lifecycle of a job or the relationship between individual jobs

– It does not address job management (BES does that)– It does not consider Workflows– It does not mandate how resource matching should be

performed– ...


JSDL Document Structure

<JobDefinition id=”xsd:ID”?><JobDescription>

<JobIdentification ... />?<Application ... />?<Resources ... />?<DataStaging ... />*

</JobDescription><xsd:any##other/>*

</JobDefinition>

• JobDefinition describes the job and its requirements

• This is the root of the JSDL document

• The (optional) id attribute can be specified






</JobDefinition>

• JobDescription describes the job and its requirements

• This is the actual content of the JSDL document






</JobDefinition>

• JobIdentification is a complex XML type and contains elements to identify the job

• The multiplicity of this element is zero or one






</JobDefinition>

• Application describes the application which is meant to be executed– This description is

independent of the OS• Application has a

complex XML type• The multiplicity of this

element is zero or one– If the Application

element is missing, the null job is assumed






</JobDefinition>

• Resources describes the resource requirements of the job

• This element supports only a limited set of core requirements

• Additional requirements can be defined through extensions– Even better, using

GlueSchema?






</JobDefinition>

• DataStaging describes the data requirements of the job– Which files are used as

input (stage in)– Which files are

produced as output (stage out)

• This element can be repeated more than once if more than a single file is required/produced


JSDL Core Element Set:JobIdentification




</JobDefinition>


JSDL Core Element Set:JobIdentification

• JobName (xsd:string) is the user-defined name for this job

– Can be the same for multiple JSDL documents

• Description (xsd:string) is a textual description of the job

• JobAnnotation (xsd:string) contains informations which (unlike Description) MAY be used by the receiving system to process the job

• JobProject (xsd:string) specifies the project to which the job belongs.

– The project could be used by accounting systems or access control systems

<JobIdentification><JobName ... />?<Description ... />?<JobAnnotation ... />*<JobProject ... />*<xsd:any##other>*

</JobIdentification>?


JSDL Core Element Set:Application




</JobDefinition>


JSDL Core Element Set:Application

• ApplicationName is a string that defines the name of the application and is used to identify the application independent of the location of its executable on a host or system

• ApplicationVersion is a string that defines the version of the application to execute

• Description is a string with a textual description of the application

<Application><ApplicationName ... />?<ApplicationVersion ... />?<Description ... />?<xsd:any##other/>*

</Application>


Example

<JobDefinition id=”HelloWorldJob”><JobDescription>

<JobIdentification><JobName>Hello World</JobName><Description>This job prints the Hello World

message</Description><JobProject>OMII-EU</JobProject><JobAnnotation>uuid:090fe040e0</JobAnnotation>

</JobIdentification><Application>

<ApplicationName>Echo</ApplicationName><ApplicationVersion>1.0</ApplicationVersion><Description>This application prints the Hello

World message to the standard output</Description></Application>

</JobDescription></JobDefinition>


JSDL Core Element Set:Resources Element




</JobDefinition>


JSDL Core Element Set:Resource Element

• Contains the resource requirements of the job.

• If this element is not present then the consuming system MAY choose any set of resources to execute the job.

<Resources><CandidateHosts ... />?<FileSystem .../>*<ExlusiveExecution .../>?<OperatingSystem .../>?<CPUArchitecture .../>?<IndividualCPUSpeed .../>?<IndividualCPUTime .../>?<IndividualCPUCount .../>?<IndividualNetworkBandwidth .../>?<IndividualPhysicalMemory .../>?<IndividualVirtualMemory .../>?<IndividualDiskSpace .../>?<TotalCPUTime .../>?<TotalCPUCount .../>?<TotalPhysicalMemory .../>?<TotalVirtualMemory .../>?<TotalDiskSpace .../>?<TotalResourceCount .../>?<xsd:any##other>*

</Resources>?


CandidateHosts elements

• Specifies the set of named hosts which may be selected for running the job.

• If this element is present then one or more hosts from the set MUST be chosen to run the job.

• A named host may be a single host (e.g., a machine name), a logical group of hosts (e.g., a named logical group or cluster), a virtual machine, and so on.

<CandidateHosts><HostName .../>+

</CandidateHosts>?

<jsdl:CandidateHosts><jsdl:HostName> bach.example.com </jsdl:HostName><jsdl:HostName> mozart.example.com </jsdl:HostName><jsdl:HostName> beethoven.example.com </jsdl:HostName>

</jsdl:CandidateHosts>


FileSystem element

• Describes a filesystem that is required by the job.

• It may contain the location where the filesystem should be made available, the required amount of disk space and the type of the filesystem.

• The filesystem may be local to the resource (e.g., on a local disk), or may be remote (e.g., an NFS mount).

<FileSystem name=”xsd:NCName”><Description ... />?<MountPoint ... />?<MountSource ... />?<DiskSpace ... />?<FileSystemType ... />?<xsd:any##other/>*

</FileSystem>*


FileSystem element

• Textual description of the FS

• xsd:string that describes a local location that MUST be made available in the allocated resources for the job.

• xsd:string that describes a remote location that MUST be made available locally for the job.

• This is a range value (see later) that describes the required amount of disk space (in bytes) on the containing FileSystem element for the job.

• This is a token that describes the type of filesystem (see next slide)

<FileSystem name=”xsd:NCName”><Description ... />?<MountPoint ... />?<MountSource ... />?<DiskSpace ... />?<FileSystemType ... />?<xsd:any##other/>*

</FileSystem>*


FileSystemType enumeration

Normative JSDLName

Definition

swap Conventional swap space for paging out memorytemporary Conventional temporary space for a file that is

periodically removed. The space is unavailable after thejob completes

spool Temporary space for a file that MAY persist after the jobcompletes

normal This is a normal space for files to be written to and readfrom. Files are not deleted after the job completes unlessthe user explicitly requests it


FileSystem element examples

<jsdl:FileSystem name="ROOT"><jsdl:Description>

The root of the main system filesystem.Not safe to assume that it is writeable. The actual root of the filesystem depends on the OS

</jsdl:Description><jsdl:FileSystemType>normal</jsdl:FileSystemType>

</jsdl:FileSystem>

<jsdl:FileSystem name=”HOME”><jsdl:Description>Ali’s home</jsdl:Description><jsdl:MountPoint>/home/ali</jsdl:MountPoint><jsdl:DiskSpace>

<jsdl:LowerBoundedRange>1073741824.0</jsdl:LowerBoundedRange></jsdl:DiskSpace><jsdl:FileSystemType>normal</jsdl:FileSystemType>

</jsdl:FileSystem>


Resource Element: ExclusiveExecution

• boolean that designates whether the job must have exclusive access to the resources allocated to it by the consuming system.– True—run

exclusively on the resources

– False—others can run at the same time


</Resources>?


Resource Element: OperatingSystem

• Complex type that defines the operating system required by the job.


</Resources>?

<OperatingSystem> <OperatingSystemType ... />?<OperatingSystemVersion ... />?<Description ... />?<xsd:any##other/>*

</OperatingSystem>?


Operating System EnumerationNormative JSDL Names

Unknown WINNT LINUX HP_MPEOther WINCE Lynx NextStepMACOS NCR3000 XENIX PalmPilotATTUNIX NetWare VM RhapsodyDGUX OSF Interactive_UNIX Windows_2000DECNT DC_OS BSDUNIX DedicatedTru64_UNIX Reliant_UNIX FreeBSD OS_390OpenVMS SCO_UnixWare NetBSD VSEHPUX SCO_OpenServer GNU_Hurd TPFAIX Sequent OS9 Windows_R_MeMVS IRIX MACH_Kernel Caldera_Open_UNI

XOS400 Solaris Inferno OpenBSDOS_2 SunOS QNX Not_ApplicableJavaVM U6000 EPOC Windows_XPMSDOS ASERIES IxWorks z_OSWIN3x TandemNSK VxWorksWIN95 TandemNT MiNTWIN98 BS2000 BeOS


Resource Element: CPUArchitecture

• xsd:string specifying the CPU architecture required by the job in the execution environment.


</Resources>?

<jsdl:CPUArchitecture><jsdl:CPUArchitectureName>other</jsdl:CPUArchitectureName> <tns:OtherCPUArchitectures>cell</tns:OtherCPUArchitectures>

</jsdl:CPUArchitecture>


CPUArchitecture Name Enumeration

Normative JSDLName

Definition

sparc A SPARC architecture processorpowerpc A PowerPC architecture processorx86 An Intel Architecture processor derived from the 8086 chip setx86_32 An x86 processor capable of 32-bit processing modex86_64 An x86 processor capable of 64-bit processing modeparisc A PARISC architecture processormips A MIPS architecture processoria64 An Intel Architecture 64-bit processorarm An ARM processorother A value not defined by this enumeration


Range specification

• A range value is a complex type that allows the definition of exact values (with an optional “epsilon” argument), left-open or right-open intervals and ranges.

• All numbers given are of type xsd:double. UpperBoundedRanges and LowerBoundedRanges are limited to the upper or lower bound, respectively.

• Ranges may be “unlimited” to either negative or positive infinity


Ranges pseudoschema

<UpperBoundedRange exclusiveBound="xsd:boolean"?>xsd:double

</UpperBoundedRange>?<LowerBoundedRange exclusiveBound="xsd:boolean"?>

xsd:double</LowerBoundedRange>?<Exact epsilon=”xsd:double”?>

xsd:double </Exact>*<Range>

<LowerBound exclusiveBound="xsd:boolean"?>xsd:double

</LowerBound><UpperBound exclusiveBound="xsd:boolean"?>

xsd:double</UpperBound>

</Range>*


Example

• The pseudo-expression “5, 6.7777, 7.0, [50.3,99.5), [100-” (which indicates a disjoint range consisting of the values 5, 6.7777, 7.0, all values inclusive of from 50.3 up to, but not including, 99.5 and all values from 100 on up) can be encoded in a RangeValue as follows:

<jsdl:LowerBoundedRange> 100.0 </jsdl:LowerBoundedRange><jsdl:Exact> 5.0 </jsdl:Exact><jsdl:Exact epsilon=”0.00001”> 6.7777 </jsdl:Exact><jsdl:Exact> 7.0 </jsdl:Exact><jsdl:Range>

<jsdl:LowerBound> 50.3 </jsdl:LowerBound><jsdl:UpperBound exclusiveBound=”true”> 99.5 </jsdl:UpperBound>

</jsdl:Range>


Matching semantics• A numeric value, N, matches a RangeValue, R, if and

only if at least one of the following conditions is true:– R contains an UpperBoundedRange, U, with a false exclusiveBound

attribute and where N ≤ U.

– R contains an UpperBoundedRange, U, with a true exclusiveBound attribute and where N < U.

– R contains a LowerBoundedRange, L, with a false exclusiveBound attribute and where N ≥ L.

– R contains a LowerBoundedRange, L, with a true exclusiveBound attribute and where N > L.

– R contains an Exact, E, with an epsilon attribute e, where E-e ≤ N ≤ E+e.

– R contains a Range with LowerBound, L, and UpperBound, U, such that both of the following are true:

• L has a false exclusiveBound attribute and N ≥ L, or L has a true exclusiveBound attribute and N > L.

• U has a false exclusiveBound attribute and N ≤ U, or U has a true exclusiveBound attribute and N < U.


Resource Element: Individual*, Total*

• Requirements on resources provided by each single node, or by the aggregate allocated resource pool


</Resources>?

Example

<Resources><IndividualCPUSpeed>

<Exact> 1000000000 </Exact></IndividualCPUSpeed><IndividualCPUCount>

<LowerBound> 4.0 </LowerBound></IndividualCPUCount><TotalCPUCount>

<UpperBound> 16.0 </UpperBound></TotalCPUCount>

</Resources>?

Each resource has at least 4 CPUs at 1Ghz each

All the resources must provide at most 16 CPUs

Beware of unsatisfiable requirements!

• Here we request:– At most 8 resources...– ...each one providing

exactly 1 CPU...– ...with a total CPU count

of exactly 16

<Resources><IndividualCPUCount>

<Exact> 1.0 </Exact></IndividualCPUCount><TotalCPUCount>

<Exact> 16.0 </Exact></TotalCPUCount><TotalResourceCount>

<UpperBound> 8.0 </UpperBound></TotalResourceCount>

</Resources>?


JSDL Core Element Set:DataStaging Element




</JobDefinition>


Data Staging Elements

• Data staging defines the files that should be moved to the execution host (stage in) and the files that should be moved from the execution host (stage out).

• Files are staged in before the job starts executing. Files are staged out after the job terminates. – More complex file transfers (e.g., conditional transfers

based on job termination status or pre-warming of grid-enabled file-systems) are out of scope of the JSDL specification.

• Permission and access control for the staged files should be handled by the implementation and is out of scope of the JSDL specification.


DataStaging element

• Note: the ordering of the DataStaging elements in the JSDL document is not significant <DataStaging name=”xsd:NCName”?>

<FileName ... /><FileSystemName ... />?<CreationFlag ... /><DeleteOnTermination ... />?<Source ... />?<Target ... />?<xsd:any##other/>*

</DataStaging>*


DataStaging element:FileName

• This element is a string specifying the local name of the file (or directory) on the execution host. The FileName MUST be a relative path

<DataStaging name=”xsd:NCName”?><FileName ... /><FileSystemName ... />?<CreationFlag ... /><DeleteOnTermination ... />?<Source ... />?<Target ... />?<xsd:any##other/>*

</DataStaging>*


DataStaging element:FileSystemName

• If the FileSystemName is specified then the FileName is relative to the specified FileSystem declaration referenced by the name.

• In this case there MUST also be a FileSystem element with the same name.


</DataStaging>*


DataStaging element:CreationFlag

• This element determines whether the file created on the local execution system can overwrite or append to an existing file.

• A typical value for this element, expected to be commonly supported, is “overwrite.”


</DataStaging>*


DataStaging element: DeleteOnTermination

• This is a boolean that determines whether the file should be deleted after the job terminates. – If true the file is deleted after

the job terminates or after the file has been staged out.

– Otherwise the file remains, subject to the persistency of the FileSystem it is on.

• If not present, behavior is unspecified


</DataStaging>*


DataStaging element:Source

• A Source element contains the location of the file or directory on the remote system.

• This file or directory MUST be staged in from the location specified by the (optional) URI before the job has started.


</DataStaging>*


DataStaging element:Target

• A Target element contains the location of the file or directory on the remote system.

• This file or directory MUST be staged out to the location specified by the (optional) URI after the job has terminated.


</DataStaging>*


Examples

• Staging in a file

<jsdl:DataStaging><jsdl:FileName>control.txt</jsdl:FileName> <jsdl:Source>

<jsdl:URI>http://foo.bar.com/~me/control.txt</jsdl:URI></jsdl:Source> <jsdl:CreationFlag>overwrite</jsdl:CreationFlag> <jsdl:DeleteOnTermination>true</jsdl:DeleteOnTermination>

</jsdl:DataStaging>


Examples

• Staging in a file relative to a filesystem

<jsdl:FileSystem> ... </jsdl:FileSystem>...<jsdl:DataStaging>

<jsdl:FileName>control.txt</jsdl:FileName> <jsdl:FileSystemName>HOME</jsdl:FileSystemName><jsdl:Source>

<jsdl:URI>http://foo.bar.com/~me/control.txt</jsdl:URI></jsdl:Source> <jsdl:CreationFlag>overwrite</jsdl:CreationFlag> <jsdl:DeleteOnTermination>true</jsdl:DeleteOnTermination>

</jsdl:DataStaging>


Examples

• It is possible to define both Source and Target elements so that the file is first staged in before the job starts execution and staged out after the job finishes.

<jsdl:DataStaging><jsdl:FileName>state.txt</jsdl:FileName> <jsdl:Source>

<jsdl:URI>http://node1/~me/state.txt</jsdl:URI></jsdl:Source><jsdl:Target>

<jsdl:URI>http://node2/~me/state.txt</jsdl:URI></jsdl:Target> <jsdl:CreationFlag>overwrite</jsdl:CreationFlag> <jsdl:DeleteOnTermination>true</

jsdl:DeleteOnTermination> </jsdl:DataStaging>


Examples

• Multiple stage out operations may be specified by using the same FileName (on the same FileSystem) in separate DataStaging elements.

<jsdl:DataStaging><jsdl:FileName>result.txt</jsdl:FileName> <jsdl:Target>

<jsdl:URI>http://node1/~me/result.txt</jsdl:URI></jsdl:Target>...

</jsdl:DataStaging>...<jsdl:DataStaging>

<jsdl:FileName>result.txt</jsdl:FileName> <jsdl:Target>

<jsdl:URI>http://node2/~me/result.txt</jsdl:URI></jsdl:Target>...

</jsdl:DataStaging>


JSDL attribute extensions• Every JSDL element allows for additional attributes, as

many as needed (provided they have a namespace other than the normative JSDL one)

• Example: define an attribute called “order” that defines the order of staging in files

<?xml version="1.0" encoding="UTF-8"?> <jsdl:JobDefinition xmlns="http://www.example.org/"

xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/11/jsdl" xmlns:o="http://www.example.org/order-of-execution">

<jsdl:JobDescription> <jsdl:DataStaging o:order="1">

<jsdl:FileName>foo</jsdl:FileName> <jsdl:CreationFlag>overwrite</jsdl:CreationFlag> <jsdl:Source>

<jsdl:URI>http://www.nowhere.com/foo-file</jsdl:URI></jsdl:Source>

</jsdl:DataStaging> <jsdl:DataStaging o:order="2">

<jsdl:FileName>bar</jsdl:FileName> <jsdl:CreationFlag>overwrite</jsdl:CreationFlag> <jsdl:Source>

<jsdl:URI>http:://www.nowhere.com/bar-file</jsdl:URI></jsdl:Source>

</jsdl:DataStaging> </jsdl:JobDescription>

</jsdl:JobDefinition>

JSDL element extensions

• Similarly for elements

...<jsdl:Resources>

<jsdl:TotalCPUCount><jsdl:Exact>1.0</jsdl:Exact>

</jsdl:TotalCPUCount> <jsdl:TotalDiskSpace>

 <jsdl:LowerBoundedRange>1073741824.0</jsdl:LowerBoundedRange>

</jsdl:TotalDiskSpace> <res:Reservation xmlns:res="http://www.example.org/reservation">

<res:Ticket>h933fsolenri900wnmd90mm34</res:Ticket> </res:Reservation>

</jsdl:Resources> ...


Normative extension: POSIX Application

• This normative extension defines a schema describing an application executed on a POSIX compliant system.

• The PosixApplication element describes a POSIX style Application and its requirements. – It contains Executable, Argument, Input, Output, Error,

WorkingDirectory, Environment, various POSIX limits elements as well as User and Group names.

– If it is present as a sub-element of the JSDL Application element it MUST appear only once.


POSIX Application Normative Schema<POSIXApplication name=”xsd:NCName”?>

<Executable ... />?<Argument ... />*<Input ... />?<Output ... />?<Error ... />?<WorkingDirectory ... />?<Environment ... />*<WallTimeLimit ... />?<FileSizeLimit ... />?<CoreDumpLimit ... />?<DataSegmentLimit ... />?<LockedMemoryLimit ... />?<MemoryLimit ... />?<OpenDescriptorsLimit ... />?<PipeSizeLimit ... />?<StackSizeLimit ... />?<CPUTimeLimit ... />?<ProcessCountLimit ... />?<VirtualMemoryLimit ... />?<ThreadCountLimit ... />?<UserName ... />?<GroupName ... />?

</POSIXApplication>?

Final JSDL Example / 1<?xml version="1.0" encoding="UTF-8"?><jsdl:JobDefinition xmlns="http://www.example.org/"

xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/11/jsdl" xmlns:jsdl-posix="http://schemas.ggf.org/jsdl/2005/11/jsdl-posix" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<jsdl:JobDescription><jsdl:JobIdentification>

<jsdl:JobName>My Gnuplot invocation</jsdl:JobName><jsdl:Description> ... </jsdl:Description>

</jsdl:JobIdentification><jsdl:Application>

<jsdl:ApplicationName>gnuplot</jsdl:ApplicationName><jsdl-posix:POSIXApplication>

<jsdl-posix:Executable>/usr/local/bin/gnuplot

</jsdl-posix:Executable><jsdl-posix:Argument>control.txt</jsdl-posix:Argument><jsdl-posix:Input>input.dat</jsdl-posix:Input><jsdl-posix:Output>output1.png</jsdl-posix:Output>

</jsdl-posix:POSIXApplication></jsdl:Application><jsdl:Resources>

<jsdl:IndividualPhysicalMemory><jsdl:LowerBoundedRange>2097152.0</jsdl:LowerBoundedRange>

</jsdl:IndividualPhysicalMemory><jsdl:TotalCPUCount>

<jsdl:Exact>1.0</jsdl:Exact></jsdl:TotalCPUCount>

</jsdl:Resources>

Final JSDL Example / 2<jsdl:DataStaging>

<jsdl:FileName>control.txt</jsdl:FileName><jsdl:CreationFlag>overwrite</jsdl:CreationFlag><jsdl:DeleteOnTermination>true</jsdl:DeleteOnTermination><jsdl:Source>

<jsdl:URI>http://foo.bar.com/~me/control.txt</jsdl:URI></jsdl:Source>

</jsdl:DataStaging><jsdl:DataStaging>

<jsdl:FileName>input.dat</jsdl:FileName><jsdl:CreationFlag>overwrite</jsdl:CreationFlag><jsdl:DeleteOnTermination>true</jsdl:DeleteOnTermination><jsdl:Source>

<jsdl:URI>http://foo.bar.com/~me/input.dat</jsdl:URI></jsdl:Source>

</jsdl:DataStaging><jsdl:DataStaging>

<jsdl:FileName>output1.png</jsdl:FileName><jsdl:CreationFlag>overwrite</jsdl:CreationFlag><jsdl:DeleteOnTermination>true</jsdl:DeleteOnTermination><jsdl:Target>

<jsdl:URI>rsync://spoolmachine/userdir</jsdl:URI></jsdl:Target>

</jsdl:DataStaging></jsdl:JobDescription>

</jsdl:JobDefinition>


Basic Execution Service (BES)

• The BES specification defines Web Services interfaces for creating, monitoring, and controlling computational entities – UNIX or Windows processes, Web Services, or parallel

programs—called activities—within a defined environment.

• Clients define activities using JSDL• A BES implementation executes each activity that it

accepts on an appropriate computational resource– the resource may be a single computer; a cluster; a

Web Service hosting environment; or even another BES implementation.


BES

• Basic Execution Service• Latest version is Draft v35

– Right out of the public comment phase– Should become the final specification v1.0

• Specification (draft v35) available at http://forge.ogf.org/sf/go/doc14658?nav=1

• OGSA-BES Working Group http://forge.ogf.org/sf/projects/ogsa-bes-wg


BES

• The BES specification defines three Web Services port-types, aimed at different types of client: – the BES-Factory port-type allows clients to create, monitor, and

control sets of activities, and to monitor BES attributes, such as a characterization of the resources the BES makes available to activities and the number of activities it currently has instantiated. This port-type is intended for use by ordinary clients.

– the BES-Activity port-type allows clients to create, monitor, and control individual activities. This port-type is intended for use by ordinary clients.

– the BES-Management port-type allows clients to monitor the details of and control the BES itself. This port-type is intended for use by system administrators.


BES service port-types

BESService

StopAcceptingNewActivitiesStartAcceptingNewActivities

CreateActivityGetActivityStatusesTerminateActivitiesGetActivityDocuments GetFactoryAttributesDocument

GetStatus (?)Terminate (?)GetDocument (?)GetActivityAttributesDocument (?)

BE

S-

Ma

na

ge

me

nt

po

rt-t

yp

e

BE

S-F

ac

tory

po

rt-t

yp

e

BE

S-

Ac

tiv

ity

po

rt-t

yp

e(o

pti

on

al)


Basic State Model• All activities MUST traverse the following set of states:

– Pending: The service has created a record for an activity but not yet instantiated it on a suitable computational resource or enabled it to start execution on such a resource.

– Running: The activity is executing on some computational resource.

– Finished (a terminal state): The activity has terminated successfully. Successful termination implies that the activity exited of its own accord rather than due to some failure in the BES or of the computational resources on which the activity was running. A successfully terminating activity may nevertheless return an error code as its return value.

– Terminated (a terminal state): The client – which might be some system administrator (and hence not necessarily the client who originated the request to create the activity) – has issued a TerminateActivity request.

– Failed (a terminal state): The activity has failed due to some system error/failure event, such as failure of a computational resource that the activity was running on.


Basic State Model

Pending Running

Failed

Terminated

Finished

TerminateActivity request

Succesful termination of activity

System error or failure

event


State Specialization

• It is possible to define “state profiles” providing detailed sub-state information to compliant clients

• Example:

Pending Running:Stage-in

Failed

Terminated

FinishedRunning:Execute

Running:Stage-out


State Specialization Rules

• A specialization can introduce sub-states only by replacing a state in the state model that it is specializing with a graph of sub-states and transitions among those sub-states.

• Transitions MUST maintain consistency with the basic state model (see example in next slide)


Invalid State Specialization example

PendingRunning

:S1

Failed

Terminated

FinishedRunnin

g:S2

NO!!


BES-Management port-type

• StopAcceptingNewActivities– This operation is used to request that the BES stop

accepting new activities.– No input or output data is required

• StartAcceptingNewActivities– This operation is used to request that the BES start

accepting new activities.– No input or output data is required


BES-Factory port-type

• The BES-Factory port type contains operations that support the creation and manipulation of activities controlled or maintained by the BES.

• It also contains an operation for retrieving attribute information about the BES itself which effectively provides an abstract characterization of a particular BES-Factory.


BES-Factory operations:CreateActivity

• This operation is used to request the creation of a new activity• Input

– ActivityDocumentType ActivityDocument: An XML document describing a single activity that is to be created

• Output– CreateActivityResponseType Response: On success,

Response contains an ActivityIdentifier (EPR) identifying the requested activity.

• Faults– NotAuthorizedFault– NotAcceptingNewActivities– UnsupportedFeatureFault– InvalidRequestMessageFault

<bes:ActivityDocument> <jsdl:JobDefinition> ... </jsdl:JobDefinition> <xsd:any/> *</bes:ActivityDocument>


BES-Factory operations:GetActivityStatuses

• This operation queries the status of a set of activities• Input

– EPR[] ActivityIdentifier: A vector of one or more EPRs (obtained from previous CreateActivity operations), indicating activities from which we require status information.

• Output– GetActivityStatusResponseType[] Response: An XML

document containing a vector of ActivityStatus elements (see below), one for each EPR provided in the input. Note that because many activities can be queried, this document may become large.

• Faults– UnknownActivityIdentifierFault: The specified EPR cannot be

matched to an activity known to the container.


BES-Factory operations:GetActivityStatuses

• This operation requests that a specified set of activities be terminated. The BES attempts to terminate each activity specified

• Input– EPR[] activities: A vector of one or more EPRs identifying the

activities that are to be terminated.• Output

– TerminateActivityResponseType[] Response: A vector detailing the responses of the BES to the termination requests. The Terminated element is a boolean value: If true is returned, then the associated activity is now in the Terminated state. If false then the activity MAY eventually transition to the Terminated state.


matched to an activity known to the container.


BES-Factory operations:GetActivityDocuments

• This method is used to get the JSDL associated to an activity• Input

– EPR[] ActivityIdentifiers: A vector of one or more EPRs identifying the activities for which ActivityDocuments are requested.

• Output– GetActivityDocumentResponseType [] Response. The output

from this operation returns a vector of ActivityDocumentResponse elements.


matched to an activity known to the container. The EPR may reference an activity that has come to an end and been removed from the system.


BES-Factory operations:GetFactoryAttributesDocument

• This operation is used to request a document containing the BES-Management attributes.

• Input– None

• Output– BESResourceAttributesDocumentType

BESResourceAttributesDocument: An XML document containing the various attributes listed in the next slide

• Faults– None


BES-Factory attributesName Type Number Data Type Description

IsAcceptingNewActivities BES 1 Boolean True if BES is accepting new activitiesCommonName BES 0 or 1 String Short human-readable name for BESLongDescription BES 0 or 1 String Longer human-readable description.TotalNumberOfActivities BES 1 Integer Number of activities currently managed in BESActivityReference BES ≥0 EPR EPRs to activities currently active in BES TotalNumberOfContainedResources

BES1

Integer Number of contained resources accessible bythe BES

ContainedResource BES≥0

anyType Currently expected to be either of typeBasicResourceAttributesDocumentType orFactoryResourceAttributesDocumentType

NamingProfile BES ≥1 URI URIs of Naming profiles used by BES BESExtension BES ≥0 URI URIs of supported BES extensionsLocalResourceManagerType BES 1 URI Resource’s local resource manger typeResourceName BR 0 or 1 String Resource’s nameOperatingSystem BR 0 or 1 String Resource’s operating system typeCPUArchitecture BR 0 or 1 String Resource’s CPU architecture typeCPUCount BR 0 or 1 Double Resource’s CPU countCPUSpeed BR 0 or 1 Double Resource’s CPU speed, in HertzPhysicalMemory BR 0 or 1 Double Resource’s physical memory, in bytesVirtualMemory BR 0 or 1 Double Resource’s virtual memory, in bytes

Type: BES denotes a BES specific attribute, while BR denotes a basic resource attribute (i.e., one that applies only to underlying resources)


BES-Activity port-type

• The BES-Activity port-type defines operations for monitoring and managing individual activities. – These operations are intended to be applied to EPRs

returned by previous CreateActivity operations.• No operations are defined for the BES-Activity port

type. – These would be expected to include ‘single’ activity

versions of the operations on the BES-Factory port type, i.e. GetStatus, Terminate, GetDocument, GetActivityAttributesDocument.


Optional Extensions:Idempotent Execution Semantics

• If idempotent execution is required, the following element MUST be used to uniquely identify an activity to the BES using a client-generated identifier

• If present in an activity description, the BES MUST NOT execute the activity containing this identifier more than once– Should a BES receive a second CreateActivity request that

includes the same identifier as a previously received request, the BES MUST NOT create the requested activity a second time if it already created the activity for the first request.

<bes-ext:IdempotentActivityID> wsa:AttributedURI</bes-ext:IdempotentActivityID>


Optional Extensions:Subscription to Notification Events

• A BES that allows its clients to subscribe for messages concerning activity state changes MUST do so using either the WS-Eventing or WS-Notification protocols.

• Compliant BESs implementing this extension MUST support the (optional) inclusion of one of the following subscription request elements as an extension to the ActivityDocument element in the input to the CreateActivity request:

• Then, one of the following associated subscription reference elements MUST be added as an extension to the ActivityDocument element returned in the response to CreateActivity

<wsnt:Subscribe/> OR <wse:Subscribe/>

<wsnt:SubscribeResponse/> OR <wse:SubscribeResponse/>


Optional Extensions:Lifetime Management

• A BES that implements the WS-ResourceLifetime operation SHOULD use the following element to indicate the requestor’s suggestion for the initial setting of the termination time resource property (WS-ResourceLifetime) of the activity:

<bes-ext:InitialTerminationTime> xsd:dateTime</bes-ext:InitialTerminationTime>


Security considerations

• A BES will typically want to control who can invoke its various operations.

• There may also be a need for identity mapping between a grid credential and the BES’s authorization and authentication space.

• These considerations are outside the scope of this document. – The WS-I Basic Security Profile provides advice to

implementers as to how to provide interoperable security solutions.

• This problem is being addressed in OMII JRA3/Security


Conclusions

• The BES specification is succesful in addressing the most basic functionalities required by job submission and management, with at least two weak aspects

• Resource Modeling– Resource description and matching is extremely basic– Users cannot express complex requirements– Focus in OMII-EU/JRA2 GlueSchema

• Security– No autentication/authorization mechanism is mandated– Nobody is willing to implement BES without any security– Focus in OMII-EU/JRA3


Thank you!

• This tutorial is available athttp://grid.pd.infn.it/omii