web and grid services

USC Viterbi School of Engineering

Web and Grid Services

Slides taken from a variety of sources:GT4 tutorial, by Borja Sotomayorhttp://gdp.globus.org/gt4-tutorial/

International Summer school on Grid Computing 2007m by Malcolm Atkinson

http://www.iceage-eu.org/issgc07/index.cfm HP introduction to WSRF by Sanjay Dahiya

http://foss.in/slides/lb2004/wsrf-intro.ppt


• Service Oriented Architecture• Web Services• WS-RF• Globus implementation of WS-RF• OGSA-DAI


Service Oriented Architectures:Three Components

Registries

Service Consumers

Services

Register an available serviceSend name & description


Three Components

Registries

Service Consumers

Services

Request a serviceSend a description


Three Components

Registries

Service Consumers

Services

Set (possibly empty)of matching

services


Three Components

Registries

Service Consumers

ServicesRequest service operation


Three Components

Registries

Service Consumers

ServicesReturn result or Error


Composed behaviour

• Services are themselves consumers– They may compose and wrap other services

• The registry is itself a consumer• A federation of registries may deal with registry

services reliability & performance• Observer services may report on quality of services

and help with diagnostics• Agreements between services may be set up

– Service-Level Agreements

– Permitting sustained interaction


Web Services


Web Services

• Enable communication between machines• Provide any number of functionality• Can be found and invoked• Self-describing—tell you how they can be invoked


• Web Services are platform-independent and language-independent, since they use standard XML languages

• Most Web Services use HTTP for transmitting messages (such as the service request and response) – Good for going through firewalls

• Issues:– Overhead: XML is not as efficient for data transmission as using a

proprietary binary code. What you win in portability, you lose in efficiency.

– Limited in functionality, no explicit state management or lifecycle management

Web Services


Typical Service Invocation


The Web Services architecture


A typical web service invocation

• Whenever a client needs to communicate with a service, it calls a client stub

• The client stub will turn this 'local invocation' into a proper SOAP request

• The SOAP request is sent over a network using the HTTP protocol. The server receives the SOAP requests and hands it to the server stub which invokes the service implementation


Web Server• Web service: software that exposes

a set of operations

• SOAP engine: knows how to handle SOAP requests and responses--Apache Axis

• Application server: provides a space for applications that multiple clients can access—a container--Jakarta Tomcat server

• Http server: knows how to handle HTTP messages

Sometimes a container is described as: SOAP engine + application server + HTTP server


Limitations of Web Services

• No explicit state management or lifecycle management

• Web services are usually stateless• Many services do

not need state


Adding state to services

• Need to allow clients to access appropriate state


“Stateless” vs. “Stateful” Services

• Without state, how does client:– Determine what happened (success/failure)?– Find out how many files completed?– Receive updates when interesting events arise?– Terminate a request?

• Few useful services are truly “stateless”, but WS interfaces alone do not provide built-in support for state

Client

FileTransferService

move (A to B)move


Open Grid Services Architecture


Before OGSA

• Grid services before OGSA– Resource management (Globus GRAM)

– Resource discovery (Globus MDS)

– Data Management (Globus GridFTP, RLS)

– Security

• All had – Different mechanism for access

– Different mechanism for discovery

– Different mechanisms for management


OGSA

• Brings “order” to distributed services• Promotes “open” standards: defined in GGF (now OGF),

OASIS• Enables Virtualization

– Encapsulation behind a common interface of diverse implementations

• Allows the composition of lower-level services to form more sophisticated services

• Defines common behaviors that all services must have:– Naming– Lifetime management– State management– Notification


WSRF


FileTransferService (without WSRF)

• Developer reinvents wheel for each new service– Custom management and identification of state: transferID

– Custom operations to inspect state synchronously (whatHappen) and asynchronously (tellMeWhen)

– Custom lifetime operation (cancel)

Client

FileTransferService

move (A to B) : transferIDmove

statewhatHappen

tellMeWhen

cancel


WSRF in a Nutshell• Service• State representation

– Resource– Resource Property

• State identification– Endpoint Reference

• State Interfaces– GetRP, QueryRPs,

GetMultipleRPs, SetRP

• Lifetime Interfaces– SetTerminationTime– ImmediateDestruction

• Notification Interfaces– Subscribe– Notify

• ServiceGroups

RPs

Resource

ServiceGetRP

GetMultRPs

SetRP

QueryRPs

Subscribe

SetTermTime

Destroy

EPREPR

EPR


FileTransferService (w/ WSRF)

• Developer specifies custom method to createResource and leaves the rest to WSRF standards:– State exposed as Resource + Resource Properties and identified by

Endpoint Reference (EPR)

– State inspected by standard interfaces (GetRP, QueryRPs)

– Lifetime management by standard interfaces (Destroy)

ClientFileTransferService

createResource (A to B) : EPRcreateResource

RPs

Transfer getRP

queryRPs

destroy


Grid Infrastructure: Open Standards

Web services(WSDL, SOAP, WS-Security, WS-ReliableMessaging, …)

WS-Resource Framework & WS-Notification*(Resource identity, lifetime, inspection, subscription, …)

WS-Agreement(Agreement negotiation)

WS Distributed Management(Lifecycle, monitoring, …)

Applications of the framework(Compute, network, storage provisioning,

job reservation & submission, data management,application service QoS, …)


WS-Resource Properties

• Each resource has a Resource Properties document.• Resource Properties document is referred in

service portType in WSDL.• Defined in XML schema.• Each element in the Resource Properties document

is a Resource Property (RP).• Resource properties can be queried using multiple

query dialects.• Independent of back-end implementation.


Accessing Resource Properties

• Pull– Client can query the RP document, using query engines.

• GetResourceProperty• GetMultipleResourceProperties• QueryResourceProperties

• Push– Allows services to send changes in their resources’ RPs

to interested parties.• WS-Notification


WS-Notification

• Subscriber indicates interest in a particular “Topic” by issuing a “subscribe” request

• Broker (intermediary) permits decoupling Publisher and Subscriber

• “Subscriptions” are WS-Resources• Publisher need NOT be a Web Service• Notification may be “triggered” by:

– WS Resource Property value changes– Other “situations”

• Broker examines current subscriptions • Brokers may

– “Transform” or “interpret” topics– Federate to provide scalability


WS-Notification

• Characteristics of WS-Notification:– Web services integration of traditional enterprise

publish/subscribe messaging patterns• Composes with other Web services technologies• Facilitates integration between different messaging middleware

environments– Standardizes the role of Brokers, Publishers,

Subscribers and Consumers– Provides two forms of publish/subscribe:

direct publishing and brokered publishing– Standardizes Web service message exchanges for

publishing, subscribing and notification delivery– Defines XML model of Topics and TopicSpaces to

categorize and organize notification messsages


WS-Resource Lifetime• Creating new resources.• Destroying old resources.

– Immediate destruction.– Scheduled destruction using termination time.

• Soft-state lifetime management– Lifetime extension.

• Example :– jobs in a batch submission system could be represented as

resources

– submitting a new job causes a new resource to be created

– when the job is completed, the resource is destroyed.


WS-ServiceGroups

• Service groups maintain information about collection of services or resources.

• ServiceGroupRegistration– Add new members to group using WS invocation.

• Represent service groups as resources.• MembershipContentRules

– Imposes restrictions on services that can become part of service group like implementing an interface.

• WS-Notifications for service group changes.• For example – Resource registries etc

WS-BaseFaults• XML based fault transmission.• Associated with an operation in WSDL.• Includes standard datatypes for transmitting webservice faults

– Originator, Timestamp etc..

Example :

<wsdl:portType name="pt">

<wsdl:operation name="op">



<wsdl:input … />

<wsdl:output … />

<wsdl:fault name=“aFault" message="tns:aFaultMessage"/>

<wsdl:fault name="BaseFault" message="wsbf:BaseFaultMessage"/>

</wsdl:operation>

</wsdl:portType>


Globus Toolkit implementation of WSRF


Data Mgmt

SecurityCommonRuntime

Execution Mgmt

Info Services

GridFTPAuthenticationAuthorization

ReliableFile

Transfer

Data Access& Integration

Grid ResourceAllocation &

ManagementIndex

CommunityAuthorization

DataReplication

CommunitySchedulingFramework

Delegation

ReplicaLocation

Trigger

Java Runtime

C Runtime

Python Runtime

WebMDS

WorkspaceManagement

Grid Telecontrol

Protocol

Globus Toolkit v4www.globus.org

CredentialMgmt

Globus Toolkit: Open Source Grid Infrastructure

Tools for building WSRF

services


GT4 WS Core in a Nutshell

RPs

Resource

ServiceGetRP

GetMultRPs

SetRP

QueryRPs

Subscribe

SetTermTime

Destroy

EPREPR

EPR

Implementation of WSRF: Resources,

EndpointReferences, ResourceProperties

Operation Providers: pre-build implementations of

WSRF operations

Notification implementation: Topics, TopicSet, Embedded

Notification Consumer service


Service Container


RPs

Resource

ServiceGetRP

GetMultRPs

SetRP

QueryRPs

Subscribe

SetTermTime

Destroy

EPREPR

EPR

ResourceHome

RPs

Resource

ServiceGetRP

GetMultRPs

SetRP

QueryRPs

Subscribe

SetTermTime

Destroy

EPREPR

EPR

ResourceHome

RPs

Resource

ServiceGetRP

GetMultRPs

SetRP

QueryRPs

Subscribe

SetTermTime

Destroy

EPREPR

EPR

ResourceHome

Service Container: host multiple services in container; one JVM

process

…more details: based on AXIS service

container, processes SOAP messages


Service Container


RPs

Resource

ServiceGetRP

GetMultRPs

SetRP

QueryRPs

Subscribe

SetTermTime

Destroy

EPREPR

EPR

ResourceHome

RPs

Resource

ServiceGetRP

GetMultRPs

SetRP

QueryRPs

Subscribe

SetTermTime

Destroy

EPREPR

EPR

ResourceHome

RPs

Resource

ServiceGetRP

GetMultRPs

SetRP

QueryRPs

Subscribe

SetTermTime

Destroy

EPREPR

EPR

ResourceHome

Secure Communication: Transport, Message,

Conversation (Transport demonstrates best

performance)

PIP

PDP

Configurable Security Policies: Policy Information

Points (PIPs), Policy Decision Points (PDP) -- chained

Example authorization PDPs: GridMap, SAML

implementations


Service Container


RPs

Resource

ServiceGetRP

GetMultRPs

SetRP

QueryRPs

Subscribe

SetTermTime

Destroy

EPREPR

EPR

ResourceHome

RPs

Resource

ServiceGetRP

GetMultRPs

SetRP

QueryRPs

Subscribe

SetTermTime

Destroy

EPREPR

EPR

ResourceHome

RPs

Resource

ServiceGetRP

GetMultRPs

SetRP

QueryRPs

Subscribe

SetTermTime

Destroy

EPREPR

EPR

ResourceHome

PIP

PDP

WorkManager DB Conn Pool JNDI Directory

WorkManager: “thread pool”, site independent

“work” manager

Apache Database Connection Pool library

(JDBC “DataSource” implementation)

JNDI Directory: manages internal, shared objects

(ResourceHomes, WorkManager,

Configuration objects,…)


Apache Tomcat

Service Container


RPs

Resource

ServiceGetRP

GetMultRPs

SetRP

QueryRPs

Subscribe

SetTermTime

Destroy

EPREPR

EPR

ResourceHome

RPs

Resource

ServiceGetRP

GetMultRPs

SetRP

QueryRPs

Subscribe

SetTermTime

Destroy

EPREPR

EPR

ResourceHome

RPs

Resource

ServiceGetRP

GetMultRPs

SetRP

QueryRPs

Subscribe

SetTermTime

Destroy

EPREPR

EPR

ResourceHome

PIP

PDP

WorkManager DB Conn Pool JNDI Directory

Deploy Service Container “standalone”

or within Apache Tomcat


Relationship Between OGSA, GT4, WSRF & Web Services


Dealing with Data OGSA-DAI


OGSA-DAI

• An extensible framework

• accessed via web services

• that executes data-centric workflows

• involving heterogeneous data resources

• for the purposes of data access, integration, transformation and delivery within a Grid

• and is intended as a toolkit for building higher-level application-specific data services


Motivation• Grid is about sharing resources

• OGSA-DAI is about sharing structured data resources

RelationalDatabase

XMLDatabase

IndexedFile


Sharing data via website download

• ZIP up data and put it on a website• Pros

– Easy distribution for providers

– Easy access for consumers

• Cons– Consumers have to download all the data

– Consumers have to load data into local databases to use it

– Static snapshot

– Security


Sharing data via direct access

• Providers tell consumers– Database URL – mycomputer.epcc.ed.ac.uk:3306– Username – userID – Password – password

• Pros– Consumers have direct access, so it should be faster

• Cons– Firewall issues– User and password management is hard– No consistent security model– Hard to use in grid/web service workflows– Continued on next slide…


Sharing data via direct access

• Cons (continued)– No server-side layer in which to standardize database

heterogeneities

– Client needs to know, and have installed, correct driver for the database.

– Different drivers for Java, C#, C++, Fortran etc.

– Totally different API for different database types, e.g. JDBC for Relational, XMLDB for XML, Lucene for indexed files.


• Manipulate data using domain-specific operations, e.g.– Book findByISBN(ISBN)

– List<Book> findByAuthor(Author)

– List<Book> findByKeyword(Word)

• Pros– Fits with grid/web service approach

– Abstraction hides back-end database details

– Web services are programming language neutral

– Operations likely to map well to authorization policies

Domain-specific web services


• Cons– Slower than direct access

• Web service layer

• SOAP transport overhead – especially for large result sets

– Domain-specific API prevents use of generic data exploration, mining and manipulation tools

Books Cancer

UniversityEmployees

Generic Data LinkingApplication

Books writtenby Universityemployees

Universityemployees in

1932 who havesince died of

cancer

Domain-specific web services


OGSA-DAI generic web services• Manipulate data using OGSA-DAI’s generic web services

• Clients sees the data in its ‘raw’ format, e.g.– Tables, columns, rows for relational data

– Collections, elements etc. for XML data

• Clients can obtain the schema of the data

• Clients send queries in appropriate query language, e.g. SQL, XPath

RelationalDatabase

XMLDatabase

IndexedFile

OGSA-DAI

request

data


OGSA-DAI

• Pros:– Fits with grid/web service approach.

– Web services are programming language neutral.

– Access to schema and raw data supports generic tools.

• Cons:– Slower than direct connection mainly due to SOAP

overhead.

– One more layer between client and data

– Data not transferred in efficient binary format.


Reducing the SOAP effect – workflows

TransformWeb Service

OGSA-DAI

FTP Server

OGSA-DAI

FTP Server

Query ->Transform ->DeliverToFTP

web and grid services

Documents