grid architectures steve fisher/ral [email protected] gridpp, edinburgh, november 2001

Grid Architectures

Steve Fisher/RAL

<[email protected]>

GridPP, Edinburgh, November 2001

mailto:[email protected]

mailto:[email protected]

Edinburgh, November 2001Grid architectures - Steve Fisher/RAL 2

GGF - GPA• I will be talking about the GGF Grid Protocol

Architecture Working Group (http://www-itg.lbl.gov/GPA/)– It is THE body to address standardisation of Grid

Architectures– Chaired by Bill Johnston, Ian Foster and Reagan

Moore – Two meetings so far

• Some of my slides derived from:– Bill Johnston’s summary of GGF3 – and talks given there

http://www-itg.lbl.gov/GPA/








Revised Proposed Charter• The role of the Grid Protocol Architecture Working

Group is to provide a conceptual framework for discussing the interrelationships, completeness, and minimality of the protocol approach to Grid services that is coming out of GGF

• The GPA will initially have five work areas:– produce an initial overview architecture to provide a starting

point– explore the issue of different “views” of Grids– establish a methodology for representing the architecture– document the relationship of the the GGF architecture to

other, existing, Grid-like architectures from outside GGF– evolve the initial overview architecture


Issues• How do we answer the question “what is a minimal

set of Grid functions/protocols” and how do we represent their inter-relationships?– characterise existing Gird protocols and interfaces and

where GF WGs are currently working– look at existing architecture representations– Is UML suitable?

• Important to make sure that we are considering a sufficient set of services– examine what is currently bring done, or trying to be done,

with Grids as the first approach

• There are different views of Grids– will these all admit to a common architecture?– If not, how do they differ?


Is UML suitable?• What is architecture?

– divisions into sub-systems– and the control model

• Often represented by simple box and line diagrams– Just show the sub-systems and main communication– Easy to understand

• UML Component and Deployment diagrams– More notation

• More explicit

• Harder for the casual reader


UML Component Diagram

Component

Dependency


UML Deployment Diagram

Shows software deployed in hardware

Node

Connection


More UML• UML also offers class

diagrams to expose an API• And Sequence and

Collaboration diagrams to show possible interactions of objects

• Oxford have EPSRC funding to work with DataGrid – they will be looking at the applicability of UML for architecture

• Will facilitate more UK input to GGF


Are all Grids the same?• Grids originated for “supercomputers”• Knowledge Grids?• Grid and the web• DataGrid(s)


Knowledge Based Data Grids• The knowledge based Data Grid (Moore)

grew out of the need to manage very complex data and inter-relationships– Data objects from many sources

• e.g. different imaging modalities and anatomical data about the brain

– Information about the data objects (attributes)– Knowledge about the data objects (relationships)

• e.g. how do the observations from the different types of imaging - e.g. compositional and metabolic – relate to anatomical structure


Use Cases• NIH Biomedical Informatics Research

Network– Federation of multiple existing digital libraries– Support information discovery, data access, data

movement, and data analysis on distributed resources

• NARA Persistent Archive– Build a data collection that maintains authenticity

of digital data while technology evolves– Support information discovery, data access, and

migration to new data encoding standards

AttributesSemantics

Knowledge

Information

Data

Ingest Services

Management AccessServices

(Model-based Access)

(Data Handling System)

MC

AT

/HD

F

Gri

ds

XM

L D

TD

SD

LIP

XT

M D

TD

Rul

es -

KQ

L

InformationRepository

Attribute- based Query

Feature-basedQuery

Knowledge orTopic-Based Query / Browse

KnowledgeRepository for Rules

RelationshipsBetweenConcepts

FieldsContainersFolders

Storage(Replicas,Persistent IDs)

National Partnership for Advanced Computational Infrastructure

Knowledge-Based Data Grids


Queries across data sources from a common interface

@SENSELAB: X1 := select output from parallel fiber ;@MEDIATOR: X2 := “hang off” X1 from Domain Map;

@MEDIATOR: X3 := subregion-closure(X2);

@NCMIR: X4 := select PROT-data(X3, Ryanodine Receptors);

@MEDIATOR: X5 := compute aggregate(X4);

"How does the parallel fiber output relate to the distribution of Ryanodine Receptors?"

Sources: NCMIR UCSD / Yale Senselab

KIND Mediator


Web Services Grids• Web based Grid services represent the

intersection of the commercial Web services (e.g. IBM’s WebSphere) and scientific and engineering oriented Grid services

• Also driven by the desire to use commercial tools to facilitate the construction of “portals” – application / domain specific frameworks that present “packaged” Grid services toGrid end-users (e.g. scientists and engineers)


So are they different?• I have seen nothing so far to distinguish

these other types of grids– They may have some distinguishing services– But no fundamental differences


Layering• A general layering approach based on

increasing levels of abstraction (abstract machines) is useful for conceptualizing the services

=Globus service

Problem SolvingEnvironment

Applicationsand Supporting

Tools

Tools to implement the human interfaces, and the mechanisms to express, organize, and manage the workflow of solving a problem

ApplicationDevelopment

Support

GridCommonServices

LocalResources

Ap

plic

atio

nC

od

es

Vis

ual

izat

ion

To

olk

its

Co

llab

ora

tio

nT

oo

lkit

s

Inst

rum

ent

Man

agem

ent

To

olk

its

Dat

a P

ub

licat

ion

and

Su

bsc

rip

tio

nT

oo

lkit

s

Grid Enabled Libraries

Glo

bu

s M

PI

CO

RB

A

Co

nd

or

Java

/Jin

i

PR

E/C

OR

BA

OL

E/D

CO

M

Gri

d

Info

rmat

ion

S

ervi

ce

Un

ifo

rmR

eso

urc

eA

cces

s

Bro

keri

ng

Glo

bal

Q

ueu

ing

Glo

bal

Eve

nt

Ser

vice

s

Co

-S

ched

uli

ng

Dat

a C

atal

og

uin

g

Un

ifo

rm D

ata

Acc

ess

Co

llab

ora

tio

n

and

Rem

ote

In

stru

men

t S

ervi

ces

Co

mm

un

icat

ion

S

ervi

ces

Au

then

tic

atio

n

Au

tho

riza

tio

n

Sec

uri

ty

Ser

vice

s

Au

dit

ing

/

Acc

ou

nti

ng

Fau

lt

Man

ag

emen

t

Mo

nit

ori

ng

Communication Services

ResourceManager

CPUs

ResourceManagerTertiary Storage

ResourceManagerOn-Line Storage

ResourceManagerScientific

Instruments

ResourceManagerNetwork Monitors

ResourceManager

Highspeed Data

Transport

ResourceManager

QoS

collective

resource

fabric

layers of increasing abstraction - taxonomy #1 (“Johnston”)


EDG Architecture (as circulated initially to GPA)


Comments on layering

• “things” in each layer use only those in same layer and below– BUT not only immediately below– i.e not an abstract machine model


EDG Architectureanother view

Collective ServicesCollective Services

Information & Monitoring

Information & Monitoring

Replica ManagerReplica Manager Grid SchedulerGrid Scheduler

Grid Application LayerGrid Application Layer

Job ManagementJob Management

Local ApplicationLocal Application Local DatabaseLocal Database

Fabric servicesFabric services

ConfigurationManagement

ConfigurationManagement

Node Installation &Management

Node Installation &Management

Monitoringand

Fault Tolerance

Monitoringand

Fault Tolerance

Resource Management

Resource Management

Fabric StorageManagement

Fabric StorageManagement

Grid

Fabric

Local Computing

Grid

Data ManagementData Management Metadata Management

Metadata Management

Object to File Mapper

Object to File Mapper

Underlying Grid ServicesUnderlying Grid Services

Computing Element Services

Computing Element Services

Authorisation, Authentication

and Accounting

Authorisation, Authentication

and Accounting

Replica CatalogReplica Catalog

Storage Element Services

Storage Element Services

SQL Database Service

SQL Database Service

Service Index

Service Index

Slightly simplified version of a WP2 diagram


Comments on layering• This diagram has layers referencing BOTH

ways• When you look at each service you rarely find

a clean layering – especially as the services get more complex, they are implemented using many other services both “above” and “below”


Layering and client server• It was a principle EDG defined early on that all

services should be available – only restricted by policies.– Good

• anyone can compose new services easily

– Bad • policies are needed to keep things under control – e.g.

modifying replica catalog

• Should we: – just say that these constraints are policy issues and not

represent them in the architecture?– Or try to come up with a simple representation of constraints

for the conceptual representation of the Grid architecture?


Set of Services• EDG Architecture has been criticised as “just

a set of services”• This may be good however

– Provided that we demonstrate that they are sufficient

• e.g. sequence or collaboration diagrams (UML)

• For each EDG service we define upon what services it depends– IS IT ARCHITECTURE OR IMPLEMENTATION?

• Architecture from EDG middleware standpoint• But implementation from a user standpoint


Service and Protocols• To a client the protocol is uninteresting:

– only the API– service could be local

• Need to allow for migration to and/or integration of new services

• A service could respond to >1 protocol• A service could support >1 security

mechanism• Makes it easy to introduce new services


Need for openness• We should have lots of overlapping services and use

the ones we want – typically a VO would choose– Though a lot is dictated by the RB

• Do we want InterGrid or OpenGrid?• We don’t want to tie ourselves down in a fast moving

world• Need to watch industry

– IBM– SUN– Microsoft

SOAP, WSDL

JINI, UDDI

.NET, Passport etc.

grid architectures steve fisher/ral [email protected] gridpp, edinburgh, november 2001

Documents