10 April 2003 Deploy Grid in Israel Universities 1

Deploy Grid testbed in Israel universities

Lorne Levinson

David Front

Weizmann Institute

10 April 2003 Deploy Grid in Israel Universities 2

The Grid Problem

• Seamless, flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resources

• Enable virtual organizations (VO)s to share geographically distributed resources as they pursue common goals assuming the absence of:

central location, central control, trust relationships.

10 April 2003 Deploy Grid in Israel Universities 3

10 April 2003 Deploy Grid in Israel Universities 4

DataGRID Job submission

UIJDL

ResourceResourceBrokerBroker(Match requests to (Match requests to resources)resources)

Job SubmissionJob SubmissionServiceService (Condor G) (Condor G)

StorageStorageElementElement (SE) (SE)

ComputeComputeElement (CE)Element (CE)

Information Information ServiceService

Job Status

ReplicaReplicaCatalogueCatalogue

DataSets info

Author.&Authen.

Job S

ub

mit

Even

t

Job

Qu

ery

Job

Stat

us

Input “sandbox”

Input “sandbox” + Broker InfoGlobus RSL

Output “sandbox”

Output “sandbox”

Job Status

Pu

blis

h

grid

-pro

xy-in

it

Exp

and

ed J

DL

SE & CE info

Logging &Logging &Book-keepingBook-keeping

10 April 2003 Deploy Grid in Israel Universities 5

File replication

Read

StorageStorageElementElement

Information Information ServiceService

Replica Catalogue (RC)Replica Catalogue (RC)Translates logical FN to physical FNTranslates logical FN to physical FNAsk for a LFN

Ask for LFN

Return nam

e & protocol

Return PFN

Pub

lish

SE & CE info

Write Writes a file

RC only tells where files areIn order to get them, one may use Gridftp

10 April 2003 Deploy Grid in Israel Universities 6

10 April 2003 Deploy Grid in Israel Universities 7

Data Management future : Reptor: The Next Generation Replica Manager

Replica Manager

Replica Metadata

Replica Location

File Transfer

Optimization

Transaction

Consistency

Preprocessing

Postprocessing

Subscription

Client

Reptor

Giggle

RepMeC

Optor

GDMP

10 April 2003 Deploy Grid in Israel Universities 8

The Grid is NOT (yet)

• Not: Super computing– The trend: Many PCs rather than a big computer

• Not: Can not guarantee fast response, mainly because of network limitations

• Not (yet) : A total solution for distributed computing– Batch rather than interactive or stream oriented:

Examples: the grid is not geared to• Support games • Show a lecture on line

• Not (yet) : Ultimate parallel computing solution– Supported class: Embarrassingly Distributed (ED)

10 April 2003 Deploy Grid in Israel Universities 9

Classes of parallel computing

Supported by Grid

10 April 2003 Deploy Grid in Israel Universities 10

Assumptions

• Grid should be functional

• Users are scientists

• Prefer open source

• Run on Linux RedHat 7.3

10 April 2003 Deploy Grid in Israel Universities 11

How?

• Define priorities for required functionality and suppliers of relevant SW packages

• Support wide functionality scope. Each institution will select its subset

• Find best, up-to-date, reasonably tested, set of SW packages

• Honor Glue - USA-Europe collaboration • The actual content and versions will change

because SW is evolving• Stick to one SW provider that does exactly that

10 April 2003 Deploy Grid in Israel Universities 12

LHC Computing Grid - LCG

• LCG is the project that selects at tests computing grid SW for LHC experiments in CERN

• LCG will supply highly integrated Grid SW• Utilizes American and European Grid SW

• Honors GLUE: America-Europe collaboration

10 April 2003 Deploy Grid in Israel Universities 13

LCG main SW suppliers• Globus: A bag of Grid services

– GSI: Grid Security Infrastructure– GRAM: Grid Resource Allocation & Management– MDS: Monitoring and Discovery Service– GridFTP: file transfer over the net

• VDT:– Condor-G: Submit jobs and monitor job consistency

• EDG:– Resource Broker: Select on what Computing Element (CE) to submit job– Data replication– VOMS: Virtual Organization Management System– TBD: LCFG: Installation and configuration

• GLUE:– Make sure that USA and Europe grid SW work together

10 April 2003 Deploy Grid in Israel Universities 14

LHC Computing Grid - LCG

Software Delivery Process

Globus NMI VDT

EDG

LCG

LCGCertificationAnd Testing

LCG IntegrationVDT

+

+

USA Europe

10 April 2003 Deploy Grid in Israel Universities 15

LCG-1 Components 1/2003

VDT 1.1.6

Globus 2.0 + patches

ClassAds: 0.9.4

Condor: 6.4.7

Condor-G: 6.4.7

EDG Certificates 0.12-1

EDG CRL Update: 1.2.5-1

MDS 2.2 + patches + static GLUE schema

EDG mkGridmap

EDG 1.4.3

GDMP 3.2.6 ( to be migrated to 4.0)

Replica Catalogue Server ( to be migrated to RLS) edg-rc-server 3.1-2

Replica Manager edg-replica-manager

Replica Catalogue API /CLI ReplicaCatalogue 3.2.3 [4]

Resource broker: Workload Management (informationindex-1.2.9-1, jobsubmission-[glue-aware], lbserver-1.2.14-1, locallogger-1.2.12-1, proxy-1.2.8-1, userinterface-1.2.15-1)

LCG

lcg-version

Configuration solutions

DataTAG 1.0

GLUE Information Providers

Edt-monitor

10 April 2003 Deploy Grid in Israel Universities 16

LCG-1 Components 3-5/2003

VDT 1.x.x 3/2003

Globus 2.2.x

RLS pre-release

Condor: 6.4.7

Info Providers (partial Glue set)

EDG 2.0 3-5/2003

R-GMA x.x.x (To be evaluated)

Reptor x.x

RepMecC x.x

Resource Broker x.x (Glue)

VOMS x.x.x (To be evaluated)

LCG 3/2003

lcg-version

Test-suite

DataTAG 1.0 3/2003

Edt-monitor

Dilemma: Should the more advanced, but less tested version be used?

10 April 2003 Deploy Grid in Israel Universities 17

Grid functionality: SecurityFunctionality Supplied by Priority

Authentication:

Certificate Authority (CA):

Allocate user and host certificates

Europe, USA: Many certification authorities

Israel: Machba to be a certification authority

Essential

single sign-on Globus: Grid Security Infrastructure (GSI)

Essential

Certificate delegation Globus: GSI Essential

Authorization:Virtual Organization (VO)

management

Globus: Mapfile - Map local users to grid users

EDG mkGridmap

EDG 2.0: VOMS

High

Firewall:

Submit grid jobs securely through firewalls

Not supplied yet High

10 April 2003 Deploy Grid in Israel Universities 18

Grid functionality: Job submission

Functionality Supplied by Priority

Job submission Globus:

Grid Resource Allocation & Management (GRAM)

Essential

Job consistency Condor-G Very high

Resource Broker (RB):

Select a Computing Element (CE) for job submission

EDG (Glue) The bigger the number of CE’s,

The more a RB is needed

Scheduling policy:

Control local resource allocation according to policy

Not supplied yet High

Job dependencies EDG 2.0

based on Condor DAGMan

Support interactive jobs EDG 2.0

10 April 2003 Deploy Grid in Israel Universities 19

Grid functionality: Data management

Functionality Supplied by Priority

Copy files over the net Globus: GridFTP Essential

Data mirroring EDG: GDMP Very high

Replica

Catalog+ management

EDG:

Replica catalog

EDG 2.0: reptor

The bigger the number of (big) files to be read at many sites,

The more replica management is needed

Virtual Data Not available

10 April 2003 Deploy Grid in Israel Universities 20

Grid functionality: Information services

Functionality Supplied by Priority

Information services

LDAP based

Globus:

Monitoring and Discovery Service (MDS).MDS resource information models physical and logical components of a compute resource as a hierarchy of elements, using LDAP

Essential

RDB based EDG 2.0: R-GMA Essential

Job monitoring Not available? High

10 April 2003 Deploy Grid in Israel Universities 21

Grid functionality: Graphical User Interface (GUI)

Functionality Supplied by Priority

Graphical (web based) User Interface (GUI)

• Genius:EDG simple portal for job submission

• EDG 2.0 will have more GUI

High

10 April 2003 Deploy Grid in Israel Universities 22

Links

Globus Project™– www.globus.org

Global Grid Forum– www.gridforum.org

European Data Grid EDG– http://eu-datagrid.web.cern.ch/eu-datagr

id

/