CERNThe Status of EU DataGrid projectThe Status of EU DataGrid project

Presented by Bob Jones CERN

Technical Coordinator

bob.jones@cern.ch

EU DataGrid 2

CERN

November 2001

Main project goals and characteristicsMain project goals and characteristics

To build a significant prototype of the LHC computing model To collaborate with and complement other European and US

projects To develop a sustainable computing model applicable to other

sciences and industry: biology, earth observation etc. Specific project objectives

Middleware for fabric & Grid management (mostly funded by the EU) evaluation, test, and integration of existing M/W S/W and research and development of new S/W as appropriate

Large scale testbed (mostly funded by the partners)

Production quality demonstrations (partially funded by the EU)

Open source and technology transfer Global GRID Forum

Industry and Research Forum

EU DataGrid 3

CERN

November 2001

Main PartnersMain Partners

CERN – International (Switzerland/France)

CNRS - France

ESA/ESRIN – International (Italy)

INFN - Italy

NIKHEF – The Netherlands

PPARC - UK

EU DataGrid 4

CERN

November 2001

ParticipantsParticipants

Main partners: CERN, INFN(I), CNRS(F), PPARC(UK), NIKHEF(NL), ESA-Earth Observation

Other sciences: KNMI(NL), Biology, Medicine

Industrial participation: CS SI/F, DataMat/I, IBM/UK

Associated partners: Czech Republic, Finland, Germany, Hungary, Spain, Sweden (mostly computer scientists)

Formal collaboration with USA established

Industry and Research Project Forum with representatives from

Denmark, Greece, Israel, Japan, Norway, Poland, Portugal, Russia, Switzerland

EU DataGrid 5

CERN

November 2001

Project ScopeProject Scope

9.8 M Euros EU funding over 3 years

90% for middleware and applications (HEP, EO and biology)

Three year phased developments & demos (2001-2003)

Possible extensions (time and funds) on the basis of first successful results:

DataTAG (2002-2003)

CrossGrid (2002-2004)

GridStart (2002-2004)

…

EU DataGrid 6

CERN

November 2001

Programme of workProgramme of work

MiddlewareWP1 Grid Workload Management F. Prelz/INFNWP2 Grid Data Management P. Kunszt/CERNWP3 Grid Monitoring services S. Fisher/RALWP4 Fabric Management O. Barring/CERNWP5 Mass Storage Management J. Gordon/RAL

TestbedWP6 Testbed Integration F. Etienne/CNRSWP7 Network Services C. Michau/CNRS

Scientific ApplicationsWP8 HEP Applications F. Carminati/CERNWP9 Earth Observation Applications L. Fusco/ESA-ESRINWP10 Biology Applications C. Michau/CNRS

Dissemination WP11 M. Lancia/CNR

Project Management WP12 F. Gagliardi/CERN

EU DataGrid 15

CERN

November 2001

Status Status

EU contract signed on December 29th, 2000

Project started on 1/1/2001

Work ramping up at CERN and the collaborating institutes (Globus

initial installation, tests and prototype production)

International test bed infrastructure being deployed

Architecture Task Force produced 2nd version of Architecture doc.

First (internal milestone) at PM9 - test-bed 1

EU DataGrid 16

CERN

November 2001

Test Bed ScheduleTest Bed Schedule

TestBed 0 (early 2001)

International test bed 0 infrastructure deployed

Globus 1 only - no EDG middleware

TestBed 1 ( now )First release of EU DataGrid software to defined users within the project:

HEP experiments (WP 9)

Biology applications (WP 10)

Earth Observation (WP 11)

TestBed 2 (Sept. 2002)

Builds on TestBed 1 to extend facilities of DataGrid

TestBed 3 (March 2003) & 4 (Sept 2003)

EU DataGrid 17

CERN

November 2001

DataGrid statusDataGrid status

Preliminary architecture definedEnough to deploy testbed 1

First M/W delivery(GDMP, first workload management system, fabric

management tools, Globus installation, including certification and authorization, Condor tools)

First application test cases ready, long term cases defined

Integration team actively building Testbed 1 WP8

WP8

WP9

WP9

WP10

PierGiorgio Cerello

Eric Van Herwijnen

Julian Lindford

Andrea Parrini

Yannick Legre

WP6

WP6

WP6

WP6

WP6

WP6

WP6

WP6

WP6

WP6

WP6

Brian Coghlan

Flavia Donno

Eric Fede

Fabio Hernandez

Nadia Lajili

Charles Loomis

Pietro Paolo Martucci

Andrew McNab

Sophie Nicoud

Yannik Patois

Anders Waananen

WP1

WP2

WP3

WP4

WP5

WP7

Elisabetta Ronchieri

Shahzad Muzaffar

Alex Martin

Maite Barroso Lopez

Jean Philippe Baud

Frank Bonnassieux

EU DataGrid 18

CERN

November 2001

Collective ServicesCollective Services

Information &

Monitoring

Information &

Monitoring

Replica ManagerReplica

ManagerGrid

SchedulerGrid

Scheduler

Local ApplicationLocal Application Local DatabaseLocal Database

Underlying Grid ServicesUnderlying Grid Services

Computing Element Services

Computing Element Services

Authorization Authentication and Accounting

Authorization Authentication and Accounting

Replica CatalogReplica Catalog

Storage Element Services

Storage Element Services

SQL Database Services

SQL Database Services

Fabric servicesFabric services

ConfigurationManagement

ConfigurationManagement

Node Installation &Management

Node Installation &Management

Monitoringand

Fault Tolerance

Monitoringand

Fault Tolerance

Resource Management

Resource Management

Fabric StorageManagement

Fabric StorageManagement

Grid

Fabric

Local Computing

Grid Grid Application LayerGrid Application Layer

Data Management

Data Management

Job Management

Job Management

Metadata Management

Metadata Management

Object to File Mapping

Object to File Mapping

Service Index

Service Index

EU-DataGrid Architecture

EU DataGrid 19

CERN

November 2001

Test bed 1 ApproachTest bed 1 Approach

Software integration

combines software from each middle-ware work package and underlying external tool kits (e.g. Globus)

performed by integration team at CERN on a cluster of 10 Linux PCs

Basic integration tests

performed by integration team to verify basic functionality

Validation tests

application groups use testbed 1 to exercise their application software

e.g. LHC experiments run jobs using their offline software suites on test-bed 1 sites

EU DataGrid 20

CERN

November 2001

Detailed TestBed 1 ScheduleDetailed TestBed 1 Schedule October 1:

Intensive integration starts

Based on Globus 2 November 1:

First beta release of DataGrid (CERN & Lyon)

(depends on changes needed Globus 1->2) November 15:

Initial limited application testing finished

DataGrid ready for deployment on partner sites (~5 sites) November 30:

Widespread deployment

Code machines split for development

Testbed 1 open to all applications (~40 sites) December 29: WE ARE DONE!

EU DataGrid 21

CERN

November 2001

TestBed 1 SitesTestBed 1 Sites

First round (15 Nov.)

CERN, Lyon, RAL, Bologna

Second Round (30 Nov.)

Netherlands: NIKHEF

UK: See John Gordon’s talk

Italy: 6-7 sites: Catania, Legnaro/Padova, Milan, Pisa, Rome, Turin, Cagliari?

France: Ecole-Polytechnique

Russia: Moscow

Spain: Barcelona?

Scandinavia: Lund?

WP9 (GOME): ESA, KNMI, IPSL, ENEA

EU DataGrid 22

CERN

November 2001

Licenses & CopyrightsLicenses & Copyrights

Package Repository and web siteProvides access to the packaged Globus, DataGrid and required external software

All software is packaged as source and binary RPMs

Copyright StatementCopyright (c) 2001 EU DataGrid – see http://www.edg.org/license.html

LicenseWill be the same (or very similar) to Globus license

A BSD-style license which puts few restrictions on use

Condor-G (used by WP1)Not open source or redistributable

Through special agreement, can redistribute within DataGrid

LCFG (used by WP4)Uses GPL

EU DataGrid 23

CERN

November 2001

SecuritySecurity

The EDG software supports many Certification Authorities from the various partners involved in the project

http://marianne.in2p3.fr/datagrid/ca/ca-table-ca.html

but not Globus CA

For a machine to participate as a Testbed 1 resource all the CAs must be enabled.

all CA certificates can be installed without compromising local site security

Each host running a Grid service needs to be able to authenticate users and other hosts

site manager has full control over security for local nodes

Virtual Organisation represents a community of users6 VOs for testbed 1: 4 HEP (ALICE, ATLAS, CMS, LHCb), 1 EO, 1 Biology

EU DataGrid 24

CERN

November 2001

ldxprofldxprof

GenericComponent

GenericComponent

rdxprofrdxprof

LCFG Components

DBM File

LCFG configuration files

mkxprofWeb ServerXML Profile

(one per client node)

Server node

HTTP

Client nodes

Node configuration and installation toolsNode configuration and installation toolsNode configuration toolsNode configuration tools

For reference platform (Linux RedHat 6.2)

Initial installation tool using system image cloning

LCFG (Edinburgh University) for software updates and maintenance

EU DataGrid 25

CERN

November 2001

Middleware componentsMiddleware components

Job Description Language (JDL)

script to describe the job parameters

User Interface (UI)

sends the job to the RB and receives

the results

Resource Broker (RB)

locates and selects the target

Computing Element (CE)

Job Submission Service (JSS)

submits the job to the target CE

Logging and Book-keeping (L&B )

records job status information

Grid Information Service (GIS)

Information Index about state of

Grid fabric

Replica Catalog

list of data sets and their duplicates

held on Storage Elements (SE)

EU DataGrid 26

CERN

November 2001

A Job Submission ExampleA Job Submission Example

UIJDL

Logging &Logging &Book-keepingBook-keeping

Job SubmitEvent

ResourceResourceBrokerBroker

Output Sandbox

Input Sandbox

Job SubmissionJob SubmissionServiceService

StorageStorageElementElement

ComputeComputeElementElement

Brokerinfo

Output Sandbox

Input Sandbox

Information Information ServiceService

Job Status

ReplicaReplicaCatalogueCatalogue

EU DataGrid 27

CERN

November 2001

Iterative ReleasesIterative Releases

Planned intermediate release schedule

TestBed1: October 2001

Release 1.1: January 2002

Release 1.2: March 2002

Release 1.3: May 2002

Release 1.4: July 2002

TestBed 2: September 2002

Similar schedule will be organised for 2003 Each release includes

feedback from use of previous release by application groups

planned improvements/extension by middle-ware WPs

use of software infrastructure

feeds into architecture group

EU DataGrid 28

CERN

November 2001

Software InfrastructureSoftware Infrastructure

Toolset for aiding the development & integration of middle-warecode repositories (CVS)

browsing tools (CVSweb)

build tools (autoconf, make etc.)

document builders (doxygen)

coding standards and check tools (e.g. CodeChecker)

nightly builds

Guidelines, examples and documentationshow the software developers how to use the toolset

Development facilitytest environment for software (small set of PCs in a few partner sites)

Provided and managed by WP6setting-up toolset and organising development facility

EU DataGrid 29

CERN

November 2001

Future PlansFuture Plans

Tighter connection to applications principal architects

Closer integration of the software components

Improve software infrastructure toolset and test suites

Evolve architecture on the basis of TestBed results

Enhance synergy with US via DataTAG-iVDGL and InterGrid

Promote early standards adoption with participation to GGF WGs

First project EU review end of February 2002

Final software release by end of 2003