Conference xxx - August 2003

Anders YnnermanDirector, Swedish National Infrastructure for Computing

Linköping University

Sweden

The Nordic Grid Infrastructure – A Grid within the GridEarly Experiences of Bridging National eBorders

NOTUR 2004, Tromsö, June 11 - 2

Outline of presentation

• TYPES of GRIDs

• Some GRID efforts in the Nordic Region• Nordic participation in EGEE

• SweGrid testbed for production

• NorduGrid - ARC

• North European Grid

• Nordic DataGrid Facility

• Identifying potential problems for Nordic Grid collaborations

• Proposed solutions

NOTUR 2004, Tromsö, June 11 - 3

GRID-Vision

• Hardware, networks and middleware are used to put together a virtual computer resource

• Users should not have to know where computation is taking place or where data is stored

• Users will work together over disciplinary and geographical borders and form virtual organizations

“The best path to levels of complexity and confusion never previously reached in the history of computing”

NOTUR 2004, Tromsö, June 11 - 4

Flat GRIDSETI@HOME

GRID

Resource

User

Resource

User

Resource

User

Resource

User

Resource

User

NOTUR 2004, Tromsö, June 11 - 5

Collaborative GRIDUK eScience

GRID

Resources

UserUser

Resources

NOTUR 2004, Tromsö, June 11 - 6

Power plant GRIDDEISA, SweGrid, …

GRIDHPC-center

HPC-center

HPC-center HPC-center

HPC-center

UserUser

NOTUR 2004, Tromsö, June 11 - 7

Hierarchical GRIDEDG/EGEE

GRIDRegional

center

Management

Local resource

Regional center

UserUserLocal

resourceLocal

resourceLocal

resource

The EGEE project Enabling Grids for E-science in Europe

NOTUR 2004, Tromsö, June 11 - 9

EGEE: Strategy

• Leverage current and planned national and regional Grid programmes, building on the results of existing projects such as DataGrid and others

• Build on the EU Research Network Geant and work closely with relevant industrial Grid developers and NRENs

• Support Grid computing needs common to the different communities, integrate the computing infrastructures and agree on common access policies

• Exploit International connections (US and AP)

• Provide interoperability with other major Grid initiatives such as the US NSF Cyberinfrastructure, establishing a worldwide Grid infrastructure

NOTUR 2004, Tromsö, June 11 - 10

EGEE: Partners

• Leverage national resources in a more effective way for broader European benefit

• 70 leading institutions in 27 countries, federated in regional Grids

NOTUR 2004, Tromsö, June 11 - 11

EGEE Activities

JRA1: Middleware Engineering and Integration

JRA2: Quality Assurance

JRA3: Security

JRA4: Network Services Development

SA1: Grid Operations, Support and Management

SA2: Network Resource Provision

NA1: Management

NA2: Dissemination and Outreach

NA3: User Training and Education

NA4: Application Identification and Support

NA5: Policy and International Cooperation

24% Joint Research 28% Networking

48% ServicesEmphasis in EGEE is on operating a productiongrid and supporting the end-users

NOTUR 2004, Tromsö, June 11 - 12

EGEE Service Activity (I)

• 1 Operations Management Centre – OMC• Coordinator for CICs and for ROCs• Team to oversee operations – problems resolved, performance

targets, etc.• Operations Advisory Group to advise on policy issues, etc.

• 5 Core Infrastructure Centres – CIC• Day-to-day operation management– implement operational policies

defined by OMC• Monitor state, initiate corrective actions, eventual 24x7 operation of

grid infrastructure• Provide resource and usage accounting, security incident response

coordination, ensure recovery procedures

• ~11 Regional Operations Centres – ROC• Provide front-line support to users and resource centres• Support new resource centres joining EGEE in the regions

NOTUR 2004, Tromsö, June 11 - 13

The Northern Region ROC

• Joint operation between SARA (Netherlands) and Swedish Infrastructure for Computing (SNIC)

• Collaboration body formed in North European Grid Cluster (NEG)• SNIC-ROC is responsible for: Sweden, Norway, Finland, Denmark and

Estonia

• Operation:• Negotiate service level agreements (SLA) with committed resource centres

(RC) in the Nordic countries and Estonia• Deploy and support EGEE grid middleware - includes documentation and

training• Monitor and support 24/7 operation of Grid resources• Ensure collaboration with other Grid initiatives in the region - Nordic Data

Grid, Nordugrid, Swegrid, NorGrid, CSC,…

NOTUR 2004, Tromsö, June 11 - 15

EGEE Middleware Activity • Hardening and re-engineering of existing

middleware functionality, leveraging the experience of partners

• Activity concentrated in few major centers

• Key services: Resource Access• Data Management (CERN)

• Information Collection and Accounting (UK)

• Resource Brokering (Italy)

• Quality Assurance (France)

• Grid Security (NEG)

• Middleware Integration (CERN)

• Middleware Testing (CERN)

NOTUR 2004, Tromsö, June 11 - 16

EGEE Implementation Plans

• Initial service will be based on the LCG infrastructure (this will be the production service, most resources allocated here)

• CERN experiments and life sciences will provide pilot applications

• Experiments form virtual organizations

• As project evolves more application interfaces will be developed and EGEE will take on its role as a general infrastructure for E-science

NOTUR 2004, Tromsö, June 11 - 17

EGEE Status

• First meeting held in Cork 300 participants

• Project management board appointed• Anders Ynnerman representing Northern Federation

• Dieter Krantzmueller elected first chair

• Routines for time reports are being set up

• ROC recruitment is under way

NOTUR 2004, Tromsö, June 11 - 18

SweGrid production testbed

• Initiative from • All HPC-centers in Sweden• IT-researchers wanting to research Grid technology• Users

Life Science Earth Sciences Space & Astro Physics High energy physics

• PC-clusters with large storage capacity• Build for GRID production• Participation in international collaborations

• LCG• EGEE• NorduGrid• …

NOTUR 2004, Tromsö, June 11 - 19

SweGrid subprojects

0.25 MEuro/year- Portals- Databases- Security

Globus AllianceEGEE - security

0.25 MEuro/year6 Technicians

Forming the core team for the Northern EGEE ROC

2.5 MEuro6 PC-clusters

600 CPUs for throughput computing

SweGridTest-bed

GRID-research Technical deployment and implementation Hardware

NOTUR 2004, Tromsö, June 11 - 20

SweGrid

• Total budget 3.6 MEuro

• 6 GRID nodes

• 600 CPUs • IA-32, 1 processor/server

• 875P with 800 MHz FSB and dual memory busses

• 2.8 GHz Intel P4

• 2 Gbyte

• Gigabit Ethernet

• 12 TByte temporary storage • FibreChannel for bandwidth

• 14 x 146 GByte 10000 rpm

• 120 TByte nearline storage• 60TByte disk

• 60 TByte tape

• 1 Gigabit direct connection to SUNET (10 Gbps)

NOTUR 2004, Tromsö, June 11 - 21

SUNET connectivity

GigaSunet 10 Gbit/s

“The snowman topology”

2.5 Gbit/s

SweGrid 1 GbpsDedicated

Univ. LAN

10 Gbit/s

Typical POP at Univ.

NOTUR 2004, Tromsö, June 11 - 22

Persistent storage on SweGrid?

Size Administration

BandwidthAvailability

1 2 3

NOTUR 2004, Tromsö, June 11 - 23

Observations

• Global user identity• Each SweGrid users must receive a

unique x509-certifikat• All centers must agree on a

common lowest level of security. This will affect general security policy for HPC centers.

• Unified support organization• All helpdesk activities and other

support needs to be coordinated between centers. Users can not decide where their jobs will be run (should not) and expect the same level of service at all sites.

• More bandwidth is needed• To be able to move data between

the nodes in SweGrid before and after execution of jobs continuously increasing bandwidth will be needed

• More storage is needed• Users can despite increasing

bandwidth not fetch all data back home. Storage for both temporary and permanent data will be needed in close proximity to processor capacity

NOTUR 2004, Tromsö, June 11 - 25

SweGrid status

• All nodes installed during January 2004

• Near line storage systems are currently being installed

• Extensive use of the resources already• Local batch queues

• GRID queues through the NorduGrid middlware

• 60 users

• Contributing to Atlas Datachallenge 2 • As a partner in NorduGrid

NOTUR 2004, Tromsö, June 11 - 26

The NorduGrid project

• Started in January 2001 & funded by NorduNet-2• Initial goal: to deploy DataGrid middleware to run “ATLAS Data Challenge”

• NorduGrid essentials• Built on GT-2• Replaces some Globus core services and introduces some new services• Grid-manager, Gridftp, User interface & Broker, information model,

Monitoring

• Track record• Used in the ATLAS DC tests in May 2002• Chosen as the middleware for SweGrid• Is Currently being used in ATLAS DC II

• Continuation • Could be included in the framework of the ”Nordic Data Grid Facility”• Middleware renamed to ARC

NOTUR 2004, Tromsö, June 11 - 29

A NorduGrid snapshot

NOTUR 2004, Tromsö, June 11 - 30

Reflections on NorduGrid

• Bottom up project driven by an application motivated group of talented people

• Middleware adaptation and development has followed a flexible and minimally invasive approach

• HPC centers are currently “connecting” large resources since it is good PR for the centers

• As soon as NorduGrid usage of these resources increases they will be disconnected. There is no such thing as free cycles!

• Motivation of resource allocations is missing – no Authorization• NorduGrid lacks an approved procedure for resource allocation to VOs and

individual user groups based on scientific reviews of proposals

NOTUR 2004, Tromsö, June 11 - 31

HPC-center

Users Country 2

Challenges

HPC-center

Users Country 4

HPC-center

Users Country 1

Funding agencycountry 4

HPC-center

Users Country 3

Funding agencycountry 3

Funding agencycountry 2

Funding agencycountry 1

Current HPC setup

NOTUR 2004, Tromsö, June 11 - 32

GRID management

UsersVO 1

VO 2

VO 3

MoU

SLAs

Proposals

AuthorizationMiddleware

Other GRIDs

Accounting

Funding agency Country 1

HPC-centerFunding agency

Country 1

HPC-centerFunding agency

Country 1

HPC-centerFunding agency

Country 1

HPC-center

NOTUR 2004, Tromsö, June 11 - 33

Nordic Status

• A large number (too large) of GRID initiatives in the Nordic Region

• HPC – centers and GRID initiatives are rapidly merging• Strong need for a mechanism for exchange of resources

over the borders• Very few existing users belong to VOs

• Most cycles at HPC-centers are used for chemistry• How will these users become GRID users?

• There is a need for authorities that grant resources to projects (VOs) • Locally• Regionally• Nationally• Internationally

NOTUR 2004, Tromsö, June 11 - 36

Nordic Possibilities

• Well defined region• Several different Grid initiatives

• SweGrid, NorGrid, DanGrid, FinGrid, …

• Similar projects• Portals• National storage• National Authentication

• Collaboration between funding agencies already exists• Limited number of people involved in all efforts• Cultural similarities

• Nordic way is to shortcut administration and get down to business

• Expansion to the Baltic new member states poses interesting challenges

NOTUR 2004, Tromsö, June 11 - 37

The Nordic DataGrid Facility

• Based on a Working paper by the NOS-N working group • The collaborative board for the Nordic research councils

• Project started 2003-01-01

• Builds on interest from• Biomedical sciences• Earth Sciences• Space and astro sciences• High energy physics

• Gradual build-up and planing

• Originally planned to be one physical facility

• Project is currently undergoing changes …

NOTUR 2004, Tromsö, June 11 - 38

The Nordic Grid

• Most likely:• NDGF will be built as a Grid of Grids

• Common services and projects will be identified

• Will serve as a testbed for Grid Economic models

• NorduGrid software (ARC) development will continue in the Framework of NDGF

• The name will change to NorduGrid?

• In this way the Nordic countries will have one interface to the outside world to the benefit of:• LCG

• EGEE

• …

NOTUR 2004, Tromsö, June 11 - 39

Conclusions

• Nordic countries are early adopters of GRID technology

• There are several national GRIDs

• The NorduGrid middleware (ARC) has successfully been deployed

• A Nordic GRID is currently being planned• Nordic level Authentication, Authorization, Accounting

• Nordic policy framework

• Nordic interface to other projects