Databases and the Grid

OGSA-DAIArchitecture & Requirements

Malcolm AtkinsonOGSA-DAI Chief Architect

Director of National e-Science Centrewww.nesc.ac.uk

30th May 2002OGSA Early Adopters’ Workshop

Argonne National Laboratories

Overview

UK e-ScienceScale, Coordination, Structure, Projects

Database Task Force & GGF DAI-WGOGSA-DAI Project

Scope, Scale, Participants, PlansArchitectureRelationship with OGSARequirements

£80m Collaborative projects

E-ScienceSteering

Committee

DG Research Councils

Director

Director’s Management Role

Director’sAwareness and Co-ordination Role

Generic Challenges EPSRC (£15m), DTI (£15m)

Industrial Collaboration (£40m)

Academic Application SupportProgramme

Research Councils (£74m), DTI (£5m)

PPARC (£26m) BBSRC (£8m) MRC (£8m) NERC (£7m) ESRC (£3m) EPSRC (£17m) CLRC (£5m)

Grid TAG

UK e-Science Programme

Tony Hey

Cambridge

Newcastle

Edinburgh

Oxford

Glasgow

Manchester

Cardiff

Southampton

London

Belfast

Daresbury Lab

RALHinxton

UK Grid Network

AccessGrid always-on video

walls

AccessGrid always-on video

walls

Nationale-

ScienceCentre

NeSC’s Roles

TAG

NeSC

eSIGSC

Application Pilots IRCs …e-Science Centrese-Scientists, Grid users, Grid services & Grid Developers

UK Core Directorate Global Grid Forum …

CS Research

DBTF ATFGNT

Coordination, Stimulation & Education

ETF

STF

UK Architectural Task Force (ATF)

Malcolm Atkinson (NeSC) Geof. Coulson (Lancaster U.)

Jon Crowcroft (Cambridge U.) David De Roure (Southampton U.)

Vijay Dialani (Southampton U.) Andrew Herbert (Microsoft)

Ian Leslie (Cambridge U.) Andrew Martin (Oxford U.)

Ken Moody (Cambridge U.) Steven Newhouse (ICSTM & LeSC)

Tony Storey (IBM)

…………… Plus consultations

UK Role in Open Grid Services Architecture, Version 0.6 11th March 2002

www.nesc.ac.uk

teams

ATFObtained Agreement: OGSA as Foundation for UK work, 18 April 2002

e-Science Institute

National e-Science Centre

Edinburgh + Glasgow Universities

Physics & Astronomy 2Informatics, Computing ScienceEPCC

£6M EPSRC/DTI + £2M SHEFC over 3 years

www.nesc.ac.uk

e-Science Institutevisitors, workshops, co-ordination, outreach

middleware development50 : 50 industry : academia

‘last-mile’ networking

e-Science Institutevisitors, workshops, co-ordination, outreach

middleware development50 : 50 industry : academia

‘last-mile’ networking

UK Pilot Projects

Research Councils Autonomy> 30 Projects$5 million to $0.3 million

Wide Range of DisciplinesIndustrial InvolvementIntegration and Access to Informatione-Science Centre Projects

> 50% Industrial Involvement

Equator: Technological innovation in physical and digital life

AKT: Advanced Knowledge Technologies

DIRC: Dependability of Computer-Based Systems

MIAS: From Medical Images and Signals to Clinical Information

IRC ‘Grand Challenge’ Projects IRC ‘Grand Challenge’ Projects

From presentation by Tony Hey

Particle Physics and Astronomy e-Science Projects

GridPPlinks to EU DataGrid, CERN LHC Computing Project, US GriPhyN and PPDataGrid Projects, and iVDGL Global Grid Project

AstroGridlinks to EU AVO and US NVO projects

From presentation by Tony Hey

OGSA-DAI Early Adopter

Comb-e-Chem:Structure-Property Mapping

Southampton, Bristol, Roche, Pfizer, IBM

DAME: Distributed Aircraft Maintenance Environment

York, Oxford, Sheffield, Leeds, Rolls Royce

Reality Grid: A Tool for Investigating Condensed Matter and Materials

QMW, Manchester, Edinburgh, IC, Loughborough, Oxford, Schlumberger, …

EPSRC e-Science Projects (1)

From presentation by Tony Hey

EPSRC e-Science Projects (2)

MyGrid: Personalised Extensible Environments for Data Intensive in silico Experiments in Biology

Manchester, EBI, Southampton, Nottingham, Newcastle, Sheffield, GSK, Astra-Zeneca, IBM, Sun

GEODISE: Grid Enabled Optimisation and Design Search for Engineering

Southampton, Oxford, Manchester, BAE, Rolls Royce

Discovery Net: High Throughput Sensing Applications

Imperial College, Infosense, …

From presentation by Tony Hey

OGSA-DAI Early Adopter

MyGrid e-Science Workbench

Goal is to develop ‘workbench’ to support:Experimental process of data accumulationUse of community informationScientific collaboration

Provide facilities for resource selection, data management and process enactmentBioinformatics applications

Functional genomics, pattern database annotation

Manchester, EBI, Newcastle,Nottingham, Sheffield, SouthamptonGSK, AstraZeneca, Merck, IBM, Sun, ...

From presentation by Tony Hey

Overview

UK e-ScienceScale, Coordination, Structure, Projects

Database Task Force & GGF DAI-WG

OGSA-DAI Project

Scope, Scale, Participants, PlansArchitectureRelationship with OGSARequirements

DBTF Web Pages

http://www.cs.man.ac.uk/grid-db

DBTF Membership

Malcolm Atkinson (NESC)Vijay Dialani (Southampton University)Norman Paton (Manchester University)Dave Pearson (Oracle UK)Tony Storey (IBM Hursley)Paul Watson (Newcastle University)

DBTF: Aims & Actions

Requirements Capture

Pilot Project MeetingsReportDave Pearson

RoadmapUK CoordinationGGF Articulation

StandardsBoF GGF4Papers GGF5

ImplementationProjectsOGSA-DAI

ArchitectureLiase with ATFLiase with Globus team

Educatione-Science InstitutePilot ProjectsGSC

EvolvingGGF DAIS WGBroader community

Overview

UK e-ScienceScale, Coordination, Structure, Projects

Database Task Force & GGF DAI-WG

OGSA-DAI Project Scope, Scale, Participants, PlansArchitectureRelationship with OGSARequirements

CambridgeOxford

Glasgow

Cardiff

Southampton

London

Belfast

Daresbury Lab

RALHinxton

OGSA-DAI Partners

EPCC & NeSC

Newcastle

IBMUSA

IBM Hurseley

Oracle

Manchester

EPCC & NeSCIBM UKIBM USAManchester e-SCNewcastle e-SCOracle $5 million, 18 months, started 1st February 2002

OGSA-DAI Scope

Definition and development of generic Grid data services which provide access to and integration of data held in databases, and the management of data within a distributed environment. Database

A stored, structured collection of dataAccessed using an API that takes account of the structure of the data stored

IncludesRelational and object databasesXML repositoriesAdequately described collections of files

Databases in the Grid

Computational Complexity

DataComplexity

Scope of Database Services

Discovery of Data by ContentQuery and Update StatementsMetadata Management & EvolutionTransactions (Flavours of)Distributed queries and updatesSpecialised typesEncapsulated (safe) Function applicationNotification (driven by triggers, etc.)

OGSA-DAI ObjectivesProduce specifications for generic data services

based on a common design framework consistent with Open Grid Service Architecture

Design specificationsas basis of standards recommendations via Database Access and Integration Services Working Group to the Global Grid Forum

Deliver Grid data services software in future releases of the Globus Toolkit (GT3 December 2002)

Refine identified requirementsevaluate design options develop demonstratorstransfer skills to the Grid community

Develop reference implementations of generic data servicesEnsure that the Grid model and OGSA standards address fully the needs of data

access and integration Ensure Grid data services meet the levels of service required

performance, scalability, resilience, availability, and manageabilityevolution and distribution

large user populations and large data volumes

OGSA-DAI Plan

Two PhasesPhase 1: Started Feb 02 ends GGF5Detailed Plan –

Requirements, Designs & Prototypes 6 Work Packages Project Management (Oracle, EPCC) Architecture (NeSC, DBTF) XML Data Management (NeSC & EPCC) Distributed Query Systems (Manchester & Newcastle) Metadata & Registries (NeSC & EPCC) Relational Databases (IBM UK)

Phase 2: 12 months Structure and Objectives to be Refined in Major Review GGF5 DAIS WG meeting a major input

OGSA-DAI Time Line

Feb ’02 May ’02 Jul ’02 Sep ’02 Dec ’02 Feb ’03 May ’03 Sep ’03

Ship for GT3 Integration

RDB + GT2 / OGSA Prototypes Available

XML + OGSA Prototype Available

Design Documents & Demos for DAIS WG @ GGF5

RDB + GT2 / OGSA Prototypes for Early Adopters

XML + OGSA Prototypes for Early Adopters

WS + GSI UK support ( > 60 downloads)

Phase 2 StartsPhase 1 Starts

Milestones & Deliverables3rd Jul 2002 GGF 5 Deliverables

1st Draft – OGSA-DAI Design SpecificationWorking Grid data service prototype with workshop materialDraft Phase 2 functional scope for each Work Package

30th Sept 2002

End Phase 1Phase 1 Review Report and recommendations including: revisions to Phase 2 streams of work, Work Package structure, content, and scopeCompleted, Tested, Work Package prototypes with evaluation report detailing functional scope and deficiencies, design options, measures for acceptanceRDBMS/Globus-2 prototype implementation Phase 2 scopeAgreed 2nd Draft – OGSA-DAI design specificationDissemination programme for UK e-Science communityTransition programme for UK Grid Support Team and Globus Development Team

31st Dec 2002

Globus Toolkit Release1st Grid data services reference implementation for Globus Toolkit 31st Grid data services specification for Globus Toolkit 3Scope of functional content for 2nd Globus Toolkit release and specification1st release training and support courses

31st Mar 2003

Interim UK e-Science community releaseInterim Grid data services implementation for UK e-Science communityRelease training and support courses, with documentation

31st Jul 2003

Globus Toolkit Release2nd Grid data services reference implementation for Globus Toolkit 32nd Grid data services specification for Globus Toolkit 32nd release training and support coursesPublications and papers to support reference implementations through WG discussions and GGF standards processesFinal Project Report

OGSA-DAI: Key Components

Grid Database Services (GDS)GXDS, GRDS, GSFDS, …Perform DB actionsExtra Data Service ElementsDB-action-Management FunctionsNotifications from Triggers

Grid Database Service Factories (GDSF)Create the aboveExtra Data Service Elements

Database Service Registries (DSR)Specialised Registries to find DBs, Services & Factories

Grid Data Transfer Services (GDTS)Described at Requirement LevelFlexible & mapped to grid-FTP, MQ Series, …

OGSA-DAI Architecture

1requestforfactory

DSR

GDSF

client

OGSA-DAI Architecture

2responsewithGDSFsGSHs

1requestforfactory

DSR

GDSF

client

OGSA-DAI Architecture

2responsewithGDSFsGSHs

1requestforfactory

3script for 3GDSs

DSR

GDSF

client

4creation of 3GDSs

OGSA-DAI Architecture

2responsewithGDSFsGSHs

1requestforfactory

3script for 3GDSs

DSR

GDSF

GDS1

GDS2

GDS3

client

4creation of 3GDSs

OGSA-DAI Architecture

5responsewith 3GSHs

2responsewithGDSFsGSHs

1requestforfactory

3script for 3GDSs

DSR

GDSF

GDS1

GDS2

GDS3

client

4creation of 3GDSs

OGSA-DAI Architecture

6scriptsrequestingDB actions

5responsewith 3GSHs

2responsewithGDSFsGSHs

1requestforfactory

3script for 3GDSs

DSR

GDSF

GDS1

GDS2

GDS3

client

4creation of 3GDSs

OGSA-DAI Architecture

6scriptsrequestingDB actions

5responsewith 3GSHs

2responsewithGDSFsGSHs

1requestforfactory

3script for 3GDSs

DSR

GDSF

GDS1

GDS2

GDS3

client

7transfer databatch to GDS2

stream to GDS3

4creation of 3GDSs

OGSA-DAI Architecture

6scriptsrequestingDB actions

5responsewith 3GSHs

2responsewithGDSFsGSHs

1requestforfactory

3script for 3GDSs

DSR

GDSF

GDS1

GDS2

GDS3

client

7transfer databatch to GDS2

stream to GDS3

8stream datato GDS2

4creation of 3GDSs

OGSA-DAI Architecture

6scriptsrequestingDB actions

5responsewith 3GSHs

2responsewithGDSFsGSHs

1requestforfactory

3script for 3GDSs

DSR

GDSF

GDS1

GDS2

GDS3

client

9transfer databatchto client

7transfer databatch to GDS2

stream to GDS3

8stream datato GDS2

4creation of 3GDSs

OGSA-DAI Architecture

6scriptsrequestingDB actions

5responsewith 3GSHs

2responsewithGDSFsGSHs

1requestforfactory

3script for 3GDSs

DSR

GDSF

GDS1

GDS2

GDS3

client

9transfer databatchto client

7transfer databatch to GDS2

stream to GDS3

8stream datato GDS2

10stream datato specifieddestination

OGSA-DAI & OGSA <((-:}Description, e.g. portType Works Well

Adding only one portType / GDS(F) | DSR

Expect to make extensive use ofData Service Elements

Special to DBs: Static & Dynamic

Component ManagementNotificationGrid-FTPAccountingSecurity:

Authentication, Authorisation & Privacy

Reliable invocation…

OGSA-DAI & OGSA <))-:}Lifetime Issues

Conditions for terminationControlled clean-up opportunityScope of StateEvolution

Notification IssuesRegistering & using same notification system

For DBs, e.g. triggers do we have to construct a dummy Service Data Element?

Type System IssuesStandards needed for wide range of types

Service Definition IssuesHow to create / obtain standard definitions for common services

OGSA-DAI Summary

On Schedule & Going WellExpect Contributions via DAIS-WG @ GGF5Expect Contributions to GT3 ReleasesEarly Days

Testing Architectural DesignUsing OGSAWorking with Early Adopter Pilot Projects

AstroGrid & MyGrid

Planned release of prototypes

Influence OGSA-DAI directionVia DAIS-WG