web: email: [email protected] towards grid interoperability richard boardman, stephen crouch, hugo...

Web: www.omii.ac.uk Email: [email protected]

Towards Grid Interoperability

Richard Boardman, Stephen Crouch, Hugo Mills, Steven Newhouse, Juri Papay

and the OMII-UK Team

All Hands Meeting11/09/07


Contents

• Introduction & background• Interoperability and standards

o Job Submissiono Implementations

• Build and test• Job brokering using job submission

standards


Introduction: What is the Grid?

• The grid – many definitions!

“Grid computing offers a model for solving massive computational problems by making use of the unused CPU cycles of large numbers of disparate, often desktop, computers treated as a virtual cluster embedded in a distributed telecommunications infrastructure” – Wikipedia

An infrastructure for “coordinated resource sharing and problem solving in dynamic, multiinstitutional virtual organizations” – Foster, Kesselman, Tuecke.

“A service for sharing computer power and data storage capacity over the Internet.“ – CERN (European Organisation for Nuclear Research)

• Differing perceptions of the Grid:o Particle physics community: massive, loosely coupled, distributed

computing environment with computing capability, bandwidth and storage.

o Bio-informaticians: global virtual federated database of experimental data, research papers and laboratory records.

• Suggested common view:o The Grid provides secure virtualisation of resources and enables

collaboration and establishment of virtual organisations.


Background

• Many Grid infrastructures have been developed, but, traditionally, with little interoperability:

o Policies governing access/use of distributed resourceso Lack of adherence to emerging common standards

• Interoperability offers huge benefits for categories of users within user community:

o e-Infrastructure providers: easier deployment/management of software distributions

o e-Science users: freedom to choose services deployed in different Grids; based on functionality, not deployed on a particular Grid

o e-Science application developers: portability of apps across multiple Grids to increase uptake


Standards: the key to Interoperability• Adoption of common standards strongly

supported and implemented by OMII-Europe and GIN (Grid Interoperability Now) for:

o Job Submission, Accounting, Virtual Organisation Management…

o Standards from: OGF, OASIS, W3C, DMTF…o Across platforms: EGEE, Globus, UNICORE &

others

• Will focus on Job Submission (OGSA-BES, JSDL) standards


Emerging Job Submission Standards• Two key standards for two key elements:

o The Basic Execution Service interface (OGSA-BES)

• Simplified version of OGSA-EMS (Execution Management Service)

• Handles basic job lifecycle management• Defines simple (but extendable) job state model

– Pending, running, cancelled, failed or finished

o The Job Submission Description Language (JSDL)

• Specify job executable, data staging and resource requirements


Summary of OGSA-BES

BES-Management Port-typeStopAcceptingNewActivities Request that the BES stop accepting new activities

StartAcceptingNewActivities Request that the BES start accepting new activities

BES-Factory Port-typeCreateActivity Request the creation of a new activity

GetActivityStatuses Request the status of a set of activities

TerminateActivities Request that a set of activities be terminated

GetActivityDocuments Request the JSDL documents for a set of activities

GetFactoryAttributesDocument Request XML document containing BES properties

BES-Activity Port-type (optional)GetStatus Request the status of an activity

Terminate Request that an activity be terminated

GetDocument Request the JSDL document for an activity

GetActivityAttributesDocument Request XML document containing activity properties


Interface/Interaction Standards not Enough• BES & JSDL alone not enough for real interoperability

o JSDL is extensibleo Differing security models across Grid infrastructures

• HPC-Profile proposes Grid interoperability through:o Restricted OGF Job Submission Description Language (JSDL)o OGF OGSA Basic Execution Service (BES)o WS-I Basic Profile

• In addition, an agreed security framework between participants

o Via HTTPS transport (server offers certificate) & username/password (client) for user authentication

• OMII-UK has implemented the HPC-Profile within:o GridSAM – funded by OMII-UK, first to adopt BESo CROWN – with OMII-Europe, in collaboration with Beihang

University, China


Build and Test

• To gain confidence of compliance with interoperability standard, test against other standards-compliant infrastructures

o Test multiple BES/JSDL clients against multiple service endpoints

o Mechanistic process; automation advantageous• ETICS (CERN) provides test automation framework

o e-Infrastructure for Testing, Integration and Configuration of S/W

o Leverages Metronome (formerly NMI Build & Test) across Condor cluster

o Controls management of software builds and testingo Create project configurations, maintain historical records

• ETICS deployed to enable automated compliance testing


NMI Build and Test


ETICS


Build and Test

• BES method sequence for testing (core functionality):

o Get attributes documento Create jobo Query job statuso Show job outputo Get job’s JSDL documento Terminate job

• Scenarios implemented in ETICS to test interoperability

o Testing across HPC-Profile endpoints (with simple config)o Test OMII-UK client component against service


Job Brokering using the CROWN Scheduler

• CROWN Grid developed by Beihang University

• With OMII-Europe as part of Component Exchange activity with OMII China:

o Identified CROWN Meta Schedulero Integrated into OMII release (interoperation)

• Usage of OMII-UK Grimoires service registry

o Coordinated implementation of BES interface to the Scheduler


CROWN BES Scheduler

BESClient

Registrysubmit

monitor

subm

itm

onito

r

submitmonitor

CROWN Scheduler

Supported Platform• 2 jobs, different requirements• Delegate BES resource selection by

submitting both jobs to Scheduler• Monitor each until completion

BES

BES

1. requestresources

3. monitor

Grimoires

2. submit

2. submit

3. monitor

• 2 jobs, different requirements• Identify 2 appropriate BES instances• Submit to both


CROWN BES Scheduler Demo


Conclusions

• First phase of projects dealt with infrastructures, test beds and application software

o Led to greater understanding of key issues• Next phase concentrates on interoperability

and providing solutions-based approaches• Interoperability offers key benefits to the

community:o Ease of managemento Choiceo Simplicity of implementation

• Have illustrated a process of standards adoption, build and compliance testing, and usage within a scheduling application