K.HarrisonCERN, 21st November 2002

GANGA: GAUDI/ATHENA AND GRID ALLIANCE

- Background and scope- Project organisation- Technology survey- Design- Work towards first release- Conclusions

21st November 2002 2

Background and scope

- Ganga is being developed jointly by ATLAS and LHCb to provide an interface for running Gaudi/Athena applications on the Grid Deal with all phases of a job life cycle: configuration, submission, monitoring, error recovery, output collection, bookkeeping Hide Grid technicalities from user

- First ideas for Ganga put forward in summer 2001 by P.Mato (LHCb) and C.Tull (ATLAS)

- Ganga project supported in UK by GridPP Cambridge and Oxford each have one joint ATLAS/LHCb post funded until December 2004

21st November 2002 3

Ganga: schematic representation

Gaudi/Athenaapplication

GangaG

UI/C

LI

JobOptionsAlgorithms

Collective&

ResourceGrid

Services

HistogramsMonitoringResults

21st November 2002 4

Project organisation

- Current main contributors to Ganga development are: Design team: K.Harrison, W.Lavrijsen, P.Mato, A.Soroko, C.Tull Project coordinators for GridPP: N.Brook, R.W.L.Jones- Project information is available on the web: http://ganga.web.cern.ch/ganga- A mailing list has been set up: project-ganga@cern.ch - Design team nominally has a telephone meeting once every two weeks (next meeting 3rd December)- Presentations of Ganga status and plans given at various other meetings of ATLAS, LHCb and GridPP

21st November 2002 5

Technology survey (1)

- Looked at two systems for distributed production:

AliEn - Developed and used by ALICE - Has its own Grid middleware, but plan is to incorporate EDG tools in the future - Foresee possibility of distributed analysis using PROOF

SLICE - Developed and used by LHCb - Job control currently based on PVSS, but testing EDG middleware as it becomes available

21st November 2002 6

Technology survey (2)

- Looked at two web-based user-Grid interfaces:

GENIUS - Developed by NICE srl (Italy) and INFN - Implements all EDG middleware commands, and provides an easy-to-use interface to the EDG Testbed GRAPPA - Developed by GriPhyN/ATLAS group at Indiana University - Prototype allows submission of Athena Atlfast jobs to US-ATLAS Testbed - Plan evolution to a Virtual-Data Browser: act on Virtual Data, with functionality analogous to a web browser

21st November 2002 7

Design considerations

-Technology survey has given us insight into how Ganga should be implemented- Ganga should not reproduce what already exists, but should make use of, and complement, work from other projects, including AthASK, DIAL and Grappa in ATLAS - The design should be modular, and the different modules should be accessed via a thin interface layer implemented using a scripting language, with Python the current choice- Ganga should provide a set of tools that can be accessed from the command line (may be used in scripts), together with a local GUI and/or a web-based GUI that simplifies the use of these tools- Ganga should allow access to local resources as well as to the Grid

21st November 2002 8

Possible Ganga architecture

Server

EDG UI

PYTHON SW BUS

XML RPC server

XML RPC module

GANGA Core Module

OS Module

Gaudi/Athena

GaudiPython PythonROOT

PYTHON SW BUS

GU

I

DB

Remote user

(client)

LAN/WAN

GRID

LRMS

Local Job DB

ProductionDB

BookkeepingDB

JobConfiguration

DB

21st November 2002 9

Ganga components

- Core Ganga components, each corresponding to a Python base class, will include: Application – set of packages, parameters, inputs, outputs ApplicationHandler – deals with an Application Job – specification of required Application and batch/Grid resource requests JobHandler – deals with a Job Each base class has specialisations for different types of Application/Job- Idea is that Ganga will have functionality analogous to a mail system, with jobs having a role similar to mails Make configuring a Gaudi/Athena job and running it on the Grid as easy as sending a mail

21st November 2002 10

Example specialisations of base classes

ApplicationHandler

AnotherApplicationHandler

AthenaApplicationHandler

AtlfastApplicationHandler

Address different types of applications

JobHandler

AnotherJobHandler

GridJobHandler

PBSJobHandlerLSFJobHandler

LocalJobHandler

Address different types of batch systems

21st November 2002 11

Plans for first release

- Aim to have a first release of Ganga in January 2003- Minimum functionality expected of first release is as follows: It should run on the CERN Lxplus machines Implementation choices known to be incompatible with other platforms will be avoided It should handle a single Gaudi/Athena job, and one job type per experiment Target Atlfast for ATLAS Target DaVinci (Gaudi-based analysis) for LHCb Other types of jobs, and chained jobs, will come later Job submission will be to the CERN Lxbatch system Submission to other batch systems and to Grid will be added with subsequent releases (Grid submission by March 2003?)

21st November 2002 12

Current status

- First versions of Application, ApplicationHandler, Job, JobHandler base classes have been coded Working on specialisations- Have implemented an interface to the LHCb JobOptions database Use xmlrpclib module- Prototype GUI has been created Probably needs some improvement- Working on environment configuration using cmt from PythonTake AthASK as a starting point, but need to separate experiment-independent parts from workarounds specific to ATLAS

21st November 2002 13

Prototype GUI

Embedded Python

interpreter

Tree of user

jobs

Job optionsfor selected

job

21st November 2002 14

Conclusions

- Ganga project has been successfully set up as an ATLAS/LHCb joint venture- A technology survey has been carried out, and has given insight into how Ganga should be implemented- A basic design for Ganga has been formulated Component based, and will evolve with time- Making good progress towards a first release (January 2003), with job submission to a local batch system- Tentatively aim to have a Ganga release with submission to the Grid around March 2003