dial status and plans

16
David Adams ATLAS DIAL Status and Plans David Adams BNL September 25, 2003 ATLAS Software Workshop Grid session CERN

Upload: kieve

Post on 23-Jan-2016

44 views

Category:

Documents


0 download

DESCRIPTION

DIAL Status and Plans. ATLAS Software Workshop Grid session CERN. David Adams BNL September 25, 2003. Goals of DIAL What is DIAL? Design Status Development plans Integration with GANGA. Contents. Goals of DIAL. 1. Demonstrate the feasibility of interactive analysis of large datasets - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: DIAL Status and Plans

David Adams

ATLAS

DIAL Status and Plans

David AdamsBNL

September 25, 2003

ATLAS Software WorkshopGrid session

CERN

Page 2: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 2

David Adams

ATLAS

ContentsGoals of DIAL

What is DIAL?

Design

Status

Development plans

Integration with GANGA

Page 3: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 3

David Adams

ATLAS

Goals of DIAL1. Demonstrate the feasibility of interactive analysis of large datasets

• How much data can we analyze interactively?

2. Set requirements for GRID services• Datasets, schedulers, jobs, results, resource discovery,

authentication, allocation, ...

3. Provide ATLAS with a useful analysis tool• For current and upcoming data challenges• Real world deliverable• Like to add another experiment would show generality

Page 4: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 4

David Adams

ATLAS

What is DIAL?DIAL provides a connection between

• Interactive analysis framework– Fitting, presentation graphics, …

– E.g. ROOT, JAS, …

• and Data processing application– Natural to the data of interest

– E.g. athena for ATLAS

DIAL distributes processing• Among sites, farms, nodes

• To provide user with desired response time

Look to other projects to provide most infrastructure

Page 5: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 5

David Adams

ATLAS

In tera ctive a na lysise .g . R O O T , JA S , ...

D IA L

D istribu ted p rocessing ru nn ing da ta -specific a pp lica tion

D a ta s e t S c he d u le r A A AJ o b

Page 6: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 6

David Adams

ATLAS

DesignDIAL has the following major components

• Dataset describing the data of interest

• Application defined by experiment/site

• Task is user extension to the application

• Job uses application and task to process a dataset

• Result is the output of a job

• Scheduler creates and manages jobs

Together these define a high-level JDL • (job definition language)

Figure shows how these components interact →

Page 7: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 7

David Adams

ATLAS

UserAnalysis

Job 1

Job 2

Application Task

Dataset 1

Scheduler

1. Create or locate

2. select 3. Create or select

4. select

8. run(app,tsk,ds1)

5. submit(app,tsk,ds)8. run(app,tsk,ds2)

6. splitDataset

Dataset 2

7. create

e.g. ROOT

e.g. athena

Result9. fill

10. gather

Result 9. fill

Result Code

Page 8: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 8

David Adams

ATLAS

StatusDIAL release 0.5 last week

User interfaces• ROOT

– Uses ACLiC to make all dataset/DIAL classes accessible

• Command line– Command dial_submit processes one job

– Application, task and dataset taken from XML files

Schedulers• LocalScheduler: single local job

• MasterScheduler: distributed processing at a site– Fork, LSF, lsrun or Condor

– Run at both BNL and CERN

Page 9: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 9

David Adams

ATLAS

Status (cont)Datasets

• CbntDataset describes a single CBNT hbook file– Inherits from EventDataset

• EventMergeDataset holds multiple EventDataset

• Together these can be used build dataset which corresponds to a collection of CBNT hbook files

• A few files with XML descriptions of such datasets are available for demonstration

Results• HbookResult holds an hbook file assumed to hold a

collection of histograms

• Merge method uses PAW to add histograms

Page 10: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 10

David Adams

ATLAS

Status (cont)Application

• dial_cbnt uses PAW to fill histograms using a user-supplied fortran function

Task• For use with the above, task holds hbook file with

empty histograms and fortran code to fill them

Possible to do distributed processing of a collection of CBNT hbook files

See dial root demo 2

Page 11: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 11

David Adams

ATLAS

Development plansAdd dataset catalog

• Enable users to select dataset

• Fill with DC1 datasets

• See talk at DB meeting

Grid enable• Condor-G

– Scaling, reliability, response time issues

• Define DIAL web service– Client scheduler to connect from user interface

– User could submit jobs to BNL or CERN from anywhere

• DIAL grid scheduler– Use above DIAL service and/or “conventional” grid WMS

Page 12: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 12

David Adams

ATLAS

Development plansPython user interface

• Python busses can be used as user interface– GANGA, ATLAS interactive, PI, …

ROOT CBNT application• Similar to HBOOK version

Athena application• Enable DIAL users run distributed athena• Initial application to fill histograms• Later add support to create new dataset (production)• Need athena-compatible dataset

– ATLAS POOL event collection– Zebra, Athena-ROOT obsolete?

Page 13: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 13

David Adams

ATLAS

Integration with GANGAWe identified the following goals and tasks for DIAL/GANGA integration:

• Provide dataset access from GANGA– Add DS catalog to DIAL

– Import dataset classes to Python

• Enable DIAL job submission (CBNT datasets) from GANGA

– Modify GANGA job interface to use datasets

– Import DIAL classes into Python

– Provide GANGA GUI interface to DIAL scheduler

Page 14: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 14

David Adams

ATLAS

Integration with GANGA (cont)• Enable GANGA job monitor to track DIAL jobs

– Define common XML job description

– Incorporate in GANGA and DIAL

• Enable GANGA to used DIAL for distributed production with athena

– Add athena application support to DIAL

– Add GANGA GUI for this job submission

Page 15: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 15

David Adams

ATLAS

What about ARDA?

Information Service

Authentication

Authorisation

Audi ting

Grid Monitoring

Workload Management

Metadata Catalogue

File Catalogue

Data Management

Computing Element

Storage Element

Job Monitor

Job Provenance

Package Manager

DB Proxy

User Interface

API

7:

12:

5: 13:

8:

15: 11:

9: 10:

1:

4:

2:

3:

6:

14:

DIALdataset

GSI

Chimera

Pacman

MonaLisaAMI

Magda

RLS

GANGA

AMI?MDS

EDG RB

Condor

ROOT

Page 16: DIAL Status and Plans

September 25, 2003 DIAL ATLAS SW - Grid 16

David Adams

ATLAS

What about ARDA? (cont)Architectural Roadmap towards Distributed Analysis

• LCG RTAG

DIAL dataset• More general than HEPCAL/ARDA dataset

• Datasets and their catalogs are part of the metadata and data management services identified by ARDA

DIAL components• Define language by which user interacts with workload

management service

• May also intrude into this service– At what point in the grid do we turn DIAL datasets into files?

– How do we get interactive response?