ATLAS Distributed ATLAS Distributed AnalysisAnalysisDietrich Liko

Thanks to …Thanks to … pathena/PANDA:

T. Maneo, T. Wenaus, K. De

DQ2 end user tools: T. Maneo

GANGA Core: U. Edege, J. Mosckiki, K. Harrison, A. Murao, A. Soroko

GANGA GUIC.L. Tan

Athena AODJ. Elmshäuser

User ProductionF. Brochu

TAG NavigatorC. Nicholson, M. Keynon

• EGEE/gLite: H.C. Lee, D. Liko

NordugridB. Samset, K. Pajchet

OSG/PandaD.Liko + PANDA team

Dashbord MonitoringB. Gaidioz, S. Reddy, J. Wu

AMI IntegrationF. Fassi, C.L. Tan + AMI team

ATLAS Analysis ModelATLAS Analysis Model

Different scenariosDifferent scenarios Inside the Framework

◦ Athena◦ AOD & ESD◦ Grid based (GANGA, PANDA etc)

Outside the framework◦ ROOT◦ DPD (ntuple)◦ ROOT & PROOF

Development aims to minimize the difference◦ Migrate programs between ROOT and Athena◦ AOD access from ROOT

Some recent activitiesSome recent activitiespathena/PANDA

◦See Tadashi’s talk

GANGA◦EGEE◦Nordugrid◦PANDA (based on pathena)

GANGA statistic and use cases

Migration to DQ2 0.3Migration to DQ2 0.3

The new release GANGA 4.3.5 is available as of today. It supports DQ2 0.3

All users have to update their GANGA versions, as the changes of DQ2 are not backward compatible

EGEE – new featuresEGEE – new features

Support for very large sandboxesSupport for group areas (EventViews)Support for LFC bulk operationsAOD to ESD backnavigationSupport for xrootd in Lyon and GridKadCache readahead bufferAutomatic patches for ROOT access

to DPM and dCacheMore backends for AthenaMC

EGEE – gLite WMSEGEE – gLite WMSAt CERN we have started to use

the gLite WMS◦It will be available also at sites in

some time

Bulk submissionBetter sandbox handlingWill be aware of job priority

settings at the site when available

NordugridNordugrid

upgraded to the latest ARC middleware, 0.6.0, distributed with 4.3.3

made sure Athena-related code works with DQ2 0.3

done some work on ensuring proper use of credentials and backend commands, especially for systems without AFS

made several internal changes for better job monitoring etc.

Next Steps ◦ get DQ2 user dataset registration to work, using the

new DQ2 tools ◦ connect NG backend jobs with arda dashboard

PANDAPANDANew interface to PANDA

◦ Newly developed◦ Based on pathena

Now it is possible to send a job to all ATLAS grids◦ Not all options are supported yet …

Can be the basis of further integration between the projects

Will be available in the next release (4.3.6)

GANGA UsersGANGA Users

ATLAS: 435 users, 60 to 90 Users per week

GANGA ATLAS JobsGANGA ATLAS Jobs

Data collected by ARDA dashboard

GANGA Jobs at T1 sitesGANGA Jobs at T1 sites

Data collected by ARDA dashboard

WMS Site efficiency at WMS Site efficiency at IN2P3IN2P3

Job Distribution over the Job Distribution over the TiersTiers

Data collected by ARDA dashboard

Where are the GANGA Where are the GANGA users?users?

Some examples of GANGA Some examples of GANGA UsageUsage

Spain◦ IFIC Valencia

Italy◦ INFN-Naples

UK◦ Lancaster & Manchester

Germany◦ Munich

IFIC ValenciaIFIC ValenciaTop Physics

◦ AANT Ntuples + ROOT

Exotic Physics◦ AOD + Athena◦ HPTV Ntuples +

ROOT Marcos

Use GANGA◦ AOD analysis◦ Ntuple production

IFIC Valencia Analysis Facility

Local SE

STORM &Lustre

GANGA

Athena

AOD Analysis

ROOT/PROOF

DPD or Ntuple Analysis

Investigate StoRM with Lustre as disk based SE

INFN NaplesINFN Naples

LMU MunichLMU Munich Production of Signal MC sample with AthenaMC -

50000 additional events:◦ event generation: 5 jobs or use already validated

evgen samples where only a small fraction has officially been simulated/reconstructed

◦ simulation: 1000 jobs◦ reconstruction: 50 jobs

From this exercise a prototype for automatic job submission has evolved which will eventually will be part of Ganga

Statistics from the dashboard: 11/4-11/6: 79% Grid eff. * 77% Application eff. = 53% overall eff.

Distributed Analysis as part of the CSC homework:◦ process signal and background MC samples at well

known and maintained sites: GridKa, Lyon and LRZ ◦ This involves: SUSY signal, ttbar (5200), Di-boson

backgrounds, etc. - Using SUSYView

UK Use CasesUK Use Cases1. GANGA FOR PHYSICS VALIDATION

Require rapid validation of new releases as soon as they are deployed (usually reconstruction. EVGEN and DIGITS prepared for a variety of bench-mark channels some months back; AthenaMC used to reconstruct the digits using the new release/cache. AODs analyzed via Athena or sometimes copied directly to a local disk with DQ2 if there are a small number of files .Typical statistics: ~10 000 events

2. GANGA FOR SMALL PRIVATE PRODUCTIONS

Require small non-official samples for people developing new reconstruction code. Typical example: Eva Bouhova and Rob Henderson developing the V0 finder - this work has required several samples not included in the official production. Events generated, simulated and digitized using AthenaMC. The files are then copied to the local site with DQ2 so that they can be scanned using the reconstruction code under development Typical statistics: ~5000 events

3. GANGA FOR CSC PHYSICS ANALYSIS

Running physics analysis code over CSC datasets with the aim of producing CSC notes GANGA used to submit analysis jobs on CSC AOD datasets; resulting ntuples merged and copied to local disks for analysis in ROOT. Typical statistics: 50 000 - 250 000 events

4. GANGA FOR ANALYSIS TOOLS DEVELOPMENT

4. Using GANGA to test new tools for vertexing, tagging etc. GANGA used to submit analysis jobs on CSC AOD datasets, running new analysis tools; resulting ntuples merged and copied to local disks for analysis in ROOT. Typical statistics: 50 000 - 250 000 events

1.

SummarySummaryDistributed Analysis Use Cases in ATLAS are

growing◦ I was showing GANGA, you will hear soon about

PANDA

While we have two projects, we try to integrate them close and increase the collaboration

Datamanagment and data distribution are the crucial aspects

2nd week of July: GANGA Developer Days at CERN◦ All developers will be locked in a room …