Slide 1

LHC ATLAS users analysing data on the GridM. Lamanna, D. Van Der Ster (CERN IT ES group) and J. Elmsheuser (LMU Munich)for the ATLAS Distr. Computing project

EGEE-III INFSO-RI-222667 Enabling Grids for E-sciencEwww.eu-egee.orgEGEE and gLite are registered trademarks Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 Objective of the talkShare ATLAS experience in dealing with large data samples (data handling and data access) with other grid usersThis is one of the goal of the User ForumDescribe the commissioning procedure to match the requirements for analysis (I/O intensive activities)TrainingSite commissioningUser supportDiscuss the current status of the evolution of the system

2Massimo LamannaEnabling Grids for E-sciencEEGEE-III INFSO-RI-222667 ATLASLHC pp-collisions in the ATLAS detectorsNow on at 3.5 TeV + 3.5 TeVData distribution (DDM)ATLAS distr. Data managementData reconstruction (PanDA)ATLAS production systemAll must run on EGEE, OSG and NDGF infrastructures (WLCG)At this point, user analysis kicks inFor small data sets, download and study them at home (more later in the talk)Make use of the Grid infrastructure (PanDA/WMS+DDM on top of EGEE/OSG/NDGF)Analysis is an individual activityMany usersAnalysis is about understanding complex dataIterativeInteractiveTask-completion latency more important that global throughputI/O performances absolutely criticalWell-established practices (on batch and personal workstations)Grid is essential (not an optional nice-to-have tool) but can be complemented by the use of other resources (for example to development and tune up algorithms, debugging)

Examples of pre-analysis activitiesAnalysis activities3Massimo LamannaEnabling Grids for E-sciencEEGEE-III INFSO-RI-222667 ATLAS users accessing the dataData typeRAWESDAODdESDLHC data 2009102455527725Cosmics 200933873021MC09523590Massimo Lamanna4Number of distinct users that accessed ATLAS data on the Grid (by data type) in the period between 20 November 2009 and 23 February 2010. This includes job access and data download.

Simulated HITS (the equivalent of RAW for real data) are kept on tape and not made generally available for analysis. No dESDs were produced for simulated events.

Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 ATLAS Distributed Data ManagementLHC experiments collects data in the the 100-1000 MB/s rangeIntegrated values (RAW data) are in the PB/year rangeAdditional data have to handledSimulationDerivated types (RAW reconstruction and analysis stages)Data are replicatedAccess efficiency, data safetyThe data management system is a key component of the ATLAS distributed computingExample of a complex high-level serviceIt uses LFC, FTS and SRMComplex software product (development and operations)5Massimo LamannaEnabling Grids for E-sciencEEGEE-III INFSO-RI-222667 DDM software stackOpen science gridEgeeNdgf6Massimo LamannaEnabling Grids for E-sciencEEGEE-III INFSO-RI-222667 ATLAS Computing Model: Hierarchical Grid OrganizationTier-0 facility (CERN): Custodial and distribution of primary RAW detector dataFirst pass reprocessing of the primary event streamDistribute the derived datasets to the Tier-1s10 Tier-1 data centers: Custodial and long-term access to a subset of the RAW dataAccess to derived datasets~100 Tier-2 institutes:Analysis capacity for users & physics-groupsMonte-Carlo simulation

RAWESDAODCASTOREvent filterReconstructionRAWRAWESD, AODTier 0RAWESDAODMCAnalysisRe-ReconstructionESD, AODTier 1RAWESDAODCASTORAnalysisMonte CarloRAWRAWESD, AODTier 2RAW, ESD, AODRAW7Massimo LamannaEnabling Grids for E-sciencEEGEE-III INFSO-RI-222667

Baseline services for ATLAS dataDDM: organization, access, placement and deletion

1000+ users 500+ endpoints3 grids~2,000,000 dataset replicas~80,000,000 file replicas8Massimo LamannaEnabling Grids for E-sciencEEGEE-III INFSO-RI-222667 Performances 19Massimo LamannaEnabling Grids for E-sciencEEGEE-III INFSO-RI-222667 Performances 210Massimo LamannaEnabling Grids for E-sciencEEGEE-III INFSO-RI-222667 UsersWhat ATLAS users should doThey use the ATLAS framework on their desktop (Athena)They use Ganga or pathena to submit their Athena jobGanga has been developed also in EGEE. Pathena is a CLI built on the PanDA API (used also by Ganga)Ganga allows native access to EGEE and NDGF infrastructures (+batch resources)They run their jobs and are happy What does it take it to have users happy?Teaching the toolsCommissioning the sites where jobs runSupport users in case of problems

11

Important:500+ users (steadily growing)100+ sites (rather stable)Massimo LamannaEnabling Grids for E-sciencEEGEE-III INFSO-RI-222667 11Teaching the toolsMaterial generated by the tools developers and power usersDocumentationTraining material (part of the Ganga distribution)A lot of peer-to-peer helpLearning from your colleague in an informal wayRegular programme of tutorialBy now ran by members of the experiments. Regularly ran every 6 weeks for about 2 yearsThis is part of general ATLAS training (including using Athena)Massimo Lamanna12Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 Commissioning the sites (1)In the past major efforts to improve grid reliabilityOut of all jobs, compute ( 1 #failure/#total )Initially essentially reconstruction jobsEvidence that good sites for simulation activities might show poor behaviour for analysisLow reliability (many failures)Data access? Software distribution? User code problems? Low efficiency (wasting resources)Not enough bandwidth to the storageThese jobs are often IO-limitedFor a set of job we measure the mean value of the distribution CPU_time/Elapsed_timeOften long elapsed time indicate possible problems in reading dataOf course they can be fixed!

Massimo Lamanna13Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 Commissioning the sites (2)Reproducible benchmark (on top of Ganga)Define a set of datasets (commonly used ones by users)Define a set of user jobs (frozen applications from representative users). We call this the job cocktail.Define a procedure to load centres (submit jobs)Submit jobs, retrieve results and compute reliability and efficiencyTests can be defined both centrally and by individual sitesYou can test your site during troubleshooting or after an upgradeYou can test software (e.g. ATLAS Munich developed improved dCache libraries and tested with hammerCloud) Publish the resultsProved to be very important! And keep an eye on itSmall scale, regular, centralised-only jobsResults are published in SAM

141,2,3: in use for about one year:HammerCloudMassimo LamannaRunning for a few years now:GangaRobotEnabling Grids for E-sciencEEGEE-III INFSO-RI-222667 HammerCloud15

Massimo LamannaCollaboration with CMS to extend it to CRAB jobsEnabling Grids for E-sciencEEGEE-III INFSO-RI-222667

HammerCloud used also for stress the system real hard16

User Analysis Test (October 2009)HammerClouds measurementsUsers Feedback (formulaire)

Massimo LamannaEnabling Grids for E-sciencEEGEE-III INFSO-RI-222667 GangaRobotBased on GangaInspired from the CMS JobRobotTesting site since early 2008 Small volume of jobsRun under user credentialsCocktail of applicationsIt is also an active user protection:bad sites are automagically blacklistedThis avoid unnecessary suffering on the user sideIncreasing the integration with HammerCloud

Massimo Lamanna17Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 Support users in case of problems

This is not enoughUsers need supportExpect high pressure from users with first LHC dataLess expert usersMore pressure on the system, hence more hickupsWe decided for a shift systemGood examples in ATLAS for example in running production activitiesX. Espinal (PIC) and K. De (UTA)Experts taking shifts from their home instituteBenefit from a worldwide collaboration!What is special here:More diverse community to be supportedProtect the developers mantra

Massimo Lamanna18Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 Distributed Analysis Shift Team (DAST)Providing support to Distributed Analysis users since September 2008https://twiki.cern.ch/twiki/bin/view/Atlas/AtlasDAST

First contact point with DA users via a mail forum:hn-atlas-dist-analysis-help@cern.chLess formal that a ticketing systemEncouraging user2user supportWhich is important and we would like to stimulte it even moreBack-officeSimple (yet sophisticated) procedures: shifters are not sitting in a control room. They need to privately communicate and annotate issuesI am working on this issue, I have already asked the 2nd line expertPrototype using gmail and gcalendar

Massimo Lamanna19Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 More ATLAS users are joining DASTDAST teamTrain the shiftersFormal tutorial (sometimes videotaped)Junior shifter paired with senior onesEmpower the shifters and the usersBugs are then fed in the proper tracking systemTo developersMonitoring!Allow users to create snapshots of their environment for the support group.

Massimo Lamanna20

Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 Experience with DASTCatch all of all user problems:Conditions database access, Athena, physics analysis tools, site problems, dq2-* tools, data access at sites, data replication

Shifters: 8-hour day shifts to cover around 15h per day (5 days per week)8 (+4 junior) in the EU time zones5 (+3 junior) in the NA time zone

Massimo Lamanna21

Stable on ~ 800 messages a month

Time structure (as UTC+1)Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 Whats nextIt is difficult to make predictions, especially about the future http://www.larry.denenberg.com/predictions.htmlTrends from ATLAS analysisWill these trends be confirmed?Are they relevant for ATLAS only?HEP only?Scientific computing only?Where do we stand now?Powerful federation of data-handling infrastructuresLargely based on Grid infrastructures, notably EGEENear futureATLAS (and HEP): complement the infrastructure with Tier3 facilitiesAnalysis facilitiesSmall ones (Tier3 proper)Large ones: National Analysis Facilies (NAF@DESY, LAF@IN2P3CC, )

Massimo Lamanna22Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 ATLAS Tier3 key pointsOperationsMust be simple (for the local team)Must not affect the rest of the system (hence central operations)Data managementAgain simplicityDifferent access pattern (analysis)I/O bound, iterative/interactiveMore ROOT-based analysis (PROOF?)Truly local usagePerformancesReliability (successful jobs / total )Efficiency (CPU/elapsed) events read per secondClient-based grid sitesData are downloaded, not replicatedData are accessed via native protocolsNo catalogues (the storage *is* the catalogue, e.g. file systems)Other data (conditions, in some cases even the software) are cached on demand

Massimo Lamanna23Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 ATLAS Tier3 working groupsDDM-Tier3 linkHow to download data...S. Campana (CERN)Software / Conditions dataData distribution access and caching of auxiliary dataA. de Salvo (Roma) and A. da Silva (Victoria)Data access (Lustre/Xrootd)Main data access via file system or file-system likeS. Gonzalez La Hoz (Valencia) and R. Gardner (Chicago and OSG)Also creating an inventory and a knowledge base!Tier 3 SupportTools/infrastructure: HammerCloud, DAST, docs...D. Van der Ster (CERN)PROOF Working GroupParallel ntuple scanNeng Xu (Wisconsin) and W. Ehrenfeld (DESY)VirtualizationYushu Wu (LBNL)

Massimo Lamanna24Interim reports: due in a week (next S&CWAt CERN)Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 ConclusionsCollisions are the real new thingGet analysis activities done!Go for simple effective solutionsKeep open to new ideas and inevitable evolution on all frontsTrends/desiresSimplify the sitesless persistent services, e.g. cataloguesPhD should do more physics less system administration Simplify the global systemPrivilege client-based sites and on-demand replication (cache)Keep on investing in user supportIt will be never to much

Massimo Lamanna25Credits: material from Dan Van Der Ster (CERN), Johannes Elmsheuser (LMU), Vincent Garonne (CERN) and Nurcan Ozturk (UTA)Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667

DDM ClientsDDM

CommonModular

Framework

Production system

Analysissystem End-user

Physics meta-data

system

WLCGOPEN SCIENCE GRID LHC COMPUTING GRID NORDUGRID

Site Services:

Centrals Catalogs:

OracleDatabase

deletiontransfer consistency

repository

data usage

Data export

content

location

accounting

subscription

monitoring

Simone Campana: ATLAS Computing

ADC Review at CERN 11-12 Mar. 2009

Total ATLAS disk usage

PetaB

ytes

STEP'09: All DDM activities involved

Experience in 2008: CCRC08

5

MB/s

MB/s MB/s Nominal Rate

T0->T1s throughput 12h backlog recovered in 90 min

T1->T2 transfers 4 PB data deleted in 1 day

T0->T1s throughput

All Experiments in the game

Throughput tests: Burst subscriptions injected

every 4 hours and immediately honored

Failover tests: 12h backlog

fully recovered in 30 minutes

MB/s

MB/s MB/s