Oliver Gutschefor the
Data Operations Team
Data Operations
US CMS Collaboration Meeting 201006. May 2010
2US CMS Collaboration Meeting 2010 - Data Operations05/06/10
Outline‣ Introduction to Data Operations
‣T0 processing
‣T1 processing
‣T2 MC production
‣Transfer Operation
‣Release Validation
‣How to stay informed about samples, MC, etc.
‣Service work and other contributions
3US CMS Collaboration Meeting 2010 - Data Operations05/06/10
CMS Computing
‣ Traditional view of the tiered computing infrastructure
‣ Data Operations handles all central tasks on the T0, T1 and T2 level
‣ The project is lead by Markus Klute (MIT) and Oliver Gutsche (FNAL)
‣ More and more T3 sites (especially in the US) have been added or are going to be added
T0T0T0T0
T1: USAT1: USA
T1: ItalyT1: Italy T1: FranceT1: France
T1: SpainT1: SpainT1: T1: GermanyGermany
T1: TaiwanT1: TaiwanT1: UKT1: UK
T2T2T2T2 T2T2T2T2 T2T2T2T2 T2T2T2T2
T2T2T2T2
T2T2T2T2
T2T2T2T2
T2T2T2T2
T2T2T2T2
T2T2T2T2
T2T2T2T2 T2T2T2T2 T2T2T2T2T2T2T2T2 T2T2T2T2 T2T2T2T2
T2T2T2T2
T2T2T2T2
T2T2T2T2
T2T2T2T2
T2T2T2T2
T2T2T2T2
4US CMS Collaboration Meeting 2010 - Data Operations05/06/10
T0 processingTask leaders
Josh Bendavid (MIT)Marco Zanetti (MIT)
IntegrationDave Mason (FNAL)
Stephen Gowdy (CERN)
T0 at CERNT0 at CERN
TapeTape
‣Custodial copy of data coming from Custodial copy of data coming from
the detectorthe detector
‣Inactive “cold” copy of all data not Inactive “cold” copy of all data not
mean to be accessed, only to be mean to be accessed, only to be
archivedarchived
CPUCPU
‣Process all data coming from the Process all data coming from the
detectordetector‣ExpressExpress (latency 1 hour)(latency 1 hour)
‣Bulk: Bulk: Repacking, PromptRecoRepacking, PromptReco (latency 24 hours plus conditions hold)(latency 24 hours plus conditions hold)
‣PromptSkimmingPromptSkimming is dispatched is dispatched by the T0 infrastructure although run by the T0 infrastructure although run at the T1 sitesat the T1 sites
NetworkNetwork
‣Transfer all data from the detector Transfer all data from the detector
to T1 sites for archivalto T1 sites for archival‣Time critical otherwise buffers at T0 Time critical otherwise buffers at T0 overflowoverflow
5US CMS Collaboration Meeting 2010 - Data Operations05/06/10
Data taking 2010‣ Current acquisition era:
Commissioning10
‣ 1 physics Primary Dataset (PD)
‣ Various Secondary Datasets (SD) and Central Skims (SD)
‣ All datasets have a custodial copy at one of the T1 sites
‣ FNAL has a replica of ALL data
‣ Next acquisition era: Run2010A
‣ Will be put in place at Linst > 1E29
‣ 7 physics PDs: JetMETTauMonitor, JetMETTau, EGMonitor, EG, MuMonitor, Mu, MinimumBias
‣ FNAL will have a copy of all data
6US CMS Collaboration Meeting 2010 - Data Operations05/06/10
T1 processing
T1 siteT1 site
TapeTape
‣Custodial copy of data and MCCustodial copy of data and MC
‣Non-custodial replica of data Non-custodial replica of data
and MCand MC CPUCPU
‣PromptSkims (produce SD/CS)PromptSkims (produce SD/CS)
‣Re-reconstruction passes on Re-reconstruction passes on
data and MC including SD/CS data and MC including SD/CS
productionproduction
‣MC production if resources are MC production if resources are
free and T2 level fully utilizedfree and T2 level fully utilizedNetworkNetwork
‣Serve data and MC samples to Serve data and MC samples to
T2 sitesT2 sites
‣Archive data from T0, other T1 Archive data from T0, other T1
sites and the T2 level (MC)sites and the T2 level (MC)
Task leaders
Kristian Hahn (MIT/FNAL)Guillelmo Gomez-Ceballos
(MIT)
IntegrationJose
Hernandez (CIEMAT)Claudio
Grandi (INFN Bologna)
7US CMS Collaboration Meeting 2010 - Data Operations05/06/10
T1 operation in 2010
Jobs per day
‣ Summer09 MC: re-digitization / re-reconstruction
‣ Input:
‣ ~575 Million events, 450 TB
‣ Processing:
‣ ~500 workflows
‣ ~500,000 processing jobs
‣ 90% of the events processed in ∼5 days
‣ Tails finished after 2 weeks
‣ Output:
‣ ~1500 datasets
‣ ~400 TB RAW, ~220 TB RECO, ~65 TB AOD
‣ 2010 data re-reconstruction
‣ Requested when change in release or global tag in T0
‣ 2 passes until now: Apr1ReReco & Apr20ReReco
‣ PD: MinimumBias & ZeroBias plus associated SD/CS
‣ Also re-reconstruction of corresponding MinBias MC samples
‣ Full request currently takes ~2-3 days due to long tails
‣ NEW: train model
‣ Run a new re-reconstruction pass every week
‣ Use stable release and pick up latest conditions and all added statistics
‣ Train leaves the station on Thursday’s, 8 PM CEST
8US CMS Collaboration Meeting 2010 - Data Operations05/06/10
T2 MC production
T2 siteT2 siteCPUCPU
‣50% for analysis50% for analysis
‣50% for MC production50% for MC production‣Standard MC production in Standard MC production in multiple steps (GEN-SIM-RAW, multiple steps (GEN-SIM-RAW, GEN-SIM-RECO, … )GEN-SIM-RECO, … )‣Newer workflows using LHE Newer workflows using LHE datasets as inputdatasets as input‣PileUp workflows using MinBias PileUp workflows using MinBias or data samples for PileUp or data samples for PileUp mixingmixing NetworkNetwork
‣Archive produced MC samples Archive produced MC samples
at T1 sites for custodial storageat T1 sites for custodial storage‣Samples are moved from the Samples are moved from the various T2 sites to one T1 site to various T2 sites to one T1 site to consolidate the distributed consolidate the distributed productionproduction
Task leaders
Ajit Mohapatra (U Wisconsin)
Valentina Dutta (MIT)
9US CMS Collaboration Meeting 2010 - Data Operations05/06/10
T2 MC production in 2009/2010
‣ Record of over 300 Million events in 1 month
‣ Production with lots of variation over the year due to request situation
‣ Currently requests gets produced quickly as there is literally no queue
10
US CMS Collaboration Meeting 2010 - Data Operations05/06/10
Transfer Operation
PhEDExPhEDExCentralCentral
‣Handles dataset transfers Handles dataset transfers
between all CMS sitesbetween all CMS sites‣ Database to keep track of Database to keep track of
files, blocks and their files, blocks and their location location
‣ Links between sites which Links between sites which can be used for transferscan be used for transfers
‣ Central scheduling of Central scheduling of transfers and balancing transfers and balancing between source sitesbetween source sites
‣ Infrastructure to submit Infrastructure to submit transfer or deletion transfer or deletion requests (Webpage)requests (Webpage)
Per sitePer site
‣Agents that handleAgents that handle‣ TransfersTransfers‣ DeletionsDeletions‣ Verification / consistency Verification / consistency
checkschecks
Task leaders
Paul Rossman (FNAL)
Si Xie (MIT)
11
US CMS Collaboration Meeting 2010 - Data Operations05/06/10
Transfer operation in 2009/2010
‣ Totally transferred data volume in last 52 weeks: 17 PB
‣ Site with highest incoming and outgoing volume: FNAL
Transfer volume in last 52 weeksby Destination
Transfer volume in last 52 weeksby Source
Significant effort
has to be spent to debug
transfer problems
12
US CMS Collaboration Meeting 2010 - Data Operations05/06/10
Release Validation
CERN & FNALCERN & FNALRelease validationRelease validation
‣Produce special MC samples Produce special MC samples
and data reconstructions for all and data reconstructions for all
releasesreleases‣Standard set:Standard set:‣Turnaround 24 hours on 500 Turnaround 24 hours on 500 slots at CERNslots at CERN‣Now mostly run at FNAL using Now mostly run at FNAL using opportunistic cycles in parallel to opportunistic cycles in parallel to T1 productionT1 production‣Run for all releases except Run for all releases except patch releasespatch releases‣High statistics set:High statistics set:‣Turnaround 1 week for higher Turnaround 1 week for higher statistics samplesstatistics samples‣PileUp and HeavyIon samplesPileUp and HeavyIon samples‣Produced in parallel to standard Produced in parallel to standard set outside the 24 hour windowset outside the 24 hour window
Task leaders
Oliver Gutsche (FNAL, interim)
Special OperatorDiego Reyes (U Los Andes, Colombia)
13
US CMS Collaboration Meeting 2010 - Data Operations05/06/10
Release Validation in 2009/2010
‣ Significant contribution to software validation and stability of releases for production
‣ High visibility task with reports in all major offline & computing meetings
20092009 20102010
ReleasesReleases 86 28
Events*Events*730,054,6
90377,701,9
88
Size [GB]Size [GB] 108,305 51,441
* double counting RAW, RECO and AOD events
14
US CMS Collaboration Meeting 2010 - Data Operations05/06/10
User advice
‣ To stay informed about current and future samples:
‣ Requests are submitted and acknowledged in
‣ Samples are announced in
‣ Both lists are low traffic lists
‣ We strongly discourage asking questions or replying to threads on these lists
15
US CMS Collaboration Meeting 2010 - Data Operations05/06/10
Your contribution to Data Operations
‣ Data Operations can award service credit for all its tasks
‣ Graduate students and post-docs can spend 25% of their time working for Data Operations as operators
‣ Interest in computing and high scale data processing required
‣ Training would give detailed insight into computing infrastructure and software and prepare ideally for all analysis tasks
‣ Very talented graduate students and post-docs can spend 50% of their time filling one of the 5 task leader positions
‣ High visibility in the collaboration
‣ Significant contribution to the success of the experiment
‣ Closely connected to everything related to data and MC, good for analysis
‣ Data Operations is constantly looking for talents to replenish the current manpower
‣ Urgently we are looking for leaders of the Release Validation task
‣ Please contact Markus and Oliver if you are interested or have questions.
16
US CMS Collaboration Meeting 2010 - Data Operations05/06/10
The Data Operations Team
‣ Project lead:
‣ Markus Klute & Oliver Gutsche
‣ Task leaders:
‣ Josh Bendavid, Marco Zanetti, Kristian Hahn, Guillelmo Gomez Ceballos Retuerto, Ajit Mohapatra, Valentina Dutta, Paul Rossman, Si Xie
‣ Operators
‣ Andrew Lahiff, Andrey Tsyganov, Aresh Vedaee, Ariel Gomez Gomez Diaz, Arnaud Willy J Willy J Pin, Derek Barge, Dorian Kcira, Gilles De De Lentdecker, Jeff Haas, Jen Adelman-McCarthy, Joseph Mccartin, Julien Caudron, Junhui Liao, Lukas Vanelderen, Nicolas Dominique, Dominique Schul, Petri Lehtonen, Subir Sarkar, Vincenzo Spinoso, Xavier Janssen
17
US CMS Collaboration Meeting 2010 - Data Operations05/06/10
Summary & Outlook
‣ Data Operations handles all central processing and production tasks on the T0, T1 and T2 level of the distributed computing infrastructure
‣ Current performance in the areas of data taking,skimming, re-reconstruction, MC production, transfers and release validation is excellent
‣ We are always looking for interested and talented people to help us getting the data as quickly and reliably as possible to all CMS collaborators.
‣ Don’t miss the Computing Shift presentation on Friday
18
US CMS Collaboration Meeting 2010 - Data Operations05/06/10
Glossary‣ Express
‣ Low latency processing of extra express stream(s) from the detector (40 Hz), latency 1 hour
‣ Repacking
‣ Binary streamer files from the detector are translated into the ROOT format
‣ PromptReco
‣ RAW ROOT files are reconstructed promptly at the T0
‣ PromptSkimming
‣ As soon as blocks of data (groups of files either constraint by runs or by number of files (1000)) are completely stored on tape at the custodial T1 site, the PromptSkimming system sends jobs to this site to run the skimming workflows
19
US CMS Collaboration Meeting 2010 - Data Operations05/06/10
Glossary‣ Primary Dataset (PD)
‣ Data stream from P5 is split into PDs according to trigger selections with minimal overlap
‣ Needed for processing in distributed computing infrastructure
‣ Produced at T0, re-reconstructed at T1 sites
‣ Secondary Dataset (SD)
‣ More restrictive trigger selection than PD
‣ Produced at T1 sites with PromptSkimming system, also produced after re-reconstruction passes at T1 sites
‣ Central Skims (CS)
‣ In addition to a more restrictive trigger selection than the parent PD, reconstruction level selections are applied
‣ Same processing as SD