f cdf experiment fermilab experiment support stephan lammel we, sep26th, 2007

f

CDF ExperimentFermilab Experiment

Support

Stephan LammelWe, Sep26th, 2007

2007-Sep-26 DoE Prog Review 2

f

CDF, Fermilab Experiment Support, St.Lammel

Overview

• Fermilab staff from all around the Laboratory collaborate in the CDF Experiment:– Particle Physics Division, Computing Division,

Accelerator Division and Technical Division

• This brings a lot of expertise to the CDF Experiment

• And gives the experiment access to a large variety of expert resources,– for consulting and in early stages of a project– experts to lead/support a project– testing/manufacturing places– contingency resources


f


Fermilab Experiment Support• Basic support as host laboratory:

– installation, maintenance, technical, engineering, electronics, gas, networking, video-conferencing, safety, etc...

• and beyond in especially three areas:– Upgrades/Replacements– Detector Operation– Offline Computing


f


• Upgrades/Replacements– Data Logger Replacement

• Detector Operation– Fermilab Responsibilities– Three Person Shift Crew– Remote Consumer Operator

• Offline Computing– Fermilab Responsibilities– Expedited N-tuple Production– Glidein Processing Facilities


f


Data Logger Replacement• Original data logger for Run II was

designed and built end of the 90’s for Run IIa luminosity by a team from Rochester and Tsukuba:– SGI/IRIX based, long term maintainability

concern– single points of failure, redundancy not

affordable– custom setup, loss of expertise concern– good but tight match to needs, little flexibility– data center buffer space, expensive to

increase– complex, retro fitted to evolved data handling


f


Data Logger Replacement• Features of new data logger:

– Use commodity hardware/solution where possible and appropriate

– Share archival part of solution with D0– Clean data handling interface, eliminate intermediate

steps, collect and declare metadata directly for production use

• Additional benefits of new data logger:– increased bandwidth to match DAQ/trigger bandwidth

• 100 MB/s (from max 24 MB/s of original data logger)

– Additional flexibility for high luminosity trigger environment

– Increased buffer space and ability to archive data faster than to record them


f


Data Logger Replacement


f


Data Logger Replacement• Data logger replacement was a joint

effort of Tsukuba, Rochester, Fermilab-PPD and Fermilab-CD– Fermilab-PPD brought in DAQ expertise

– Fermilab-CD brought in metadata management and tape archival expertise

– Tsukuba and Rochester brought in the expertise of the old data logger and did a lot of the integration work

• under the leadership of Fermilab-PPD


f






f


Detector Operation

• Fermilab groups have agreed responsibilities for:– solenoid (sole responsibility)– silicon detector– COT (sole responsibility)– hadron calorimeter– muon systems– trigger (level-2 Pulsar, muon)– DAQ (sole responsibility)– consumer/monitoring


f


Detector Operation

• Last year the operations department reviewed data taking, how to streamline operations, gain efficiency, etc.

• The control room shifts were re-organized from four people to three of which one can be remote:– automate more common tasks to make the

job easier and allow the shift to focus on critical decisions

– transfer responsibilities to achieve more natural grouping and to better balance work load


f


Detector Operation

• Three person shift crews started Dec 29th after several months of planning, review and preparation:– training of shift crew– computer console re-arrangement– automation of routine monitoring tasks– review of alarms and make selected ones audible– enhance interface to Tevatron– automate DAQ recovery for more frequent inhibits– two week trial run

• Total savings of 4 FTE (3 shifts, 7 days/week) !

• Shift crew reduction was initiated and organized by Fermilab scientists


f


Detector Operation

• Consumer operator, CO, monitors the data taking and quality.– He interacts with the scientific

coordinator and detector/data taking expert but does not himself control the detector or data taking

• With many collaborators in Europe and Asia participating in the detector operation from off-site is attractive to both the experiment and foreign institutions.


f


Detector Operation

• Steps to realize a remote consumer operator:– The CO setup of the control room was duplicated in

Pisa– All monitoring plots made web based by the

consumer/monitoring team– IP based video link setup– A week long trial run with a remote CO and shadow

CO in the control room in November– Since January one week-long remote owl shift per

month from Pisa– Remote CO shifts from Tsukuba after the shutdown

• Remote CO was initiated by a Fermilab scientists and setup together with a colleague in Pisa.


f


Detector Operation

• Both the current operations heads are from Fermilab.

• Three of the four associate heads are from Fermilab.

• Half of the detector/online groups are lead by Fermilab people.


f






f


Offline Computing

• Fermilab group has agreed responsibility for:– data handling, including metadata

administration, tape archival and disk caches– full event reconstruction– central processing facilities– analysis framework and infrastructure– databases (software, interfaces,

management)– offline operations– software release management/support– various reconstruction/analysis packages


f


Offline Computing

• The CDF analysis strategy is an evolution from earlier hadron collider experiments:– validate/tune reconstruction on small data

subset– central store of reconstructed data– analysis through iterative selections

• Although offline computing is based on commodity hardware/software, resources are nevertheless tight.– avoid duplication as much as possible– shared solutions to share

support/maintenance– short latency/fast turn-around


f


Offline Computing

• CDF Computing Model (data flow view):– Raw data are written to tape (base of all physics)– Detector/data are calibrated (4-6 weeks effort)– Full reconstruction using on-site production

facilities– Make highly-reduced analysis datasets, N-tuples– Monte Carlo generation/simulation/reconstruction

•with run config matching data taking conditions

– N-tuple selection for individual analyses needs– Further analysis on user desktop and at home

institutions


f


Offline Computing

• CDF Computing Model (hardware view)– Enstore system provides IP-access tape storage– On-site processing facilities are used for:

•detector/data calibration• full reconstruction•N-tuple making•analysis data selections (of the N-tuples)

– Off-site processing facilities (CDF specific and Grid)•Monte Carlo

generation/simulation/reconstruction– Disk cache of tape, disk pool, and project space

hold the data analysis accessible


f


Offline Computing

• Data are grouped into few months periods by the offline to handle detector re-calibration needs:

Data Recording Integ Lumi

Total Lumi

Available

Dec 2004 - Mar 2005

130 pb^-1

680 pb^-1

Jul 2005

Mar 2005 - May 2005

130 pb^-1

810 pb^-1

Aug 2005

May 2005 – Jul 2005

100 pb^-1

910 pb^-1

Sep 2005

Jul 2005 - Sep 2005

95 pb^-1

1005 pb^-1

Oct 2005

Sep 2005 - Nov 2005

135 pb^-1

1140 pb^-1

Jan 2006

Nov 2005 - Jan 2006

110 pb^-1

1250 pb^-1

Mar 2006

Jan 2006 – Feb 2006

50 pb^-1

1300 pb^-1

Apr 2006

Jun 2006- Sep 2006

210 pb^-1

1510 pb^-1

Oct 2006

Sep 2006 – Nov 2006

180 pb^-1

1690 pb^-1

Jan 2007

Nov 2006 –Jan 2007

280 pb^-1

1970 pb^-1

Apr 2007

Jan 2007 –Mar 2007

264 pb^-1

2234 pb^-1

Jun 2007

Apr 2007 - May 2007

185 pb^-1

2419 pb^-1

Sep 2007

May 2007 - Aug 2007

317 pb^-1

2736 pb^-1

– datasets for calibration produced within ½ week

– calibration plus validation within 3 weeks

– full reconstruction finishes a month later


f


Offline Computing

• N-tuple (highly reduced analysis dataset) making issues and problems:– making N-tuples is very I/O intensive– it is non-trivial (need to learn N-tuple

creation tricks)– physicists need to use the heavy analysis

framework– making N-tuples takes quite a bit of CPU

(latest analysis-level corrections are applied)– significant load on the central processing

facilities– significant effort per analysis for each

physicists


f


Offline Computing

• New N-tuple production:– common framework produces the 3 most used formats– shared expertise and resources– reduced load on on-site processing facility (N3)– fast, 3 day turn-around– off-loads N-tuple making from analyses

• Expedited N-tuple production great success:– individual analyses start from much smaller production

N-tuples– analyses are expedited

• Fermilab group developed and manages expedited N-tuple production(from the need to reduce load on central processing

facilities)


f


Offline Computing

• Currently three on-site processing facilities:– production farm (does full event reconstruction)– “CAF” (executes user analysis jobs, CDF-only)– “FermiGrid” (latest user facility, unused cycles

are being shared with other Fermilab experiments)

• Working on evolving processing facilities into Grid based facilities with one on-site and one off-site job submission point for users:– idea and early development of glide-in

mechanism by Fermilab-CDF person– development into common Grid tool by

Fermilab-CMS


f


Example, Higgs Search

• The search for the Higgs is the top priority for the CDF Experiment.

• Additional sensitivity, beyond the increased integrated luminosity, needs to come from:

– increased lepton acceptance– forward b-tagging– improved di-jet mass resolution


f


Example, Higgs Search• A Fermilab scientist is leading the effort to

trigger on muons that fall into the phi gaps between wedges.

• Forward, BMU muons have been pioneered by an ex-University of Wisconsin Fermilab scientist.

• Fermilab postdoc modified tracking chamber digitization for increased hit efficiency at high luminosity.

• A Fermilab scientist has reworked the tracking software for increased acceptance in the forward region and to get good efficiency at high luminosity:– forward b-tagging– lepton acceptance

• A group to work and improve the di-jet mass resolution was setup by CDF last autumn. It is co-lead by a scientist of Fermilab.


f


Sumup

• Fermilab contributes with 25 FTEs (of 65 total) to the detector operation.

• Fermilab contributes with 12 FTEs (of 25 total) to the offline computing.(three of the five top leaders are Fermilab)

• Of the top experiment leaders/managers 5 (of 7) are Fermilab scientists.


f


Summary

• Fermilab continues to support the CDF experiment way beyond its host laboratory duties.

• With upgrades winding down Fermilab scientist are looking for ways to increase the efficiency, shorten analysis turn-around, and reduce manpower.

• Strong Fermilab participation in all main support areas of the experiment including collaboration leadership.

f cdf experiment fermilab experiment support stephan lammel we, sep26th, 2007

Documents