Installation Schedule and StatusGeorge Ginther

University of Rochester

for the DØ Collaboration

4 May 2006

4 May 2006 2

Run IIb Shutdown Activities

• Install and commission Layer 0 detector• Replace current L1Cal trigger with L1Cal trigger

upgrade• Replace current DFEAs in L1CTT with DFEA2• Complete cabling/commissioning of L1Cal Track

Match trigger• Complete modifications to L2STT to accommodate

Layer 0 inputs• Complete hardware modifications to muon PDT

readout to accommodate latency shift– 90% COMPLETE –AWAITING PARTS Currently expected 10 MAy

• Refine V15 trigger list

• Complete platform testing of pre-production AFEII-t

4 May 2006 3

Some Additional Shutdown Activities

• Preparation for latency shift In PROgress• Routine subsystem maintenance and improvements In

Progress– SAFETY SYSTEM TESTS/Tightening BOLTS….– Operating system upgrades – CONTROLS PROCESSOR UPGRADES

• Individual channel recoveries IN PRogress

• Luminosity Monitoring COMPLETE– Detector maintenance– Improve grounding, new preamps

• SMT IN PROGRESS– HDI recovery effort– BLM upgrade?

• CFT COMPLETE– New SBC in crate 0x52

• Calorimeter IN PROGRESS– ICD preamp modifications (and some phototube replacements)– Cable harness checks– Noise studies

• Muon IN PROGRESS– Source calibrations, HV calibrations, cable improvements, modifications to VME power supplies to

improve radiation resistance• L1Muon COMPLETE

– Improve power supplies

4 May 2006 4

Installed Luminosity Monitor

4 May 2006 5

Improving A Layer PDT Grounds

4 May 2006 6

Schedule• Installation schedule implemented in MS Project

– Scheduled 5 day work week – Includes a lab holiday (and treats it as such)– Parallel tasks implemented where detector configurations allow– 14 week duration (69 working weekdays)

• Our estimate of Layer 0 installation effort originally generated 14 week duration

• Second shift activities when appropriate• 8 weekdays of explicit schedule contingency• Weekends serve as additional contingency opportunity

• 51 working days into shutdown– No major upgrade related surprises (yet)– Minor adjustments to keep things moving– Working longer hours when necessary to maintain pace– So far, only one Saturday has been explicitly scheduled– Four week days of contingency have been used to insure that the

Layer 0 and re-installed H disk readout is as complete as possible

4 May 2006 7

Layer 0 Installation Milestones

ID TASK NAME ActualCurrent Forecast

Pre-ShutdownForecast

Milestone

Dates

2 Beginning of RunIIb Tevatron Shutdown 2/23/06 2/27/06 2/27/06

28 Detector Open, Ready for Access 3/1/06 3/2/06 3/07/06

34 RunIIa Be Beam Pipe Disconnected 3/3/06 3/7/06 3/9/06

47 H Disks Removed 3/15/06 3/17/06 3/23/06

56 RunIIa Be Beam Pipe Removed 3/21/06 3/27/06 3/31/06

75 Layer 0 Tooling and Mounts Ready 4/5/06 4/10/06 4/17/06

79Layer 0 Installed (including junction card mounts)

4/7/06 4/12/06 4/19/06

91 RunIIb Be Pipe Connected, Layer 0 Cabled 4/26/06 4/26/06 5/03/06

94 Inner H Disks Re-Installed (including cabling) 5/4/06 5/2/06 5/9/06

99Silicon Cold and Ready for Technical Commissioning

5/8/06 5/5/06 5/12/06

103

Complete Technical Commissioning of Silicon 5/12/06 5/11/06 5/18/06

118

Detector Closed for Tevatron Resumption 5/31/06 6/02/06 6/4/06

Allocated 4 days of contingency toLayer 0 and H Disk readout verification

4 May 2006 8

L1 Cal and L1 CTT Installations

ID TASK NAME ActualCurrentForecast

Pre-Shutdow

nForecast

2 Beginning of RunIIb Tevatron Shutdown 2/23/06 2/27/06

Level 1 Calorimeter Trigger Upgrade

158

Retire Run IIa L1 Cal Trigger Electronics 3/6/06 3/6/06

172

Complete Clean-out of L1 Cal Trigger Racks 3/9/06 3/29/06

180

L1 Cal Trigger Racks Ready for Trigger Installation

3/20/06 4/13/06

189

L1 Cal Trigger Ready for Technical Commissioning

5/10/06 5/10/06

Level 1 Central Track Trigger Upgrade

206

DFEA Crates Extracted 3/13/06 3/13/06

213

DFEA2 Installation Complete 4/25/06 4/4/06

217

L1 CTT Technical Commissioning Complete 5/17/06 5/17/06

Switched order of some installation andverification operations to optimize cable access

30-Apr  

0700-0730 Site-wide power outage

Cable N inner H disk

First production AFEII bare board

deliveredCable S inner H

disks

Complete installation of L1Cal LVDS

cablesVerify H disk connections

SAM V7 TestsComplete H disk

readout verificationInstall beampipe

supportsMake EC Be

beampipe jointsInstall Tedlar membrane  

7-May   Cool silicon SMT checkout

L1Cal Upgrade Ready for Technical

CommissioningSMT checkout

Muon PDT mod complete

SMT checkout

Remove gap hardware

Close ECs, EFs  

14-May   Install SNEGs

100 AFEII bare boards delivered

Leak checkInstall heat tape on

SNEGs

L1CTT Technical Commissioning

CompleteActivate SNEGs

Open EFsInstall BLMs

Survey ECActivate SNEGs contingency  

21-May

Begin detect

or verification

Detector checkoutResume captain

coverage on evening shifts

Test new latency?detector checkout

Test new latency?detector checkout

Tevatron coldTest new latency?Close CF and EF

Remove SNEG heaters

Install veto countersClose clamshellsSurvey Detector

Centerbeam  

28-May   lab holiday

AFEII bare board production complete

Survey Detector CenterBeam

Search and secure collision hall

Test magnet power supplies

StickMic SurveySupervised Access

Ends?      

4-Jun

shutdown

endsround the clock shift coverage resumes?          

11-Jun    First stuffed AFEII boards delivered      

first store

18-Jun HEP Collab Meeting Collab Meeting Collab Meeting Collab Meeting Collab Meeting  

25-Jun              

2-Jul  Trigger Latency

Change          

9-Jul    

First set of production AFEII

installed        

16-Jul              

4 May 2006 10

What might happen to the schedule contingency ?

• Detector Maintenance Issues NO IMPACT TO DATE

• Beampipe related – Radiation survey results Items near beampipe were Class 1– Beryllium wipes results Wipes were clean– Leak checking complications NO IMPACT SO FAR

• Maintain purges during installation whenever practical

• North EC beampipe conditions – Interior Temperature 8C—WEATHER WAS COOPERATIVE– Weld diameter ADEQUATE CLEARANCE– Surface irregularities ADEQUATE CLEARANCE

• Detector alignment ADEQUATE AlIGNMENT

• Unanticipated complications VLPC CRYOSTAT LEAKSCLOCK INSTABILITIESFragile H DISK MOUNTSSMOKE IN THE COLLISION HALLPERSONNEL MATTERS

4 May 2006 11

A Few Potential Concerns• Low energy high occupancy noise in ECs

– Currently in remission– Addressing this unidentified noise may be challenging– IMPROVING GROUNDING OF A LAYER PDTS

• Possible schedule conflicts since Dave Butler has been assigned to Mechanical Operations crew for duration of shutdown– Expect to call on Ken Schultz or Bert Gonzalez if Dave becomes

effectively overcommitted HAS NOT BEEN An ISSUE• Recent loss in L1Cal installation team

– Learned that Alan Stone will not be available for L1Cal installation on 31 Jan 2006

– Exploring responses with assistance of PPD– SELCUK HAS BEEN Very USEFUL CONTRIBUTOR– HAVE MANAGED WITHOUT MUCH ADDITIONAL SUPPORT

• Recent confirmation of loss for Layer 0 installation team– Sasha Leflat and Evgueni Zverev will not be available before May – OTHER COLLABORATORS HAVE PITCHED IN TO HELP FILL THE

GAP, AND CONTINGENCY HAS BEEN DEVOTED TO VERIFYING READOUT

4 May 2006 12

Personnel Complications

• Personnel issues generated challenges– Unanticipated resignations/transfers– Illnesses/other personal issues

• Surprising number of sick days early in shutdown

– Alan S (CMS)– Selcuk C (personal)– John A (ANL)– Youri O (TD)– Mike M (AFE maintenance)– Russ R (personal)– Vic M (personal)– Bill C (recovery from ILC trip)– Dave B (personal)– Rich S (personal and CMS)– Linda B (personal and CMS)– Steve J and Todd N (other assignments)

4 May 2006 13

Power Outages

• Power Outages for power distribution maintenance at DZero and master substation– Keep solenoid (and VLPCs) cold while helium

compressors are offline• Feeder 46B maintenance (8 March)

– Electrical power distribution maintenance was performed on 1 April

• VLPC cryostat leaks developed during this maintenance

– Site-wide power outage 3 April• Readback for L1CTM crate 0x14 +3.3V supply failed• CAL preamp power supply 11 primary failed to recover• Two online cluster nodes suffered hardware failures

– Site-wide power outage 1 May• Two BLS power supplies failed to recovery

4 May 2006 14

Approaching end of shutdown

• Aim to have all possible subsystems back to full operational status to probe for outstanding issues before detector is fully closed– Need to complete H disk cabling, testing– Make EC to Be beam pipe joints– Dry and cool silicon tracker– Re-establish silicon readout systems– Complete detector maintenance activities

– End Cap Calorimeters scheduled to be closed ~12 May to proceed with SNEG beam pipe installation and leak check

• Re-install BLMs and veto counters– Aim at having subsystems fully functional by ~21 May

where possible– Toroids scheduled to be closed ~25 May– Survey detector position– Solenoid and Toroid tests ~30 May

4 May 2006 15

Recovery Plans• Re-establish detector operations

– Come out of shutdown with current trigger latency• L1Cal Track Match results too late to participate in trigger

– Checkout detector performance (online and offline verification)– Verify beam spot location

• Adjust location and firmware accordingly• Trigger latency change to be implemented as soon as detector

performance is stable (~2 to 3 weeks)– Will need offline verification that timing adjustments have been properly

implemented in all subsystems• Prompt offline analysis will be essential to minimizing the luminosity

cost of implementing the upgrades (and recovering from the installation and shutdown)– Alignment– Calibration

• AFE II installation/commissioning in parallel– Currently anticipate delivery of stuffed boards between late-June and early

August– Expect adiabatic installation in batches– Operate in AFE I mode (no timing information)– Dependent upon offline analysis (trigger, tracking and object ID) to certify

system as it is installed

4 May 2006 16

More on Recovery

• In addition to optimizing the trigger once the new elements are technically commissioned…

• Tracking consequences– Optimize tracking using layer 0 data

• Online and offline– Since outer H disks will no longer be in place, forward

tracking may require adjustments• Luminosity Monitoring

– Luminosity monitors have been dismounted and serviced and will require scrutiny

– Material in front of luminosity monitors changed– Re-examination of luminosity constant

• ICD maintenance

• Preparing for even larger instantaneous luminosities

4 May 2006 17

Summary

• Shutdown involves substantial parallel efforts to install Run IIb upgrades of the DØ detector and make a smooth transition to commissioning and operations

• Need to complete the shutdown activities in a safe and timely manner– 51 weekdays days into 69 weekday shutdown (74%)– Most activities progressing well– No major upgrade related surprises detected (yet)– Collaborators continue to make important contributions– Lab is providing significant support for these activities– Second shift support and activities have been very valuable in

maintaining schedule • ~Four weeks remaining until end of shutdown• Commissioning upgrades (and recovering from the shutdown)

must be completed reliably and efficiently to ensure that the post-shutdown data is as fruitful as possible– Addressing these challenges in a timely manner requires

significant contributions from the lab and a broad range of collaborators

4 May 2006 18

Closed Configuration

4 May 2006 19

Clam shells and muon shielding open on north end

4 May 2006 20

North EF open too

4 May 2006 21

South EF and EC open allowing access to the south gap (assuming CF is also open)

GAP

4 May 2006 22

Mechanical Support• Ops personnel will be working two shift schedule

– First shift 0700-1530• Russ Rucinski• Pete Simon• Bob Kubinski• Chris Tolian• Dave Butler (Technical Centers)• John Cornele (Mechanical Department on an as needed basis)• Ops Shifter• Dan Markley (for detector moves)

– Second shift 1500-2230• Mike Sarychev• Jim Fagan• Bob Barger• Rolondo Flores• Chuck McNeal• Steve Jakubowski (Technical Centers)• Ops Shifter• Todd Nebel (Technical Centers--at start and end of shutdown)

– Ops Shifters• Bill Frank, Bryan Johnson, Pat Healey, John Najdzion

4 May 2006 23

Electrical Support

• Operations and Shutdown support– John Anderson/Mike Matulik– John Foglesong– Mike Cherry– Victor Martinez– Shoua Moua– Johnny Green (Electrical Engineering Department)– Tim Martin (Electrical Engineering Department)– Bruce Merkel (Electrical Engineering Department)

• Plus ongoing support from EED Colliding Beam Experiments group– Lyn Bagby, Jamieson Olsen, Mike Utes

• AFE II activities must continue in parallel

4 May 2006 24

Additional Manpower

• Alignment• Layer 0 Installation

– Handling SNEGs, gate valves, leak checking (AD)– Dave Butler, Mike Roman, Joe Howell, Youri Orlov (PPD)– Dan Olis (TD)– Bert Gonzalez and Ken Schultz (PPD)– Randy Wyatt (Alignment) as backup for Mike Roman (PPD)– Welder for EC beampipe flanges (~1 day)– Ron Davis (Vacuum) to make beampipe joints (PPD)

• Trigger Upgrades– CTT Mixer Stefano Rapisarda (CD)– L1CalTrack visitor support– PDT modifications for latency shifts

• Sten Hansen• electrical tech for 6 weeks