S. Zimmermann, ATLAS Post-LS1 Operations Workshop 1

Post-LS1 Shift Scheme and Shift Organization

24 June 2013

S. Zimmermann, ATLAS Post-LS1 Operations Workshop 2

Recap: 2012 shift scheme

24 June 2013

• 2012 shift crew consisted of 9 (detector operations) + 1 (ADC, distributed computing) shifters

• Detector shifters: Muon, ID + BCM, Calorimeter + Forward (Lucid, less ALFA/ZDC)• Trigger• Run Control• Data Quality, including luminosity monitoring• Slimos and Shift leader• Distributed Computing, including database monitoring also for ONLINE

S. Zimmermann, ATLAS Post-LS1 Operations Workshop 3

Shift organization 1Questions asked to systems as part of the questionnaire1. Were your shifters required to do a minimum number of shifts in a certain

period of time? If yes, what was the requirement? 2. Did you have problems to enforce or meet the requirement? 3. Were institutes of your system generally ok with the imposed requirement? 4. Do you think a minimum requirement on shifts helped with good shift and

shifter quality and competence? Answers:5. Most systems had a requirement on a minimum number of shifts, varying from

simply asking shifts should be done in blocks (Trigger), to 2 (DQ) - 6 (LAR, SCT, PIX) blocks in xxx weeks, with xxx typically 2-3 months. The exception was run control with no requirement on a minimum number of shifts.

6. The general answer was NO, with 2 exceptions: Muons („yes“) and LAR („sometimes“)

7. General answers from essentially all systems was “yes”.

24 June 2013

S. Zimmermann, ATLAS Post-LS1 Operations Workshop 4

Shift organization 24. Do you think a minimum requirement on shifts helped with good shift and shifter quality and competence? The unequivocal answers from ALL systems was “yes, definitely” !!

24 June 2013

A look at the reality (thanks to Wainer for extracting the info from OTP) – non-detector desks

Fraction of shifters having done 1-5 shifts is significant !

S. Zimmermann, ATLAS Post-LS1 Operations Workshop 5

Shift organization 3

24 June 2013

Detector desks LAR req. 6 blocks

LUCID req. 3 blocks

TIL req. 1 block ?

Situation even less pronounced for detector desks ...

A remark: Official ATLAS rule, still stated in the shift booking pages, is min. 20 shifts per person doing shift work ....

S. Zimmermann, ATLAS Post-LS1 Operations Workshop 6

Shift organisation 4

24 June 2013

Gaps between shifts for the same shift person: many people followed the rule of doing shifts within a 2-3 month period, but there is also a significant tail to many many weeks(the plot is somewhat biased due to the different number of shifts done on average per system)

Conclusions:• Definitely should require a certain minimum number og shifts after post-LS1

from shifters, and enforce it – subdetector communities seem all ok with it• Many systems remarked tools to check/enforce shift requirements are lacking,

maybe we should see globally (OTP) what can be done for it.

S. Zimmermann, ATLAS Post-LS1 Operations Workshop 7

Post-LS1 shift scheme 1Questions asked to systems:1. DCS operations: Move to a shift crew where all DCS monitoring and DCS actions

are being handled by a common DCS shifter for all sub-detectors. 2. DAQ operations: In a merged shift crew, the DAQ shifter will in charge to handle

ANY stopless removal, recovery, the busy monitoring of all sub-detectors, these tasks no longer being done by subdetector shifters. Would there be tools missing to allow such?

3. Separating DAQ from DCS operations: – Do you see typical issues in your subsystem where DAQ and DCS interfere,

that would be less visible if not monitored by the same shifter?– Do you fear having separate shifters handling DCS and DAQ/Run Control will delay taking

the correct action in case of a major problem?4. What is the minimum number of shifters in the control room you think one could

arrive at to safely and efficiently run ATLAS? 5. Do you see compelling reasons which absolutely require to keep sub-detector

specific shifters? If yes, which ones? 6. What tools do you think are most missing in order to reduce the number of

shifters?

24 June 2013

S. Zimmermann, ATLAS Post-LS1 Operations Workshop 8

Post-LS1 shift scheme 25. Do you see compelling reasons which absolutely require to keep sub-detector specific shifters? If yes, which ones?

24 June 2013

The answer to this question was „No“ or for some non-detector systems, no answer which we took as „no/not applicable, eg DQ, with the following exceptions/caveats:

DAQ: We do not see major technical reasons. However, as discussed above, a lot of effort will have to be put into standardizing procedures, including e.g. calibrations. It is not clear this feasible in basically one year when a lot of effort will be put into having ATLAS running again. One should probably consider how quickly we want to get there and be prepared to invest the needed amount of resources.Accounting and responsibility are to be considered. Who will be the responsible of a merged desk? There seems to be wider implications on the shift model used so far.

Lumi: A: competence –or lack thereof – over too broad a range of concepts, jargon, issues, actions.

LAR (*): No if the shifters are of the proper level of qualification. (*) LAR advocats strongly a professional shifter scheme with people on shifts for ~ 6months

Muons: Yes. (At least the first few months of Run II, we would like to keep the Muon shifter since we need to optimize our tools and recovery procedures. We can reduce the shifter only if we reach the stable operation state.)

Comment: this sounds more like what an expert would do in the intial running phase, which will be the case for all systems ...

S. Zimmermann, ATLAS Post-LS1 Operations Workshop 9

Post-LS1 shift scheme 31./2./3. A shift scheme were all DCS operations is in the hand of a common DCS shifter, all DAQ operations in the hand of a common run control shifter

24 June 2013

Systems generally seem to agree IF one goes for a recduced/merged shift crew, this is the natural way to organise it ....

There is strong objection against such a scheme by LAR:

„In a direction of reducing the ACR Crew, it is important to raise the level of expertise of the crew with a detector global vision meaning a global vision connecting Physics, data flow/data quality and detector behavior. This really does not go in the direction of having a central DCS crew but more to have a reduced expert crew in all the above aspects ( so several run manager/shift leader experts). So no central DCS shifters.”

S. Zimmermann, ATLAS Post-LS1 Operations Workshop 10

Post-LS1 shift scheme 41./2./3. A shift scheme were all DCS operations is in the hand of a common DCS shifter, all DAQ operations in the hand of a common run control shifter

24 June 2013

• Systems generally (LAR excepted (*)) had no concerns if DCS actions would be executed in a merged scheme by somebody not from their own system. • PIX actions are currently restricted to experts, which is planned to remain so.• Lumi made the comment that luminosity monitoring while via the DCS

infrastructure, should in a merged scheme not be under the DCS shifter, but rather coupled to data quality or shift leader, since it frequently plays a role in communication with the LHC/CCC.

• All systems declared they would see the DCS shift task then open to anybody, with proper training, not requiring specific detector experience etc.

• In general, systems did not see too many tools in the area of DCS missing to hand all DCS operations to a common shifter.

• Handing all DAQ operations, incl. Removals and recoveries to a common shifter. Some systems stressed this would require standardized removal/recovery mechanisms, and could only work if the recovery does not require detector specific knowledge. Better BUSY monitoring was mentioned (L1Calo). Trigger rate checking would have to be automatised, which is alread planned. Some trigger monitoring as L1Calo TT maps would have to be seen how to handle.

S. Zimmermann, ATLAS Post-LS1 Operations Workshop 11

Post-LS1 shift scheme 51./2./3. A shift scheme were all DCS operations is in the hand of a common DCS shifter, all DAQ operations in the hand of a common run control shifter

24 June 2013

• Systems are generally not concerned about issues where DAQ and DCS interplay, and which potentially could be less visible if not monitored by the same shifter.• PIX answered some revision in the handling of the pre-amps off may be needed,

which at present result in a harmless DCS NOT_READY state whenever the run is stopped and the ATLAS partition goes below CONNECTED.

• Systems are generally not concerned that having separate shifters handle DAQ and DCS operations may delay taking correct actions in case of a major problem. (we had thought to eg a rack or crate trip leading to a stopless removal of a large part of the detector). Example answers given were• Systems are fully separate.• Any relevant DAQmessages related to DCS are displayed in MRS (TIL)• Not really, since for major issues an SMS/email is issued to experts anyway (TIL)

S. Zimmermann, ATLAS Post-LS1 Operations Workshop 12

Post-LS1 shift scheme 6

24 June 2013

4. What is the minimum number of shifters in the control room you think one could arrive at to safely and efficiently run ATLAS? (note: we exclude the ADC shifter here which is separate)

System Number Details

TIL 5-6 RunCtrl, SL, Slimos, DQ, DCS

DAQ/HLT Ultimately 3;At start 3+3

SL (incl. Lumi, DQ), RunCtrl (incl. Tigger), DCSThe above + ID, Calo, Muon

DCS 5 Slimos, DCS, RunCtrl(incl. Trigger), DQ, SL

DQ 3-5 (7?)

L1Calo 4-5 SL, RunCtrl, DQ, DCS, (calibration ? Slimos?)

LAR 3-4 Must be at the level of a run manager or run coordinator/professional shifter

LUCID 5 SL, RunCtrl, DCS, TDAQ (?), DQ

Muon 7 SL (incl. Slimos), RunCtrl, DCS, trigger, ID, Calo, Muon

PIX 5 SL, Slimos, RunCtrl (incl. Trigger), DCS, DQ

SCT 4 SL incl. Slimos, DAQ, DCS, DQ

TRT 5 Should aim for fewer but better trained shifters, more experience and cleearer docu – professionals ?

BCM 4

S. Zimmermann, ATLAS Post-LS1 Operations Workshop 13

Outlook

24 June 2013

Discussion