made by : maria chamizo andrea barisone · 10/16/2012 shift leader training meeting 10 phones:...

10/16/2012 1 Shift Leader Training Meeting

Shift Leader Tutorial

Made by :

Maria Chamizo Andrea Barisone

Last Revision : September 24 , 2012


Outline

1. Become a Shift Leader 2. Main Responsibility 3. Documentation 4. Shift Leader : Location and Tasks 5. Monitoring Tools 6. Overview of LHC (Modes, Fill-Cycle) 7. Shift Leader Check List 8. Handshake 9. Injection and Injection Inhibit 10. Ramp, Flat top, Squeeze and Adjust

11. Basic Checks on Stable Beam

12. Pixel and Strip HV Procedure

13. BRM Displays 14. Adjust Handshake 15. End of Fill 16. Cosmic Run 17. Magnet Ramping 18. Conclusion


To become a full operational Shift Leader remember you must follow : Shift Leader Tutorial (ok , which we are doing now !! ) DCS Tutorial (two Wednesday each month at 14:00 , P5 meeting room) SLIMOS Tutorial

Before do alone his first Shift every new SL will be scheduled for 2 training Shift.

Become a Shift Leader

Please contact Andrea Barisone to be signed up : [email protected]

mailto:[email protected]


1) The Shift Leader is also the SLIMOS (Shift Leader in Matter of Safety) You will discover all the details about the SLIMOS duties in the specific

tutorial, at the moment, just remember that you are responsible for the SAFETY of :

People Experiment 2) As a Shift Leader you’re the Reference for the CMS Control Room Activities : The SL coordinates the implementation of the daily run plan in the most efficient way The SL is the contact person between CMS and LHC control room (CCC)

Important note : In case of doubt, he must call the Run Field Manager (75575) or Run Coordinator (165765 - 161931)

Main Responsibilities


Much more detailed descriptions about the SL task are available in the dedicated online page, the “Shift Leader Duties and Instructions” : ( inside the CMS Workbook, here below the link) https://twiki.cern.ch/twiki/bin/view/CMS/OnlineWBShiftLeaderInstructions It contains an exhaustive description of: Shift Leader General Information Safety and Technical Interventions Instructions Associated to LHC Instructions Associated to Magnet Operations Tools (Elog, Run Time Logger, Online Monitoring Site)

Documentation (1)

https://twiki.cern.ch/twiki/bin/view/CMS/OnlineWBShiftLeaderInstructions


Important note : At the beginning of the Shift, the SL has to verify first of all the breaking news : You will find them on the top of the “SL Duties and Instructions” page.

Documentation (2)


Another really important document is the Contacts List : https://twiki.cern.ch/twiki/bin/view/CMS/OnlineWBContactList Where you will find all the telephone numbers you could need to call during your shift !

Documentation (3)

https://twiki.cern.ch/twiki/bin/view/CMS/OnlineWBContactList


Shift Leader : Location and Task (1)

DAQ Shifter

DCS Shifter

DQM Shifter

Trigger Shifter

This is the Shift Leader

Place

Control Room Overview

BRM Displays

Injection Button



SL Station Overview LHC main screen

Events Displays

Tandeberg (Videoconference)

SL Operational Displays


Phones: There are two phones at the Station of the SL 1) Mobile phone mapped to the virtual phone number 77705. This is used by

the CCC or people outside CMS to contact the CMS control room. 2) Cordless phone: a fixed line (77111 )to receive calls made by CMS members. Remember, the most relevant contacts are available here: https://twiki.cern.ch/twiki/bin/view/CMS/OnlineWBContactList Important: Mobile calling SHOULD ALWAYS BE ANSWERED If you carry this phone with you ... don't forget to leave it there at the end of your shift.


https://twiki.cern.ch/twiki/bin/view/CMS/OnlineWBContactList


Communication : An important part of the SL job is a good and clear communication between him\her and the Shift Crew, that means : Interact with the other Shifter (DCS,DAQ,DQM and Trigger) to know what is going on Decide (or be aware) when an expert on-call must be contacted Filter the subsystem request in order to preserve the Run Be the reference for the LHC Control Room (CCC)

Many – but not all – DQM shifters are relatively inexperienced As shift leader we expect you to interact with the DQM shifter In particular: at start of new runs work with the DQM to make sure that there are no problems.



There’s one simple question that could be come in your mind, this one : “Which are the the means at my disposal to fulfill my responsibilities?” First of all, you should open the CMS Run Coordination Page : https://cms.web.cern.ch/org/run-coordination

Monitoring Tools (1)

In this section you will find the Shift Leader Instructions and the Tutorial

On his left, there are listed all the main tools to check the CMS activity, in the next pages we will explore some of them.

Here there is the Phone List … discover by yourself all the other links!

https://cms.web.cern.ch/org/run-coordination





Run Time Logger (RTL) : The RTL is used to monitor the downtime/lifetime status of CMS during data taking. It automatically creates a downtime entry (provided SCAL FM is in the readout) when there are no L1 triggers in CMS. The SL has to choose which subsystem is responsible for the downtime and report it in the RTL.




E-log: The SL is responsible for entering information in the shift e-log to describe : Start/End of the shift / Data taking conditions/problems / Hourly updates (during

the Run)



10/16/2012 15 Shif Leader Training Meeting

You can check the expected status of every CMS subdetectors (according to the LHC beam mode) looking to the Automation page. You can find it in the CMS Run Coordination page as “ Detector Safety Mode” Or directly clicking on this link :

http://cmsonline.cern.ch/portal/page/portal/CMS%20online%20system/DCS/Automation

http://cmsonline.cern.ch/portal/page/portal/CMS online system/DCS/Automation



At your station, in front of you, little above your main displays, there’s a big screen where are reported all the relevant information about LHC status : We will discover later how to get quickly the main information you often need to check. As usual you have a link at these pages also in the Run Coordination Page.


LHC : Diagram

10/16/2012 18

LHC : Modes

Shift Leader Training Meeting

The LHC modes are summary of the machine state. Three main Modes are defined : • Accelerator Mode =summary of the LHC machine state • Beam Mode = description of the main phases of the accelerator cycle • Sector Operation Mode = is a given LHC sector operational or not? ( We won’t consider this part in our Tutorial)

Next two pages contains a more exhaustive description about the first two Modes : We leave you alone to read and if you like get really detailed information download the following document inside the EDMS Portal https://edms.cern.ch/document/1070479/

https://edms.cern.ch/document/1070479/

https://edms.cern.ch/document/1070479/


LHC Accelerator Mode Accelerator mode Description Beam Status

SHUTDOWN Usual winter status. Cold magnets floating. NO BEAM

COOLDOWN* Coming back from shutdown. The principal activity during this mode is cryogenics related

MACHINE CHECKOUT* Check the simultaneous functioning of the various LHC sub-systems in the final configuration following the LHC hardware commissioning

ACCESS* Access

MACHINE TEST* Operations’ tests without beam. Ad hoc tests without beam during normal running periods

CALIBRATION* Power Converter calibration. No external interference with circuits

WARM UP* One or more sectors warming up for repair.

RECOVERY* Typically quench recovery, or recovery from cryogenics plant disturbance.

SECTOR DEPENDENT This Accelerator Mode tries to cope with the situation in which different sectors of the machine can be in different states at the same time

BEAM SETUP Machine setup with one or both beams BEAM

PROTON PHYSICS Beam based operation aimed at proton physics.

ION PHYSICS Beam based operation aimed at ion physics.

SPECIAL OPTICS PHYSICS Beam based operation with special optics, like for example the one for TOTEM.

MACHINE DEVELOPMENT Beam based machine development.


Beam mode Description

SETUP Possibly beam in transfer lines with transfer line dumps in. Includes pre-injection plateau and injection plateau - no beam in ring.

ABORT Recovery mode following beam permit drop. This mode can be entered from any state if there is no beam in the machine.

INJECTION PROBE BEAM

If either ring 1 or ring 2 will be injected with or have safe beam circulating. In this mode a number of checks will be done for the different accelerator sub-systems before injecting higher intensities. The aim will be to establish a circulating safe beam with a given lifetime

INJECTION SETUP BEAM

In order to make more precise measurements before filling for physics, a SETUP BEAM will be used.

INJECTION PHYSICS BEAM

At this stage the machine has been optimized. It proved to be able to have circulating beam with appropriate lifetime and it is ready to accept higher intensities needed for physics

PREPARE RAMP Injection complete, preparing for ramp.

RAMP Ready to ramp or ramping or immediate post ramp.

FLAT TOP Ramp finished - pre-squeeze checks.

SQUEEZE Preparing for or squeezing.

ADJUST Preparing for collisions or adjusting beams after the squeeze.

Beam mode Description

STABLE BEAM Stable conditions with collisions in the experiments, backgrounds and life time under control. Small adjustment of beam parameters permitted.

UNSTABLE BEAM

Emergency mode entered from stable beams in case of sudden beam degradation. The UNSTABLE BEAMS mode may be entered without prior warning to the experiments

BEAM DUMP Requested or emergency dump. It will be verified that all the machine protection equipment performed correctly

RAMP DOWN Ramp down and cycling after programmed dump at end of physics fill.

CYCLING

Pre-cycle before injection following access, recovery, etc. The objective of this mode is to reset the magnetic history of the machine and prepare the machine for a new cycle.

RECOVERY Within this mode the reason of the abort, quench, emergency dump, etc, will be diagnosed by the post mortem analysis system.

INJECT AND DUMP

This mode may be used during first commissioning or for injection studies. I

CIRCULATE AND DUMP

Dump after large number of turns following injection

NO BEAM In a machine mode where there is no beam or no preparation for beam.

LHC : Beam Mode

10/16/2012 21 Shif Leader Training Meeting

LHC : Fill Cycle Flat Top

10/16/2012 Shift Leader Training Meeting 22

LHC : Beam Mode (Sequence)

1

2

3

4 5 6 7

8

9

10

1

3

2

Handshake (Injection) LHC CMS

Inhibit Injection

Start of a Physic Run

Start of a Cosmic Run

Stop the Cosmic Run


LHC : Get Information

The best way to be in touch with the LHC operation is checking regularly its Elog : https://ab-dep-op-elogbook.web.cern.ch/ab-dep-op-elogbook/

https://ab-dep-op-elogbook.web.cern.ch/ab-dep-op-elogbook/














Shift Leader Check List

To help the Shift Leader we maintain a detailed checklist that he must follow during his shift.


How to use the Check List

It’s frequently updated to reflect latest changes so should be the most up-to-date document : Always make sure you are using the latest version !!! (print it from to the SL instruction page , on the right of the Breaking News window) As an LHC fill can cover several shifts, please continue the same checklist, but note where you have take over. At the end of the fill put the checklist into the red binder. If you find problems with the checklist please inform the RFMs or the run coordinator. No “oral traditions” are allowed.

Take care : there are several different checklists

( Proton Physics 8TeV,Heavy Ions, Machine Development … ) so chose the right one !


Handshake

The Handshake Operation : Before LHC issues the Injection Warning handshake (seen in DCS console) you should be contacted by LHC control room (CCC) : they will ask you to lift the beam injection inhibit button.

Here you can see the part of the DCS Main panel dedicated to the Handshake operation : once pulled the injection button this image should change showing its released form

Once performed all the steps in your Check List dedicated to this operation you must pull the injection inhibit button to allow LHC to inject.


Injection inhibit. Green light off - injection inhibited

Beam abort

Injection Inhibit


When LHC issues Injection Warning handshake (seen in DCS console): Detectors will automatically be brought to a safe state If pixels or strips are off – call the pixel and strip DOCs Stop the Cosmic Run Once all checks in the checklist are performed the SL pulls the injection inhibit

button to allow LHC to inject. (PAY ATTENTION OF DOESN’T ACT ON THE WRONG BUTTON, THE BEAM ABORT !!)

LHC leaves setup beam mode and goes to injection probe beam

No major action required (follow the Check-List)

Then LHC changes mode again to Injection Physics Beam. No major action required (follow the Check-List)

Injection (1)


After injection is complete LHC goes to Prepare Ramp mode Shift leader pushes the injection inhibit button to prevent further

injections (PAY ATTENTION OF DOESN’T ACT ON THE WRONG BUTTON, THE BEAM ABORT !!) then LHC announces on LHC page 1 that next fill is for physics

Verify that CMS inhibits injection (green light off - button pushed in)

Follow checklist to make sure all subsystems are ready. Since the beginning of 2012 if the injection is allowed Pixels and

Tracker will not be able to switch on the HV. LHC Accelerator Mode should be proton physics

Injection (2)


Ramp: No major action required (follow the Check-List) Flat Top: Start a Physics Run Make sure we are using LHC clock – check with DAQ Check the cogging Ask DAQ shifter to destroy RCMS configuration and recreate it. Start a run

with the collision configuration and check that the correct combination of keys are used.

Squeeze and Adjust : At Squeeze DCS automatically turns HV for ZDC, DT, and RPC. In case the LHC skips the Squeeze mode the DCS shifter should manually turn on HV for DT and RPC.

Ramp, Flat Top, Squeeze and Adjust


Stable Beam: Immediately after LHC has declared the Stable Beam the Shift Leader and the

DCS shifter check that the beam conditions are acceptable to turn ON the Pixel and Strip HV (more details in the next page).

When HV is ramped for strips and pixel the DAQ automatically do a Pause & Resume.

DAQ receives the PHYSICS_ON flag.

If the LHC does not perform a miniscan shortly after stable beams call CCC and request a scan. Also if CMS lumi is more than ~5% lower than ATLAS request a lumi scan.

Stable Beam


Pixel and Strip HV Procedure (1)

When the beam mode reaches STABLE BEAMS with machine mode of PROTON PHYSICS, the Shift Leader and DCS shifter should check the state of the Pixel and Strips detectors and be sure that they have automatically turned ON. If after ~1.5 minutes of STABLE BEAMS the Pixel and Strips detectors have NOT turned to 100% ON and/or are stuck in GOING_TO_PHYSICS state:

DCS shifter send the GO_TO_PHYSICS command once (to the corresponding partition) if sending the GO_TO_PHYSICS command once does not bring the specific sub-detector

ON, the Shift Leader must call the corresponding Pixel and /or Strips DOC.

If they stay in STANDBY, the automated turn-on did not work, maybe because the backgrounds were not OK when Stable Beams were declared. The DCS shifter follow the instructions to manually turn ON the Tracker (see next page)…

Important: the communication between the DCS shifter and Shift Leader is crucial for good cross checking !


Pixel and Strip HV Procedure (2)

If after ~1.5 minutes of STABLE BEAMS (machine mode of PROTON PHYSICS ) the Pixel and Strips detectors are still in STANDBY the Shift Leader ( with the support of the DCS shifter ) checks if the beam conditions are acceptable to manually turn ON the Pixel and Strip HV. Here is the check list with the controls that must be performed (the main part):

oINJECTION must be inhibited oBKGD1 and BKGD2 must be below 20 oBKGD3 must be below 40 oBEAM MODE must be STABLE BEAMS

When the conditions for turning on the HV are met, the DCS shifter turns ON their HV Channels. (remember : he should do that only after your agreement ) The complete procedures are available in the usual spot: https://twiki.cern.ch/twiki/pub/CMS/PixelOperations/PIXEL-HV-Procedures.pdf

If any of these conditions are not met call BRM 165501 and Pixel 165502

Since 2012 if the injection is allowed (the injection inhibit button is released and its green light is on) Pixels and Strip Tracker won’t be able to switch on their HV (as additional security measure)

.


BRM Displays

Top Left BRM 1hr Summary

Middle Left BCM

Bottom Left BSC/BCM1F

& DIP browsers

Top Center Shift leader display

Middle Center BPTX scope snapshots?

BCM1F bunch ID?

Top Right Lumi Display

Middle Right LHC Bunches

Bottom Right Post Mortem

SCX5SCR27-CMS SCX5SCR28-CMS


BRM Screen

Currents and Energy Bunches

Luminosity CMS Rates

BKGDS sent to LHC

Timing

BCM. BCM.


Basic Checks to do during Stable Beam (1)

Record in the Elog at the beginning and every hour:

(Initial) beam intensity per beam Instantaneous luminosity

Background levels for BKGD1, BKG2 and BKG3 Total L1 rate (you can ask the Trigger shifter)

Stream A rate Express Stream rate Stream A Event size

DQM status (check done with the DQM shifter) Trigger status (check done with the Trigger shifter) DAQ status (check done with the DAQ shifter)



Initial Beam Intensity

Backgrounds Levels Instantaneous Lumi



Stream A

Lv1 Rate Event Size

Express Rate



Here you can see how many slices of the PC farm are in or out the data taken

Here you can see the State of

each Subdetector

and the number of FED involved in the

data taken


The LHC might want to go out of stable beams to make orbit adjustments etc. This is done by them issuing an Adjust_Warning

DCS automatically sends pixels, strips, and ZDC to standby. As shift leader you inform pixel and strip DOCs that an adjust handshake is

ongoing.

When stable beams again declared follow the procedure to turn on the HV for pixel and strips

Adjust Handshake


When LHC issues the Beam Dump Warning : a) Three minutes after the dump warning, DCS sends PIXEL, STRIP and ZDC

automatically to STANDBY, if any of the them are ON. The DCS Shifter cross checks the sub-detector status with the Safe Mode Detector Automation.

b) The beams are automatically dumped five minutes after the Warning was sent. Continue the run until the beams have been dumped.

c) At this point a Cosmic Run (with possible calibration activities) can start.

End of Fill


After the beams have been dumped we want to start a cosmic run as soon as possible in order to have continuous monitoring of the ECAL. Reconfigure for cosmic data taking using the local clock. Make sure to include RPCs, CSCs, and DTs in the run. Confirm that CMS is inhibiting injection. When LHC goes to beam mode RAMP DOWN: The pixels HV will turn on automatically. Ask C-DCS shifter to switch ON STRIP TRACKER End the run when LHC starts the injection handshake

(detectors will automatically go to standby).

Cosmic Run after Beam Dump


When the CMS solenoid is ramping several subsystems needs to be put into a safe state:

ECAL off HCAL standby (HF can be on)

CSC off DT standby CASTOR off

Strips and pixels can be on.

As shift leader you should make sure that these detectors are in the off state before any intentional ramp takes place.

Magnet Ramping

10/16/2012 44

Conclusions

You are the SLIMOS (Shift Leader in matter of Safety) You coordinate the implementation of the Daily Run Plan in the most efficiently way You are the contact person between CMS and LHC control room (CCC) At the start of your Shift, verify here the breaking news :

https://twiki.cern.ch/twiki/bin/view/CMS/OnlineWBShiftLeaderNews Interact with the other Shifter (DCS,DAQ,DQM and Trigger) to know what’s going on Decide (or be aware) when an expert on-call must be contacted Filter the subsystem request in order to preserve the Run

Your most important tasks :

Your most powerful tools to achieve a good result:

Check List Contact Phone List Run Time Logger

Elog Shift Leader Online Instructions Run Coordination Page

Shift Leader Training Meeting

https://twiki.cern.ch/twiki/bin/viewauth/CMS/OnlineWB

https://twiki.cern.ch/twiki/bin/view/CMS/OnlineWBShiftLeaderNews

made by : maria chamizo andrea barisone · 10/16/2012 shift leader training meeting 10 phones:...

Documents