summary of the mini workshop on atlas counting room (atcr)

ATLAS Week Plenary Meeting - 26 February 2004Beniamino Di Girolamo - CERN PH/ATD

Summary of the mini Workshop on ATLAS Counting Room (ATCR)

Beniamino Di GirolamoCERN


Mini Workshop Program (part I)

Expe

rienc

e an

d fu

ture

ATLAS-LHC interfaces


Mini Workshop Program (Part II)

ATL

AS

syst

ems

requ

irem

ents

Experience and discussions from WG


Summary

Summarizing has been difficult Many slides Lively discussions Going back to the original material is

suggested to have more detailed answers on specific questions

All slides are available athttp://agenda.cern.ch/fullAgenda.php?

ida=a04353


Goal

Start discussions inside ATLAS on the detector operating model

Survey the experience gained up to now in/outside HEP in this domain

Collect first requirements and needs

Define a “control room” ATLAS project from commissioning to steady running of the experiment

M. Nessi


Experience talks: J. Proudfoot on D0&CDF

experience and hints useful because very near to our future needs: Access system based on microchip-equipped

badges Shift crew: two different approaches

• D0: based on sub-detector partitions• CDF: not based on sub-detector partitions

In both cases geographical divisions of functions of the stations with a shift captain in the middle

Binary decision making where possible Monitoring of data with extremely simple

automatic flagging of the quality, refined manually as soon as possible. Results logged in database• It takes 5 minutes to start a run, the same to stop.

Run is stopped only when reaching the limit record size on disk

J. Proudfoot


D0 Detector Monitoring Stations – arranged by

detector subsystem

D0 Data Acquisition Control and Monitoring

Station

It works efficiently. Nothingspecial just keep in mind future

upgrades and leave plenty of extra space.

J. Proudfoot


CDF “Slow Controls” Station

•Low & High Voltage

•Cooling

•Beam Losses

CDF Data Acquisition System control and Monitoring Station

Revamped between Run 1 and Run2

J. Proudfoot


Additional hints: different choices

D0 shift crew Captain DAQ calorimeter/muon central tracker Global monitor mechanical support

CDF shift crewSciCo, DAQ ACE, Monitoring ACE, COPlus 1 Operations Manager (either in control room or on call)Plus 1 Offline shifter on day shiftShift crew focus is to take data not to solve specific problems of sub-

detectors CDF choice

People brought into the game choosing among the best people available, not always tied to institutes responsibilities. No volunteers. 95 % resident at Fermilab

J. Proudfoot


Recipes

ACE - 12 wks. - 1/2 time Scientific Coordinator (Scico) - THREE - 8 day shifts in a yr. Consumer Operator (CO)- ONE - 8 day shift - maybe every 2

yrs. Everyone sends in availability for 6 months to a yr and we

have someone (DeeDee Hahn)work out the schedule

Training, - everyone gets safety training - rad worker, loto, controlled and supervised access before they come on shift. We have a CDF training officer (DeeDee Hahn) to give the training, though it can be taken through Fermilab ES&H.

One day of overlap shift for CO and Scico, so they get on the job training.

Aces have 2 weeks of overlap, plus 2 - half days of in class training

There is a large amount of web-based training material

J. Proudfoot


Remote access

No possibility to do remote shifts Remote checks and monitoring by experts A lot of material on the web Critical systems not available for remote

login

Efficiency

85 % efficiency of the operations, struggling to go up to 91 %

J. Proudfoot


Experience from ESO: A. Wallander

Model of operations Local visitor: classical highly interactive

operation Remote operations

• Negative experience: stopped

Support service operations• All observations fully specified in advance

(months) and stored in a queue• Execution done by professionals (staff

astronomer and operator) with minimum human interaction

A. Wallander


A. Wallander

Commissioning and operation experience

Commissioning of a new telescope site in Paranal

Commissioning plan with well defined tasks

Strong team on site (temporary relocation, missions) Day to day decisions on site (fast decision making)

Strict access control (no remote engineering)

IMPORTANT: strict policy on standards All PCs are from firm X and run OS Y Everybody becomes expert because everybody uses the

same material


Paranal Observatory aerial viewParanal Observatory aerial view

La Silla

A. Wallander


OperationOperation

A. Wallander


Different remote control experience: Far remote control of accelerators - F. Willeke

Completely different strategy from ESO Targeted to Linear Collider

From past experience: LC will be continuously in commissioning• It is also a way to keep the attention high

Far remote control strategy Not to save money, but to keep expertise in various

places not only in a central site Could follow time zone switch to change shifts

Sociological aspects: not discussed, but under careful analysis

F. Willeke


Collaboration Models

HERA/LHC Model GAN Model

Project

Host Laboratory

In kind Contributions

Partner Inst.

Project

Partner Labs

Site Laboratory

In kind Contributions

Special responsibilities

Site lab instead of host lab

F. Willeke


Experience from HERA, LEP, SLC...Experience from HERA, LEP, SLC...

Maintenance, Trouble Shooting RepairMaintenance, Trouble Shooting Repair:: essentially “REMOTE FACILITIES”,: problems diagnosed remotely before intervention, interventions by non-experts successful in 90% of the cases, experts help via telephone suffices or via remote access unscheduled presence of experts on-site is an exception

Commonality with ESOCommonality with ESO:: Very reliable equipment

If the intervention of the remote expert is needed on site, it may take a week

Therefore careful MTBF analysis and spare policy

F. Willeke


CERN Control Center

D. Manglunki

Integrate (NOT aggregate) the functions of: MCR PCR QCR TCR

… into ONE CERN Control Centre: the CCC

D. Manglunki


System requirements

Standardised consoles for AB, AT,TS: system allows any operation from anywhere Reconfigurable room

Fixed displays / CATV Access systems

Presently 4 different ones; some hardwired Fast analog signals observation and processing

(FFT, BTF,…) Administration PC Telephone Intercom

D. Manglunki


Current building extension plan

625m2 control roomat ground level

with5.6m ceiling height

Tele-com

Venti-lation

Servers

Operators services(kitchen, meeting room,

rest room, toilets, showers,lockers, ….)

Recep-tion

Visitors’balcony

Repair lab,Meeting room,

Videoconference/remote MD, temporary offices

[staged] …

•625m2 Control room40 console modules including 4 access systems40 fixed displays–Reconfigurable working space–Easy access–Comfortable light, acoustics, and temperature–Outside view–Combine visibility and privacy

D. Manglunki


Relations with the LHC

N. Ellis: Signal exchange These signal can be available in the ATCR Issue: policy of possible action on sub-

detectors based on info from machine Mutual machine <-> experiment interlock

B. Chauchaix Overview on safety system Implications for ATLAS


ATLAS to machine - illustration

N. Ellis

BEAM ZONES LHC TUNNELS/EXPERIMENTS

Point 7

Point 8

Point 6

Point 5 Point 4

Point 3.3

Point 2

Point 1

SPS

TI8

TI2

PX16 PX

14 PX15

PM15

USA15

UX15

US15 UL14

PM18

UJ18 UJ16

UJ14 UL16

RB14 RB16 RR13

UJ23 UJ24

UA23 RA23

PX24 UL2

4 US25 UW25 PM25 UL26

UX25 UJ26

PGC2 UA27

RA27

UJ27

R32

UJ32 TZ32 P

M32

R33

R28

R22

PZ33

UJ33 UP33 R34 R36

R42 UJ43 UJ44 UA43 RA43 UL44

US45

UX45

PM45

PX46 PZ

45

UL46 RB44 RB46 UJ46

UA47 RA47 UJ47 R48 TX46

UW45

UJ53 UL54

PM54

UXC55 USC55 UJ56

PM56

PX56

UL56 UJ57 RZ54

TD62 R62

UJ63 UA63 RA63

UJ64 UL64

UL66 UJ66 RB66

PM65

PX64

PZ65

UW65 US65 UX65 TX64

UA67 RA67

UJ67 UJ68

TD68 R6

8

UJ76 TZ76

PM76

R76

R77

UJ83 UA83

RA83 UJ84

PX84 PZ

85 PM85

UL84 UL86

UW85

US85

UX85 TX84 UJ86

RB84 RB86 UA87

RA87 UJ87 UJ88 R88

Point 3.2 SERVICE ZONES

LHC SERVICE GALLERIES/CAVERNS/SHAFTS

NON-INTERLOCKED AREAS

US32

Syst

em

Overv

iew

Controled and Interlocked AreasControled and Interlocked Areas Access control takes

place upon entering or leaving zones : Non-interlocked

areas Service zones Beam zones

ATLAS specificsATLAS specifics Monitor access – simple card reader (person ID):

Site entrance ATLAS Control Room -SCX1 Entrance of SDX1 Computer barrack inside SDX1

PAD at surface level - shaft PX15: All people entering must be identified with ID + Biometrics No safety token needed Log and display at ATCR the ID of the people entered

Date and time of entry and access duration Number of people present – (a maximum of ~100 people)

Personnel Access in UX15 (cavern) via a Personnel Access Device (PAD) at ULX15 & UPX16

Material Access in UX15 via a Material Access Device at ULX15 Tracing radioactive material – INB obligation

At the ULX15 & UPX16 access points At the PX14, PX16 (when opened)

Imp

lem

en

tati

on

at

Poin

t-1

(A

TLA

S)

Subdetector operational model

Functions to be done by shift crew (non-experts): Normal data-taking operation, monitoring of DAQ/DCS, radiation levels, …

Monitor DCS Warnings and Errors Monitor MINBIAS rates Monitor Calibration Triggers Monitor Basic Histograms Call LAr experts


Functions to be done by subdetector people locally at IP1 / expt area In normal datataking: monitoring as above but in more detail In local calibration running/dedicated studies (hardware or software), control ID

from control room or US(A)15 depending on tasks being performed. Maintain USA15 electronics - Tilecal Maintain DAQ Code (ROD crates) - Tilecal LAr

(initially, all actions are here at the ATLAS pit area) Detailed monitoring of histograms, DCS etc. for each LAr sub-detector

(EMB, EMEC, HEC, FCAL) Detailed status and checks of FEC and BEC electronics systems Need ability to control local runs (pulsing, calibration) as part of these

checks and diagnostics Local repairs in USA15 and EMF (Electronics maintenance facility)


LVL1 Quiet work area

Stations for preparing trigger configurations, etc. Stations for in-depth (offline) prompt analysis of trigger performance, efficiencies,

backgrounds, etc.

Local stations in USA15 Also some tables, chairs, etc.

Lab space work area full test-rig (or elsewhere on site?)

Space for use of laptops wireless networking

Storage cupboards for spares, tools, etc.


Functions to be done by subdetector people from their CERN offices Monitoring, and offline tasks not impacting the detector hardware Don’t expect to ‘take control’ from offices at CERN (ID) Calibration Coefficient Calculations (Tilecal) Database Updates Monitor TileCal Performance Physics analysis

LAr (this is at a later stage during more stable running)

Monitor subsystem performance for typical events/triggers Inform shift crew if something abnormal Expert control possible


Functions to be done by subdetector people at home institutes Monitoring and offline tasks not impacting the detector hardware (ID, LAr and Tilecal) Note: offsite but on-call experts will be very important – communicate with the local

team at IP1 (ID) Same as those at CERN offices Monitor TileCal Global Performance Physics analysis

Subdetector needs at IP1

In UX15 cavern: ID

Not defined in detail, but (wireless?) network access for laptops at UX15 platforms and close to patch panels PP2 and PP3 will be needed

Already (and even more importantly) during commissioning phase, and in shutdowns

TILECAL Cabinet(s) for storage, working place (table) Visual alarms (needed for Cs scans, part of DSS ?)

LAr Access to Front End Crates (FEC ) during standard (long) access times

Scaffolding must be provided Tools from LAr experts

Access to ELMBs on cryo platform area during short access


In US15 and USA15 areas ID

Again, not defined in detail. PC/network access (also in gas and cooling areas) Local DAQ running from terminals in rack areas

Throughout experiment lifetime, but especially during commissioning and initial running

ID has around 50 racks to commission and keep running – no small task Mobile phone coverage in all underground areas

TILECAL Electronics test items: scopes, … Space to work Monitors and keyboards in/near racks Cabinet(s) for storage (tools, cables, power supplies, ...), books, documentation etc.

LAr Nothing in US15 USA15:

Special permanent cupboards for LAr-specific equipment, documentation, tooling, ...

Carts to move heavy equipment


on surface, outside control room ID

Each ID system will need workplaces, preferably close together bearing in mind combined studies and need for communication between systems

2,3,4 workplaces with 2-4 PCs each – certainly more than one Three ID subdetectors, barrel + endcaps All will want to calibrate/test/develop at same time when there is no beam

TILECAL Electronics Room Mechanical workshop Analysis area with PCs (including Tilecal DCS Control Station), general workarea Meeting room(s) Cabinets for storage

LAr No LAr-specific work area is needed if all requested workstation places in the ATCR are provided

If not possible then equivalent LAr-specific places for workstations are needed A few small meeting/discussion rooms with whiteboards for detailed technical discussions

Control room functionality

General purpose equipment Coffee machine, fridge, small kitchen with water, Office supplies nearby, Telephone, fax

machine and paid telephones for outgoing calls nearby, Video conferencing nearby, Printers, Whiteboards, bottled water, vending machines (nearby), Coat rack

Permanent displays Audible effects: Warnings, end of run, … Magnet systems, general cooling and gas status Detector status and alarms, primary services (water, electricity, gas) Beam conditions and radiation levels LVL1, HLT, Data logging parameters Event Display Webcam/video of different parts of detector in UX cavern and of electronics in USA15 Note: will need to duplicate (some of) this information for local running

Need to be able to launch additional local displays if work is not being operated from control room

LAr Cryogenics status

Control room functionality

TILECAL workstations for

Single event display Histograms Analysis of data General purpose for shift crew (e.g. mail) Run Control HLT development

LAr One for each sub-detector for detailed data quality tests, event display, ability to drive local

pulsing/calibrations runs, ... EM barrel EM end-cap Hadronic end-cap Forward calorimeter

One for detailed FEC/BEC electronics monitoring One for each critical DCS system

LAr DCS SCS + temperature + purity displays LAr HV displays, monitoring and control LAr FEC LV and PS monitoring and control

Separated Functionality for Shift Work

Additional terminal room

Det. TerminalsVisitors

area

Control area for

shift

Meeting table with computer

screen

G. Mikenberg

Separate Control and Safety functions

• Safety functions should be in permanent display in a part of the control room and be constantly supervised.– The safety elements include power, magnet,

cryogenics, cooling, gas as well as gas and fire alarms.

• DAQ, detector power, histograms should be controllable and displayed at various terminals in the Control Rom.

G. Mikenberg

Experts on call should be able to perform work via the network.

• The best way to solve problems is to have it handle by an expert and not by the shift-crew.

• Once a problem is found, it should be reported to the expert on-call.

• The experts should be able to find the problem via the network.

• A secure access system with a firewall should be available for controlling sub-detectors from outside.

G. Mikenberg


HLT requirements

David R. Quarrie: Control Room Aspects of Tier-0 Operations

41Mini Workshop on ATLAS Control Room Design - 25 Feb 2004

Prompt Reco

Goal is to minimize latency between events being available at output of Event Filter to being available for Physics Analysis

Propose that prompt reconstruction be operated as an extension of the TDAQ/HLT system

Operators in the control room And office space nearby

Good communication with primary operators

Rapid feedback of problems In both directions

Hardware requirements in control room are not large ~2 workstations

Multiple slave displays

Subsystems Operation (USA15 or US15)

• Hardware monitoring and control• Readout/Write data from/to the front-end• Calculations (calibration, conversions, etc.)• Triggering of automatic actions (inc. feedback)• Archiving of raw data into the PVSS DB

Sub-detector Operation (USA15 or surface)

• Full operation of the detector• Summary status of the detector• Archiving of summary information • Coordination and synchronization of services or sections of the detector• Verification of commands• Logging of commands• Execution of automatic procedures or actions• Receive commands from DAQ• Export data to DAQ• Send messages to DAQ• Connect to services in the layer above

ATLAS

Tilecal

EB- B- B+ EB+

Cooling LV HV SAlone

MDTPixel …CIC

Global Operation

SCS

LCS

LHC DAQCERN

MagnetDSS

BE Hierarchy FunctionsBE Hierarchy Functions

Operator Tools (1b/7)Status display

Services (1/3)

• Detector Safety System– Highly reliable system for detector equipment safety– PLC-based FE independent from DCS– BE implemented with the same tools as the DCS– Graphical interface in the control room

• Underground access control– List of people in the different ATLAS zones– Retrieved from the grant access or the Find People in ATLAS

(FPiA) systems

F. Varela

• Web server– DCS on a private network– Publishes info which can be checked via the web– Allows for a limited set of actions

• Remote access to the DCS– Regulates the access to the DCS via remote login– Authentication will be provided– Access granted by the shift operator (?)– Session will be logged– Allowed actions to be decided

Services (2/3)

F. Varela

• Databases– Where will the Conf. and the Cond. DB server be?

• GSM– Alarms in the system will be reported to the expert via SMS

• Reporting system– Produces statistics of incidents happened for a time interval

• Miscellaneous– Web browser, – e-mail– etc.

Services (3/3)

F. Varela

Summarizing

Will we adopt CDF or D0 model? Many requests of space for tools, documentation in USA15 Requests for possibility of easy working in USA15

Chairs and small tables near racks Number of workstations going to infinite

Not everybody will work at the same time Functions will be configurable on workstations Remote monitoring in nearby barracks Otherwise… remember 40 consoles in 625 m2 …

Function of control room will evolve with time Commissioning Sub-detector debugging with first events “1st year of beam” ~ 3 years? Stable operations

Remote access for monitoring from the very beginning to help debugging Building the concept in the design to allow further utilizations

Other points will be summarized in the next two talks by Marzio and Ilias

summary of the mini workshop on atlas counting room (atcr)

Documents

day of overlap shift

monitoring ace

shift captain

schedule training

job training

monitoring stationrevamped

monitoring stationit

subdetector partitionsin