atlas hlt/daq stato e prospettive
DESCRIPTION
CSN1 Settembre 2005. ATLAS HLT/DAQ Stato e prospettive. Valerio Vercesi. Outline. Pre-series Status in USA15/SDX1 Commissioning and exploitation Large Scale Test Activities, experiences Lessons learnt Activities Monitoring, ROD Crate DAQ Algorithms development and deployment Finance - PowerPoint PPT PresentationTRANSCRIPT
ATLAS HLT/DAQ ATLAS HLT/DAQ Stato e prospettiveStato e prospettive
Valerio VercesiValerio Vercesi
CSN1 Settembre 2005CSN1 Settembre 2005
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 2
OutlineOutline
Pre-series Status in USA15/SDX1 Commissioning and exploitation
Large Scale Test Activities, experiences Lessons learnt
Activities Monitoring, ROD Crate DAQ Algorithms development and deployment
Finance Accounting 2006 requests
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 3
S. Falciano (Roma1) Coordinatore Commissioning HLT A. Negri (Pavia) Coordinatore Event Filter Dataflow A. Nisati (Roma1) TDAQ Institute Board chair e Coordinatore
Muon Slice PESA F. Parodi (Genova) Coordinatore b-tagging PESA V. Vercesi (Pavia) Deputy HLT leader e Coordinatore PESA
(Physics and Event Selection Architecture) Attività italiane
Trigger di Livello-1 muoni barrel (Napoli, Roma1, Roma2) Trigger di Livello-2 muoni (Pisa, Roma1) Trigger di Livello-2 pixel (Genova) Event Filter Dataflow (LNF, Pavia) Selection software steering (Genova) Event Filter Muoni (Lecce, Napoli, Pavia, Roma1) DAQ (LNF, Pavia, Roma1) Monitoring (Cosenza, Napoli, Pavia, Pisa) Pre-series commissioning (LNF, Pavia, Roma1)
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 4
ATLAS TDAQ system ATLAS TDAQ system
Muon
ROD ROD ROD
LVL1
LVL2
Event builder network
Storage: ~ 300 MB/s
ROBROB ROBROB ROBROB
Calo Inner
PipelineMemories
ReadoutDrivers
ReadoutBuffers~1600
High-Level Trigger
LEVEL-1 TRIGGER• Hardware-Based• Coarse granularity from calorimeter & muon systems
LEVEL-2 TRIGGER• Regions-of-Interest “seeds”• Full granularity for all subdetector systems
• Fast Rejection “steering”
EVENT FILTER • Possibly “seeded” by Level 2 • Full event access• Algorithms inherited by offline
RoI
EF farm~1000 CPUs
1 selected event
every millionTDAQ ≅Rates
40 MHz
~75 kHz
~2 kHz
~200 Hz
~2 ms
~10 ms
~ 1 s
Latency
EF
LVL2farm
( )
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 5
TDAQ TDAQ
Trigger e Data Acquisition hanno da sempre in fase di commissioning un doppio ruolo Come “server” per il commissioning dei rivelatori Come “client” per utilizzare le informazioni realistiche
dell’esperimento per i propri studi di funzionalità e performance
La situazione si è già presentata durante il Combined Testbeam 2004
Il TDAQ di ATLAS è un progetto in piena evoluzione in cui development/commissioning/exploitation sono ancora fasi molto miscelate Presentazione di risultati e indicazione delle prospettive Maggiore enfasi alle componenti con forte partecipazione
italiana Descrizione del piano di commissioning generale
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 6
Pre-series designPre-series design“Module-0” of final system
8 racks (~10% of final dataflow)
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 7
Pre-series realityPre-series reality
LVL2 rack
EF rackSwitch rackOnline rack
6 racks SDX1
ROS rack
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 8
Commissioning and exploitationCommissioning and exploitation
Fully functional, small scale, version of the complete HLT/DAQ Equivalent to a detector’s ‘module 0’
Purpose and scope of the pre-series system Pre-commissioning phase
To validate the complete, integrated, HLT/DAQ functionality To validate the infrastructure, needed by HLT/DAQ, at point-1
Commissioning phase To validate a component (e.g. a ROS) or a deliverable (e.g. a Level-2
rack) prior to its installation and commissioning TDAQ post-commissioning development system
Validate new components (e.g. their functionality when integrated into a fully functional system)
Validate new software elements or software releases before moving them to the experiment
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 9
Pre-Series CommissioningPre-Series Commissioning
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 10
Commissioning LVL2+ROSCommissioning LVL2+ROS
First measurements with full LVL2 rack feeded by ROS data
Using separate Control and Data networks
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 11
Commissioning EFCommissioning EF
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 12
Pre-series Status in USA15/SDX1 Commissioning and exploitation
Large Scale Test Activities, experiences Lessons learnt
Activities Monitoring, ROD Crate DAQ Algorithms development and deployment
Finance Accounting 2006 requests
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 13
Large Scale TestsLarge Scale Tests Pre-serie work will help understanding the TDAQ system in terms
of functionality Forms the basis for future deployments/exploitations
Complexity of ATLAS TDAQ system arises also from the size of bulk components involved Topology of communications, size of LVL2/EF farms, software, …
Test scalability of HLT system using presently available large installations Understand issues like configuration, startup time, communication, control,
error reporting, … UCB/TRIUMF WestGrid Cluster (http://www.westgrid.ca)
60 racks x 14 nodes = 840 Dual-CPU nodes(3 GHz CPUs / 2-4 GB RAM)
CERN LXSHARE Cluster (http://batch.web.cern.ch/batch) Up to ~700 nodes (various flavours)
Reference page for all tests http://atlas-tdaq-large-scale-tests.web.cern.ch
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 15
State transitionsState transitions
Luke Warm Start Luke Warm Stop
USR_RUNNING_TIME(default is 30 s)
RUNNING
CONFIGURED
INITIAL
ABSENT
Configure
Boot
Unconfigure
Shutdown
Setup Close
Cold Start Cold Stop
configure: load configure
start: prepareForRun startTrigger
stop: stopTrigger stopFrontEnd stopDataCollection stopEventFilter stopRecording
unconfigure: unconfigure unload
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 16
LVL2 transition times LVL2 transition times
State transistion timing quite acceptable No significant differences between 2 and 3 tier Run Control
2 T ier Run Control
0
10
20
30
40
50
60
70
80
0 5 10 15 20 25
Configuration Id (# L2PU nodes 8 -> 256)
Tim
e (
se
cs
)
setup
boot
conf
start
wait
stop
unconf
shutdown
close
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 17
EF results @ LSTEF results @ LST
EF TrigMoore with Oracle/MySQL
0
100
200
300
400
500
600
700
800
0 100 200 300 400 500 600 700
Number of processes (1EFD+2PTs)
con
fig
. /
tota
l ti
mes
, s config Oracle
config MySQL
Total test Oracle
Total test MySQL
Timings HelloWorld / TrigMoore
0
50
100
150
200
250
0 100 200 300 400 500 600 700
Number of processes (1EFD+2PTs)
tim
e,
s
config HelloWorld
config TrigMoore MySQL
backend setup HelloWorld
backend setup TrigMoore MySQL
Effect of realistic algorithm:TrigMoore vs HelloWorld EF standalone 1EFD+2PTs / node up to 200 nodes MySQL as geometry DB used significant slow down due to
access/reading geom. DB MySQL vs Oracle DB in TrigMoore
EF standalone, 1EFD+2PTs / node Oracle DB – up to 160 nodes MySQL DB – up to 200 hosts MySQL works faster at “small
scales”, while Oracle looks better at higher scales - to be investigated more
not able to on higher than 200 nodes with any of both partitions – to be investigate further (do we need to replicated DBs ?)
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 18
Pre-series Status in USA15/SDX1 Commissioning and exploitation
Large Scale Test Activities, experiences Lessons learnt
Activities Monitoring, ROD Crate DAQ Algorithms development and deployment
Finance Accounting 2006 requests
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 19
Cosmics Tile setupCosmics Tile setup
MobiDAQ (Mobile DAQ):
read out of 8 drawers in
the pit with temporary
RODemu but real TDAQ
(tdaq-01-02-00), tests of
electronics, cosmic
muons runs
CSN1 Settembre 2005 V. Vercesi - INFN Pavia
GNAM Monitoring chainGNAM Monitoring chain
Framework per monitoring on-line a basso livello Core: trasporto di eventi, istogrammi e comandi Plugin dinamici: decodifica e istogrammazione Possibilita’ di correlazione fra diversi rivelatori
Comandi asincroni (reset, rebin, update)
Status Incluso in TDAQ da aprile Utilizza i servizi disponibili
per il monitoring Validato al CTB04; in uso in
alcuni siti di commissioning
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 21
GNAM al commissioningGNAM al commissioning Acquisizione dati nell’ambito del software TDAQ Commissioning di MDT usa GNAM per monitoring online ed analisi dati Sampling completo a livello di ROS [~2 KHz per noise test, ~200 Hz per pulser] Integrazione di librerie: completato per MDT, in via di sviluppo per RPC Stato del monitoring MDT:
Richiesta minimale di informazioni allo shifter (nomi camere) Output: file di istogrammi e file di testo con risultati di analisi dati per ciascuna camera e per
ciascun run In sviluppo:
ottimizzazione dell’analisi dati on-line installazione del presenter per la visualizzazione degli istogrammi on-line event display
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 22
PMP PresenterPMP Presenter
Visualizzazione asincrona on-line di istogrammi Interattivo (reset, rebin, zoom, fitting, ecc...) Operazioni grafiche su istogrammi (ROOT canvas) Grafica configurabile
Status Incluso in TDAQ da settembre Utilizza i servizi disponibili
per il monitoring Pienamente funzionale
al CTB04 Riprogettato per nuove
funzionalità e maggiore scalabilità
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 23
GNAM&PMP: sviluppi futuriGNAM&PMP: sviluppi futuri
GNAM Completare l’integrazione nel framework del TDAQ
Configurazione di Core e librerie dal database del TDAQ Messaggistica e gestione degli errori software
Supporto per la generazione di allarmi automatici Livelli di severità, routing
Verifica delle prestazioni e delle risorse necessarie CPU, Memoria, Banda
Studio della scalabilità
PMP Completare la nuova versione
Minimizzazione del traffico di rete Adattamento della IGUI alla nuova struttura
Generazione allarmi Plugin di analisi degli istogrammi
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 24
ROD Crate DAQROD Crate DAQ RCD is used as interface with the RODs
for Control Configuration Monitoring
Statistics Event sampling
Data readout (through VME) User guide for detectors developers
available Validation system in Bld. 40 DAQ Commissioning – Phase 1:
The ROD Emulator system will be used in order to validate all common RCD software and infrastructure
After adding and validating the detector sw and hw specific items, multi crate event building will be used in the absence of the full DAQ chain
VMEbus memory + CORBO
=
Memory +
Registers +
Interrupt capability
RCC
MEMORY
CORBO
Config & Control
Data readout
REB
Event Fragments
ROD Emulator
ROS
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 25
RCD exploitationRCD exploitation
Experience from Combined Testbeam extremely useful Recall almost all detector used it in the CTB Successful workshop to put forward new requirements
As a consequence, several improvements during last months Configurable interrupt handling Simplified user interface to access ordered event fragments Data driven event building for multicrate acquisition in the commissioning
phase Simplified ROD emulation Hardware trigger distribution
All sub-detector commissioning (but LAr…) sites use RCD MDT and RPC on the forefront
BB5 integration, Point 1 with MROD, Lab testing with ROD emulators
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 26
Algorithms: Muon sliceAlgorithms: Muon slice LVL1 simulation is the fundamental input for the measurement of
the full muon vertical slice performance LVL2 and EF Muon algorithms have been extensively tested on
data simulated in ATLAS Rome Physics Workshop: June 2005
LVL2: Fast Confirm the LVL1 trigger with a more precise PT estimation within a
Region of Interest (RoI) Global pattern recognition, track fit, fast PT estimate via Look Up Table
(LUT) with no use of time consuming fit methods Event Filter: TrigMoore
Based on offline reconstruction algorithm Moore Can run seeded (reconstruction starting from RoI of previous levels) Precise PT determination
General goal is now to achieve more realistic estimate of trigger selections and corresponding rates Real geometry, configuration and conditions database usage, …
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 27
LVL1 Coincidence WindowsLVL1 Coincidence Windows
Athena release 10.0.4 Low pT 6 GeV Threshold
Low-pT Inefficiency mapEfficiency curve
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 29
MuFast endcap extensionMuFast endcap extension Early stages of project Endcap differs from Barrel
M and O station are outside B field Inhomogeneous B field – bending is local
Algorithm Pattern recognition and fit in TGC → position and slope in EM Extrapolate segment into MDT EM/EO → Roads in EM/EO, find hits, fit
Next pattern recognition and fit in MDT as in mFast – not done yet Extrapolation into EI and LUT
-1.40E-05
-1.20E-05
-1.00E-05
-8.00E-06
-6.00E-06
-4.00E-06
-2.00E-06
0.00E+00
0 10 20 30 40
PT
Slo
pe
Forward Endcap
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 30
MuFast: MDT miscalibrationMuFast: MDT miscalibration
Commissioning the algorithms: realistic approach to data handling The plot shows the muFast resolution for two different scenarios:
the correct MDT r-t function is used, red points a systematic shift of + 0.2 mm is added to the radius returned by the correct
r-t function, blue points
+10% degradation @ 50 GeV, to be compared with a + 5% expected by a naive calculation
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 32
pT resolution (Moore)
The relative degradation in (pT) is +5% for muons with a 6 GeV/c transverse momentum, increases to +13% around 50 GeV/c
This MDT miscalibration leaves almost unaffected resolutions.
TrigMoore: MDT miscalibrationTrigMoore: MDT miscalibration
Single muons (with pT = 6, 10, 20, 40, 100 GeV/c, produced for the Rome Initial Layout) have been reconstructed in two different scenarios Using the correct MDT r-t relation function (red squares in pictures) Applying a systematic +0.2 mm shift on the radius obtained with the correct MDT
r-t function (blue circles in pictures)
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 33
LVL2 tracking: SiTrackLVL2 tracking: SiTrack
Preliminary results obtained on DC1 b-jet samples at initial luminosity
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 34
B-tagging @ LVL2B-tagging @ LVL2 Results obtained with the “standard” SiTrack algorithm on DC1 data b-tagging: likelihood ratio using transverse and longitudinal impact parameters Upgraded version to be tested soon: should improve both efficiency and track
parameters resolution and hence significantly improve the b-tagging performance
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 35
Algorithms steeringAlgorithms steering
Cluster60
eg25
eg15
eg60
iSe60
iSg60
iSg25
l2g60i
l2g25i
eg60Hy
eg25Hy
eg15Hy
T2Calo
iSg15
iSe25
iSe15
EMtrackSoft
l2e25i
l2e15i
gIsol60
gIsol25
eIsol15
eIsol25
g60Hy
g25Hy
g15Hy
e60Hy
e25Hy
e15Hy
EMtrackHard
TrackSoft15EM
l2e60ieIsol60
eg20
iSg20 l2g20i
eg20Hy
iSe20 l2e20i
gIsol20
eIsol20
g20Hy
e20Hy
EM60Hy
EM25Hy
EM15Hy
EM20Hy
EM25
EM15
EM60
EM20
l2e15ieIsol15
TrackSoft02EM
EMROI
Cluster15
Cluster25
Cluster20
TrackSoft25EM
TrackHardEM
T2Calo
T2Calo
T2Calo
EMtrackSoft
EMtrackSoft
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 36
ATLAS Commissioning PhasesATLAS Commissioning Phases
Commissioning means bringing ATLAS systems from “just installed” to “operational”. It is broken in 4 phases Subsystem standalone commissioning Integrate subsystems into full detector Cosmic rays, recording data, analyze/understand, distribute to
remote sites Single beam, first collisions, increasing rates, etc…
A consistent part of commissioning activities will be done during the installation itself
Phases will overlap since different systems may be at different development levels For the barrel calorimeter commissioning will start soon Tile calorimeter is already taking data
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 37
HLT CommissioningHLT Commissioning Commissioning is a set of activities which spans the time interval from the
installation of the HLT racks and nodes … A rack is the elementary unit for commissioning OS, Dataflow and Online software are installed
... to the phase when the HLT is filtering physics data and recording them HLT selection algorithms are installed and running stably The complete trigger menu (at least for early physics) is configured The trigger selection efficiencies and background rejection rates are understood and
can serve as input for physics measurements Phase-1 Commissioning definition is the most urgent
Heavily use the Pre-series to exercise the procedures for installation and commissioning
Important steps will cover the integration of detectors into full system Involve operations that have a very strong coupling with the offline commissioning
activities Development of specific algorithms looking at simple data decoding (cabling,…)
Final commissioning phases extend far beyond the data-taking startup (interface with run coordinator team)
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 38
Cosmic muons in ATLASCosmic muons in ATLAS
Rock ~ Silicon
600m x 600m x 200m deep
(2.33 g/cm3)
AirConcrete
Surface building
PX14/16 shielding
(2.5 g/cm3)
PX14
(18.0 m Inner Ø)
PX16
(12.6 m Inner Ø )
ATLAS
Geant Simulation Initial detector
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 39
Overall planOverall plan
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 40
OutlookOutlook A lot of work during this year, system entering phase of complete
deployment Purchase plan proceeding as scheduled, with some minor delays (Wo)manpower situation not always healthy More help and support welcome
Three big tasks awaiting us in the next months Commissioning the pre-series and extract a coherent and complete set of
system performance measurements Based on previous experiences and on already established partial results
On-line trigger selections evaluation (rates, efficiencies, physics coverage,…) evolving towards more realistic approach
Calibration, geometry “as installed”, mis-alignment, error handling, complete trigger menus, physics analysis based on trigger objects
Prepare for cosmic run next year “Cosmic” slices and trigger menu (Tile, LVL1, “digital” LVL2)
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 41
Pre-series Status in USA15/SDX1 Commissioning and exploitation
Large Scale Test Activities, experiences Lessons learnt
Activities Monitoring, ROD Crate DAQ Algorithms development and deployment
Finance Accounting 2006 requests
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 42
AccountingAccounting Contributo INFN alla Pre-serie
Read-Out System: 51 kCHF (ROS Racks) Online Computing System: 40 kCHF (Monitoring, Operations) Online Network System: 44 kCHF (Switches, FileServer)
Inviati al CERN a Dicembre 2004 VV riceve in copia tutte le fatturazioni dei singoli acquisti ed un sommario
mensile dello stato finanziario Contributo CORE 2005
Online Computing System: 45 kCHF (Monitoring, Operations) Inviati al CERN a Maggio 2005 Già acquisiti due file server
Read-Out System: 275 kCHF (ROS Racks) Questo acquisto si espleta secondo una gara e non con un semplice market
survey o price inquiry come fino ad ora avvenuto Richiesta alla Giunta l’autorizzazione per partecipare alla gara
Grazie a Speranza che si è prodigata per espletare le pratiche necessarie Il CERN preferisce gestire la gara su un periodo di due anni
Omogeneità dei componenti vs miglioramento delle prestazioni Si sommano i 275 kCHF del 2006
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 43
Cost Profile (kCHF)Cost Profile (kCHF)
2004 2005 2006 2007 2008 2009 Total
Pre-series 140 0 0 0 0 0 140
Detector R/O 0 275 275 0 0 0 550
LVL2 Proc 0 0 65 195 230 160 650
Event Builder 0 0 50 50 110 70 280
Event Filter 0 0 170 180 570 380 1300
Online 0 45 135 0 0 0 180
Infrastructure 0 0 80 80 20 20 200
INFN Total 140 320 775 505 930 630 3300
TDR Total 1048 3357 4087 4544 7522 4543 25101
INFN Percentage(%) 13.4 9.5 19.0 11.1 12.4 13.9 13.1
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 44
Missioni EstereMissioni Estere
LNF Commissioning HLT/DAQ Pre-serie e pit: 6 m.u.
Ferrer, Kordas (+ Miscetti, Giovannella)
Pavia VV coordinatore PESA e duputy HLT: 1 m.u. Negri A. responsabile Event Filter: 1 m.u. Scannicchio D. commissioning HLT: 2 m.u.
Roma1 Speranza responsabile commissioning HLT: 2 m.u. Leandro chair IB, coordinatore slice mu: 1 m.u. ROD crate DAQ e HLT/DAQ muoni : 4 m.u.
Pasqualucci, Di Mattia, …
CSN1 Settembre 2005 V. Vercesi - INFN Pavia 45
MilestonesMilestones 30/06/2005
TDAQ - Installazione, test e uso della "Pre-serie" (~ 10% TDAQ slice)
“ragionevolmente” raggiunta: ritardi accumulati soprattutto sugli acquisti delle componenti
24/12/2005 TDAQ - Installazione e test dei ROS di Pixel, LAr, Tile, Muon
(interfacciamento al ROD Crate e integrazione nel DAQ) Parte del piano di commissioning in esecuzione: piccola dipendenza dalla data
di consegna dei ROS 30/04/2006
Completamento dei test sulla pre-serie e definizione delle funzionalità per il supporto al commissioning TDAQ
31/08/2006 Commissioning delle slice di ROS dei rivelatori utilizzando le funzionalità
della pre-serie (modulo-0 del sistema finale) 31/12/2006
Presa dati integrata dei rivelatori nel pozzo con raggi cosmici