![Page 1: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/1.jpg)
Online monitor for L2 CAL upgrade
Giorgio Cortiana
Outline: Hardware MonitoringNew Clusters Monitoring
http://www.pd.infn.it/~cortiana/TrigMonOut/TrigMonOut.html
![Page 2: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/2.jpg)
Hardware Monitor results:All mezzanine firmware has been recently updated. As a results, the error rate has been further reduced.
Error rate is reduced of more than a factor 30!
For each run a global pulsar canvas informs about the system status.
Pulsar 00Old
mezzanine firmware
New mezzanine firmware
Error rate 9x10-5 <2 x 10-6
How the monitor looks like:
The code runs online almost every day, and stays inside the PulsarMon package
![Page 3: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/3.jpg)
From this information we can easilyfind out the location and the kind of error from log files:
In this case the error is limited to Pulsar 12 and due to ADMEM-DCASmismatches.
![Page 4: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/4.jpg)
Follows a summary report of the online monitor during last week-end. All this is after the mezzanine firmware upgrade:
Analyzed runs: 243711, 243712, 243713, 243715, 243730, 243732
Hardware Monitor: results from online running over the week end:
Plots and tables will refer to all this runs…
![Page 5: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/5.jpg)
![Page 6: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/6.jpg)
Run 243730: Event 19,751,876phi, eta = (20,20), (20,22), (20,23)
ADMEM-DCAS mismatch
![Page 7: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/7.jpg)
Analyzed runs: 243711, 243712, 243713, 243715, 243730, 243732
DCAS connectortower phi,eta = 4,0
Hardware Monitor: results from online running over the week end:
![Page 8: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/8.jpg)
Analyzed runs: 243711, 243712, 243713, 243715, 243730, 243732
DCAS connectortower phi,eta = 4,0
Pulsar
DCAS
ADMEM
X
ADMEM-Pulsars OKADMEM-DCAS BAD
Hardware Monitor: results from online running over the week end:
Problem solved byChanging the connection cable
![Page 9: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/9.jpg)
Analyzed runs: 243711, 243712, 243713, 243715, 243730, 243732
DCAS connectorstower phi,eta = 4,0
ADMEM-DCAS mis-matches
Hardware Monitor: results from online running over the week end:
![Page 10: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/10.jpg)
Analyzed runs: 243711, 243712, 243713, 243715, 243730, 243732
DCAS connectorstower phi,eta = 4,0
ADMEM-DCAS mis-matches
Pulsar
DCAS
ADMEMX
ADMEM-Pulsars BADADMEM-DCAS BAD
Hardware Monitor: results from online running over the week end:
![Page 11: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/11.jpg)
Analyzed runs: 243711, 243712, 243713, 243715, 243730, 243732
DCAS connectorstower phi,eta = 4,0
Control DAQ buffer problemFrom run 243715, 243730, 243732
Replaced problematic board.
ADMEM-DCAS mis-matches
Hardware Monitor: results from online running over the week end:
![Page 12: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/12.jpg)
Hardware Monitor: current results
All hardware problems foundduring the week end are nowsolved as demonstrated by thepreliminary online monitor resultson the current run 243787
![Page 13: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/13.jpg)
Clustering algorithm monitor:
We have all the tools to do a massive debugging of the new clustering algorithm.
Follows some plots produced on runs 243715 (which has a large number of ADMEM-DCAS mismatches), 243787 (from this night)
for these runs the TL2D bank contains new clusters information.
All the code is ready for online running, and stays inside the L2GlobalTrigger package.
What we do is essentially to create a simTL2D bank based on DCAS bank and to run the “same” algorithm as the L2CPU.
Up to now, we have a qualitative agreement between simulation and data. We are investigating the sources of the discrepancies.
![Page 14: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/14.jpg)
Clustering algorithm monitor: Pass2run 243715:
Sim N of pass2 clusters
Rea
l N p
ass2
Rea
l met
X
Rea
l met
Y
Sim met X Sim met Y
![Page 15: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/15.jpg)
Clustering algorithm monitor: Pass2run 243715:
Sim N of pass2 clusters
Rea
l N p
ass2
Rea
l met
X
Rea
l met
Y
Sim met X Sim met Y
The large discrepancies onNpass2, metx and mety are related to ADMEM-DCASmismatches.
![Page 16: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/16.jpg)
Clustering algorithm monitor: Pass2run 243787:
Sim N of pass2 clusters
Rea
l N p
ass2
Rea
l met
X
Rea
l met
Y
Sim met X Sim met Y
![Page 17: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/17.jpg)
Clustering algorithm monitor: Pass2run 243787:
Sim N of pass2 clusters
Rea
l N p
ass2
Rea
l met
X
Rea
l met
Y
Sim met X Sim met Y
In this case, there are no ADMEM-DCAS mismatches:There is a qualitativematch in the Npass2, but a perfect agreement in metx and mety
![Page 18: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/18.jpg)
Clustering algorithm monitor: Pass2run 243787:
Sim N of pass2 clusters
Rea
l N p
ass2
Rea
l met
X
Rea
l met
Y
Sim met X Sim met Y
In this case, there are no ADMEM-DCAS mismatches:There is a qualitativematch in the Npass2, but a perfect agreement in metx and mety
The cluster monitor can track upstream problemsand returns consistent results with respect to the hardware monitor
![Page 19: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/19.jpg)
Clustering algorithm monitor: Pass2run 243787
Rea
l Em
Rea
l Had
Sim Em Sim Had Sim SeedEta
Rea
l See
dEta
Sim SeedPhi
Rea
l See
dPhi
Rea
l NTo
wer
s
Sim NTowers
Pass2 cluster variables:There is a qualitativematch between data and simulation.disagreements probably due to outdated LUTs in the simulation.… investigating.
![Page 20: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/20.jpg)
We have all the tools to validate the complete system.
The hardware is really stable and with the new mezzanine firmware the error rate is of O(10-6)
We have all the methods to debug the information provided by the new clustering algorithm, based on the comparison between real data and the simulation. We are debugging the simulation code, and we expect to finalize it in few days or so.
We are ready to start the integration of our monitoring with the default TrigMon in control room. We will start next week.
Conclusions and plans
![Page 21: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/21.jpg)
- Backup -
![Page 22: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/22.jpg)
Inputs for the simulation code
We use ADMEM (calorimeter banks) information to compute trigger Tower information.
This info are then used to create a simulatedPulsar bank (TP2D) to be used to check the data:
![Page 23: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/23.jpg)
Hardware setup
M5 used for timing studies.It receive the same data
sent to L2CPU
Full path up to the CPUis monitored!
![Page 24: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/24.jpg)
M0
Mezzanine M1
Mezzanine M2
Mezzanine M3
cable 0cable 1
cable 2cable 3
DA
TAIO
1D
ATA
IO 2
CO
NTR
OL
Pulsar Board 0DAQ buffer 1DAQ buffer 2
L2 C
AL
syst
em o
verv
iew
For each LVDS pulsar,All DAQ buffers are monitored
Comparing real data to simulation
![Page 25: Online monitor for L2 CAL upgrade Giorgio Cortiana Outline: Hardware Monitoring New Clusters Monitoring](https://reader035.vdocument.in/reader035/viewer/2022081513/5a4d1adb7f8b9ab059974ba4/html5/thumbnails/25.jpg)
Clustering algorithm monitor: Pass4
N pass4 clusters
Em pass4 Iso pass4
Phi pass4 Eta pass4 NTowers pass4