tdaq news and issues
DESCRIPTION
TDAQ news and issues. M. Sozzi NA62 TDAQ WG meeting CERN – 20/10/2010. Outline. TEL62 TDCB Clock L0 processor Interfacing to PCs Simulation. TEL62 - overview. Newer FPGAs and memories Doubled bus bandwidth with SL-FPGA - PowerPoint PPT PresentationTRANSCRIPT
TDAQ news and issues
M. Sozzi NA62 TDAQ WG meeting
CERN – 20/10/2010
1. TEL622. TDCB3. Clock4. L0 processor5. Interfacing to PCs6. Simulation
Outline
TEL62 - overview• Newer FPGAs and memories• Doubled bus bandwidth with SL-FPGA• Fewer I/O pins in new chosen PP-FPGA +
constraints for compatibility with LHCb cards =
After price confirmation for 500 FPGAs:increased size of PP-FPGA.
All 5 FPGAs on the TEL62 of the same type:EP3SL110F1152C4N (4x the original)
• Additional pins used for communication bus between PP-FPGAs (useful for LKr/L0)
• Internal note NA62-10-06 distributedComments from Lausanne group received
TELL1PP
PP
PP
PP
SL GbE
TEL62PP
PP
PP
PP
SL GbE
TEL62 - status• Some trouble with power distribution
Difficult to estimate power consumption before firmware is ready
• Design finished and start layout work at CERN at beginning of November
• Components for 2 prototypes being procured
• Production tests, firmware development, etc.?
TDCB – status and plans• Some delays in mounting
2 prototypes of V4 expected next week• Firmware debugging ongoing• Detailed documentation being written• Need to prepare test-bench system
• Preliminary tests in Pisa• Collect orders for first batch• First batch production (<16):
[these might be used in a synchronization run]• Tests and validation by sub-detector groups
TDCB – validation# boards
(first batch) Test where? Test who?
CEDAR
CHANTI
LAV
RICH Perugia M. Piccini
CHOD Perugia ? M. Piccini ?
STRAWS (?) 0
IRC/SAC
MUV
ComponentsNA62 TDAQ
Fallback
CEDAR GTK CHANTI LAV STRAWS RICH CHOD IRC/SAC LKR LKR/L0 MUV TDAQ TOTAL Spares STRAWS TOTAL SparesBirmingh
am Ferrara Napoli Pisa CERN Perugia Perugia ? Bulgaria CERN Roma2 Mainz Pisa CERN
Channels 128 200 2496x2 7360 2000 128 196 0 7360Channels/station 260x2 1840 1000 64 0 1840Particle rate 10 MHz 10 MHz 0Overall total rate 256 MHz 250 MHz 20 MHz 1 MHz 256 MHzFE-DAQ distance 5m <5m <6m <5m
TEL62 3 0 1 15 3 6 1 2 2 46 2 4 85 20 102Of which: spares 1 0 0 3 1 2 0 1 1 8 1 2 20 0.24 4 23 0.23Prototypes 0 0 0 0 0 0 0 0 0 1 0 1 2 0 2First batch 0 0 0 2 0 1 0 0 1 0 1 2 7 0 7Custom crates 1 0 1 0 1 1 1 1 2 0 1 1 10 0 9Of which: spares 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00 0 0 0.00Full TEL62 crates 0 0 0 14 0 0 0 0 0 3 0 0 17 5 22Of which: spares 0 0 0 1 0 0 0 0 0 1 0 0 2 0.12 1 3 0.14VME crates 0 0 0 0 0 0 0 0 0 1 0 0 1 0 1Of which: spares 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00 0 0 0.00TDC boards 10 0 4 50 0 20 2 4 0 0 4 10 104 80 184Of which: spares 2 0 0 7 0 4 1 2 0 0 0 6 22 0.21 16 38 0.21(first batch) 2 0 0 2 0 2 0 1 0 0 1 6 14 4 18LTU 1 3 1 2 2 1 1 1 2 3 1 2 20 2 20Of which: spares 0 0 0 0 0 0 0 0 0 0 1 1 0.05 1 2 0.10TTCex 1 3 1 2 2 1 1 1 2 4 1 2 21 2 21Of which: spares 0 0 0 0 0 0 0 0 0 0 1 1 0.05 1 2 0.10# of lasers/module 2 3 10 10 4 10 2 10 2 6 10 4 73 4 73# of lasers 2 9 10 20 8 10 2 10 4 24 10 8 117 8 117TTCoc 0 0 0 2 0 0 0 0 0 3 0 1 6 0 6Of which: spares 0 0 0 0 0 0 0 0 0 0 0 0 0.00 0 0 0.00Optical attenuators 3 1 1 2 20 6 1 2 2 2 2 4 46 20 46Of which: spares 0 0 0 0 0 0 0 0 0 0 2 2 0.04 0 2 0.04TTCrq 0 35 0 0 0 0 0 0 0 0 0 0 35 0 35Of which: spares 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Please check and update!
Clock - modules• 2 NA62 LTU prototypes available, ready for
testing at CERN (Birmingham)Then: full production (19+ modules), paid
• 19 new TTCex modules for NA62 in production• Need to finalize number of lasers: Birmingham
will distribute a final poll• Payment to NA62 account
• 5 old TTCex available in E-POOL
Clock - infrastructure• NA48 clock generator seems OK (spare?)• Organize central purchase of splitters,
attenuators, fibres (Birmingham)• Finalize “officially supported” VME CPU• Install central crate(s) + CPU• Sub-detectors: define TTC distribution network• Lay down fibres in ECN3
Burst sequence
• Start of burst signal (via TTC special command):reset timestamp counters
• Start of burst trigger (via TTC):can send data
• Normal triggers…send data
• End of burst signal (via TTC special command):store timestamp counter value
• End of burst trigger (via TTC):send end-of-burst data, stop data-taking
Sub-detectors only see 2 hardware signals: START OF BURST and END OF BURST
Calibration triggers• Sub-detector drives sub-detector calibration
system and sends signal pulse to L0TP• L0TP delivers (if possible, if allowed) calibration
trigger (after fixed, small delay)
• In-burst vs. Out-of-burst calibrations:L0TP knows when the beam really ends (EE), and can:- Deliver special control trigger (e.g. to instruct sub-detectors to change pulser parameters)- Modify acceptance rules for calibration triggers
This scheme avoids need for additional trigger and synchronization lines
Central (L0) Trigger System(logical)
CHODMUV
LKR/L0LAV
(RICH)
All s
ub-
dete
ctor
sPulser
L0 p
rim
itiv
es
(GbE
)Ch
oke/
Erro
r (L
VDS)
All s
ub-
dete
ctor
s
Cloc
k, L
0, B
urst
(T
TC)
Gen
SPS
Farm
Farm
Burst (GbE)
Dat
a (G
bE)
Burst
40MHzTime-
match L0
primitives,decide L0
and trigger type, check
latency, downscale, monitor
Monitor & record
choke/error,
send special triggers
Re-sync,
encode
Record L0
data
Generate special triggers
Gen
SPS
Farm
FarmBu
rst
(G
bE )
Dat
a (G
bE)
Burst
40MHzCHO
DMUV
LKR/L0LAV
(RICH)
All s
ub-
dete
ctor
s
Spare
L0 p
rim
itiv
es
(GbE
)Ch
oke/
Erro
r (L
VDS)
All s
ub-
dete
ctor
s
Cloc
k, L
0, B
urst
(T
TC)
Custom FPGA-basedboard
ORReal-time capable
PC
Re-
sync
hron
izer
PC (DIM)
PC interface
Central (L0) Trigger System(functional)
TTC
mod
ules
Pulser
Gen
SPS
Farm
FarmBu
rst
(G
bE )
Dat
a (G
bE)
Burst
40MHz
All s
ub-
dete
ctor
s
Cloc
k, L
0, B
urst
(T
TC)
Re-
sync
mem
ory
PC (DIM
)
GbE
RX
Corelogic
GbE
TX
Fan-out
L0
Fan-
out
CHODMUV
LKR/L0LAV
(RICH)
All s
ub-
dete
ctor
s
Spare
L0 p
rim
itiv
es
(GbE
)Ch
oke/
Erro
r (L
VDS)
Burs
t Fa
n-ou
t
PCIePC
Ie
Central (L0) Trigger System(implementation)
PCIe custo
m board
Custom
board or PC
LTU TTCex
Pulser
L0TP: Real-time tests (Ferrara)
PCI-express
FPGA
CPU
Involvement of very experienced group working on APE.Using a custom board with 2 Intel CPUs (standard Linux, so far) and FPGAs to test real-time response of a basic L0TP algorithm
L0TP: Real-time tests (Ferrara)
First naïve results: latency FPGA → CPU → MEM → FPGAwith 0 computation timeand standard Linux(depending on output buffering)
Now making more tests with more realistic time-matching algorithm
11μs
18μs
25μs
Further tests using standard PC and Pisa Altera Stratix IV PCIe development board (can such system be a L0TP?)Expect report at next meeting
L0 Trigger Processor
Rather limited progressDetailed descriptive note in preparationTemporary solution might be quite different from final system
Interfacing to PCsStarted development in Pisa of buffer manager code for readoutUser-space independent processesFixed-format shared memory
CTRL
PROD
PROD
PROD
PROD
CONS
CONS
CONS
CONS
GbE
GbE
GbE
GbEProcs
Sharedmemory
To be used for: L1 PCsMultiple processes either
running L1 (single sub-detector) algorithm or reading sub-detector data to L2 farm(on different event sets)
L1 algo
L1 algo
L1 algo
Readout
GbE L1 Trigger Processor PC
GbE L2 farm
Sub-detector
data
Sub-detector
data
L1primitives
Maybe also: software L0 ?Multiple GPUs running L0 primitive generation
(for some sub-detector).Algorithm time and latency not an issue.Raw data-transfer time not an issue.Control time?
From CPU thread without operating system?From hardware PCIe sequencer?
CondensedSub-
detector data
L0primitives
GPU
GPU
GPU
GPU
GbE L0 Trigger Processor
Online software / run control
• Existing solutions (DIM, SMI++) seem adequate• Possible help from CERN DCS group to port some of the above solutions to NA62• Need coordination and expertise within NA62• Write note describing NA62 requirements (volunteers willing to help?)
Towards a run controlCommon interface to run control:• Single machine per sub-system• Respond to the following commands:
- Enable/disable global control- Cold-start initialization- Start run <number> <global_conf> <local_conf>- End run- Query status- Reset
• No need of start/end burst (special triggers)
Simulation - Tools
Some work started (see talks).
- Do we have everything we need for trigger simulation in the official MC?Can it be ran in a fast enough mode?- Do we need a fast parameterized MC?
Simulation - Issues- Verification of L0 rates- How much below 1 MHz can we stay?- Effect of LAV?- Which control triggers can be allowed?- CHOD/RICH vs. MUV acceptance matching- Can CHOD be used in L0? Efficiency (accidentals)?- How much can RICH add? (At L1?)- How much can STRAWS contribute at L1?- What can be done realistically with LKr tiles at L0? Can something more be done with tiles at L1? Is some (summary) single-cell information useful at L1?- Most significant correlation cut at L2?Which sub-detectors are required at L2?With which resolution?
Synchronization run?TEL62 test and firmware• No help available within NA62
- Roma Tor Vergata cannot help in common TEL62 firmware development- Still missing sub-detector involvement in firmware
• Possibility of help from a PhD student in the STRAWS group
L0 Trigger Processor• Need commitments and tests