lhcb daq review, september 11-12 lhcb timing and fast control system tfc team: arek chlopik, warsaw...
TRANSCRIPT
LHCb DAQ Review, September 11-12
LHCb Timing and Fast
Control System
TFC Team:Arek Chlopik, WarsawZbigniew Guzik, WarsawRichard Jacobsson, CERNBeat Jost, CERN
• Introduction to the TFC system
• Progress and status
2LHCb DAQ Review, September 11-12
LHCb Read-out
Read-out Network (RN)
RU RU RU
6-15 GB/s
6-15 GB/s
50 MB/sVariable latency
L2 ~10 msL3 ~200 ms
Control &
Monitoring
LA
N
Read-out units (RU)
Timing&
FastControl
Level-0
Front-End Electronics
Level-1
VDET TRACK ECAL HCAL MUON RICH
LHC-B Detector
L0
L1
Level 0Trigger
Level 1Trigger
40 MHz
1 MHz
40 kHz
Fixed latency 4.0 s
Variable latency <1 ms
Datarates
40 TB/s
1 TB/s
1 MHzFront End Links
Trigger Level 2 & 3Event Filter
SFC SFC
CPU
CPU
CPU
CPU
Sub-Farm Controllers (SFC)
Storage
Th
rott
le
Front-End Multiplexers (FEM)
Unique feature : Two levels of high-rate triggers• Level-0 (40 MHz --> 1.1 MHz Accept rate)
• Level-1 (1.1 MHz --> 40-100 kHz Accept rate)
3LHCb DAQ Review, September 11-12
Timing and Fast Control
TTCrx TTCrx
TTCrx
L1E
FEchipFEchipL1 buffer
ADCADCADC
ADCADCDSP
Con
trol
FEchip
TTCrx
L0E
FEchipFEchipFEchip
TTCrx
L0E
FEchipFEchipFEchip
TTCrx
L1E
FEchipFEchipL1 buffer
ADCADCADC
ADCADCDSP
Con
trol
Clock fanoutBC and BCR
LHC clock
L0 trigger L1 trigger
Readout Supervisor Readout
Supervisor
Local trigger (optional)
L0
L1
Readout Supervisor
L0
L1
TFC switch L1 Throttle switch
L0 Throttle switch
SD1 TTCtx SD2 TTCtx SDn TTCtx L0 TTCtx L1 TTCtx
Optical couplers Optical couplers Optical couplers Optical couplers
TTCrx
L1E
FEchipFEchipL1 buffer
ADCADCADC
ADCADCDSP
Con
trol
FEchip
TTCrx
L0E
FEchipFEchipFEchip
TTCrx
L0E
FEchipFEchipFEchip
TTCrx
L1E
FEchipFEchipL1 buffer
ADCADCADC
ADCADCDSP
Con
trol
DAQ DAQ
Th
rott
le O
R
Th
rott
le O
R
TTC system
1717
L1 trigger system
GPS receiver
Consists of:
- RD12 TTC distribution system:
•TTCtx’s
•Tree-couplers
•TTCrx’s
- Components specific to LHCb:
•Readout Supervisors•TFC Switch•Throttle Switches•Throttle ORs
4LHCb DAQ Review, September 11-12
Use of TTC Timing, Trigger and Control distributed using the TTC system:
Channel A used to distribute (accept/reject signal)
• L0 trigger (40 MHz --> 1.1 MHz accept rate)
Channel B used to distribute (short broadcasts):
• L1 trigger (1.1 MHz --> 40-100 kHz accept rate)
• Bunch/Event Counter Resets
• Control commands (FE resets, calibration pulses)
Broadcast order is handled according to a priority scheme
Usage of the 6 (+2) user bits in the short broadcasts:
Bit/Broadcast 7 6 5 4 3 2 1 0
L1 Trigger 1 Trigger type EI D(2-bit) 0 0
Reset 0 1 R L1 EI D L1 FE L0 FE ECR BCR
Calibration 0 0 0 1 Pulse type 0 0
Command 0 0 R R R R 0 0
5LHCb DAQ Review, September 11-12
LHCb specific components Readout Supervisor “Odin”
all mastership in single module
TFC switch Clock, trigger and command distribution and support partitioning
Throttle switches (L0 & L1) and Throttle ORs Throttle feed-back
6LHCb DAQ Review, September 11-12
Readout Supervisor “Odin”
- Single module
- Clock distribution
LHC clock
- L0 handling & distribution
L0
- Auto-trigger generator
Trigger generator
- Trigger controller
Trigger controller
Throttles
- Reset/cmd generator
Reset/commandgenerator
- “RS Front-End”RS Front-End
L0/L1
DAQ- ECS interface
ECS interface
ECS
TTC encoder
Ch A/B
- TTC encoding
- L1 handling & distribution
L1
- L1 derandomizationL1
Derandomizer
Designed with emphasis on:• Versatility - to support many different types of running modes
• Functionality easily added and modified.
8LHCb DAQ Review, September 11-12
LHCb partitioning Partition (TFC) Def. Generic term for a configurable
ensemble of parts of a sub-detector, an entire sub-detector or a combination of sub-detectors that can be run in parallel, independently and with a different timing, trigger and control configuration than any other partition.
Option: 16 or 32 concurrent partitions
Crucial: Equal internal propagation delays. If skew too large, FE will suffer from timing alignment problems when using different RS’.
INPUT CHANNELS
OUTPUT CHANNELS
Control Interface
Pool of Readout Supervisors
Partition A Partition B
Front-Ends grouped by TTCtx/Optical couplers to partition elements
TFC Switch
TTC
info
rmati
on e
nco
ded
ele
ctri
cal
9LHCb DAQ Review, September 11-12
Buffer overflows Throttle signals fed back electrically to the RS in control of the partition
with data congestion
Two Throttle Switches: Throttle signals to the L0 trigger Throttle signals to the L1 trigger
All Throttle Switches and ORs will log throttle history
INPUT CHANNELS
OUTPUT CHANNELS
OR OR
Control Interface
Pool of Readout Supervisors
Partition A Partition BThrottle Switch
Front-Ends etc grouped by Throttle ORs i.e. ~Throttle Switches
Thro
ttle
sig
nals
10LHCb DAQ Review, September 11-12
Progress and Status In view of the TDR, the aims of this year are to:
Design the TFC components specific to LHCb Review the TFC architecture and components Layout the first prototype of the TFC Switch and the RS Simulate the RS at several levels Test the way the TFC architecture exploits the TTC system Produce a first prototype of the TFC Switch and the RS Test critical points of the TFC Switch and the RS
Overview of work schedule for 2001:
Except for delays in the area of testing TTC system, schedule well maintained.
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
Testing TTC system
Testing TFC switch and TTC system
Testing RS
11LHCb DAQ Review, September 11-12
TFC Switch (Progress and Status)
Reviewed November 8, 2000 (together with TFC architecture and Throttle Switches)
very well received
First prototype was ready in April-May. Main aim with prototype was to measure the two crucial quantities:
Time skew between paths (aimed at <100 ps). Jitter (aimed at 50 ps)
12LHCb DAQ Review, September 11-12
TFC Switch (Progress and Status)
All measurements carried out successfully Jitter at the output ~80 - 100 ps. Jitter from generator ~50 ps Maximum skew between all inputs to each multiplexer is between 100 - 400 ps Skews between between output paths (multiplexers to output) was very large (-
> 4ns) A few mistakes were discovered in the routing when equalizing the paths.
• The mistakes + dielectric characteristics can account for the skews measured on the input and the ouput paths.
The intrinsic propagation delay of the multiplexers vary between 400 - 1000 ps. Specs claim maximum 850 ps.
Comparing line lengths with measured propagation delays shows that the signal speedis ~40% slower than the “ideal” 5ns/m. This is consistent in all measurements.
The measurements show that the performance with respect to skew is not satisfactory. Solution:
Route all lines on board layers with equal dielectric characteristics Add appropriate delay lines at the outputs to compensate for the inevitable
intrinsic skew due to the components.• Problem with temperature dependence of delay chips• Each board needs calibration.
Input and output coupling capacitors with less tolerance to improve signal shape.
Switch has still to be tested with CC-PC and in full TTC setup. The first prototype will be sufficient for tests of the first prototype of the RS
13LHCb DAQ Review, September 11-12
RS “Odin” (Progress and Status)
Specifications ready end of last year - almost entirely based on FPGA Specs, logical design and first draft of schematics reviewed April 4, 2001.
Very well received Importance of simulation emphasized.
Specs have been simulated in a high level behavioral model with a behavioral model of the LHC machine, trigger decision unit, and FE, using Visual HDL
14LHCb DAQ Review, September 11-12
RS “Odin” (Progress and Status)
The FPGA designs have been simulated using MaxPlus To check the FPGA designs and crosscheck the MaxPlus simulation, some
of the blocks have been simulated at gate level using Leapfrog.
The behavioral model of the LHC machine, the trigger decision units, and the FE have been refined in order to support a simulation of the real RS design. The behavioral model of the RS is currently being replaced in Visual HDL block by block by the FPGA designs at gate level including all delays.
The entire L0 path (except TTC encoder) has been simulated. Shows that the current design, using three or four clocks (different level of pipelining) works.
I/O
L0 pipeline
L0 handling
15LHCb DAQ Review, September 11-12
RS “Odin” (Progress and Status)
The interface to the L0 and the L1 trigger Decision Units have been agreed on.
RS Minimal Version currently in production: Almost all functionality but not the “RS internal FE” and fewer counters. Aim with first prototype is:
• Verify that the FPGAs are sufficiently fast with safe margin for the functions requiring synchronous operation.
• Measure performance and check concurrent operations of the many functions
16LHCb DAQ Review, September 11-12
TTC tests (Progress and Status)
The need for 1.1 MHz short broadcast transmission on channel B is a crucial point to test. Lacking RS, a test bench was setup using existing equipment:
Using a scope (before the TTCpr was available) shows no problem transmitting 1.1MHz short broadcasts. 1.6 MHz was measured. Data integrity not tested!
Since the encoder circuit in the TTCvx will implemented in the RS and we will use TTCtx the test bench has also allowed us to gain experience and study the performance.
TTCpr is designed to receive ATLAS L1A: Help from ATLAS to modify the code of the onboard FPGA to receive short
broadcasts. Two problems remain:
• The transfer of the short broadcasts into the host PC does not work properly.• Testing the same throughput (1.1 MHz * 16 bits) using the ATLAS version of
the FPGA (long broadcasts) shows problems above ~100 kHz. EventIDs show jumps. PC not capable to cope?
ALEPH FIC TTCvi TTCvx TTCtxTTCpr
VME
17LHCb DAQ Review, September 11-12
Conclusions LHCb TFC system architecture and specific components have been
reviewed in two reviews.
The partitioning concept well integrated.
The first designs and layouts of the LHCb specific components are ready.
Detailed simulation of RS continuing.
The first prototype of the TFC Switch built and the first RS in production.
The results of the tests of the first TFC Switch are very useful.
The first tests with the TTC system show positive results. Work going on with the TTCpr.
When the RS is ready, it will replace the TTCvi + TTCvx in the test bench