btev trigger
DESCRIPTION
BTeV Trigger. Erik Gottschalk, Fermilab (for the BTeV Trigger Group). Overview. Introduction Brief overview of the BTeV detector Level-1 trigger algorithm Trigger architecture Level-1 trigger hardware Original baseline design (digital signal processors – DSPs) - PowerPoint PPT PresentationTRANSCRIPT
BTeV Trigger
Erik Gottschalk, Fermilab
(for the BTeV Trigger Group)
Erik GottschalkBeauty 2005: BTeV Trigger 2
Overview
• Introduction• Brief overview of the BTeV detector• Level-1 trigger algorithm• Trigger architecture• Level-1 trigger hardware
– Original baseline design (digital signal processors – DSPs)– New baseline design (commodity hardware)– Proposed change to the new baseline design (upstream
event builder & blade servers)
Not covered:• Level-1 muon trigger• Level-2/3 trigger (high-level trigger – “HLT”)• Real Time Embedded Systems (RTES) Project
Erik GottschalkBeauty 2005: BTeV Trigger 3
Introduction
• The challenge for the BTeV trigger and data acquisition system was to reconstruct particle tracks and interaction vertices for EVERY proton-antiproton interaction in the BTeV detector, and to select interactions with B decays.
• The trigger system was designed with 3 levels, referred to as Levels 1, 2, and 3:“L1” – look at every interaction and reject at least 98% of minimum bias background“L2” – use L1 computed results & perform more refined analyses for data selection“L3” – reject additional background and perform data-quality monitoring
Reject > 99.9% of background. Keep > 50% of B events.
• The data acquisition system was designed to save all of the data in memory for as long as was necessary to analyze each interaction, and move data to L2/3 processors and archival data storage.
• The key ingredients that made it possible to meet this challenge:– BTeV pixel detector with its exceptional pattern recognition capabilities– Rapid development in technology – FPGAs, processors, networking
Erik GottschalkBeauty 2005: BTeV Trigger 4
BTeV - a hadron collider B-physics experiment
BTeV at C0CDF
D0 pp
Tevatron
Fermi National Accelerator Laboratory
Erik GottschalkBeauty 2005: BTeV Trigger 5
BTeV Detector in the C0 Collision Hall
Erik GottschalkBeauty 2005: BTeV Trigger 6
BTeV Detector
MuonEM Cal
Straws &Si Strips
Dipole Magnet
RICH
30 StationPixel Detector
Erik GottschalkBeauty 2005: BTeV Trigger 7
Multichip module
Silicon Pixel Detector
50 m
400 m5 cm
1 cm
6 cm
10 cm
pixel sensors
sensor module
5 FPIX ROC’s
128 rows x22 columns
14,080 pixels (128 rows x 110 cols)
380,160 pixelsper half-station
total of 23Million pixelsin the full pixel detector
Sensor module
HDI flex circuit
Wire bonds
Readout module
Bump bonds
Pixel detector half-station
TPG substrate
Erik GottschalkBeauty 2005: BTeV Trigger 8
Counting RoomCollision Hall
Pixel Data Readout & 1st Part of L1 Trigger
FPGAsegment tracker
(ST)
to neighboring FPGAsegment tracker
to neighboring FPGAsegment tracker
Pixel stationsPixel data combiner boards
Pixelprocessor
Pixelprocessor
Pixelprocessor
1542 channels @140Mbps from3 pixel stations (257/half-plane)
Optical links
12 channels @2.5Gbps
Row (7bits) Column (5bits) BCO (8bits) ADC (3bits)
sync (1bit)
Chip ID (13bits)
FPIX2 Read-out chip
Pixel processor
time stamp ordering
pixel clustering
xy coordinates
Internal bondpads for chip ID
Data outputinterface
LVDS drivers& IO pads
Registers & DAC’s
Commandinterface
Debuggingoutputs
128x22Pixel array
End-of-columnlogic
Erik GottschalkBeauty 2005: BTeV Trigger 9
BTeV Trigger and Data Acquisition System
500 GB/s(200KB/event)
2.5 MHz
L1 rate reduction: ~50x
L2/3 rate reduction: ~20x
12.5 GB/s(250KB/event)
50 KHz
2.5 KHz 200 MB/s(250KB / 3.125 = 80KB/event)
Erik GottschalkBeauty 2005: BTeV Trigger 10
Simulated B Event
Erik GottschalkBeauty 2005: BTeV Trigger 11
Simulated B Event
Erik GottschalkBeauty 2005: BTeV Trigger 12
L1 Vertex Trigger Algorithm
1) Segment tracking stage: Use pixel hits from 3 neighboring stations to find the beginning and ending segments of tracks. These segments are referred to as triplets.
Two stage trigger algorithm:1. Segment tracking2. Track/vertex finding
Erik GottschalkBeauty 2005: BTeV Trigger 13
1a) Segment tracking stage: phase 1 Start with inner triplets close to the interaction region. An inner triplet represents the start of a track.
Segment Tracker: Inner Triplets
Erik GottschalkBeauty 2005: BTeV Trigger 14
1b) Segment finding stage: phase 2 Next, find the outer triplets close to the boundaries of the pixel detector volume. An outer triplet represents the end of a track.
Segment Tracker: Outer TripletsTrack/Vertex Finding
Erik GottschalkBeauty 2005: BTeV Trigger 15
2a) Track finding phase: Match inner triplets with outer triplets to find complete tracks.
2b) Vertex finding phase:• Use reconstructed tracks to locate interaction vertices• Search for tracks detached from interaction vertices
Track/Vertex Finding
Erik GottschalkBeauty 2005: BTeV Trigger 16
• Generate Level-1 accept if “detached” tracks going into the instrumented arm of the BTeV detector with:
2
2.0
6
25.02
b
b
pT
(GeV/c)2
cm
L1 Trigger Decision
b
p p
B-meson
Execute Trigger
Erik GottschalkBeauty 2005: BTeV Trigger 17
GlobalLevel-1
ITCH
Information Transfer Control Hardware
GL1
Level-1 Buffers
12 x 24-port Fast Ethernet Switches
Level 2/3Processor Farm
Pixel Processors
FPGA Segment Finder
Track/Vertex Farm
Gigabit Ethernet Switch
Data Combiners +Optical Transmitters
OpticalReceivers
BTeV Detector
Front End Boards
8 Data Highways
Data Logger
Cross Connect Switch
BTeV Trigger Architecture
Erik GottschalkBeauty 2005: BTeV Trigger 18
~0.5TB L1 Buffers Based on Commodity DRAM
PC motherboard(L1 buffer server)
Gigabit Ethernet(to L2/3 switch & farm)
Output buffer(after L1 accept)
L1 buffer module
Optical Receivers(2 X 12 chan X 2.5 Gbps)
FPGA
L1 Buffer Memory(DDR DRAM)
Erik GottschalkBeauty 2005: BTeV Trigger 19
Prototype L1 Buffer
PCI card based L1 bufferboard with commodity DRAM
PC motherboardwith Gigabit ethernet acting as L1 buffer server
Erik GottschalkBeauty 2005: BTeV Trigger 20
Original Baseline: DSP-Based L1 Track/Vertex Hardware
Block diagram of prototype L1 track/vertex farm hardware
DSP
DSP
DSP
DSP
DSP
RO
M
RAM
RAM
RAM
RAM
RO
MR
OM
RO
M
Hitachi H8SRAM
Hitachi H8SRAM
ArcNetController
ArcNetController
CompactFlash
64 KBFIFO
Buffer Manager(BM)
TriggerResults
Manager(TM)
Host PortGlue Logic
(HPGL)On-boardPeripheralsGlue Logic
(OPGL)
LCD Display
Data fromsegmentfinder
Processedresults toL1 buffers
To GL1
To externalhostcomputer
JTAG
McBSP lines (SPI mode)
HPI bus
Erik GottschalkBeauty 2005: BTeV Trigger 21
DSP-Based Track/Vertex Hardware Prototype
Erik GottschalkBeauty 2005: BTeV Trigger 22
L2/3 Farm
New L1 Baseline using Commodity Hardware
30 Pixel Stations
Pixel Processors
FPGA Segment Finders (56 per highway)
Track/Vertex Farm
56 inputs at ~45 MB/s each
Level 1 switch
33
ou
tpu
ts at ~
76
MB
/s ea
ch
Trk/Vtx
node #1
Trk/Vtx
node #2
Trk/Vtx
node #N
GL1
ITCH
Front ends
L1 muon
L1 buffers
Other detectors
L2/3 Switch
PTSM network
33 “8GHz” Apple Xserve G5’s
with dual IBM970’s
(two 4GHz dual core 970’s)
Infiniband switch
Xserve identical to
track/vertex nodes
Ethernet switch
1 Highway
Erik GottschalkBeauty 2005: BTeV Trigger 23
Prototype L1 Track/Vertex Hardware
16 Apple Xserves
Infiniband switch
Front Rear
Erik GottschalkBeauty 2005: BTeV Trigger 24
Collision HallCounting Rm
30 Station Pixel Detector
Level 1 Crossing Switch
To Global Level 1
1
Time Stamp Ordering
Pixel Pre-processing
Segment Finding
Event Building
Vertex Finding
2
3
4
Pixel Data Combiner Boards
TSO TSO TSO TSO TSO
Vertex Farm
DCB DCB DCB DCB DCB DCB DCB DCB DCB DCB
28 x 2 FPGA Segment Trackers
Pixel Pre-processor Modules
Time StampOrdering Modules
Current baseline architecture
30 Station Pixel Detector
Collision HallCounting Rm
Pixel Data Combiner Boards
DCB DCB DCB DCB DCB DCB DCB DCB DCB DCB
TSO TSO TSO TSO TSO
1
Time Stamp Ordering
Pixel Pre-processing
Segment Finding
+Event Building
Vertex Finding
2
3
FPGA Segment Trackers
Pixel Pre-processor Modules
PP PP PP PP
Vertex Farm
To Global Level 1
Time StampOrdering Modules
New integrated upstream event builder architecture
Proposed Upstream Event Builder Architecture
described in:
BTeV-doc-3342
described in:
BTeV-doc-3342
Erik GottschalkBeauty 2005: BTeV Trigger 25
TV
STST STST ST STST ST
TV TV TV TV TV TV TV
To Global Level 1
ST
TV
L1 Event Building Switch
Segment Trackers have complete
events. No need for a single large
switch for event building.
L1 Farm Transformation
ST: Segment Tracker
TV: Track/Vertex Finder
Erik GottschalkBeauty 2005: BTeV Trigger 26
TV
STST STST ST STST ST
TV TV TV TV TV TV TV
To Global Level 1
switch
ST
TV
switch switch
However, we may still need
a switching function to deal
with TV node failures. This
can be handled by smaller
switches.
L1 Farm Transformation
Erik GottschalkBeauty 2005: BTeV Trigger 27
TV
STST STST ST STST ST
TV TV TV TV TV TV TV
To Global Level 1
TV
STOr, the switching function
can be handled by a
“Buffer Manager” as it wasdone in our DSP prototype.
Which could possibly be
integrated into the Segment
Trackers (ST).
L1 Farm Transformation
Erik GottschalkBeauty 2005: BTeV Trigger 28
Each 7U crate can hold up to 14 dual-cpu blades which
are available with Intel Xeon’s (>3.0GHz) or IBM970PPC’s.
Blade Server Platform
Intel/IBM blade server chassis
6 of these crates will fit into a standard 42U rack for a
total of 168 CPU’s (on dual cpu blades) per rack.
In the proposed architecture we replace AppleXserves with blade servers.
Erik GottschalkBeauty 2005: BTeV Trigger 29
front view
Blade Server Platform Features
2 network interfaces are provided
on the mid-plane of the chassis:
- primary/base network: each slot
connects blade’s on-board gigabit-
ethernet ports to a switch module
in the rear
- secondary hi-speed network: each
slot also connects ports on the
blade’s optional I/O expansion card
to an optional hi-speed switch module
(myrinet, infiniband, etc.)
rear view
- the I/O expansion card might even be
a custom card with an FPGA to implement
a custom protocol.
Erik GottschalkBeauty 2005: BTeV Trigger 30
L1 Hardware for 1 Highway with Blade Servers
Complete segment tracking and track/vertex hardwarefor 1 highway housed in 4 blade server crates
Erik GottschalkBeauty 2005: BTeV Trigger 31
L1 Trigger Hardware in a Blade Server Crate
Module 4Module 3
156 MB/s out of
each ST board
(use 2 pairs)
From pixel pre-processor
Segment tracker
boards
Dual CPU Track & Vertex blades
78 MB/s into L1tv
node (use 1 pair)
0.5 MB/s out of each
L1tv node to GL1
(use 1 pair)
4 MB/s to GL1
Erik GottschalkBeauty 2005: BTeV Trigger 32
Level 1 Hardware Rack Count
Complete L1 trigger hardware for
one highway can fit in one 42U rack
Entire L1 trigger for all 8 highways
in 8 42U racks
Segment finder & track/vertex hardware
in 4 blade server crates
Upstream event builder/pixel pre-processor
Erik GottschalkBeauty 2005: BTeV Trigger 33
Summary
The BTeV trigger evolved from an original baseline design that satisfied all BTeV trigger requirements to a new design that was easier to build, required less labor, had lower cost, and lower risk.
When BTeV was canceled we were on the verge of proposing a new design to the Collaboration. This design was expected to:
– reduce the cost of the trigger system– reduce the amount of space needed for the hardware– improve the design of the system architecture– improve system reliability (redundancy & fault tolerance)– improve integration of L1 & L2/3 trigger systems
Ideas that we developed for the BTeV trigger are likely to be used in future high-energy physics and nuclear physics projects.
Erik GottschalkBeauty 2005: BTeV Trigger 34
End
End
Erik GottschalkBeauty 2005: BTeV Trigger 35
Level 1 Vertex Trigger
FPGA segment trackers
Merge
Trigger decision to Global Level 1
Event Building Switch: sort by crossing number
track/vertex farm(~528 processors)
30 station pixel detector
Erik GottschalkBeauty 2005: BTeV Trigger 36
Backup slides
Backup slides
Erik GottschalkBeauty 2005: BTeV Trigger 37
Data Rate into L1 Farm
Using the triplet format proposed in BTeV-doc-1555-v2:
internal triplets: 128 bits = 16 Bytesexternal triplets: 74 bits = 10 Bytes
Extrapolating linearly to 9 interactions/crossing and applying a safety factor of 2:
2 x (4.5 x 910 Bytes = 4095 Bytes) = 8190 Bytes
Then assumed 35 internal and 35 external triplets for events with <2> interactions/crossing:
35 x 26 Bytes = 910 Bytes
Total rate going into L1 track/vertex farm in all 8 highways :
2.5 MHz x 8190 Bytes ~ 20 GBytes/s
Erik GottschalkBeauty 2005: BTeV Trigger 38
Required L1 Farm Computing Power
• Assume an “8.0 GHz” IBM 970:
– L1 trk/vtx code (straight C code without any hardware enhancements like hash-sorter or FPGA segment-matcher) takes 379 s/crossing on a 2.0 GHz IBM 970 (Apple G5) for minimum bias events with <6> interactions/crossing
– assume following:• L1 code: 50%, RTES: 10%, spare capacity: 40%• 379 s + 76 s + 303 s = 758 s
– 758 s ÷ 4 = 190 s on a 8.0 GHz IBM 970
– Include additional 10% processing for L1-buffer operations in each node• 190 s + 19 s = 209 s
• Number of “8.0 GHz” IBM 970’s needed for L1 track/vertex farm:– 209 s/396 ns = 528 cpu’s for all 8 highways, 66 cpu’s per highway– 33 dual cpu Apple Xserve G5’s per highway
Erik GottschalkBeauty 2005: BTeV Trigger 39
L1 Segment Tracker on a PTA Card
Uses Altera APEX EPC20K1000instead of EP20K200 on regular PTA
Modified version of PCI Test Adapter card developed at Fermilab for testinghardware implementation of 3-station segment tracker (a.k.a. “Super PTA”)
Erik GottschalkBeauty 2005: BTeV Trigger 40
Prototype L2/3 farm
Prototype L2/3 farmusing nodes fromretired FNAL farms
Erik GottschalkBeauty 2005: BTeV Trigger 41
Real Time Embedded Systems (RTES)
Global Manager
Regional Level-1 Regional Level-2/3
DatabaseAnalysisOperators
Worker PCFarmlet
Worker DSP
Generic Modeling Environment (GME) Vendor APIs
Adaptive Reconfigurable MobileObjects of Reliability(ARMOR)
Very Light WeightAgent (VLA)
1
25
1100
1
4
1
6
1
100
• RTES: NSF ITR (Information Technology Research) funded project
• Collaboration of computer scientists, physicists & engineers from: Univ. of Illinois, Pittsburgh, Syracuse, Vanderbilt & Fermilab
• Working to address problem of reliability in large-scale clusters with real time constraints
• BTeV trigger provides concrete problem for RTES on which to conduct their research and apply their solutions