jin huang (bnl) thanks to discussions from nils · 02/04/2018 · sphenix use a next-generation...
TRANSCRIPT
Jin Huang (BNL)Thanks to discussions from Nils
Versatility of EIC event topology calls for trigger-less DAQ◦ 0.5 MHz interaction at top luminosity
Start using sPHENIX-EIC full detector simulation to estimate trigger-less DAQ◦ Total data rate on order of 100 Gbps
Match well with sPHENIX TPC/MVTX DAQ through-put rate◦ Similar architecture with ATLAS/LHCb/ALICE DAQ
upgrade in 2020+
Leverage streaming readout and EIC-related detector R&D at sPHENIX
Cold QCD TG meetingJin Huang <[email protected]> 2
I was asked to give a presentation in EIC stream readout meeting about PHENIX/sPHENIX experience
Taking the opportunity to look into a possible stream readout for the EIC detector based on sPHENIX. ◦ What is the data volume? ◦ Reuse parts of sPHENIX DAQ?◦ Event building or stream readout?
Event building is not easy since eRHIC has 10ns crossing and BBC+calorimeter do not cover all interested cross section
Cold QCD TG meetingJin Huang <[email protected]> 3
sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA ASIC : 20 Mhz 10-bit ADC , hit wavelet forming in chip 3.6 Tbps bi-direction fiber link to counting house
Cold QCD TG meetingJin Huang <[email protected]>
Buffer box
FEE data stream: 600 6.6+ Gbps bidirectional fibers linksMax continuous: 4 Gbps / fiberAverage continuous: 1.6 Gbps x 600 fibers
TPC DAM L3 Scope, 1.2.6
Output data stream to buffer box: 24 x 10 Gbps Ethernet via fiberAfter FPGA based triggering and clustering
Total continuous output to disk: 80 Gbps
DAQ L2 ScopeWBS 1.6
Clock/Trigger input: Optical linksClock = 9.4 MHzTrigger Rate = 15 kHz
FEE L3 Scope1.2.5
Structure/GEM L3 Scopes
WBS 1.2.1-4
Transfer to RCF continuously
150k chan
600 FEEs
24 sectors
Continuous
readout
4
Cold QCD TG meetingJin Huang <[email protected]> 5
ATLAS DAQ structure for mid-2020s:
• Using high performance large-FPGA optical link cardto bridge custom FEE and commodity computing
• Besides ATLAS, similar architecture also proposed for ALICE and LHCb upgrades
• 48 x 10 Gbps bi-directional optical link to FEE• Kintex7 Ultrascale FPGA• 100 Gbps PCIe Gen3 x16 link to CPU
Front-End Link eXchange (FELIX)FPGA-based PCI express card
FELIX 2.0 card in pre-production
Currently used in sPHENIX test stands
• You can also consider it as a 48-link network switchrunning specialized network protocol for 1 FEE/linkData reduction on FPGA, optimized for cost / link
Cold QCD TG meetingJin Huang <[email protected]> 6
BNL EIC taskforce studieshttps://wiki.bnl.gov/eic/index.php/Detector_Design_Requirements
Considering the case of 20+250 GeV e+p collisions at luminosity of 1034 cm-2 s-1
◦ 100x p+p luminosity
Cross section: 54 µb◦ Based on EIC task-force simulation
/gpfs/mnt/gpfs04/sphenix/user/nfeege/data/copy_eic_data_PYTHIA_ep/TREES/pythia.ep.20x250.1Mevents.RadCor=0.root
◦ 0.1% of p+p cross section
Collision rate: 500 kHz◦ 10% of p+p collision rate
Multiplicity: 1 /dη @ η=0◦ Fraction of p+p multiplicity
Total particle rate: a few M particle /s◦ sPHENIX AuAu records 30M particle /s
sPHENIX DAQ bandwidth may be able to accommodate full EIC data without triggering (full stream readout)◦ Event forming and reconstruction either in
online / offline computing farm
Cold QCD TG meetingJin Huang <[email protected]> 7
DIS collision @ Q2 ~ 100 (GeV/c)2
Cold QCD TG meetingJin Huang <[email protected]> 8
Multiplicity check for all particlesMinimal bias Pythia6 e+p 20 GeV + 250 GeV53 µb cross section
BNL EIC taskforce studieshttps://wiki.bnl.gov/eic/index.php/Detector_Design_Requirements
Based on EIC task-force pythia set /gpfs/mnt/gpfs04/sphenix/user/nfeege/data/copy_eic_data_P
YTHIA_ep/TREES/pythia.ep.20x250.1Mevents.RadCor=0.root
Cold QCD TG meetingJin Huang <[email protected]> 9
Raw data: 16 bit / MAPS hit
Raw data: 3x5 10 bit / TPC hit + headers (60 bits)
Raw data: 3x5 10 bit / GEM hit + headers (60 bits)
Cold QCD TG meetingJin Huang <[email protected]> 10
Raw data: 31x 14 bit / active tower +padding + headers ~ 512 bits / active tower
Cold QCD TG meetingJin Huang <[email protected]> 11
Raw data: 31x 14 bit / active tower +padding + headers ~ 512 bits / active tower
Cold QCD TG meetingJin Huang <[email protected]> 12
Obviously this is very preliminary, and evaluated at top lumi 1034 cm-2 s-1: Tracker+ calorimeter FEE output signal only = 32 Gbps, with header ~40 Gbps
◦ minimal account to bring out of tracker+ calo FEE via fiber ◦ No noise, and no PID detectors
FEE output with headers ~ 40Gbps * 1.5(+PID) * 2 (noise) ~ order of 100 Gbps ◦ A earlier estimate by BNL EIC taskforce BeAST detector:
Record 2.5M track / s → 160 Gbps (T. Ljubicic). In the same ball park◦ sPHENIX TPC accommodate a 4 Tbps FEE -> counting house traffic
sPHENIX TPC/MAPS DAQ is designed to record 100 Gbps @ 30M track /s The sPHENIX DAQ may serve as foundation for stream readout in EIC, that record all
detector hit at EIC without need of event building
EIC detector’s data rate for all collisions (~100 Gbps) seem fit into the sPHENIX DAQ bandwidth for disk recording◦ Online/offline computing farm perform collision ID, event
association and reconstruction
4 Tbps (FEE-link) / 100 Gbps (disk) DAQ for sPHENIX TPC/MAPS can provide a solid foundation that accommodate stream readout of an EIC detector◦ On-going: advanced R&D and beam tests◦ FEE -> fiber -> PCIe stream readout for TPC&MAPS FEE,
consistent with ATLAS/ALICE/LHCb in 2020+
May become a section in the new LOI
Cold QCD TG meetingJin Huang <[email protected]> 13
Cold QCD TG meetingJin Huang <[email protected]> 14
Cold QCD TG meetingJin Huang <[email protected]> 15
sPHENIX use a next-generation gate-less TPC for outer central tracking, Require stream readout FEE by design
SAMPA ASIC : 20 MHz 10-bit ADC , hit wavelet forming in chip. 3.6 Tbps bi-direction fiber link to counting house
Triggered readout for sPHENIX (save 75% of physics data size), Plan to also support full stream readout with firmware switch on FELIX FPGA (for commissioning / testing)
Cold QCD TG meetingJin Huang <[email protected]> 16
sPHENIX calorimeter: T1044
INTT: T1439 sPHENIX Si strip test
MVTX: T1441 sPHENIX MAPS test
TPC prototype being prepared tooMAPS Online monitoring with 4 staves in beamPreliminary alignment, ~O(100um)
Cold QCD TG meetingJin Huang <[email protected]> 17
100kHz collision in continuous DAQ triggerIn TPC DAQ simulation
FEE -> DAM limit : 6 Gbps x 8b/10b per FEEReference design rate: 1.9 Gbps, far lower than limitMax rate: 200kHz + 48 rings → max 7.2 Gbps @ module 1
― All collisions― Triggered collisions
Cold QCD TG meetingJin Huang <[email protected]> 18
Collect data from 600 bi-directional 4+ Gbps fiber links to FEEs at rate of 940 Gbps
Reducing data via triggering, clustering and compression to reduce data to ~80 Gbps
Driven to use large-FPGA based data acquisition cards with multi-10Gbps input and PCIe Gen3 output, hosted on commodity servers.
◦ Similar architecture adopted for ATLAS, LHCb, ALICE upgrade for 2020+
Default implementation taking advantage of BNL-developed FELIX PCIe card
◦ 48-bidirectional 10 Gbps optical link, large FPGA: Xlinx Ultrascale (XCKU115)
◦ PCIe Gen3 x16 link to server, demonstrated to 101 Gbps (require ~10 Gbps)
ServerFELIX v1.5 card FELIX in serverMTP <-> LC BreakoutFEE Prototype
Cold QCD TG meetingJin Huang <[email protected]> 19
Online display
PHENIX event builder/ Data storage
Standalone data(calibration, etc.)
FPHX Chip
Sensor
HDI
Ionizing Hit
IR
DA
Q R
oom
768 fibers
1.9 Tb/s
17k LVDS
3.2 Tb/s
48 fibers
~8kHz trigger
8 fibers
Data cable/bandwidth shown on this slide only
PHENIX Timing
Timing / Trigger
Stream
Triggered d
ata to d
isks