r&d for readout, trigger & daq · 3/2/2018 · technologies to efficiently provide...
TRANSCRIPT
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
R&D for Readout, Trigger & DAQ
Hucheng Chen
Hucheng Chen 2018/03/02 - BNL-Yale Meeting 2
R&D Goal: continuously develop expertise in select detectors and electronics technologies to efficiently provide integrated solutions for future experiments
Front-End ReadoutDetectors
LAr Front-End Trigger Digitizer Board (LTDB)
Liquid Argon Electromagnetic Calorimeter (LAr)
Front-End Link Exchange (FELIX)Global L1 Calorimeter
Trigger (gFEX)
Anode Plane Array (APA) for ProtoDUNE-SP Time Projection
Chamber (TPC)
Front-End Link Exchange (FELIX)at ProtoDUNE-SP
Low Gain Avalanche Detectors (LGAD)
FELIX & CaRIBOu readout boards in a
CERN test beam
Control & Readout Board (CaRIBOu)
Signal Proc.
Sensor Front-End Board
Cold Front-End for ProtoDUNE -SP TPC
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
Front-End Signal Processing
● BNL focuses on overall system optimization for integrated detector readout
● LAr Trigger Digitizer Front-End Board○ ATLAS Phase-I LAr Calorimeter Front-End (for 2021)
3LTDBLAr Calorimeter FELIXgFEX
FELIX
LTDB
○ combined analog & digital board■ analog and digital parts designed
by different institutes⇒ Gerber files merged together
■ BNL leads architectural design■ collaboration with Saclay & Milan
○ 320 calorimeter signals (supercells)■ 80 custom 12-bit ADCs @ 40 MSPS digitization■ 20 custom MTx devices for data processing■ 5 GBTx/GBT-SCA for clock distribution, control & monitoring■ 40 optical links @ 5 Gb/s ⇒ 200 Gb/s output
○ pre-production module installed in ATLAS last month
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
● Preamplifier & Shaper ASIC (HLC ASIC)○ ATLAS HL-LHC Upgrade LAr Calorimeter Front-End (for 2026)
○ fully differential low-noise amplifier■ simultaneous readout of two gain settings■ built-in driver to interface COTS or ASIC ADCs for testing■ built-in pulse generator & mask bit for individual channels
to calibrate electronic response and perform crosstalk studies
○ 65 nm CMOS process (TSMC)
○ engineered by BNL Instrumentation Division■ development continued
from VMM ASIC for ATLAS Phase-I UpgradeMuon New Small WheelDetector Front-End
4
Front-End Signal Processing
LTDBLAr Calorimeter FELIXgFEX
ZC706
FETB
test board
HLC1
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
High-Throughput Trigger
● BNL is designing electronics that maximizes throughput while also runningfast trigger algorithms:○ FPGA with many transceivers satisfies high-bandwidth, fixed-latency & processing requirements
● Global Feature Extractor Module (gFEX)○ ATLAS Phase-I Level-1 Calorimeter Trigger Module (for 2021)
■ single board system receives data from full calorimeter■ hardware designed by BNL
○ ATCA blade:■ 30 layer Megtron-6 PCB■ 3 VU9P FPGA and 1 ZU19 MPSoC■ 128 Gb DDR4 RAM■ 420 optical fibers in 35 miniPODs
○ capacity:■ input: 3.9 Tb/s on 312 fibers■ output: 1.4 Tb/s on 108 fibers■ inter-FPGA: 1.6 Tb/s (high-speed & low-speed)
○ pre-production module to be installed in ATLAS in March 2018
5LTDBLAr Calorimeter FELIXgFEX
25.6 Gb/s in PCB
Collaboration between BNL and Chicago, Indiana, Lund, Oregon, Pittsburgh, Stockholm
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
High-Throughput Trigger
● Global Common Module○ ATLAS HL-LHC Trigger/DAQ Upgrade (for 2026)
○ Global Trigger concentrates data for a full event from detector & trigger systems onto a single processor for analysis■ critical part of the TDAQ Upgrade■ up to ~100 Tb/s into Global Trigger■ exploits data aggregation and
serial-to-time multiplexing■ processes data with low latency (<4 s)
○ Global Common Module (GCM) is centerpiece of the Global Trigger■ single electronics module — based on gFEX —
used for all elements■ differences in functionality implemented via firmware
6LTDBLAr Calorimeter FELIXgFEX
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
High-Bandwidth Readout
● BNL developing electronics for high-bandwidth detector readout○ factorize front-end electronics from data handling with compact,
high-density, scalable, low maintenance, easily upgradeable, commodity-based solution
○ reduces amount of custom hardware in favor of scalable detector-independent COTS hardware & software
○ paradigm shift adopted by ALICE, ATLAS, LHCb,ProtoDUNE-SP, DUNE and sPHENIX
7
custom commodityFELIX-711 used byProtoDUNE-SP
● Front End LInk eXchange (FELIX)○ baseline for ATLAS Phase-I and HL-LHC Upgrades○ ProtoDUNE-SP for readout of one APA○ expected baseline for DUNE readout○ sPHENIX for TPC and MVTX readout○ NSLS-II with Vertically Integrated Photon Imaging Chip (VIPIC)○ proposed for Belle II, JLab SoLID, test beams, eRHIC, ANL Light Source
TPC
FELIXCold Front-End
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
High-Bandwidth Readout● Front End LInk eXchange (FELIX)
○ developed for ATLAS Phase-I Trigger/DAQ Upgrade (for 2021)■ BNL designed hardware & co-developed firmware
○ generic PCIe card with Kintex Ultrascale FPGA ■ 48-channels Tx & Rx links in 8 miniPODs■ PCIe Gen3 x16 lanes interface to host■ supports versatile line rates & timing systems
● TTC; TTC-PON; White Rabbit…
○ capacity:■ 460 Gb/s input/output via optical fiber■ 128 Gb/s to host
○ planned upgrades for ATLAS HL-LHC:■ Ultrascale+ FPGA■ PCIe Gen4 x16 lanes
8
Collaboration between BNL and ANL, CERN, Irvine, NIKHEF, UCL, Weizmann [with FNAL (art_daq)]
LTDBLAr Calorimeter FELIXgFEX
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
● Modular readout system for CMOS sensor R&D (CaRIBOu)○ open architecture with ZYNQ SoC to simplify firmware & software○ easily adapted to various sensors under development○ carefully defined interface to minimize design revisions○ used in several test beams with FELIX readout at CERN
■ trigger rate reaches 60+ kHz — much faster than previously available readout system (4 kHz)
○ outgrowth of FELIX & LDRD efforts at BNL
9
Control and Readout
LGAD CaRIBOuSensor FE
CaRIBOu& FELIX
Control & ReadoutBoard (CaRIBOu)
Sensor Front-End BoardMounted Sensor & Readout ChipTelescope at CERN Test Beam with CaRIBOu & FELIX
Hucheng Chen 2018/03/02 - BNL-Yale MeetingHucheng Chen 2018/03/02 - BNL-Yale Meeting
ATLAS HL-LHC ITk Demonstrator Readout with FELIX● CaRIBOu readout with FELIX adapted for
ATLAS HL-LHC ITk Pixel Demonstrator○ Important for final development of pixel readout chain
with FELIX ○ Setup for pixel demonstrator readout development○ Successful integration test of IBL stave0b + FELIX in
Wuppertal in November 2017
● BNL is a major contributor to the ATLAS ITk Strip detector○ Expertise in both ITk strip and DAQ systems
● An ITk Strip Demonstrator is being built at BNL with FELIX readout○ Successful integration test of the Strip Hybrid module
with FELIX readout○ Benefit from previous FELIX based CaRIBOu and pixel
demonstrator readout development○ Aim to expand FELIX readout for ITk Strip module/stave
production test stand
10LTDBLAr Calorimeter FELIXgFEX
Hucheng Chen 2018/03/02 - BNL-Yale MeetingHucheng Chen 2018/03/02 - BNL-Yale Meeting
sPHENIX Detector Readout with FELIX● sPHENIX TPC readout plans to use FELIX
as DAM (Data Aggregation Module) in DAQ system○ TPC test stand with FELIX readout has been built at
BNL○ TPC module with FEE prototype is being read out by
FELIX○ 24 FELIX modules will be needed to instrument the
full TPC for sPHENIX
● A MAPS based Vertex Detector (MVTX) has been proposed for sPHENIX○ FELIX will serve as BEE (Back End Electronics) for
MVTX readout○ MVTX test stand with FELIX readout has been built at
LANL○ FELIX has been used in the readout of MVTX
prototype at Fermilab test beam facility○ 8 FELIX modules will be needed to instrument the full
MVTX for sPHENIX
11
TPC module
FELIX server for TPC
FELIX server for MVTX @ Fermilab
LGAD CaRIBOuSensor FE
CaRIBOu& FELIX
Hucheng Chen 2018/03/02 - BNL-Yale MeetingHucheng Chen 2018/03/02 - BNL-Yale Meeting
● BNL pioneered LAr-based detector technology in 1974○ expertise in detector physics & engineering essential to many
experiments, ○ Close collaboration between Physics & Instrumentation○ unique experience in cryogenic electronics and low-noise
micro-electronics
● R&D → Experiments → R&D○ designs developed well ahead of project funding in many
cases○ basic R&D funding enables us to take leadership roles in
these projects○ a strong cold electronics team is built up as a core BNL
competence
● Strong connections with University & Laboratory partners○ MicroBooNE: Columbia, FNAL, MIT, SLAC, Syracuse, Yale ○ SBND: FNAL, Chicago, Columbia, LANL, Syracuse, Yale, Bern,
CERN, Liverpool, Manchester, Sheffield○ DUNE: Chicago, CSU, Duke, FNAL, LLNL, Penn, Princeton,
SLAC, SMU, Stony Brook, Syracuse, Wisconsin, Yale, CERN, Bern, ...
12
HELIOS
DZero
NA48/62
ATLAS
BooNE
SBND
proto-DUNE
DUNE
Nob
le L
iqui
d R
&D
at
BN
LR806
1974
1980
1990
2000
2010
2020
Long History of Liquid Argon R&D at BNL
JFET FE
JFET FE
CMOS FE
CMOS FE+ADC+FPGA
CMOS FE+ADC+FPGA
CMOS FE+ADC+COLDATA
Advancement of Cold Electronics
TPC
FELIXCold Front-End
Hucheng Chen 2018/03/02 - BNL-Yale MeetingHucheng Chen 2018/03/02 - BNL-Yale Meeting
● Front-End ASIC: ○ ~5.5 mW/channel (input MOSFET 3.9 mW)○ ~15,000 MOSFETs○ 16 channels, programmable○ adjustable gain & filter time constant○ build in pulse generator with 6-bit DAC
● ADC ASIC○ 16 channels, programmable○ sample/hold○ 12-bit ADC at 2MS/s sampling rate○ current-mode domino architecture○ FIFO 192s bit wide x 32 bits deep○ multiplexer 8:1 or 16:1○ serializer 12:1○ adjustable offsets○ ~4.5 mW/ch.○ ~300,000 MOSFETs
● Full cold readout chain with FE ASIC, ADC ASIC and cold FPGA is being used to equip the ProtoDUNE-SP LAr TPC
13
● R&D of CMOS cold electronics started in 2008○ designed for 77K-300K operation○ designed for long lifetime○ tech. CMOS 180 nm, 1.8 V, 6M, MIM,
SBRES● Cold FPGA to format digitized signal and
drive data out of cryostat through serial links
Front End ASIC for Cold Electronics at BNL
TPC
FELIXCold Front-End
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
Single Phase LAr TPC Development towards DUNE
BNL is leading TPC readout electronics SYSTEM design for MicroBooNE, SBND and ProtoDUNE-SP
14TPC
FELIXCold Front-End
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
Integral System Design Concept
Cold electronics module and its attachment to the APA frame
15
A necessary (but not sufficient!) condition to achieve a good system performance, the integral design concept of APA + CE + Feed-through, plus Warm Interface Electronics with local diagnostics and strict isolation and grounding rules will have to be followed
ProtoDUNE-SP
TPC
FELIXCold Front-End
Hucheng Chen 2018/03/02 - BNL-Yale MeetingHucheng Chen 2018/03/02 - BNL-Yale Meeting
● BNL designed front end readout electronics system for MicroBooNE experiment● Analog front end ASIC designed in 180nm is running in LAr (~89 K) to achieve optimum signal
to noise ratio● MicroBooNE is the first experiment instrumented with cold CMOS ASICs
16
MicroBooNE Front End Electronics
TPC
FELIXCold Front-End
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
MicroBooNE Front End Electronics
17
H. Cold Mother Board
V. Cold Mother Board
ASIC Configuration Board
Intermediate Amplifier
Receiver ADC Board
Service Board
Signal Feed-through Assembly
TPC
FELIXCold Front-End
Hucheng Chen 2018/03/02 - BNL-Yale MeetingHucheng Chen 2018/03/02 - BNL-Yale Meeting
● Total: 11,264 channels● 704 FE ASICs, 11,264 ADC channels, 88 Cold
FPGAs● 88 Front End Mother Board assemblies● 4 sets of cold cable bundles, 4 sets of signal
feed-throughs● ~28 warm interface boards● Lifetime study of COTS ADC is being carried out
at BNL○ Sample has been stress tested for > 700 hours with 5.25V in
LN2○ Preliminary lifetime of COTS ADC extrapolated from past
study would be more than 108 years
18
TPC Readout Electronics
Warm Interface Electronics
Cold Electronics
SBND TPC Electronics
Sigma of DNL with 5.25V @ 2Msps
TPC
FELIXCold Front-End
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
SBND Front End Electronics Prototypes
19
Signal Feed-through & WIEC Assembly
FEMB Flange Board WIB
PTB
MBB
Cold Cable
TPC
FELIXCold Front-End
Hucheng Chen 2018/03/02 - BNL-Yale MeetingHucheng Chen 2018/03/02 - BNL-Yale Meeting
● 15360 channels● 960 FE ASICs/960 ADC ASICs/120 Cold FPGAs● 120 Front End Mother Board assemblies● 6 sets of cold cable bundles, 4 sets of signal
feed-throughs● 6 warm interface electronics crates● ~36 warm interface boards● Share many common development of front end
readout electronics with SBND
20
FEMB+CE box assemblies have been installed on three APA @ CERN as of today
ProtoDUNE-SP TPC Front-End Readout Electronics
TPC
FELIXCold Front-End
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
ProtoDUNE-SP Front End Electronics
21Cold Side Warm Side
7m Cold Cables
20 CE boxes on APA
FEMB (inside CE box)
Signal Feed-through Assembly
TPC
FELIXCold Front-End
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
● BNL focuses on overall system optimization for integrated detector readout○ inspired by experimental requirements and our physics interests
■ common considerations: flexible, modular, scalable, high-density, high-throughput
○ front-end ASIC design optimized for detector technologies
○ front-end board design tailored for difficult experimental environments■ e.g., cryogenic temperatures, high-radiation environments
○ trigger/DAQ designs that exploit COTS solutions■ eg., high-speed optical links, DSP in FPGA firmware, SoC (ZYNQ+)
Summary
22
LTDBLAr Calorimeter FELIXgFEX
LGAD CaRIBOuSensor FE
CaRIBOu& FELIX
TPC
FELIXCold Front-End
Hucheng Chen 2018/03/02 - BNL-Yale Meeting 23
Backup Slides
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
Silicon Sensors● only U.S. Laboratory pursuing both
HV-CMOS Monolithic Active Pixel Sensors (MAPS) and Low Gain Avalanche Detectors (LGAD)
● depleted CCDs & single-photon detectors● Energy, Intensity, & Cosmic Frontier
experiments and Photon Sciences, Medical Imaging
● collaboration between Physics, Instrumentation, and Photon Sciences with ANL, Bern, Cambridge, CERN, CNM, Geneva, IFAE Barcelona, NYU, Pennsylvania, RAL, UC Santa Cruz, Stony Brook, Toronto, Wellesley, Yale
● supported by BNL LDRDs and DOE Early Career Award
24
The Generic Detector R&D research program relies on the unique capabilities and synergies available at BNL
Microelectronics● world-class development ASIC
development● low-power low-noise front ends● cold electronics enabling large
cryogenic detectors● high-functionality front-end ASICs● radiation-tolerant electronics● Energy & Intensity Frontier
experiments● collaboration between
Instrumentation & Physics with Arizona, Brigham Young, CEA Saclay, FNAL, LBNL, IFIN-HH Romania, IN2P3-LAL, INFN Milano, Michigan, NRL/NASA, Pennsylvania, Sao Paulo, SBU, SMU, Toronto, Washington Univ.
● supported by BNL LDRDs Data Acquisition & Trigger● readout for noble liquid environments● high-throughput data acquisition● high-bandwidth trigger processors● Energy & Intensity Frontier, Nuclear
Physics, Photon, Light Sources● collaboration between Physics &
Instrumentation and ANL, Bern, CERN, Chicago, Cincinnati, Columbia, FNAL, INFN Bologna, Indiana, UC Irvine, LANL, Lund, MSU, Oregon, NIKHEF, Pennsylvania, PNNL, Pittsburgh, SMU, Stockholm, Syracuse, UT Arlington, Weizmann, Wuppertal, Yale
Liquid Argon Detectors● BNL pioneered LAr-based detector
technology● precision sampling calorimeters● time-projection chambers● LAr properties● Energy & Intensity Frontier experiments● close collaboration between
Instrumentation & Physics with Bern, CERN, Chicago, Columbia, CSU, Duke, FNAL, LANL, LLNL, Liverpool, Manchester, MIT, Pennsylvania, Princeton, SLAC, Sheffield, SBU, SMU, Syracuse, Wisconsin, Yale
● supported by BNL LDRDs andtwo DOE Early Career Awards
Water Based Liquid Scintillator● unique facility developed by BNL● LZ, PROSPECT, 0νββ decay (SNO+),
Medical Imaging, Nonproliferation● collaboration between Chemistry &
Physics with Australia National, UC Berkeley, UC Davis, LBNL, Oxford, Penn, SBU, Tsinghua, Yale
● supported by BNL LDRDs
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
Microelectronics at BNL
● ASIC development successful because of concentration of resources & expertise and close collaboration with experimental teams○ advances in radiation detectors tightly
coupled to advances in front-end ASICs○ front-end ASICs rapidly becoming very
high functionality systems-on-chip (SOC)○ system integration is crucial!
25
ASIC for ATLAS Upgrade
Cryogenic Analog Low-Noise Front-End ASIC
adopted by MicroBooNE, Argontube, CAPTAIN, LArIAT, SBND, 50 L
ICARUS,DUNE 35 ton, proto-DUNE, DUNE
Cryogenic Front-End(8 FE ASICs, 8 ADC ASICs, 1 FPGA)
Hucheng Chen 2018/03/02 - BNL-Yale Meeting
Radiation-Tolerant Electronics
26
● Extensive studies on ADCs & FPGAs○ performance, survivability, single- and
double-event upsets○ commissioned 14 MeV neutron source @
BNL for initial component screening○ further tests at other facilities
● Critical need for HEP experiments○ many industrial components not extensively tested by manufacturers○ space- and military-qualified components are expensive○ significant interest from other fields: Silicon Engineering,
Nuclear Physics, Photon Sciences, Medical
● Systematically explore impact of radiation○ close collaboration with university and industry partners:
BYU (M.J. Wirthlin), MSU, SMU, CERN, INFN MilanoAltera, Analog, Hittite, Linear, Microsemi, TI
radiation testing of ABC130 ASIC
at BNL-SSIF