upton, new york wbs 6.7.1 & 6.7.3 global common module & felix · 09/05/2018 · global...
Post on 31-Aug-2020
0 Views
Preview:
TRANSCRIPT
Michael Begel
WBS 6.7.1 & 6.7.3Global Common Module &
FELIXMichael Begel & George Redlinger
Level-3 Managers & CAMsBrookhaven National Laboratory
U.S. ATLAS HL-LHC Upgrade Project DOE CD-1 Director’s Review Brookhaven National Laboratory
Upton, New York May 7-9, 2018
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Technical Details: GCM
2
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Global Trigger for HL-LHC
3
● ATLAS planning major upgrade of triggerand data acquisition systems for HL-LHC
■ Global Trigger is crucial element○ new functionality for TDAQ
○ strong U.S. leadership in ATLAS● M. Begel (BNL)
Global Trigger (ATLAS L3)
● Hucheng Chen (BNL) Global Common Module
● W. Fisher (MSU)Firmware Interfaces
● C. Dudley (Oregon)Trigger Framework Firmware
● S. Majewski (Oregon)Trigger Signatures
WBS 6.7.1WBS 6.7.4WBS 6.8.3
WBS 6.7.3
WBS 6.7.4WBS 6.8.2
WBS 6.6.5
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Global Trigger for HL-LHC
4
>2300 incident optical fibers9.6 to 25.8 Gb/s link speeds
several link protocols
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Global Trigger for HL-LHC
● Concentrates data for full event onto single processor for analysis at 40 MHz■ approx. 50 Tb/s into Global Trigger■ exploits data aggregation and serial-to-time multiplexing■ features common hardware
5
GCM: Global Common ModuleMUX: GCM for data aggregation & time multiplexingGEP: GCM for event processing & trigger algorithms
Mu
ltip
lexi
ng
WBS 6.7.1
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
● Features common hardware — Global Common Module (GCM)■ different functions implemented in firmware
rather than in hardware
■ two module configurations: MUX & GEP○ differ by choice of pin-compatible FPGA
● MUX: Xilinx Ultrascale+ VU9P● GEP: Xilinx Ultrascale+ VU13P
○ 24 of each configuration
■ each GCM features two nodes○ a node is a single FPGA○ nodes operate independently
● can communicate with each other
■ SoC for command, control, mon. & readout○ Xilinx ZYNQ Ultrascale+ MPSoC
Global Trigger for HL-LHC
6
Mezzanine
(optional)
Processor Processor
Command & Control & Readout
Proof of principle layout for costing purposes
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Hardware Deliverables
7
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
● WBS 6.7.1 BNL firmware deliverables:■ Frame
○ module operation and low-level hardware interfaces■ Detector Down-Link
○ link protocols■ Unpacking & Processing
○ rearrange data to optimize trigger processing → unpacking/repacking○ aggregate data across multiple links○ serial-to-time multiplexing
■ MUX→GEP Interface & data transport
Firmware Deliverables
8
WBS 6.8.3
WBS 6.8.3
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Technical Challenges and R&D
9
● High-speed high-density optical-electrical modules■ Phase-I electronics rely on Broadcom/Avago miniPODs
○ 12 simplex links up to 14 Gb/s
■ HL-LHC requires similar density but with ~25 Gb/s○ technology starting to become available in industry○ R&D required for very high-density PCB configurations○ see risk discussion for additional details
● Data aggregation and time multiplexing■ trades latency for increased bandwidth
○ asynchronous with beam clock
■ maximize throughput while maintaining stability under adverse conditions○ noise bursts, intermittent links, etc
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Technical Details: FELIX
10
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Overview: FELIX in Phase-I
11
Ethernet
Ethernet
Adapted fromF.SchreuderSlides at TWEPP17
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Overview: FELIX in HL-LHC
12
Ethernet
Ethernet
Adapted fromF.SchreuderSlides at TWEPP17
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Deliverable Overview
13
• FELIX is a system of servers and software that interface the ATLAS sub-detectors to the DAQ system. The FE electronics are connected through custom I/O cards (housed in the servers) which make up the hardware deliverable for this WBS.
US Institutes involved: ANL, BNL
• An upgrade of FELIX for Phase1▪ Higher bandwidth (~ x2)▪ Higher trigger rate (~x10)▪ Interface to the new TTC system (LTI)▪ Possibility of subsystem-specific options
(e.g. tracker calibration)
• Key features▪ 16-lane PCIe Gen-4 (~200 Gb/s)▪ Will support up to 48 links ▪ Several input protocols will be supported
• ~575 FELIX cards needed for ATLAS▪ BNL will deliver 110 cards
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Deliverable Overview
14
From the TDAQ TDR
The ITk alone takes up ~55% of the number of FELIX cards required for ATLAS.
An early R&D task is to demonstrate that FELIX meets the requirements for the ITk.
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Deliverable Overview
• Hardware: 220 FELIX cards▪ BNL: schematic design and PCB layout
• Firmware modules▪ ANL: interfaces to TTC and busy system
▪ BNL: interfaces for different front-ends
• Production board testing: ANL and BNL
15
Block diagrams from Phase-I FELIX prototype
BNL
ANL
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Technical Challenges
• Primary challenges▪ Hardware design is more complicated than Phase-I due to higher PCIe bandwidth
and higher input data rate
▪ Holding FPGA cost down while keeping up with increased data routing capability
▪ TTC-PON is a new timing system from CERN to which we must interface.
16
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
R&D Status
• R&D Accomplishments to date▪ Pixel IBL stave readout test (FE-I4B chip via VLDB cards and GBT links) with Phase-I
FELIX completed (Nov 2017)
17
Occupancy maps from threshold scan
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
R&D Status
• R&D Accomplishments to date▪ Achieved strip module readout (HCC130, ABC130)
with FELIX. Calibration test and gain measurements performed. (Jan 2018)
18
Distribution of thresholds before and after calibration
ABC130Gain distribution
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
R&D Status and Plans
• R&D Goals/Plans to Pre-Construction▪ CY18: continue ITk strips and pixels readout tests with Phase-I FELIX
o We are about 50% complete on our schedule for CY18
▪ CY19-20: design and evaluation test of FELIX demonstrator board, focusing on basic functionality of hardware, firmware, software and integration with ITk.
▪ CY21-22: two iterations of prototype boards
• Key decisions▪ FPGA selection: Dec 2022
19
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Project Management
20
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
WBS and Management Structure
21
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
L3 Project Management
22
● Management Structure■ Jinlong Zhang (ANL) deputy: Michael Begel (BNL)
■ 6.7.1 L3: Michael Begel■ 6.7.1 IC: Shaochun Tang
■ 6.7.3 L3: George Redlinger■ 6.7.3 IC: Kai Chen
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Bio Sketch of L3 Manager
● U.S. ATLAS HL-LHC Project■ deputy L2 manager for DAQ and Data Handling (WBS 6.7)■ L3 manager and CAM for Global Common Module (WBS 6.7.1)■ EVMS training completed in July 2016
● U.S. ATLAS■ Phase-1 Program Management Group■ Joint Integration Group
● International ATLAS■ co-convener of the TDAQ Global Trigger
○ Global Common Module & Data Aggregation are key deliverables
■ collaborator on ATLAS since 2008■ member of the extended Trigger/DAQ Steering Group■ manager for the Phase-I Upgrade Global Feature Extractor (gFEX)
in Level-1 Calorimeter Trigger
23
I have been working in Trigger/DAQ since 1989 with key leadership, operations, and production roles on E706, DZero, and ATLAS.
I have served on many ATLAS review panels as well as reviewing other experiments and programs.
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Bio Sketch of L3 Manager
• George Redlinger, staff physicist, BNL
• Years on ATLAS: 12
• On ATLAS I have worked mainly on physics analysis till Summer 2017
• On BNL-E949 (1999-2003) I was part of a team of 3 scientists responsible for trigger and DAQ. Supervised the BNL team comprising one EE and 3 technicians.
• For the KOPIO experiment (2003-2005) proposed at BNL, I was involved in DAQ design/R&D (bench tests of PCIe and Infiniband) and was the L2 manager for DAQ for the NSF proposal.
• I have also worked on the construction of the original CDF trigger, construction/installation/commissioning of the BNL-E787 low-mass drift chamber, and built a lead-scintillator photon veto system for E787
24
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
● BNL’s Omega Group contributions to ATLAS:■ Original Construction
○ LAr Calorimeter & Muon Detector○ Electronics, Trigger, Software○ Technical Coordination○ Leadership & Management○ System Integration○ Installation & Commissioning○ Operations
■ Phase-I Upgrade○ LAr Calorimeter electronics○ Muon ASIC and electronics○ Calorimeter trigger electronics○ Data Acquisition electronics○ Leadership & Management○ System Integration○ Installation & Commissioning
Experience of Team
25
■ HL-LHC Upgrade○ ITk Silicon Strip Tracker○ LAr Calorimeter ASIC and electronics○ Trigger & Data Acquisition electronics○ Technical Coordination○ Leadership & Management○ System Integration○ Installation & Commissioning
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
● Omega Group has a deep pool of talent in electronics■ synergistic development of electronics for LAr & Trigger/DAQ
○ common elements and interfaces ○ engineers learn about each others boards
● weekly meeting to discuss detailed status and technical challenges
● all boards internally reviewed by group prior to
BNL Instrumentation Division review ● mitigation against single-points-of-failure
■ work closely with BNL Muon and ITk Strip efforts
■ focus on overall system optimization for integrated detector readout○ gFEX used as data source & sink for FELIX integration○ FELIX integration with ITk Pixel and Strip ASICs
● Additional engineering available from Instrumentation Division■ collaborated on Original Construction and in both Phase-I and HL-LHC Upgrades
Personnel
26
TDAQ
Phase-IgFEX
Phase-IIGCM
TDAQ
Phase-IFELIX
Phase-IIFELIX
LAr
Phase-ILTDB
Phase-IIFEB2
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
ES&H
27
● Safety is of the highest priority within the Project■ Work at BNL adheres strictly to our ES&H policy
■ Our ES&H officers:○ periodically inspect our laboratories and offices○ provide oversight and advice○ interface with CERN ES&H
● Main hazards for this deliverable■ Electrical Benchtop
○ low voltage, soldering irons, noise, etc■ All work done in compliance with safety policies at BNL or CERN
Position Name
HL-LHC Point of Contact Achim Franz
Lori Stiegler (acting)
Omega Group Gerrit van Nieuwenhuizen
https://www.bnl.gov/esh
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Cost and Schedule
28
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
ATLAS L1Calo gFEX ModuleArchitecture:
ATCA blade3 VU9P & 1 ZU1935 miniPODs with 420 fibers250 W with Phase-I firmware
Maximum capability:input: 3.9 Tb/s on 312 fibersoutput: 1.4 Tb/s on 108 fibersinter-FPGA: 1.6 Tb/s (high-speed & low-speed)
Basis of Estimate
29
● GCM design based on Global Feature Extractor Module (gFEX)
■ ATLAS Phase-I L1 Calorimeter Trigger Module○ full calorimeter on a single module!○ capable of 3.9 Tb/s input rate
■ conceived, designed, developed, prototyped, and produced at BNL○ collaboration of BNL, Chicago, Indiana,
Lund, Oregon, Pittsburgh, Stockholm ○ Installation & Commissioning on-going!
■ controlled by U.S. ATLAS Phase-I Upgrade Project○ two L3 and CAMs: Module & Algorithms
25.6 Gb/s in PCB
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Basis of Estimate
30
• Material cost and schedule estimates are based on the experience of the Phase1 FELIX project
• M&S cost is dominated by the FELIX cards▪ Estimation of cost of components, PCBs and assembly come from a recent Bill of
Materials for the pre-production of the Phase1 FELIX card▪ Choice of FPGA and optical modules is still ongoing. Cost estimate is based on the
choices for the Phase1 FELIX
• Labor for hardware, firmware and integration testing based on adjusting Phase-I numbers
▪ See BOE for details
• Cost of travel to CERN based on recent experience
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Resource Optimization
31
● BNL is the most efficient and cost-effective institute for DAQ■ designed and produced the enabling technologies
○ same skilled engineers contributing to Global Trigger & FELIX○ no comparable commercial PCI or ATCA modules with large number
of high-speed optical links
■ talented group of experienced hardware & firmware designers○ Phase-I gFEX, FELIX, LTDB, ADDC and many other electronics modules○ synergistic development with other ATLAS electronics○ expertise in system integration ○ additional skilled labor available from BNL Instrumentation Division if needed
■ strong collaborations with key U.S. and International institutes
■ significant project management experience for similar deliverables
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
● Labor estimated in analogy with Phase-I gFEX module■ 1.0 FTE/year engineering of GCM hardware■ 0.5 FTE/year engineering of GCM Frame firmware■ 0.5 FTE/year engineering of Data Aggregation firmware
Effort Estimate
32
BNL is sole institute contributing to WBS 6.7.1.
0.2 FTE/year of uncosted scientist (me) supported by Research Program forL3 Management
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Effort Estimate
33
Uncosted labor of 0.2 FTE/year (funded by the Research Program) forL3 project management
Labor estimated in analogy with Phase-I FELIX
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Closing Remarks
34
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Closing Remarks
35
● BNL is taking a leading role in designing significant parts of the hardware and firmware of Global Trigger and FELIX■ hardware based on existing work (gFEX & FELIX) for ATLAS Phase-I TDAQ Upgrade■ schedule resource loaded & risks assessed
● Areas of risk have been identified. We have confidence in the workflow and the estimates of costs and uncertainties.
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Backup
36
Michael Begel
US ATLAS HL-LHC CD-1, May 7-9, 2018
Global Trigger for HL-LHC
37
WBS 6.7.1 — BNLConceptual block diagram of Global Trigger firmware and software for estimation of resource needs
top related