concepts for detector data acquisition interface ... (hls/matlab) standard reusable ... - per vc...

Concepts for Detector Data AcquisitionRyan HerbstDepartment Head, Advanced Electronics Systems

Trigger/Streaming Readout Workshop - January 27, 2017

([email protected])

SLAC TID-AIRTechnology Innovation Directorate

Advanced Instrumentation for Research Division

TID-AIR

2

Outline

Trigger/Streaming Workshop - January 27, 2017

● SLAC Overview● Detector Front End

○ PGP & RSSI (Reliable UDP)● Inline Data Processing

○ RCE and Its Applications● Detector Data Handling● Detector-DAQ Interface

○ Timing Interface○ Readout

● Online Analysis● Accelerator Controls● Software

3

Overview of SLAC Structure

LSST

ATLAS

Business &Operations

Fermi-GLAST Gamma-Ray Telescope

LCLS

Biosciences, Chemical Science, Computer ScienceElementary Particle Physics, High Energy Density ScienceMaterial Science, Particle Astrophysics & Cosmology


1.65 m

4

SLAC’s Core Instrumentation R&D Capability in AIR provides Systems & Components for SLAC & National/International Projects

Technology Innovation Directorate

• Provides engineering for SLAC directorates

• All SLAC developed LCLS/LCLS-II detectors

• HEP: ATLAS, CDMS, HPS, etc.

• LSST, CMB, Fermi …• Non-SLAC Projects

LCLS Directorate

SSRL Directorate

Science Directorate

LCLS-II Project

LSST Project

Accelerator Directorate

Super CDMS Project Advanced

Instrumentation for Research

(AIR)Division

Gunther Haller

LSST: World’s largest digital camera

Work for OthersEuXFEL, KEK,

LANL, commercial companies incl. start-ups, etc.


TID-AIR

5

Accelerator & DAQ Overview


DetectorFront End

Inline Processing DAQ

Gateway

OnlineAnalysis

OnlineData Cache

To Offline

LocalTimingGateway

AcceleratorControls & Timing

● Items in blue should be provided by facility● Allows for a common set of APIs

○ Detector control○ Scripting API○ Online analysis plug ins○ Offlines analysis APIs

TID-AIR

6

Detector Front End


DetectorFront End


Gateway

OnlineAnalysis

OnlineData Cache

To Offline

LocalTimingGateway


● A variety of detectors must be supported by facility DAQ system

● Helps to have a small set of standard interfaces

○ PGP (SLAC)○ Aurora (Xilinx)○ Cameralink○ USB○ Ethernet

TID-AIR

7

Detector Front EndKPIX ASIC For ILC Calorimetry & Tracking


KPIX ASIC bump bonded to Si sensor (6 inch wafer)1024 pixel sensor

Readout motherboardsupporting 30-layers1Gbps Ethernet optical readout

Readout flex cable bonded to Si sensorfor calorimeter

● Power pulsing● Auto ranging● Timestamped analog memory for bunch interactions● Designed for bunch structure with multiple pulses

followed by readout cycle during idle time

Test beam single electron shower event

Minimal interface:CLK, RST, CMD, DOUT

TID-AIR

8

Typical Front End Firmware


Reg Control(Master)

AXIL Crossbar

ADC Readout(Slave)

ADC Config(Slave)

Stream Tester(Slave)

EPIX Regs

(Slave)

Microblaze(Master)

SPI

LVDSPGPOr

Ethernet

SACI

VC

ASIC Acquisition

Control

To ASICs

ASIC Readout Control

VC

Embedded processor allows for automatic ADC & detector sample point tuning

High speed link for data path & timing/event delivery. Standard facility interface.

Camera specific data processing, reorganization & compression (HLS/Matlab)

Standard reusable blocks for accelerated development

Common flexible ASIC configuration interface

TID-AIRPGP2B


• PGP = Pretty Good Protocol- PGP2 = Version 2- PGP2B = Version 2 with AXI-Stream interface (compatible with PGP2)

• Architecture independent – can be deployed on any 8B/10B SERDES- Copper or optical- Speed defined by SERDES and internal data path

• Unlimited frame size• 4 virtual channels, each with separate firmware interface

- 4 frames in flight at any given time- Avoid head of line blocking for configuration messages

• Built in flow control support• Cell based protocol

- Large frames segmented into 512byte cells for transport- Guaranteed cell ordering- CRC protected cells, errors forwarded to end point - Per VC arbitration at the cell level

• Low overhead (96% efficient after 8B/10B conversion)• Can be used to transport timing and trigger

- Low latency, deterministic trigger transport interface• Unidirectional & bidirectional protocol

- Example: 1 downstream link and 4 upstream links- Upstream and downstream links can be different line rates

• Supports lane bonding for wider data path• Standard interface for many experiments

- LCLS, SID, LSST, ATLAS & Others

SLAC PGPCard4 lane x 3.125GbpsFull duplex 9.6Gbps

TID-AIRRSSI


Application FPGAApplication FPGAIOC CPU

• Reliable SLAC streaming interface• Reliable message delivery over UDP: RSSI

• Firmware to firmware as well as firmware to software• Based upon RUDP protocol

- RFC-908, RFC-1151, draft-ietf-sigtran-reliable-udp-00• Error detection and retransmission• Flow control support• Additional features added to support flow control• Facilitates breakup of large transfers into MTU sized messages

IP/UDP

Ethernet Switched Network

RSSI

FirmwareApplication

IP/UDP

RSSI

FirmwareApplication

IP/UDP

RSSI

SoftwareApplication

TID-AIR

11

Inline Data Processing


DetectorFront End


Gateway

OnlineAnalysis

OnlineData Cache

To Offline

LocalTimingGateway


● Inline data processing for larger cameras and complex front ends

○ Additional level of processing between detector front end and back end DAQ

○ Can provide data reorganization or image pre-processing

○ Can generate trigger decisions for trigger based systems

TID-AIR

12

Optional Inline Data ProcessingHigh Density / High Performance Processing In ATCA

On board 10G Ethernet switch with 10G to each processing FPGASupports 14 slot full mesh backplane interconnect!

Data processing daughter board with dual Zynq 045 FPGAs

(modular for camera integration)

Front panel Ethernet2 x 4, 10-GE SFP+

Application specific RTM for experiment interfaces 96 High Speed bi-dir links to SOCs


SOC platform combines stable base firmware / sw with application specific cores

● HLS for C based algorithms & compression

● Matlab for RF processing

Deployed in numerous experiments● LSST● Heavy Photon Search● LSST● Dune● ATLAS Muon● Koto

High performance platform with 9 clustered processing elements (SOC)

● Dual core ARM A-9 processor ● 1GB DDR3 memory● Large FPGA fabric with numerous

DSP processing elements

13

DUNE 35-Ton

• SLAC RCE Platform is readout for 35Ton detector• Fully integrated with timing system• Integrated with ARTDAQ backend system• Successful deployment of zero suppression algorithm

- Written in C++ via Vivado HLS design flow- Support for 128 channels with 25% FPGA utilization- Fully adjustable to optimize performance depending on

S/N conditions- Successful handling of data quality issues

Noisy front end data links Cold ASIC stuck bit problem


14

Heavy Photon Search

Hybrids:Pulse shapingPulse sampling

Buffering

FE boards:Amplification

Analog to digitalHybrid control/power

DAQ PlatformSLAC RCE (ATCA blade)

JLab DAQ

Flex cables:Impedance controlled, low mass signal/bias/control to

hybrids

Power suppliesLow voltageSensor bias

• Integrated with JLAB’s timing and back end DAQ system (CODA)

• Took data during 2015 & 2016• Expect more data runs in 2018

- Add new layer 0 hybridsNew layer 0

hybrid design for 2018 run


TID-AIR

15

Planned Upgrades To RCE Platform

Upgrade current 24 port 10G switch to 96 port 40G capable switch.Support 10Gbps or 40Gps to DPMsSupport 120Gbps front connectionCost reduction and lower power

DPM Upgrade:Upgrade Zynq-7000 to Zynq Ultrscale+ MPSoC

3 layers of processing, CPU, RPU & GPUAdditional processor memory up to 32GBAdd direct attached memory to Fabric (Collaboration with Oxford)

Cost reduction re-spin of COB coincides with core switch upgrade. Less layers & component cost optimization.


TID-AIR

16

Detector Data Handling


DetectorFront End


Gateway

OnlineAnalysis

OnlineData Cache

To Offline

LocalTimingGateway


● Online data cache provides temporary storage for user experiment data

○ Solid state low latency storage○ Cumulative data > 210GBps

● Allows for first level analysis soon after data taking

● Allows users to filter data before moving to costly long term retention or transferring to home institution

Large cameras at high data rates create data handling challenges

● 1Mpixel @ 100Khz ~ 210GBps

TID-AIR

17

Data Path Processing


Detector Segment

Detector Segment

Detector Segment

DAQ Node

Detector Segment

DAQ Node

DAQ Node

DAQ Node


Timestamped beam conditions

Detector segment data

Event data stored in HDF5 format

Event reconstruction & analysis

PC based flash storage allowing direct DMA from receiving blade & low latency data access

TID-AIR

18

Detector - DAQ Interface & Timestamps


DetectorFront End


Gateway

OnlineAnalysis

OnlineData Cache

To Offline

LocalTimingGateway


● Tight coupling to timing system required to properly timestamp detector data

○ Method for applying timestamps depends on the type of detector

○ Custom cameras are more flexible○ Commercial cameras come with

challenges○ Feedback path for multiple camera

coordination (busy, almost full, etc)

TID-AIR

19

Methods Of Timestamping Data For A Variety Of Detector Types


Three models of sensor integration:

1) CommercialSeparation of trigger, data, and timestamp path. Minimal support for busy, almost full. Data & timestamp combined at software level.

2) Custom-IPaths joined by an interface card. Timestamp added in hardware as data arrives at PC. PC card provides buffering, generates almost full etc. Camera uses facility approved protocol.

3) Custom-IISensor receives timing directly. Sensor contains buffering and provides almost full directly to trigger system. Data read out with custom PC card. Camera uses facility approved protocol.

Fast Control& Timing

Sensor CPU

TTL Timestamp

Data

TID-AIR

20

LCLS-2 PGP Interface CardFor Custom Cameras With Fiber Interface


Dual QUAD SFPs for 8 lanes of up to 6.125Gbps

(upgrade to 12Gbps)

LCLS-1 or LCLS-2Timing Interface

● PCI-Express card supports bi-directional links to camera front end○ Hardware handles protocol layers to reduce CPU loading

● Supports bi-direction timing interface supporting busy and trigger feedback○ Timing receiver firmware built into FPGA

● Timestamp and trigger forwarded down optical link with PGP○ Timestamp applied directly on camera

● Firmware can also apply trigger on incoming data○ Gate event acceptance based upon event code / trigger information

● Optical interface supports a number of detector protocols

○ PGP (SLAC)○ Aurora○ CameraLink○ Raw Ethernet○ User Custom!

TID-AIR

21

LCLS-1 / LCLS-2 Timing ReceiverFor Commercial Cameras With TTL Triggers


LCLS-I Timing input (119MHz)

LCLS-II Timing input (186MHz)

TTL Outputs

Receive andtransmit (feedback)

Feedback inputs can be forwarded as “almost full” or L1 Trigger inputs.

Presumes sensor can generate “almost full” and L1 Trigger signals. Sensor “almost full” could otherwise

be modeled but not guaranteed.

FeedbackInputs

Card developed for use in LCLS-I and LCLS-II controlsto interface to non-SLAC electronics.

TID-AIR

22

Online Analysis


DetectorFront End


Gateway

OnlineAnalysis

OnlineData Cache

To Offline

LocalTimingGateway


● Online analysis is extremely important● Provides confidence that experiment is

taking proper data● First feedback about timing, noise and data

quality● Large, high rate detector data bandwidth

creates a challenge

TID-AIR

23

Online Analysis Coordination MinimizesAnalysis Load & Network Traffic


● Facility based online analysis uses local timing system to coordinate online analysis

○ Event codes direct readout machines to send data for analysis

○ Minimizes network traffic and analysis load

● Analysis engines define rules for event selection at user request

● Timing system tags events for analysis by online nodes

● User API allows experiment specific analysis

○ Facility supports generic analysis types

TID-AIR

24

Accelerator Controls


DetectorFront End


Gateway

OnlineAnalysis

OnlineData Cache

To Offline

LocalTimingGateway


● A flexible accelerator controls platform ensure long term maintainability and ease of facility upgrades & expansion

● A common platform architecture for multiple controls sub-systems reduces both development costs and maintenance costs

TID-AIR

25

SLAC High Performance Accelerator Controls


Analog / RFCommon Platform

Architecture

● TID-AIR developed an ATCA based common platform for accelerator controls○ LCLS-2, LCLS-1 and SSRL○ Combines high performance digital processing with low noise RF design

LCLS-II High Performance Controls Systems● Beam Position Monitor (BPM)● Bunch Charge Monitor (BCM)● Bunch Length Monitor (BLEN)● Machine Protection System (MPS)● Timing Delivery System

LCLS-I Controls Upgrade Systems● Beam Position Monitor (BPM)● Bunch Charge Monitor (BCM)● Bunch Length Monitor (BLEN)● Low Level RF (LLRF)

Other Systems & Experiments● SSRL Low Level RF Upgrade● Transition Edge Sensor Detector

TID-AIR

26

ATCA Based Platform ForLCLS-I & LCLS-II High Performance Controls

Trigger/Streaming Workshop - January 27, 2017LCSL-2 BPM board

LCLS-1 LLRF Down Convert

ATCA AMCCarrier CardSupporting 2Analog AMCs

ATCA provides the space, power & cooling required for LCLS-2!

10Gbps EthernetBackplane

TES Readout Electronics

Trigger/Streaming Workshop - January 27, 2017 27

• Each TES element is coupled to a superconducting resonator with a SQUID that results in a resonant frequency dependent on current• Continuous current ramp induces sinusoidal frequency driver• TES sensor current modifies the frequency vs ramp current

• “Comb” of frequencies sent down to TES array• Triplett of frequencies per TES sensor

- One below and above resonant frequency for reference

• ATCA based readout electronics generates comb of frequencies• 1 at TES resonant point (dip in received

frequency)• 2 sideband frequencies for reference

• Design goal of 4K channels per FPGA• 12K total tx and rx frequencies (resonant

plus 2 reference)

EvalBoardManagement

Tree

Rogue

● Provide a system to facilitate hardware development and daq systems for interfacing to hardware

● Support a number of interface technologies between hardware and software

○ Including ones that don’t yet exist● Easy to understand mechanisms for connecting

independent modules together using a set of well defined, easy to understand interfaces

● Allow data paths to exist in independent high performance threads

● Flexible structure for creating a hierarchy of system components

○ Independent of the network and hardware hierarchies

● Direct interface to hardware development○ Generate variable definitions directly from

VHDL/Verilog● Non-professional software engineers must be able

to maintain the system○ Both core rogue components and application

software○ Easy to add functions, device control routines

and run control sequences

PGPVC0

SRPV0

AxiVersion

PGPVC1

AxiPrbsTx

PGPVC2

dataWriter

PRBS Rx

Register Access

PRBSData

MicroBlaze Console

Con

figur

atio

n S

tream

PyrogueMesh EPICS

GUIs Scripts

OtherServers

EPICSClients

RunControl

Lines in red are pure C++ threads & data paths

TID-AIR

29

The End


concepts for detector data acquisition interface ... (hls/matlab) standard reusable ... - per vc...

Documents