daq for sensor r&d at fnal · 2014. 10. 28. · captan: applications 5 ryan a. rivera | daq for...

DAQ for Sensor R&D at FNAL

Ryan A. Rivera

2014 Detector R&D DOE Review

29 October, 2014

Comprehensive Approach to Tracking Detector R&D

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 2

Sensor

Front End

Data Transmission

L1 Trigger

Radiation hardness in new sensors (3D columnar + Diamond)

New ASIC developments to give integrated track segment information

Multi-wavelength optical DAQ

Track trigger

Track Fitting FPGA, GPU’s and Associative Memory based on xTCA

Creating a tracking and trigger system that can withstand the projected HL-LHC luminosities is perhaps the most important detector challenge in the field of High Energy Physics. There must be a comprehensive approach (emphasis on FNAL/SCD contributions):


• Motivation simple DAQ

It is 6” x 6” and, for many systems, the only external connections needed are a 3.5V power supply and a standard Ethernet cable.

flexible DAQ

The user can stack compatible boards in different combinations to give unique functionality.

scalable DAQ

In addition to the vertical stacking, the stacks can be repeated arbitrarily and connected with one or many PCs in an Ethernet network.

Compact And Programmable daTa

Acquisition Node (CAPTAN )


CAPTAN User Template

OPTICAL BUS

COOLING CHANNEL VERTICAL BUS

LATERAL BUS MOUNTING HOLE

ELECTRONICS

• The CAPTAN architecture consists of a few core boards but is intended to be augmented by custom boards designed and built by users.

CAPTAN: Applications


• Developed in 2008/2009, the CAPTAN system was

designed to handle common data acquisition, control,

and processing challenges within high energy physics.

• Examples of such applications are tracker readout

systems, R&D test stands, and parallel data processing.

• As the CAPTAN system is a modular system it can be

used for a wide range of applications, from very small to

very large.

• Quite a number of groups at Fermilab and other

institutes in the US, China, and Europe have acquired

the system for their test-stands. We work with them to

provide hardware and software support.

• We are currently working on the next generation

CAPTAN with a new FPGA.


QIE10 Single Event Upset Testing


Tevatron: T980 Crystal Collimation Telescope


CAPTAN Stacks Power Supply DUT HV Supply Pixel Telescope

Frame

Ethernet Router Scintillator

The CAPTAN pixel telescope is 8 silicon pixel planes leftover from CMS, with

space for 2-4 DUTs in the middle. Pixel size is 100 µm x 150 µm. Data acquisition

with the CAPTAN system.

Fermilab Test Beam Facility


Fermilab Test Beam continued: • Old pixel telescope DAQ is based on CAPTAN

• Triggered, 2.5cm2 coverage, and 8µm track resolution • New strip telescope is based on CAPTAN too.

• Dead-timeless, 16cm2 coverage, and 5µm track resolution • For the last 6 years CAPTAN supported all versions of the CMS pixel chip • Recently tested the VIPIC Read Out Chip from FNAL/PPD using CAPTAN


• Telescope is part of the FTBF facility and has been used by many experiments as a high resolution tracking tool to characterize different Devices Under Test (DUTs)

• List of Fermi Test Beam Facility Experiments using the Telescope

T992 - Radiation-Hard Sensors for the HL-LHC (ongoing)

T995 - Scintillator Muon/Tail Catcher R&D with SiPM Readout

T979 - Fast Timing Counters – PSEC Collaboration

T1004 - Total Absorption Dual Readout Calorimetry R&D

T1006 - Response and Uniformity Studies of Directly Coupled Tiles

T1017 - CIRTE (COUPP Iodine Recoil Threshold Experiment)

T958 – FP420 Fast Timing Group

T1038 – PHENIX Muon Piston Calorimeter

• We will continue to support test beam experiments that want to use the pixel telescope

FTBF Telescope User Community


FNAL/PPD Collaboration

• 3D ASIC test stand and test beam efforts, another run in November scheduled.

• SCREAM (Single CCD Readout Module): compact low-cost, CCD readout system. Using CAPTAN firmware/software


• New CMS pixel digital ROC Test Stand • Distributed to US collaborators in Colorado, Purdue and to

Italian collaborators in Milan, Lecce and Torino.

Collaboration with CMS


Leveraging Event Building and xTCA for CMS

• Data Processing in an FPGA • Receives all CMS Calorimeter data

• 276 Gbps in to a single FPGA (Xilinx Virtex 7) • Must aggregate/summarize information and

pass to next stage in < 400ns • 20 Gbps out from FPGA


• xTCA (Telecommunications Computing Architecture) Spec put forth by PICMG (PCI Industrial Computer

Manufacturers Group: a consortium of over 250 companies).

xTCA encompasses MicroTCA and ATCA.

• Large experiments are already using xTCA or planning to: – CMS

– ATLAS

– LHCb

– LBNE

xTCA


• Completed a test beam project at Fermilab

Real-time event assembly conducted in MicroTCA form factor.

CAPTAN and xTCA


• MicroTCA.4 Standard

Specification finalized in 2012 for the physics community.

We are collaborating with CERN to use MicroTCA cards developed in Europe: GLIB, MP7, FC7.

8U 12-slot MTCA.4 shelf

MicroTCA effort


ATCA effort

12U 14-slot ATCA shelf

• ATCA

Advanced Telecommunications Computing Architecture

• More space for I/O

• Possible collaboration with SLAC on RCE development for LBNE

• Work led by Ted Liu


Other Data Processing Platforms


• A set of applications running extensible software components to be customized by experimenters to create a DAQ system.

• Lariat, DarkSide-50, LBNE and Mu2e experiments; partial use in Nova. Chosen for “Off-the-Shelf” DAQ.

• Recompilation is not needed in to change parameters – done through configuration scripts that load plug-ins

• artdaq-demo allows users to get a toy artdaq-based DAQ system up-and-running from scratch in about 10 minutes

What is artdaq?


• For CAPTAN: Update FPGA

Further develop software on LINUX - web based graphical user interface using HTML5 and JavaScript.

Build out artdaq support

Exploit parallel processing power and mTCA integration

Provide user support for their applications

Work with possible users on new applications

Next Steps for DAQ for Sensor R&D


• For xTCA: Continue to explore big data

applications

There are possibilities for event aggregating and tracking based trigger systems.

• For parallel processing: Compare xTCA with GPU and co-

processor fronts

Intel PHI

CUDA

OpenCL

Next Steps cont.


• For DAQ systems: Proceed with “Off-the-Shelf” DAQ concept

This proposal is intended to demonstrate the feasibility of a low-cost, high-bandwidth, commercial approach to data acquisition based on standard networking technology.

We can no longer afford the costs associated with developing new back-end systems from scratch for each new experiment. Experiments are asking for an “off-the-shelf”, commodity solution. The Computing Sector is all about “service” and perhaps it’s time FNAL consider a “DAQ as a service” approach.

Effort will leverage CAPTAN, artdaq, and test beam experience.

Next Steps cont.

Conclusion


Sensor

Front End

Data Transmission

L1 Trigger

Radiation hardness in new sensors (3D columnar + Diamond)

New ASIC developments to give integrated track segment information

Multi-wavelength optical DAQ

Track trigger

Track Fitting FPGA, GPU’s and Associative Memory based on xTCA

The efforts related to DAQ for Sensor R&D are intimately tied into every piece of the puzzle for a tracking and trigger system that can withstand the projected HL-LHC luminosities .


Thank you.


Backup slides…

CAPTAN Core Boards

“Green Board” NPCB – Node Processing and Control Board “Blue Board”

DCB – Data Conversion Board

“Red Board” PDB – Power Distribution Board


Design Philosophy

• Commercial Slow Control hardware and software is used for development – needed early, can’t wait for custom HW/SW – hopefully, utility extends into production and running – LabVIEW or EPICS

• Embedded Processors – use commercial modules if possible to allow early software

development – embedded Ethernet and HTTP with eye towards debugging ease


artdaq

Best Future DAQ

R&D Avenues

• Ethernet/InfiniBand: InfiniBand prevalent as the interconnect between nodes

of the highest performing super computers

• Off-the-shelf components

• xTCA

• GPUs: Fastest computer in the world, Tianhe-1A in China uses 7,168

NVidia Fermi GPUs and 14,336 Intel Xeon CPUs. Would require 50,000

CPUs for same performance with CPUs alone.

• Optics: Commonly 100 Gbps with distances over 100 m. Becoming

interconnect of choice in high speed data centers and HPC clusters. There

are 40 Gbps and 100 Gbps optical Ethernet standards.

• FPGAs

• EPICS: The tool is designed to help develop systems which often feature

large numbers of networked computers providing control and feedback.

Data Acquisition Systems - Ryan Rivera 29 Detector R&D

Retreat


DAQ Program Resource Issues

• Limited manpower – Has hindered xTCA effort

• xTCA development accessibility – Community MMC design effort

• FPGA power/expense/expertise

31

Conclusions

• For medium-small project and test stands we think that the CAPTAN should be supported as any other commercial option.

• We believe that especially for small DAQ most of the overhead comes from designing of the software and firmware. Having an integrated software/firmware with an infrastructure and a user configurable part must be the goal.

• The advantages that we see with the CAPTAN is the fact that the board is really simple with a direct Ethernet connection and a lot of IO.

• For any DAQ system we have done in the past we always had to create a custom board that was the interface between the detector and the FPGA.

• We would like to make a new CAPTAN version and to support it as an Internet Of Things DAQ board using ARTDAQ.

daq for sensor r&d at fnal · 2014. 10. 28. · captan: applications 5 ryan a. rivera | daq for...

Documents