Status of the LLTT(TPU) project

G.Punzi (Pisa)on behalf of the LLTT group

Meeting with INFN referees20/3/2014

Brief recap- Feb. 2013: LHCb Trigger workshop: First LHCb presentation

- June 2013: Feasibility study presented @LLT workshop- Received an official list of questions from management

- 4 Oct 2013: LLT workshop: Answers to questions + simulation of a baseline system- Asked to be reviewed to become a project for the upgrade

- LHCb outlook for the online evolves considerably

- December 2013: Presentation to LHCb week.

- 1-2 Feb. 2014: Presentations to LHCb Trigger Workshop

- 1 Mar 2014: Talk at international instrumentation conference (INSTR-14)

- 10 Mar 2014: Internal note presented for review

- 18 Mar 2014: Presentation to Technical Board meeting

TODAY: presentation to INFN referees.

- 31 Mar 2014: Presentation to LHCb external review committee

Effects of changes in readout design DAQ structure evolved to bi-directional EB Baseline readout has moved FPGA cards to EB LLT hardware shrunk, most (or all) functions moved into EB

CPUs (“software LLT”) BIG strategy choice: push investments upward

But HLT wants to keep a “safety net” (LLT) LLTT more substantial hardware:

→ Track Processing Unit (TPU), connected to EB Regarded by trigger group mostly as

HLT co-processor, or pre-processor Raises the bar considerably !

Can still fuel a software-LLTT functionality Safety net as well...

Hardware Status

Architecture

Cellular

Engines

switching network

Fitter

Tracking layers

Separate trigger-DAQ path

Custom switching networkdelivers hits to appropriate cells

Data organizedby cell coordinates

Blocks of cellularprocessors

Track finding and parameter determination

To DAQ

For each cellular unit in the parameter space (u,v) calculate a weighed response summing over all hits and all layers.

Tracks are peaking structures in the parameter space. Find a track as clusters of excited cells

Trigger&Tracking Workshop- M.J. Morello

Basic Principles

Hit delivery by the switching logic

Hits must be delivered only to the cell that need them (they can be more than one)

The switch network “knows” where to deliver hits

All information about the network of connections is embedded in the network via distributed LUTs

Cellular engine

Performs calculation of weights for a hit into a cell

Deals with surrounding cells as well.

Handles time-skew between events

In second stage performs local clustering in parallel, and queues results to output

Track parameter estimation by cluster Center-of-Mass

-1,0,1

-1,0,1

REG REG

16

16

22

REG

PIPELINED DIVIDER

(ENGINE OUTPUT)

z +

p -

z +

p 0

z +

p +z 0

p -

z 0

p 0

z 0

p +z -

p -

z -

p 0

z -

p +

d

+0

-

EN

GIN

E

EN

GIN

E

EN

GIN

E

EN

GIN

E

EN

GIN

E

EN

GIN

E

EN

GIN

E

EN

GIN

E

EN

GIN

E

EN

GIN

E

CoM UNIT

REQ ACK DATA

EN

GIN

E

EN

GIN

E

MUX MUX

MUX

Due to data reduction out of the engine, a 1:12 ratio is sufficient to keep up with the data flow

Final parameter determination can be done of EB CPUs to achieve full “offline-compliance”

Studies of Stratix-V capacityTPU completely implemented in firmware

All main components: - Switch

- Engines

- CoM

implemented in VHDL and placed in FPGA

Fit ~750 engines/chip on Stratix-V• exact number depends on details

(time-ordering of pixel data, etc.)

Arria 10 allows double the logic at the same price, with lower power consumption

Simulation and Timing

CELL

Hit_dataLAY

XY

addressIntersect_x_y d_x

d_y d_x^2 d_y^2

sum_square weight

Ready

Exceed 350 MHz clock freq

40MHz throughputTotal latency <1µs !

40MHz throughputTotal latency <1µs !

(Not accounting for I/O)

Much better than AM : Originally intended as

“Low Level Track Trigger”

Tracking ConfigurationVELO+UT, 16+2 layers

- Split into two separate telescopes for ease of cabling- Covers longable tracks

Layer configurations

Layer configuration acceptances

Integration in the DAQ

to Event Builder

to Event Builder

- ATC40 scheme (not the baseline anymore)

- Shows the map of connections

- Additional optical links needed to copy data to the TPU cards

TPU

TPU integrated in the EB

TPU appears to the EB as an additional “virtual detector” producing tracks

Data flow inside EB

- Local CPUs can be used to refine FPGA output - Availability of TRACKS in the Event Builder

– Can control rate by confirming LLT muon(hadron) with stiff track– In the “partial reconstruction” scheme, could have HLT1 inside EB

TPU behaves as a virtual “track detector”

Tracks inpre-EB

Tracks inpost-EB

Small flows in TPU boxes

Lab test with TEL62- We have a plan for testing the retina algorithm with real FPGAs

(Stratix-3), in a simplified configuration.

- This is lower-speed, but helps us demonstrate that we can put together and operate a complete system

- We exploit TEL62 boards, that are compatible with current LHCb DAQ, and can be easily inserted in the system (agreement with local DAQ experts)

- TEL62 boards have been ordered together with NA62 order, and will arrive soon (~month).

- Pisa has lab space for a bench test. TEL62 is used for both sequence generation and “retina” implementation.

- Work ongoing in Pisa on connection boards and logistics.

Preparations for test on silicon telescope @Milano

- Second stage of testing planned with CR in a Si telescope being built in Milano

- Details in UT talk yesterday.

Document presented to LHCb Technical Board10/3/2014

Performance parameters

Simulation

• At least 3 VELO hits in the last 8 VELO stations • At least 1 hit in each axial UT layer.

Fiducial region:|u|<0.35, |v|<0.35 (about theta< 50mrad);|z| < 15 cm;Electron rejection.

Some details on LHCb simulation used:Ebeam = 7TeVnu = 7.6 (L=2x1033) and nu=11.4 (L=3x1033), bunch crossing: 25 ns, with spilloverGeometry: DDDB : dddb-20131025, CONDDB : sim-20130830-vc-md10.VeloUT offline reconstruction Brunel v44r9 with default setting.Performances on small angle telescope with 8VELO +2UT.

Mapping of detector to receptor cell array

Intersection of “base tracks” with detectors gives a map of “nerve endings”. This encodes the information about the geometry

Every hit on the detector produces a signal on nearby receptors, depending on distance

(I skip on several subtleties. For instance, effective operation require distribution to be non-uniform)

Not unlike the distribution of photoreceptors in visual system – but it is all virtual in our case, that is, implemented in the internal LUT of the system.

Simulated full LHCb events (µ=7.6)

Out of acceptance

Generated

Used ~45,000 cell enginesC++ code, can be inserted in standard analysis code

Efficiency/Uniformity

p , pT

Efficiency/Uniformity

z , IP

Efficiency/Uniformity

(u,v) ~ (θx , θ

y)

Momentum resolution

σk = 0.0102 σ

k = 0.0126

Physics performanceand robustness

COSTING

Detailed cost estimate from the online group

Estimated at current prices: 940 kCHF Does not account for savings from moving to Arria-10 Assumes using identical boxes to the EB for simplicity

– Some further savings still possible

Is the TPU cost effective, TODAY ?Timing of piece of codeyielding the performanceswe have been comparing to:3.8 ms (standalone CPU-2012) It was later understood thatthis piece of code performs further extra work(backward VELO layers)

We have no piece of codedoing exactly the TPU workon the same sample,with the same performance

Various estimates: (16/26)*2.3+1.5 = 2.9 ms

(%GEC) 60% * 3.8 = 2.3 ms3.8ms -(VELO10) = 2.4 ms

Cost of (naked) CPU: ~120SWF/core→ 1ms@40MHz = 4.8 MCHF TPU equivalent: 10 ÷15 MCHF

Is the TPU cost effective, TODAY ?Timing of piece of codeyielding the performanceswe have been comparing to:3.8 ms (standalone CPU-2012) It was later understood thatthis piece of code performs further extra work(backward VELO layers)

We have no piece of codedoing exactly the TPU workon the same sample,with the same performance

Various estimates: (16/26)*2.3+1.5 = 2.9 ms

(%GEC) 60% * 3.8 = 2.3 ms3.8ms -(VELO10) = 2.4 ms

TPU clearly cost-effectivesolution at present time

Cost of (naked) CPU: ~120SWF/core→ 1ms@40MHz = 4.8 MCHF TPU equivalent: 10 ÷15 MCHF

Other similar works...

Projections to 2020

“It is always difficult to make predictions,especially about the future”

– Yogi Berra

Timings presented by HLT group

How the HLT group plans to use the TPU

Does not include:- Multicore inefficiency- Data/MC effects

CPU cost projections from online group

Assumptions behind the 8ms

- TPU price 2020 = 2014- CPU price drop 16x- No inefficiency factor for 400 jobs/node- Additional 2x to CPU for other uses- Full cost of TPU vs. “scalable” cost of CPU

Conclusions of TB (18/3/2014)

Summary

• We designed a system capable of track reconstruction at 40MHz with offline-like performance and ~1µs latencies.

• Cost of TPU is an order of magnitude smaller than today's CPU solutions

• Projections to the upgrade era made by the HLT group and online group predict that the CPU solution will become more convenient.Based on some assumptions.

• The TB recommended a CPU-only solution as baseline for TDR.

People

• A. Abba (MI)• F. Bedeschi (PI)• F. Caponio (MI)• M. Citterio (MI)• D. Corbino (CERN)• A. Cusimano (MI)• A. Geraci (MI)• S. Leo (PI)• F.Lionetto (PI) • P.Marino (PI)

• M.J. Morello (PI)• N. Neri (MI)• A. Piucci (PI)• G. Punzi (PI)• L. Ristori (PI)• F. Ruffini (PI)• F. Spinella (PI)• S.Stracka (PI) • D.Tonelli (CERN)• J.Walsh (PI)

Many thanks to all people who contributedto the development of this design

BACKUP

Basic principle

3/17/14

44

We inject real hits (xr,yr)k in the detector layers. For each cellular unit ith in the parameter space (u,v) calculate Ri response summing over all hits and all layers.

Tracks are peaking structures in the parameter space. Find a track as clusters of excited cells

Trigger&Tracking Workshop- M.J. Morello

Tracking EfficiencyReconstructed “offline” VELO+UT tracks using Official LHCb-MC

Bs→φφ with mu=7.6.

— Require on offline reconstructed tracks

p >3 GeV/c

pT > 500 MeV/c

— and a geometrical acceptance (retina acceptance)

20 < theta < 60 mrad

Found that ~95% of offline tracks have a compatible match within the geometric acceptance of our track processor.

— All VELO and UT hits without any requirements sent to the LLTT.

Simulated full LHCb events (µ=7.6)

Out of acceptance

Out of acceptance

Generated

Full LHCb-MC

NB: it is simulable with 100% accuracy , C++ code available to users

Software simulation 4/10/13: Benchmark study on 6 VELOPIX layers + 2 UT planes

Used 36,000 cell units in (r,j) parameters

(r,j) polar coordinates on virtual plane of track intersection.

Mapping using LHCb-MC ParticleGun.

Tracks from Official Production Bs→φφ LHCb-MC.

L=2 × 1033 cm-2s-1, sqrt(s)=14 TeV, mu=7.6

DDDBtag    =“dddb-20130408“.

CondDBtag  = “simcond-20121001-vc-md100”

No kinematics cuts applied.

No requirement on hits.

Layer configurations

Layer configuration acceptances