a proposal for the readout of the na62 lkr calorimeter

15 April 2008 B.Hallgren, M.Piccini and H.Wendler 1

A Proposal for the Readout of the NA62 A Proposal for the Readout of the NA62

LKr CalorimeterLKr Calorimeter


New NA62 LKr readout

• Preamplifiers (kept cool).

• Transceivers and Cables (faulty devices should be repaired).

• CPD to be upgraded to 1 MHz readout rate (3*106 events/burst?).

• The new CPD (nCPD) must be able to operate together with the present CPD.

• This means that a staged replacement procedure can be implemented module by module.

• The CPD module mechanics is kept.

• Fastbus crates and cooling are kept.

• Optical links (high failure rate no spare parts) are being replaced by the Gigabit Smart Link Module (SLM).

• Data Concentrator to be replaced with Network Switch and FPGA or PC FARM.

using EXISTING HARDWARE as much as possible


Calorimeter Pipeline Digitizer CPD

CPD Motherboard -> remake digital parts but carefully copy the analog layout.

Digital Subcard -> new design.

32xCPDAS cards -> kept

Analog Trigger Subcard


Present Readout of the CPDAS

• The circuit which connects the ADC to the pipeline memory the MIGA gatearray operates together with the address and control signals in the following way:– The memory operation cycle is divided into 2 x 25 ns time

periods performed in parallel on all channels.– The 1st period is used for 2 ADC words that are written into the

memory every 50 ns (Circular Buffer) via the MIGA.– The 2nd period can be used to write or read two words from the

memory (Linear Buffer) via the MIGA.– The readout from the MIGA done is done channel by channel

and takes 64 x 0.8 = 51.2 µs + overhead.


new CPD BLOCK DIAGRAM


• In the new CPD each MIGA is connected to a FPGA (per 4 CPDAS or 8 channels). The readout of 2 words is done all channels in parallel and takes 100 ns. A complete event consisting of 8 words takes 400 ns.

• Transfer of data from the channel FPGA to the module FPGA is done with 8 x LVDS 12 bit buses operating at 80 MHz (using ADC clock).

• An event consisting of 8 x 12 bit words/channel will be transferred in 800 ns (8 channels) from the channel FPGA (cFPGA) to the module FPGA (mFPGA).

• Transmission to the SLM is done with LVDS board 12bits @ 80 MHz.

• Update of FPGAs and loading of CPDAS constants can be done via the SLM using the Gigabit Ethernet (bidirectional) and two lines of the LVDS bus.

New Readout of the CPDAS


Module with 2 SLM mounted on the back of the LKr Rack

CPD-LVDS card replacing optical links

2xSLM


Mass Market Key ComponentsMass Market Key Components

10/100/1000Ethernet

FPGA

LAPTOP DDR2 SODIMM


Field Programmable Gate Array FPGA

• No Gates anymore but Logic Elements.

• Hardware blocks such as Memories, Transceivers, Programmable Delay Lines, LVDS receivers, ADC, JTAG, Logic Analyzers …

• Software blocks (Encrypted) Ethernet and DDR2 memory controllers ..

• Design is hardware oriented using “MegaWizzard Plug-In Manager”.

• FPGA system clock is only 100 to 200 MHz due to routing delays.

• Latest FPGAs are available in 65nm technology (faster, smaller packages, lower dissipation and cost).

• Example for nCPD in Cyclone III EP3C5_164MBGA 8x8 mm $17 x 8

• EP3C40_780FBGA 29 x 29 mm $133 x 2 (4)

• But high-end FPGAs Stratix III such as EP3SL150_780FBGA costs > $ 2185

• The Stratix III EP3SL340_1517FPGA costs > $11168


Double Data Rate 2 DDR2 MemoryDouble Data Rate 2 DDR2 Memory

• Analog technology using capacitors as storage elements• With no power the memory content is be kept for a few sec @25 ºC• DDR2 memories was introduced year 2003 • 2 Gbytes of DDR2 @ 667MHz SODIMM costs CHF 47 (14/4-08)• Burst speed is 667 MHz but only for 4 or 8 clock periods! • Can operate with R/W cycle times of about 100 ns. • SODIMM DDR2 memory works with 64 bits, but for example the

ALTERA Cyclone III DDR2 controller uses words of 256 bits.• Data from one CPD channel is stored in 128 bits to simplify

addressing.• 64 CPD channels requires 1 kB DDR2 memory• 2 Gbytes/CPD = 2 sec of NA62 data taking @ 1MHz trigger rate

Kingston ValueRAM DDR2 667MHz CL5 - 2 GB

KVR667D2N5/2G - 2048 MB - DDR2 - PC2-5300 - DIMM - Bus 667 MHz

http://www.1000ordi.ch/productsDisplay.php?product_display=3&productID=26369


Gigabit Ethernet

• 1 Gigabit Ethernet with 4 twisted pairs was standardized 1999.• Fully duplex with each port using 17 analog levels at 125 MHz.

– Present price of Gigabit Ethernet (10/100/1000BASE-T)– Components per module port – Marvell 88E1111 + RJ45 jack with trafo CHF 16.-– CAT 6 cable 20 m Halogen free CHF

47.-– Price/port in a Gigabit managed Network Switch CHF 100.-

• 10 Gigabit Ethernet using 4 twisted pairs was standardized 2006

Price of 100/1000/10000 BASE-T is what 2011?


Trigger Timestamp to the nCPD

• In contrast to LHC there is no TØ in the NA62 • The same type of trigger as the old system is needed for the nCPD• A timestamp which gives the time or address of the pipeline DDR2

memory to read out• Distribution of the timestamp can be done using the FASTBUS

backplane as a Broadcast Message• The connections between crates are OK with Segment

Interconnect• The VME TIC connections must be changed for 1 MHz


Trigger outputs10 m cable

Trigger outputs10 m cable

Trigger Outputs


Trigger Outputs• Fast Trigger summing outputs are available at 10 m from the LKr

crates, see Vienna Trigger Descriptions

4 in X and 4 in Y per CPD

Special backplane in the crate for the analog trigger partial summing

In total 864 in X and 864 in Y outputs

Differential analog currents 2 x 20mA


Example of using the nCPD in NA62(based on the ANTARES experiment)


NA62 nCPD SLM READOUT

• Each nCPD has one LVDS link to SLM with 120 Mbytes/s.

• The SLM can receive up to 480 Mbytes/s (4 links from nCPD).

• Each SLM has one bidirectional Gigabit Link = 80 Mbytes/s (max).

• Data reduction needed (150 active channels of 13200 in average):

• Example of readout of the nCPD:

1. Data reduction (like the Data Concentrator but with Network Switch and TELL1 clone).

2. “JPEG” picture of the calorimeter.


Data reduction like the Data Concentrator 1. For each event the nCPD determine the channels which have

values above threshold and store these channels in a bit pattern consisting of 8 bytes per nCPD and 32 bytes per SLM.

2. SLM assembles one GbE packet with i.e. 25 events.

3. Use Commercial Network Switch, i.e. as HP ProCurve:

1. to distribute packets of the same event from all CPDs to one HALO FPGA processing node.

2. to send data from clusters to DAQ channels.

4. Halo processing is still difficult, two alternatives:

1. With 12 TELL1 clones with 4 special subcards the HALO processing could be made in FPGAs by loading a matrix consisting of 128 x 128 logic elements, (better than custom hardware but needs electronics engineers).

2. With a PC farm and Gigabit Switch (should be more flexible and easier to program).

5. The channels to read out (the results from the HALO processing) are sent to all nCPDs as coordinates and readout.


Data reduction (based on ANTARES)


Number of SLMs 54 pc

Number of events per internet frame 25 pc

Size of frame packed with 25 events 832 bytes

Latency of switch 970 µsec

Switch port buffer size needed 45 Kbytes

Rate by Switch input port (from SLM) ~33 MBytes/s

Number of FPGA Halo processing channels ≥ 40 pc

Halo processing time /event 39 µsec

nCPD to SLM transfer rate (LVDS bus) 1.5 bytes/80 MHz 120 Mbytes/s

Rate of time stamps and readout commands with FASTBUS 4 bytes/10 MHz

40 Mbytes/s

Readout rate per SLM during respectively after burst (4.8s/12s) ~ 47/80 Mbytes/s

Total amount of LKr Halo Zero Suppressed Data per CPD and burst with DDR2 memory and readout on and after the burst.

~300 Mbytes

“Example” of HALO PROCESSING Parameters with SLMs and Network Switch


Other use of the FPGAs

in the nCPD


“JPEG” picture readout of LKr as independent check of data quality

4 colors = 2 bits/channel sent in Jumbo packets of 128 events gives ~ 64 Mbytes/s per SLM. One event of the whole LKr is 3.3 Kbytes + the ~150 channels of the cluster.

The FPGA calculates the mean value xm of 8 samples xi and the sum Σ(xi –xm)2

White Grey Blue Red


Increased resolution with a digital filter?

The frequency spectrum of the noise of the LKr electronics is such that it could be efficient to use a digital filter to decrease the bandwidth and thereby increase the resolution for pedestals. FPGAs are very suitable for digital filters.


ExamplesExamples

of implementation with commercial electronics


Modular Managed Network SwitchModular Managed Network Switch

The cost is about $14000 for a switch with 144 10/100/1000 ports

Modular chassis with dual power supplies with place for up to 12 modules (i.e. 288 ports)

Example of modules


>$100000 ?


Part of a PC farm with 44 x 1U PCPart of a PC farm with 44 x 1U PC

1 PC =$1000

2.5m


DiscussionDiscussion

• By using large memories and FPGA technology in the nCPD a smart readout can be implemented.

• Two cards in the CPD must be replaced to cope with the 1 MHz rate.• This should be straightforward if the old design documentation, schematics

and design files are still available. We estimate that about 2 man years are needed? One person mainly for the FPGA design the other for the design and production of the new PCBs.

• A staged replacement procedure could be implemented:– The calibration constants for the CPDAS requires that each subcard is

preferably put back into its old position on the new motherboard - module by module.

– The full capability of the system is only needed when operating at 1 MHz speed. Therefore the network switch and the PC farm should be equipped partially as needed and as late as possible to get the optimum technology i.e. multi-core processors and 10 GbE.

• A technical note is still in preparation maybe withTWiki instead.

a proposal for the readout of the na62 lkr calorimeter

Documents