instrumentation departmentj. coughlan et al.rutherford appleton laboratory23 october 2002tracker...

Instrumentation Department J. Coughlan et al.Rutherford Appleton Laboratory 23 October 2002Tracker Electronics Meeting

The Front-End Driver Card FEDv1

•VME crate requirements definition

•FEDv1 status : hardware / firmware / low-level software

•FEDv1 testing plans

http://www.te.rl.ac.uk/esdg/cms-fed/index.html


FEDv1 CratesRequirements Document

•Document needed for Tracker Crates ordering

•Aim to have common VME64x bus crates for FED and FEC

•List of special needs, connectors, power…etc.

•E.g. FED may need Transition cards (DAQ links)

•References to LHC Crate technical specification EP document

•Released for comments


FEDv1 CratesRequirements Document

CMS Tracker FED VMEbus Crate Requirements

Rob Halsall, John CoughlanInstrumentation DeptRutherford Appleton Laboratory11.10.02

This note provides a short description of the basic requirement list for the CMS Silicon Strip TrackerFront-End Driver (FED) board VMEbus crates.

The FEDs are single width 9U form factor boards [1].Each FED has an associated rear Transition module (assumed to be 6U form factor holding DAQ linkinterface).

The final system will comprise of ~440 x 9U FEDs housed in approximately 30 x 9U crates [2].

The FED crates should be identical with those housing the Tracker Front-End Controller (FEC) boards. Thespecific requirements of the latter system will be provided by those responsible for the FEC system.

The following sections are not in any order of importance.

Comments are welcome.

1. VME Spec:

Crate should conform to VME64x specs as described in ref [3].

2. Slots:

21 slots x 9U form factor (any additional 6U cards will be mounted on adapter boards)[see Costas Foudas for details of Adapter boards]

In the preliminary crate layout there are a maximum of 16 FEDs per crate [2].This allows up to 5 slots for Crate Controller(=slot1 arbiter), Monitor board, TCS interface…

Rail size : rails must accept 2 mm thick PCB boards

NOTE: FF1s will use Rittal Front Panels (VME64x standard)


FEDv1 Overview

VME-FPGA

TTCrx

BE-FPGAEvent Builder

Buffers

FPGAConfiguration

Compact Flash

PowerDC-DC

DAQInterface

12

12

12

12

12

12

12

12

Front-End Modules x 8Double-sided board

CERNOpto-

Rx Analogue/Digital

96 TrackerOpto Fibres

VMEInterface

XilinxVirtex-IIFPGA

Modularity

9U VME64x Form Factor

Modularity matches Opto Links

8 x Front-End “modules”

OptoRx/Digitisation/Cluster Finding

Back-End module / Event Builder

VME module / Configuration

Power module

Other Interfaces:

TTC : Clk / L1 / BX

DAQ : Fast Readout Link

TCS : Busy & Throttle

VME : Control & Monitoring

JTAG : Test & Configuration

FE-FPGAClusterFinder

TTC

TCS

TempMonitor

JTAG

TCS : Trigger Control System

9U VME64x


FEDv1 Project History

VME-FPGA

TTCrx


Buffers

FPGAConfiguration

Compact Flash

PowerDC-DC

DAQInterface

12

12

12

12

12

12

12

12


CERNOpto-

Rx Analogue/Digital


VMEInterface

XilinxVirtex-IIFPGA

Design Milestones

Analogue Circuit for signal match from OptoRx to ADCs

(good working relationship with Francois&Jan)

Fit 96 ADC channels on board

FPGAs choice of Xilinx Virtex-II family. Fitting FE design into target part.

But things (almost always) take longer than you think…

S-LINK Transition card / Direct?

Power module spent more effort than anticipated

FPGA de-coupling schemes.

+ many minor design decisions.


TTC

TCS

TempMonitor

JTAG


9U VME64x


FEDv1 Board Layout Primary Side

•Complex board•Analogue & Digital• & signal integrity•Pushing density limits•Modular PCB design


FED v1 Board Layout Secondary Side


FEDv1 Overview

12

12

12

12

12

12

12

12


CERNOpto-

Rx Analogue/Digital


XilinxVirtex-IIFPGA

Modularity

8 x Front-End “modules”


9U VME64x


FEDv1 Front-End module

12 FibreRibbon

PD

Arr

ay

111

2

1

23

4

35

6

2

47

8

59

10

3

611

12

5*

CLK

DCM

LVDS CLK40 from TTC

CERN Opto Rx

6

CLOCK

CONTROL

DATA

DATA OUT@ 160 MHz

3 N

FullPartially Full

RESET

ClusterFindingFPGA

DelayFPGA

DelayFPGA

DelayFPGA

Dual ADC10-bits40 MHz

OpAmp

XC2V1500XC2V40AD9218EL2140

CLK

CLK

10ASIC

4*

DataControl

Digital Processing

12x trim DAC

TempSensorLM82

* Double Data Rate I/O

Each individual ADC clock skewis adjustable in steps ~ 1nsec


FEDv1 Front-End module Primary Routed


FEDv1 Front-End module Secondary Routed


FEDv1 Full-Scale 9U Layout

TTCrx


Buffers

DAQInterface

12

12

12

12

12

12

12

12


CERNOpto-

Rx Analogue/Digital


XilinxVirtex-IIFPGA

Back-End module

For each L1-Trigger:

Collects 8 x FE variable length data fragments

Formats FED event for DAQ

Appends TTC synch information

Buffers in External QDR SRAM

Sends data via DAQ Front-end Readout Link FRL

Signals to TCS : Busy/Throttle


TTC

TCS


9U VME64x

PinDiode/Amp

ASIC


DAQFront-endReadout

LinkFRL

DAQ MezzanineCard

Transition Card

BusyThrottle

S-LINK64

FEDv1 FED-DAQ Interface

TTCrx


Buffers

12

12

12

12

12

12

12

12

TTC

TCS

FRL DAQ links use S-LINK64 standardImplementation: Channel Link800 MBytes/sec max

Average DAQ rate 200 MBytes/sec

See talk “CMS Data to surface transportation architecture”: Attila Racz

Due to mechanical constraints place DAQ link card on Transition card


Alternative scheme:“Channel Link” inVirtex-II BE-FPGA

DAQ

FEDv1 FED-DAQ Interface (alternative)

TTCrx


Buffers

12

12

12

12

12

12

12

12

TTC

TCS


FEDv1 VME module

VME-FPGAFPGA

Configuration

12

12

12

12

12

12

12

12

VMEInterface

XilinxVirtex-IIFPGA

VME64x Interface:

A32/D64 Slave & Master

DMA engine

Interrupter

Plug & Play geographic addressing

Live Insertion Transceivers

FPGA Configuration:

Xilinx System ACE Compact Flash

MPU-VME interface for in-situ reprogramming via Crate Controller

JTAG for Configuration & Test

JTAG

9U VME64x

System ACE CF


FEDv1 Power module

PowerDC-DC

12

12

12

12

12

12

12

12


CERNOpto-

Rx Analogue/Digital

XilinxVirtex-IIFPGA

Standard LHC spec crate supplies:

+3.3V, +5V, +12V

Derive: -5V, 1.5V, 2.5V on board

Board Estimate ~ 80 W

Hot Swap Controllers:

Electronic fuses

Protection against out of range voltage & current

Sequence power

Protections:

Over temperature shutdown

TempMonitor


9U VME64x

Warning


FEDv1 Testing

VME-FPGA

TTCrx


Buffers

FPGAConfiguration

PowerDC-DC

DAQInterface

12

12

12

12

12

12

12

12


CERNOpto-

Rx Analogue/Digital


VMEInterface

XilinxVirtex-IIFPGA

JTAG Boundary Scan: Digital connections

Chip Scope Integrated Logic Analyser Cores : Capture raw ADC data (without VME)

Opto-Tests or Inject electrical signals post-OptoRx

Special FPGA loads e.g. Pattern Generators

Additional Test Features:

Internal/External Clocks

External Triggers


TTC

TCS

TempMonitor

JTAG


9U VME64x


FEDv1 Status

VME-FPGA

TTCrx


Buffers

FPGAConfiguration

PowerDC-DC

DAQInterface

12

12

12

12

12

12

12

12


CERNOpto-

Rx Analogue/Digital


VMEInterface

XilinxVirtex-IIFPGA


TTC

TCS

TempMonitor

JTAG


9U VME64x

Hardware:Analogue Design CompleteDigital Design CompletePower Design CompleteFirmware:Cluster Finding CompleteEvent Building 50% DoneVME Interface 25% DoneBoard:Layout & Routing Front-End Module

CompleteBack-End Layout doneRouting started

Manufacture Tape out to PCB week 46Testing Start at RAL week 50

Board Status


FED Schedule

Schedule

2002/Q4 : 2 x FEDv1 @ for UK test Batch 1

2003/Q4 : ~10? x FEDv1 @ CERN Batch 2*

2004/Q4 : ~10? x FEDv2 manufacture

2005/Q2** : 500 x FEDv3 manufacture

(**funds permitting)

FEDv1 : Full scale PrototypeFEDv2 : Pre-productionFEDv3 : Final production

FEDv1 FEDv2

FEDv3

* Procurement started for critical parts for Batch 2


FED TIB Test Schedule

Q. Deliver 2 x FEDv1 @ CERN for TIB test starting July 2003 ??

•Very tough. Testing is a big job. Can’t promise to deliver in June?

•Use Batch1 PCBs. ie No design iteration possible.

•Risk that design has major fault.

•Prioritise firmware and testing plan.

•Require explicit list of functionality required/excluded for the TIB tests.

FEDv1 FEDv2

FEDv3

TIB?

OptoRx?


FEDv1Firmware & Low-Level Software


FEDv1 Firmware & Software

VME-FPGA

TTCrx


Buffers

FPGAConfiguration

PowerDC-DC

DAQInterface

12

12

12

12

12

12

12

12


CERNOpto-

Rx Analogue/Digital


VMEInterface

XilinxVirtex-IIFPGA

Digital Processing

Flexible Digital Logic:

Xilinx Virtex-II FPGAs 40K->3M gates*

*some in pin compatible packages

Features:

Dual Ported Block Rams

Digital Clock Managers DCM

Double Data Rate I/O DDR

Digitally Controlled Impedance I/O

Various I/O signal standards

Debugging: Logic Analyser cores

FPGAs programmed in

VHDL & VERILOG


TTC

TCS

TempMonitor

JTAG


9U VME64x


FEDv1 Front-End module

12 FibreRibbon

PD

Arr

ay

111

2

1

23

4

35

6

2

47

8

59

10

3

611

12

5*

CLK

DCM

LVDS CLK40 from TTC

CERN Opto Rx

6

CLOCK

CONTROL

DATA

DATA OUT@ 160 MHz

3 N

FullPartially Full

RESET

ClusterFindingFPGA

DelayFPGA

DelayFPGA

DelayFPGA

Dual ADC10-bits40 MHz

OpAmp

XC2V1500XC2V40AD9218EL2140

CLK

CLK

10ASIC

4*

DataControl

Digital Processing

12x trim DAC

TempSensorLM82

* Double Data Rate I/O

Each individual ADC clock skewis adjustable in steps ~ 1nsec


CMS Silicon Strip Tracker FED Front-End FPGA Logic

ADC 1 10 10

sy

nc

11

trig2

Pe

d s

ub

11

trig3

Re

-ord

er

cm

su

b

8

Hit

fin

din

g

s-datas-addr8

16

hit

Pa

ck

eti

se

r

4

averages 8header control

DP

M 16

No hits

Se

qu

en

ce

r-m

ux

8 8a

d

a

d

ADC 1210 10

trig1

sy

nc

11

trig2

Pe

d s

ub

11

trig3

Re

-ord

er

cm

su

b

8

Hit

fin

din

g

s-datas-addr8

16

256 cycles 256 cycles

hit DP

M 16

No hits

Se

qu

en

ce

r-m

ux

8 8a

d

a

d

status

averages8header status

nx256x16

trig4

Synch inSynch out

Sy

nc

h

emulator in

mu

x

Serial I/O

Se

ria

l In

t

B’Scan

Lo

ca

lIO

Config

Cluster Finding FPGA VERILOG Firmware

Full flags

data

Global reset

Co

ntr

ol

Sub resets

10

10

Ph

as

eR

eg

iste

rsP

ha

se

Re

gis

ters

2 x 256 cycles 256 cycles nx256x16

trig1Synch error

4x

Temp Sensor

Delay Line

Opto Rx

Clock 40 MHzD

LL

1x2x4x

per adc channel phase compensation required to bring

data into step

+ Raw Data mode, Scope mode, Test modes...

160 MHz


FEDv1 Full-Scale 9U Layout

TTCrx


Buffers

DAQInterface

12

12

12

12

12

12

12

12


CERNOpto-

Rx Analogue/Digital


XilinxVirtex-IIFPGA

Back-End module

For each L1-Trigger:

Collects 8 x FE variable length data fragments

Formats FED event for DAQ

Appends TTC synch information

Buffers in External QDR SRAM

Sends data via DAQ Front-end Readout Link FRL

Signals to TCS : Busy/Throttle


TTC

TCS


9U VME64x

PinDiode/Amp

ASIC


FEDv1 Back-End FPGA Logic

FIFO

FIFO

FIFO

FIFO

CircularBuffers

Frame_Syncs

Readout_Syncs

Monitor_Syncs

x8

x8

x8

TTC Rx

TCS

‘VME’

DECODECONTROL

&MONITOR

Data_stream 0

Data_stream 7

64 18Data In

20Address

18 Data Out

64FRL to DAQSLINK64

64

R/WAddress

Generator

APV hdrs

Lengths

Bx,Ex

Em Hdr

diagnostics

Data

80 Mhz

L1 100 kHz4

4

160 MHz

40 Mhz

Clock40

Reset

DCMx1

x2x4

2 x QDR SRAMx2 burst

BSCAN

160 MHz

160 MHz

160 MHz

80 MHz

Lengths

Header

FF/PF Flags 2

8

1

1

1

Control

2 MBytes

VME copy path to local buffer


FEDv1 Status

VME-FPGA

TTCrx


Buffers

FPGAConfiguration

PowerDC-DC

DAQInterface

12

12

12

12

12

12

12

12


CERNOpto-

Rx Analogue/Digital


VMEInterface

XilinxVirtex-IIFPGA


TTC

TCS

TempMonitor

JTAG


9U VME64x

Hardware:Analogue Design CompleteDigital Design CompletePower Design CompleteFirmware:Cluster Finding CompleteEvent Building 50% DoneVME Interface 25% DoneBoard:Layout & Routing Front-End Module

CompleteBack-End Layout doneRouting started

Manufacture Tape out to PCB week 46Testing Start at RAL week 50

Board Status


FEDv1 Module Testing Firmware Functionality

•VME readout A32/D32 only

•Raw Data only (unpacked)

•Scope mode for timing

•TTC interface

•Individual ADC clock skews

•Hardware throttle ?

•S-LINK

•TCS interface

•Cluster Finding

•Auto-Calibration

•VME64x dynamic addressing

Required Not Required

Q. What event rate needed for testing?Q. Do we need individual channel DAC offsets ? (or is common OptoRx offset sufficient)...


FEDv1 Low-Level Software

•XDAQ Hardware Abstraction Layer

•fedlib class API lowest level access for user programs

•e.g. fedlib_load_fe_ped(..)

•fedlib_select_clock(..)

•Plus additional layer? (see Costas’s talk)

•Permit VME A32/D32 accesses only.

•Plus CSR space

•Possibly D32 BLT

•No Interrupts

fedlib

fed codefed code


FEDv1 Address Map

Hidden from outside world…

•Address Map under construction. Implementation depends on Firmware.

•Need this to do before fedlib API

FE FPGA #0

FE FPGA #1

FE FPGA #2

FE FPGA #3

FE FPGA #4

FE FPGA #5

FE FPGA #6

FE FPGA #7

FED local CSR

TTCrx

Event Memory

FPGA confign

PedestalsSkew values...etc


FEDv1 Low-Level Software

Mechanism of FED setup hidden from User

•Eg Loading FED parameters not trivial

•Write constants eg Peds to small local FPGA buffer. Limited FPGA memory.

•Send command to load to FE FPGA

•FPGA “engine” handles serial commands

•Optional readback

•Repeat for next block

•Constraints:

•Block access only

•NB Can’t access constants during Run!

Command & Status

ParameterLoad block

ParameterReadback block


FEDv1 Firmware & Software

VME-FPGA

TTCrx


Buffers

FPGAConfiguration

PowerDC-DC

DAQInterface

12

12

12

12

12

12

12

12


CERNOpto-

Rx Analogue/Digital


VMEInterface

XilinxVirtex-IIFPGA

Digital Processing

Flexible Digital Logic:

Xilinx Virtex-II FPGAs 40K->3M gates*

*some in pin compatible packages

Features:

Dual Ported Block Rams

Digital Clock Managers DCM

Double Data Rate I/O DDR

Digitally Controlled Impedance I/O

Various I/O signal standards

Debugging: Logic Analyser cores

FPGAs programmed in

VHDL & VERILOG


TTC

TCS

TempMonitor

JTAG


9U VME64x


FEDv1 Event Readout

DAQ Trailer

FormattedFED Data

DAQ Header

Tracker Header?

Readout (via VME) Formatted Events identical? to those sent to S-LINK

Poll on Event Counter

Get length of event

(Request) Readout & Clear Event

Repeat…

Check local status registers/counters

i.e. similar to PMC (sorry)

Event Formats:

Raw Data unprocessed.

Scope Mode.

Tracker header to do...


FEDv1 Schedule

These are still guesstimates and depend on ease of Firmware implementation…

•Draft Memory Map (mid October)

•Release Event Format specs (end November)

•Release draft fedlib API spec (basic calls/no code) (end December)

•Release draft operating instructions (end December)

•Release first version of fedlib (basic calls/code) (end February)

•Caveat Emptor: It is still possible/likely that changes will be necessary to these releases.


FEDv1 Comments

•In order to achieve minimum firmware & software on TIB timescales require...

•Confirmation of test functionality.

•Close co-operation between UK and Tracker DAQ group.

•Separate small FEDv1 online meeting.

•Propose contact from UK to visit CERN for “technology transfer”.

•Keep systems h/w and s/w as close as possible (within UK) and between UK and CERN.


FEDv1 Testing


CMS Silicon Strip Tracker FED Counting Room Layout (illustration)

• 440 Boards 96 ADC/Board

• 24 Crates

• 8 Racks

• 440 Boards 96 ADC/Board

• 24 Crates

• 8 Racks

• 40 K ADC Channels 10 Bit@40MHz• Max Trigger Rate 100 kHz• Input Rate 1.5 T Byte/s• Output rate 25 Gbyte/s/%

• 40 K ADC Channels 10 Bit@40MHz• Max Trigger Rate 100 kHz• Input Rate 1.5 T Byte/s• Output rate 25 Gbyte/s/% 4 TTC Partitions

DAQFED


CMS Silicon Strip Tracker FED Crate Layout

1 212

FE 1

DAQ

TTC

FE 2

FE 3

FE 4

FE 5

FE 6

FE 7

• Crate Input Data Rate ~ 50 Gbyte/s• Crate Output Data Rate ~ 1 GByte/s per percent hit occupancy

B-Scan

F-Bus

NN Synch

100 KHz

FE 8

Throttle


1 24

E-Net Switch

FECCRATES DCS

Tracker Control WS

TTCCRATES

DAQ

VME SBCRTOS

CMS Silicon Strip Tracker FED Control & Monitoring

D-BASER/C

50K ADC Channels

FEDCRATES

instrumentation departmentj. coughlan et al.rutherford appleton laboratory23 october 2002tracker...

Documents