a rad-hard slow control network for the cms central tracker

A Rad-Hard Slow Control Networkfor the CMS Central Tracker

A. Marchioro, G. Cervelli, F. Faccio, K. Kloukinas, C. Ljuslin,

P. Moreira, E. Murer, C. Paillard, F. VaseyCERN-EP

A. Marchioro / EP 2

Outline Motivations and Requirements Architecture Network and system components:

– Token-Ring Network– Local Monitoring– Timing and Trigger Distribution

Network: High level protocol Project Status

A. Marchioro / EP 3

Motivations Requirements and Architecture no different

from traditional slow control system Merging with timing distribution: unusual ! Commercial rad-hard control components

exist, but not suitable for application and expensive

Special concern: reliability and SEU robustness Looked at commercial networks:

– Ethernet– Mil1553– CANbus– IBM Token-ring– JTAG– Other Field-buses … Either too complicated, or don’t work in

magnetic fields or unsuitable or too expensive, or…

A. Marchioro / EP 4

Requirements Must carry data and 40 MHz clock for system

synchronization Bi-directional 150m optical + ~10 m copper Compatible with opto-electronics used for data read-out Redundancy Low digital noise Easy interfaces on the FE ASICs ~120,000 FE ASICs, ~20,000 MCMs, ~2,000

controllers, ~200 links, ~50 control boards in Countingroom

Low cost (node price < 100 $ )

A. Marchioro / EP 5




A. Marchioro / EP 6

CMS Tracker R-O: General Architecture

APVs

CLK - T1

CCU

DCU

Det

PLL-Delay

A/DMemory

TTCrxI2

CFED

FEC

IV

TTCrx

PCIIntfc

CLK

& T

1

A/D

Temp

Front-endModule

Control Module

FEC cntrl

DataPathControlPath

toDAQ

CCU

CCUCCU

A. Marchioro / EP 7

Logical View of Control Ring

FEC

CCU_1

PCIOA

Node

CCU_8

Node

Channels

I2C - MBUS etc.

NodeController

Ring Protocol

Channel Specific Protocols

O/E

MBUS

Cha

nnel

Con

trolle

r

Cha

nnel

Con

trolle

r

PIO

Cha

nnel

Con

trolle

r

JTAG

Cha

nnel

Con

trolle

r

I2C

Simple JTAG, i.e.JTAG sequences setup in SW in FEC

A. Marchioro / EP 8

Full control ringWires between CCU and FE modulesCLK+T1 1 * diff pair (LVDS)I2C 2 (TTL OC)Alarm 1 (TTL OC)

Wires between CCU modulesCLK+T1 1 diff pair (LVDS)Data 1 diff pair (LVDS)Reset 1 wire (TTL)( x 2 for redundancy)

O/E converterDig. laser driverDig receiverLVDS driver

CCU moduleCCULVDS driver

FE APV moduleAPVs-PLLMux-Laser driver

A. Marchioro / EP 9

“Module Skip” Redundancy

CCU

DCU

CC

U

Primary

Secondary

DCU

CC

U

DCU

CC

U

DCU

CC

U

DIn + Clk

DOut

A. Marchioro / EP 10



Network: High level Project Status


FEC: Front-End Control Board

Con

trol

Logi

c

OpticalIntfc.

PCI/PMCConnector

FIFO

sPCI Adapter

TTC

rxFr

om T

TC n

etXT

AL

4-wa

y rib

bon

(8 w

ith re

dund

ancy

)

LVDS


CCU: Communication and Control Unit

I2C MasterIntfc.

I2C MasterIntfc.

LinkController

Protocol Adapter& Address Decoder

SCLSDA

D[0:7] A[0:7] R/W CS*

DO(A)

CLKI(A)

Power Control

Local Bus

DI(A)

DO(B)

DI(B)

Clock Distribution CLKI(B)

CLKO(A)

CLKO(B)

ST1ST2ST3ST4

TriggerDecoder

Tj Counter& other timing logic

18 x

I2C

Bus

es

PIO

Memory BusInterface

PDA[0:7] PDB[0:7]

L1

Ext Reset*


CCU-M Cabling With Redundancy

CCU

Data In-Port A

Data In-Port B Data Out-Port B

Data Out-Port A

Clk In-Port A

Clk In-Port B

ClkIn_A ClkOut_A

DIn_A DOut_A

PLL_Clk

DOut_B

ClkOut_BClkIn_B

DIn_B

PLLCKSEL CKOSEL


Coding of Clock and T1: Two Options

CLK

T1

CLK+T1

~ 25 ns

CLK+T1

T1

CLK

25 ns

~ 8 ns ~ 16 ns

Used

( )


NRZ_I (Invert 1 on Change) Coding

“0” : no transitions ( < 3 conseq.) “1” : transition at start of cycle

1 1 1 1 0 0 0 1 0 1

25 ns


Hard Reset (i.e. how to get always out of troubles)

Din

Reset

Vref

Din

Reset

Power-Up reset only must be avoided– use long sequence of 1’s (invalid data sequence) in data stream– extra comparator in receiver ASIC


4-5 Bit Encoding

4 to 5 bit NRZI encoder

5 to 4 bitNRZIdecoder

4 bit Binary Hex 5 bit Symbol0000 0 111100001 1 010010010 2 101000011 3 101010100 4 010100101 5 010110110 6 011100111 7 011111000 8 100101001 9 100111010 A 101101011 B 101111100 C 110101101 D 110111110 E 111001111 F 11101

Control Symbol Code CommentIdle 11111 IdleJ 11000 In SOF fieldK 10001 in SOF fieldH 00100 SpecialR 00111 ResetS 11001 SetT 01101 Termination


Other network components RX40: a RH 80 MHz BW digital receiver

with Automatic Gain Control Laser Driver: a RH Linear driver for

analog and digital data

preamplifier L.A. L.A. L.A. L.A.

B.F.

LVDSTx

sf

Reset block

in

reset

out

PINdiode

RX40

Exists already


DCU: Detector Control Unit

Inpu

t Mux 12 bit

A/D

TempSensor

Control / Status& Data Registers

Bias Generatorfor Modulator

Bus

Inte

rface

ControlSTM

SCLSDA

8 In

puts

• Local Monitoring

Current Sensing

Voltage Monitor

Temperatures• Designed in ¼ m CMOS

• ~ 2x2 mm2

• I2C Interface• < 10 mW• 100 sec/conversion


Timing Distribution: PLL-Delay

CLK &T1 extract

DelayLines

I2CSlaveInterface

PLL

SCLSDA

CLK

T1

I2C_Add(hardwired)

CLK&T1


Library for digital designs

Well isolation

NMOS enclosedNMOS guard-ring


Digital Library Features

0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.4 2.6

0

200

400

600

800

1000

1200

td[ps]

I [A

]

PreRad1 Mrad3 Mrad5 Mrad10 Mrad30 Mrad

Vdd

40

60

80

100

120

140

160

180

200

220

240

260

280

300

1001 Elements Ring Oscillator in 0.25 m Power: 0.05 W/gate*MHz @ 1 V

0.25 W/gate*MHz @ 2.5 V

Envisaged OperatingVoltage


LAN Protocol Inspired by Token-Ring Can work with single master and

“simpler” slaves Extensible to any local protocol

(I2C. Memory, etc.) Max data rate: 32 Mbit/sec


LAN Protocol (1) Ring-like topology

– Circulating token indicates bus available– Source node waits for token:

»Removes token, inserts data packet– Packet circulates, passed by all nodes– Destination copies packet, sets “S” symbol– Packet returns to source, is removed by

source– Source node inserts new token


LAN Protocol (2) All nodes can generate packets For simplicity:

– Only the FEC can perform network supervision» Inserts first token»Monitors token integrity»Receives all data packets from CCUs

FEC requires CPU for protocol management


LAN Protocol: Packet Format

SOF Dest Src Length Data CRC EOF

Universal

Channel SpecificCmd TR# Addr DWCh#

(Example for an I2C byte write)


LAN Protocol (3a): Write Example Transaction sequence:

– FEC sends command packet– CCU receives packet and sets “S”

symbol– Command packet returns to FEC– CCU directs data portion to channel– Channel performs action– Channel sends ACK packet to CCU

specifying TR#


LAN Protocol (3b): Read Example Transaction sequence:

– FEC sends command packet– CCU receives packet and sets “S”

symbol– Command packet returns to FEC– CCU directs data portion to channel– Channel performs read action– Channel sends data back to CCU

specifying TR#


LAN Protocol (4): Addressing CCU Address allocation

– FEC has address 0x00– Special Broadcast class: 0xf_ – Up to 254 CCUs on ring

Channel Addressing»0: CCU Node controller»1-15: I2C channels»16-31: memory channels»32-47: PIO channel (e.g. HV switches)»48-63: Event memory controller


Software Architecture Lynx-OS WinNT Linux ? Features:

– Multi-User– Multi-Task– Synchronous

(wait for Ack.)

Interrupt Handler

PCI-Hardware

User Application 1 User Application N

Sync I/O Driver

Packet Driver


Project Status: Hardware FEC: PCI-PMC version exists using FPGAs CCU: rad-soft, 6 I2C channels, 0.6 m CMOS

version is fully functional– Currently being remapped to ¼ m technology,

submission ~ Q1 ‘01– Full ASIC > 250,000 gates

PLL: exists and is OK in ¼ m CMOS DCU: design ready for submission Q1 ‘00 RX40: exists in ¼ mm, design OK Laser Driver: rad soft OK, RH submission Q1 ‘00

Cost: components well within requirements


Project Status: Software You always end up needing more

software than you expected Low level routines: OK Partitioning architecture: work

started Performance under WinNT:

– 200,000 I2C transactions/sec (limited by I2C, polling)

a rad-hard slow control network for the cms central tracker

Documents