ddl1 alice detector data link (ddl) and it’s use in star tof j. schambach
TRANSCRIPT
DDL 1
ALICE Detector Data Link (DDL)and it’s use in STAR TOF
J. Schambach
DDL 2
Presentation Outline
• Hardware Overview
• DDL Protocol / Transactions
• RORC
• DDL in TOF
• Software
• Demo
DDL 3
Readout system
Front-end electronicsFront-end electronics
DetectorData Link
DDL SIUDDL SIU
DDL DIUDDL DIU
RORCRORC
SourceInterfaceUnit
DestinationInterfaceUnit
ReadOutReceiver Card
PCPCData Acquisition
PC
Optical Fibre~200 meters
DDL 4
DDL architecture
• Source Interface Unit (SIU) (1)
– Interface to the Front-end Electronics (2)
• Destination Interface Unit (DIU) (3)
– Interface to the Readout Receiver Card (4)
• Full duplex optical link (5)
– Multimode optical cable of up to 200 m
1
2
3
4
5
DDL 5
DDL hardware
DDL 6
SIU Physical Layout
DDL 7
DDL interfaces
• SIU-FEE interface– 3.3V (LVTTL) interface
– 32-bit wide half-duplex data bus (bi-directional bus)
– Bi-directional flow control
– User defined clock (synchronous interface)
– JTAG interface
• DIU-RORC interface– 3.3V (LVTTL) interface
– 32-bit wide full-duplex data bus
– Bi-directional flow control
– User defined clock (synchronous interface)
DDL 8
DDL Interface signals
DDL 9
SIU Connector Pinout
DDL 10
SIU-FEE interface
fbD(31..0) - data lines (bi-directional)fbTEN_N - transfer enable (bi-directional)fbCTRL_N - CONTROL qualifier (bi-directional)fiDIR - bus direction (FEE input)fiBEN_N - bus enable (FEE input)fiLF_N - link full (FEE input)foBSY_N - front-end busy (SIU input)foCLK - interface clock (SIU input)TAP_TCK - JTAG clock (FEE input)TAP_TDI - JTAG data in (FEE input)TAP_TDO - JTAG data out (SIU input)TAP_TMS - JTAG mode select (FEE input)TAP_TRST - JTAG reset (FEE input)
DDL 11
Link management
DDL 12
DDL Configurations
DDL 13
Front-end Commands
DDL 14
Front-end Status Words
Data Transmission Status Word (DTSTW), produced by SIU:
DDL 15
Front-end control
on-line on-line
FECTRL
CTSTW
on-line on-line
DIU SIURORCOnlineOnline
FEE command
Report
FEE
FEE control
idle
foCLK
fiBEN_N
fiDIR
FECTRLfbD
fbTEN_N
fbCTRL_N
DDL 16
Front-end status read
on-line on-line
FESTRD
FESTW
on-line on-line
on-line
DIU SIURORCOnlineOnline
FEE command
Status and report
FEE
FEE status read
CTSTW
foCLK
fiBEN_N
fiDIR
FESTRDfbD
fbTEN_N
fbCTRL_N
HiZ
HiZ
HiZ
foCLK
fiBEN_N
fiDIR
FESTWfbD
fbTEN_N
fbCTRL_N
HiZ
HiZ
HiZ
HiZ
HiZ
HiZ
DDL 17
Event read
foCLK
fiBEN_N
fiDIR
RDYRXfbD
fbTEN_N
fbCTRL_N
HiZ
HiZ
HiZ
foCLK
fiBEN_N
fiDIR
D0fbD
fbTEN_N
fbCTRL_N
HiZ
HiZ
HiZ
fiLF_N
D1 D2 D4 D5
foCLK
fiBEN_N
fiDIR
fbD
fbTEN_N
fbCTRL_N
fiLF_N
FESTWDn-1 Dn D0min. 16 cycles
foCLK
fiBEN_N
fiDIR
fbD
fbTEN_N
fbCTRL_N
fiLF_N
FESTWDn-1 Dn HiZ
HiZ
HiZ
foCLK
fiBEN_N
fiDIR
EOBTRfbD
fbTEN_N
fbCTRL_N
HiZ
HiZ
HiZ
DDL 18
Block write
on-line on-line
STBWR
CTSTW
DIU SIURORCOnlineOnline
FEE command
Report
FEE
STBWRCTSTW
FEE data
EOBTRFEE command
Report
EOBTR
Flow control
Block data data block
foCLK
fiBEN_N
fiDIR
D0fbD
fbTEN_N
fbCTRL_N
foBSY_N
D1 D2 D4 D5STBWR
foCLK
fiBEN_N
fiDIR
fbD
fbTEN_N
fbCTRL_N
foBSY_N
EOBTRDn-1 Dn
DDL 19
Block read
on-line on-line
STBRD
CTSTW
DIU SIURORCOnlineOnline
FEE command
Report
FEE
STBRDCTSTW
FEE data
EOBTR
data block
Block data
FEE command
Report
EOBTR
Flow control
foCLK
fiBEN_N
fiDIR
STBRDfbD
fbTEN_N
fbCTRL_N
HiZ
HiZ
HiZ
foCLK
fiBEN_N
fiDIR
D0fbD
fbTEN_N
fbCTRL_N
HiZ
HiZ
HiZ
fiLF_N
D1 D2 D4 D5
foCLK
fiBEN_N
fiDIR
fbD
fbTEN_N
fbCTRL_N
fiLF_N
FESTWDn-1 Dn HiZ
HiZ
HiZ
foCLK
fiBEN_N
fiDIR
EOBTRfbD
fbTEN_N
fbCTRL_N
HiZ
HiZ
HiZ
DDL 20
RORC features
• Interface between the DIU and PCI local bus– pRORC: 32 bit/33 MHz PCI version, max. throughput 132 MB/s– D-RORC: 64 bit/66 MHz PCI version, max. throughput 528 MB/s
• PCI master capability, data push architecture– Autonomous operation with little software assistance– Supports multi-paged memory management
• Direct data transfer to the PC memory– No local memory on the board– Small elasticity buffers between different clock domains
• Built-in test capability– Internal pattern generator can produce formatted data
DDL 21
D-RORC Hardware
D-RORC with integrated DIU ports• to read out two DDL channels• to support integration with the HLT system
D-RORC with plug-in DIU• to read out single DDL channel• to support the tests of FEE readout sytems
DDL 22
RORC Roadmap
• pRORC: 32-bit, 33 MHz (PCI I/F by ASIC)– well adapted to the prototype version of the DDL
– can be used for the new version of the DDL (adapters)
– already used by several test beams (SDD, HMPID)
• D-RORC I: 64-bit, 66 MHz (PCI I/F by IP core)– well adapted to the new version of the DDL
– will integrate two DIU functions on-board
– will support the DAQ/HLT interface
• D-RORC II: 64-bit, 66/133 MHz (PCI-X I/F by IP core)– to avoid compatibility issues
– to follow the evolution of the PCs
DDL 23
Hardware Architecture
APEXFPGA
64-bit/66 MHz, PCI/PCI-X
MediaI/F 1
MediaI/F 2
BusyI/F
Conf.Flash
JTAGJTAG
P11
P12
P13
P14
CMC I/F
Opt
ical
I/F
LV
DS
I/F
Configuring
250 MB/s
250 MB/s
528 MB/s
DDL 24
Firmware Architecture
PCI core(64-bit master, memory mapped)
Controlregisters
address lengthTransmit
DMA
address lengthReceiveDMA
RX
FIF
O
TX
FIF
O
DIU I/F or DIU core
Slave I/F Master I/F
RAF
TAF
DMA control
TAF – Transmit Address FIFO
RAF – Receive Address FIFO
DIU or Media I/F
PCI bus
DDL 25
The Free FIFO
PRORC PC memory bankFirmware
readout
page address
page address
page address
Free FIFO
PC CPU
Allocation of free pages
DDL 26
Direct Memory Access
PRORCFirmware
PC memory bank
DDL
No involvement
PC CPU
DDL 27
The Ready FIFO
PRORC PC memory bank
readout
DDL
Ready FIFOFirmware
lengthpage status
lengthpage status
lengthpage status
Delivery of filled pages
PC CPU
DDL 28
Software
Event ready test:
1) SOFTWARE
start = curr;start = curr;
do {do {
while (Ready FIFO [curr].status == 0 ) curr = NEXT(curr);while (Ready FIFO [curr].status == 0 ) curr = NEXT(curr);
} while (Ready FIFO [curr].status == 0xffffffff ||} while (Ready FIFO [curr].status == 0xffffffff ||
Ready FIFO [curr].status == 0)Ready FIFO [curr].status == 0)
end = curr;end = curr;
curr = NEXT( curr );curr = NEXT( curr ); Blocks from startstart to endend are now available
2) HARDWARE
delivery of signal, enabling of semaphore or setting of flag
Release event
Firmware
FirmwareFirmware
datalength
transferstatus
Ready FIFOReady FIFO
Event ready?N
Y
Pop descriptor from Free FIFOMove data into buffer until DTSTW
or up to buffer sizeUpdate size
size = 0
DTSTW?
Y
N
size > max size?
N
Y
Throw away rest of datado not increment block length
until DTSTW receivedMark DTSTW “OVERRUN” bit
Push data length in Ready FIFO
Push transfer status = 00000000Optional: update Ready FIFO in memory
Push data length in Ready FIFOPush transfer status = DTSTWUpdate Ready FIFO in memory
Deliver “done” interruptOptional: set stop flag on error
Free FIFO empty?
Y
N
block:Reset entries of Ready FIFO
Push address & size on Free FIFO(if space available in Free FIFO)
Pull data length and transfer statusfrom Ready FIFO
Push pending free blockson Free FIFO (if any)
Load Free FIFO
Initialize Ready FIFO
Load configuration registers
Clear stop flag
Initialize internal data structuresstop flag = ON
acknowledge reset done
Transfer status possible values:ffffffffffffffff unloaded (set by sw)0000000000000000 loaded, no DTSTW (set by fw)elseelse loaded, DTSTW (set by fw)
Free FIFOFree FIFO
blocksize
Index ofReady FIFO
startaddress
stop flag ON?
N
Y
RORCRORC
DDL 29
Test equipments
• Front-end Emulator Interface Card (FEIC)– Fully functional hardware to emulate the detector front-ends
– Formatted data block generation
– Internal (free running) or external (pulse) triggering capabilities
– Adjustable parameters (using front-end control)
– Operates at the nominal speed of the DDL
• Source Interface Unit Simulator (SIMU)– Simulates the behavior of the DDL without any additional
hardware
– Eases the development and the hardware debugging
– Size is similar to the real SIU
DDL 30
Bandwidth: D-RORC to D-RORC
Testing the transfer between two D-RORC cards
• rorc_send –g ... (pattern generator)• rorc_send ... (DMA from)
DDL 31
TOF Essential Model Level 1
Start Detector with
24 Scintillators &24 PMTs
Start DetectorElectronics
Tray
32 RPCWith 6 channels
eachOn-Tray Electronics
Data, Clk, Ctrl
CANbus
30Trays
L0 Trigger
Data, Clk, Ctrl
Dual Fiber DAQTHUB
Master Clock
CA
Nb
us
Tray
32 RPCWith 6 channels
eachOn-Tray Electronics
Data, Clk, Ctrl
CANbus
30Trays
L0 Trigger
Dual Fiber DAQTHUB
Slave Clock
CA
Nb
us
OneDetector Side
CA
Nb
us
Clk
, C
trl
To other THUB’s
Clk, Ctrl
Clk, Ctrl
Control PCwith
CANBus I/F
TCD & BusyLogic
DDL 32
TOF Essential Model Level 2
Start Detector
TOF Tray
TDIG TDIG
TCPU
DIFFERENTIAL DATA AND CLOCK
TINO TINO
MRPC MRPC
COMMANDS
48 CHAN
48 CHAN
48 CHAN
TRAY CAN BUS
MULTIPLICITY
THUB
COPPER: DATA, SAMPLE CLOCK, RESET TRIGGER STROBE & DATA
TOP LEVEL CAN BUS
COPPER LINKS TO 29 TRAYS
RHIC CLOCK
SIU FIBER
CAN BUS TO 29 TRAYS
TCD
DAQ
48 CHAN
4
L0 Trigger
4
PMTs TPMT TSTD TCPU
Diff. Data & Clk
Local CANbus
Co
pper
:D
ata
, S
am
ple
Clk
, Re
set
Trg
Str
obe
& D
ata
Reset & ClockFrom other THUBs
Reset & ClockTo other THUBs
DDL 33
Run 5 TCPU
MCU
PLDALTERA STRATIXEP1S10
TDIG CABLE I/F
TRAY LEVEL CAN BUS I/F
ROM
TOP LEVEL CAN BUS I/F
TCD I/F
DDL-SIU
TDIG CABLE I/F
TDIG CABLE I/F
TDIG CABLE I/F
Clock select and distribute
Leds and switches
Local osc
Ext clock input
Leds, switches,
temperature monitor
Config eprom
Jtag for codetap
Test header
Power
JTAG multiplexer
ICD2header
JTAG for config
Ext clock output
Ext ‘reset’ output
To TDIG cables
DDL 34
DDL Interface Implementation
DDL 35
TCPU Firmware Data Path
FIFO256 X 32
5:1 MUX
FIFO
FIFO
FIFO
FIFO2048 X
32
L2 Trigger Data
MUX FIFO4 TO 32
BITDEMUX
4 TO 32 BIT
DEMUX
4 TO 32 BIT
DEMUX
4 TO 32 BIT
DEMUX
TDIG CABLE
TDIG CABLE
TDIG CABLE
TDIG CABLE
FIFO2048 X
32
MCU I/F
MCU
CAN BUS
TCD I/FFIFO
64 X 32
DDLSIU
TCD CABLE
FIBER
DDL 36
DDL Software
• PCI driver
• API routines for DATE
• Executable utility programs for test and stand-alone (= without FEE or DATE) use of the DDL, such as– Reset the RORC, DIU or SIU
– Display the status of the RORC, DIU or SIU
– Test the functionality and measure the performance of the whole DDL and RORC system
DDL 37
Installation of the utilities
• Utilities are in the directory
./rorc/Linux/• Linux kernel version: 2.4• Driver module must be inserted. As root type:
/sbin/insmod ./Linux/rorc_driver.oor insert a similar line into /etc/rc.d/rc.local
• Checked if physmem and RORC driver are loaded:
cat /proc/modules• Check if RORC card is plugged:
rorc_find
DDL 38
$ /sbin/lspci
00:00.0 Host bridge: Intel Corp. 82840 840 (Carmel) Chipset Host Bridge (Hub A) (rev 01)
00:01.0 PCI bridge: Intel Corp. 82840 840 (Carmel) Chipset AGP Bridge (rev 01)
00:02.0 PCI bridge: Intel Corp. 82840 840 (Carmel) Chipset PCI Bridge (Hub B) (rev 01)
00:1e.0 PCI bridge: Intel Corp. 82801AA PCI Bridge (rev 02)
00:1f.0 ISA bridge: Intel Corp. 82801AA ISA Bridge (LPC) (rev 02)
00:1f.1 IDE interface: Intel Corp. 82801AA IDE (rev 02)
00:1f.2 USB Controller: Intel Corp. 82801AA USB (rev 02)
00:1f.3 SMBus: Intel Corp. 82801AA SMBus (rev 02)
01:05.0 Multimedia audio controller: Cirrus Logic CS 4614/22/24 (rev 01)
01:06.0 Network controller: CERN/ECP/EDU: Unknown device 0033 (rev 01)
01:08.0 Ethernet controller: Accton Technology Corporation SMC2-1211TX (rev 10)
02:1f.0 PCI bridge: Intel Corp. 82806AA PCI64 Hub PCI Bridge (rev 02)
03:00.0 PIC: Intel Corp. 82806AA PCI64 Hub Advanced Programmable Interrupt Controller (rev 01)
03:04.0 Network controller: CERN/ECP/EDU: Unknown device 0033 (rev 02)
03:09.0 SCSI storage controller: Adaptec AIC-7892P U160/m (rev 02)
04:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200 AGP (rev 03)
Identifying the RORC cardIdentifying the RORC card
DDL 39
Program ‘rorc_find’
List all plugged and presently not used RORC devices together with their version and serial numbers.
rorc_find
The following device(s) found:
-------------------------------------------------------------------------------
Minor Channel Device type and HW identification (RORC’s FW version)
-------------------------------------------------------------------------------
0 0 D-RORC (FW: 1v35 of July 2 2003)
new DIU DDL card 2v0 LD: 20K60E SP: 2125 Mbps S/N: 00102
0 1 D-RORC
no DIU
1 0 pRORC pRORC 1v1 S/N: 00101 (FW: 1v75 of June 6 2003)
new DIU DDL card 2v0 LD: 20K60E SP: 2125 Mbps S/N: 00118
-------------------------------------------------------------------------------
3 RORC channel(s) not in use was found.
RORC driver reported 2 RORC device(s).
DDL 40
rorc_reset
Initialize the RORC card and the DDL link
rorc_reset [-{M|m} <minor> | 0] [-{D|d|B|b|S|s|F|f|O|o|E|e|C|c}]Where-D or –d reset the DIU-B or –b reset both RORC and DIU-S or -s reset SIU-F or –f clear Free FIFO-O or –o clear RORC’s other FIFOs-E or –e clear RORC error bits-C or –c clear RORC’s byte counterNo option reset RORC
DDL 41
rorc_id
Display RORC’s hardware and software identification wordsrorc_id [-{M|m} <minor> | 0] [-{D|d}{S|s}] [-{T|t} <time-out> | 100000]Where-D or –d display DIU’s firmware id as well-S or –s display SIU’s firmware id ad welltime-out time-out value for DIU or SIU replye.g.:rorc_idRORC driver version: 4.2RORC revision id: 1Hardware identity word of the RORC: pRORC 1v1 S/N:00103`, i.e. Version: 1.1, S/N: 00103. Firmware identity word of the RORC: 0x02190550, i.e. Version: 1.72 Release date : October 16 2002
Free FIFO size: 128 entries
DDL 42
rorc_send_command
Send a DDL command and receive the reply
rorc_send_command [-{M|m} <minor> | 0] -{C|c} <command> [-{T|t} <time-out> | 100000]
[-{V|v} <diu_version> | 2]Wherecommand a hexadecimal number starting with “0x”, or
an ASCII mnemonic of a DLL or pRORC command, e.g.:LBON RORC loop-back onRDYRX ready to receive message to the SIUEOBTR end of block transfer message to the SIU(see the sw manual or the program help for command codes and format)
time-out time-out value for RORC, DIU, SIU or Front-End replydiu_version 1 for prototype, 2 for final version.
DDL 43
rorc_receive
Send and receive data to the Front-End. The most important options:rorc_receive [-{M|m} <minor>|0] [-{G|g}] [-{Y|y}] [-{Z|z}] [-{X|x} <check_level>] [-{K|k} <output file>] [-{E|e} <events>] [-{P|p} <pattern>|0] [-{I|i} <init word>|0] [-Q <GBytes>|1]Where (from version 4.2: rorc_receive)-G send data using RORC’s data generator-Y do not loop-back generated data but send it via the link-Z do not send RDYRX and EOBTR commandsoutput file dump the receives the data into this file without checking itcheck_level 0: do not check the received data, 1: check only the first word,
2: not the first word, 3: check the whole event events # events to sendpattern event pattern to send or receive, it could be: c, a, 0, 1, I, dinit word the first word of each event’s payload (after the event serial number)GBytes display the number of received bytes after each received Gbytes data
(see the sw manual for further options)
DDL 44
feic.menu
Check and set the Front-End Emulator Interface Card
feic.menu [-{M|m} <minor>|0]
This script calls the rorc_send_command program several times to check the FEIC setting or to set new parameters. The following features of the FEIC’s data generator can be modified and displayed:
data pattern alternating, flying 0 or 1, incrementing, decrementing dataevent length 16, 32, 64, …., 256 Kwordstrigger mode push button, ext. trig., 16 or 128 clocks after each event,
every 10 or 100 msseed the seed value for random data length
See the description of rorcArmFeic routine in the sw manual for further details.
DDL 45
Other utilities
program name program function
diu_id DIU hw and fw identification
siu_id SIU hw and fw identification
rorc_status show RORC status
rorc_reg show RORC registers
diu_status ask and display DIU status
siu_status ask and display SIU satatus
rorc_send download data from the PC to the FEE
rorc_send_jtag download JTAG data