off-detector & software development of daq system for the eudet calorimeters tao wu on behalf of...

Off-detector & Software development of DAQ System for

the EUDET calorimeters

Tao WuOn behalf of CALICE-UK Collaboration

ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 2

EUDET DAQ Targets

Maximizing to use of off-the-shelf commercial components, cheap, scalable and maintainable

Provide well-defined interfaces between DAQ components to allow for minimizing costs and development cycles;

A control system to easily integrate the rest of sub-systems of detectors

Software to build events from bunch train data and disparate sources into single event data

Manage network and data storage


DAQ Mechanical Design targets

Targeting EUDET Module and ILC/CALICE; Aim to develop a generic system e.g. ECAL/HCAL Triggerless DAQ system: use C&C instead

All data are sent off detector within a bunch train Use bunch struct. as advantages: 1ms in bunch train, rea

d out data within 200ms of inter-train gap;

DAQ system will also control power cycling of readout ASICs

A funnel-like DAQ to collect, wrap and transmit data in stages before sending to central storage


DAQ Architecture Overview

Slab hosts VFE chips (ASICs)Slab hosts VFE chips (ASICs)

DDetector Interfaceetector Interface connected to Slabs

LDA servicing DIFs

Link/Data AggregatorLink/Data Aggregator

LDAs read out by ODR via opto-links

Off-Detector ReceiverOff-Detector Receiver

PC hosts ODR PCI-Express driver software

Local Software DAQ

Full blown Software DAQ

LDALDA

PCPC

e.g. e.g. ECALECALSlabSlab

DIFDIF

ODRODR

DriverDriver

OptoOpto

OptoOpto

Ccc

- &

Dat

a-lin

k


DAQ Architecture Hierarchy

LDA

LDAHost PC

PC

Ie

ODR

Host PC

PC

Ie

ODR

DetectorUnit

DIF

C&C

DetectorUnit

DIF

DetectorUnit

DIF

DetectorUnit

DIF

Storage

1-3Gb Fibre50-150 Mbps HDMI

cabling

10-100m0.1-1m

Det

ecto

r

Co

un

tin

g R

oo

m

Covered by this talkCovered by this talk Covered by ValeriaCovered by Valeria


Current Architecture: ODR

Assemble data into a usable form for processing

Logical tasks: Receive from LDA Process e.g. event building Store

Current input is Ethernet for testing performance Internal FPGA test data generator; External data stream via network

DAQ PCDAQ PC

LDALDA

CC

C-li

nk

ODRODR

Data-link

DIFDIF

AS

ICs

AS

ICs

DIFDIF

AS

ICs

AS

ICs

DIFDIF

AS

ICs

AS

ICs …

Clock /Clock /FastFast

ControlControl

FEFE

ODRODR

StoreStore

Machine clock


Off Detector Receiver (ODR)

Hardware: Using commercial FPGA dev-board:

PLDA XPressFX100 Xilinx Virtex 4, 8xPCIe, 2x SFP (3

more with expansion board)

SFPs for optic link

Expansion (e.g. 3xSFP)

Receives modularized data from LDA Realized as PCI-Express card, hosted in DAQ PC. 1-4 Opti-links/card (Gigabit), 4 LDAs/card, 1-2 cards/PC Buffers and transfers to store as fast as possible

Sends controls and configs to LDA for distribution to DIFs Performance studies & optimisation on-going

B.G., A.M @ RHUL


ODR Data Access Rate

transfer the data from ODR memory to the user-program memory via 4 links~700 MByte/secby 25 DMA buffers

B.G., A.M @ RHUL

25 DMA buffers

0

100

200

300

400

500

600

700

800

32 125 130 547 929 1351 2040 3800

Data Size [bytes]

Tra

ns

fer

Ra

te [

MB

/s]

25 DMA buffers

All measurements: single requester thread, no disk write, data copied to the host memory.


Clock & Control (C&C) board C&C unit provides machine clock and

fast signals to 8x ODR/LDA. A low jitter and fixed latency; Logic control (FPGA, connected via USB) Link Data Aggregator provides next stage

fanout to Detector Interfaces Eg C&C unit 8 LDAs 8 DIFs = 64

DUs. Signalling over same HDMI type cables Facility to generate optical link clock

(~125-250MHz from ~50MHz machine clock)

Board is already designed, will be build soon

LDA

LDA

Host PCHost PC

PC

Ie

ODRODR

C&C

Host PCHost PC

PC

Ie

ODRODR

Machine

Run-Control

M.P., UCL

busy


EUDET DAQ software: StateState = DeadState = Dead

Transition = PowerUpTransition = PowerUp

suceed failed

State = ReadyState = Ready

State = RunningState = Running

State = ConfiguredState = Configured

State = InBunchTrainState = InBunchTrain

Transition = PowerDownTransition = PowerDown

Transition = StartRunTransition = StartRun Transition = EndRunTransition = EndRun

Trans. = StartConfigurationTrans. = StartConfiguration Trans. = EndConfigurationTrans. = EndConfiguration

Trans. = BunchTrainStartTrans. = BunchTrainStart Trans. = BunchTrainEndTrans. = BunchTrainEnd

Preparing Run number & type

Ending/Finishing

Checks status & communications

Distributes configurations

Starts the data taking


Transition: StartRun

DAQ DAQ PCPC

DAQ DAQ PCPC

DAQ DAQ PCPC

RCRC FCFC ConfConfDBDB

filefile filefile filefile

read system status

send run number & type

send configurations

filefile filefile

Files to be written for book keeping: system status by DAQ PC run info by RC PC system status by FC


Data StorageScenario I

DAQ DAQ PCPC locallocal

diskdisk

CentralCentralstorestore

Scenario II

DAQ DAQ PCPC


Scenario III

DAQ PCDAQ PCfile in memoryfile in memory


RAIDArray

Which scenario to choose depending on the bandwidth with which the data gets produced: (1) up to 200Mbit/sec, (2) up to ~1600Mbit/sec, (3) from there on Estimation of the data rate for the EUDET DAQ prototype has to cope with ~400Mbit/s, however it depends on the detector and the choice of VFE.


DAQ Software: DOOCS framework

Software development and code base

Computer Infrastructure

Device layer

Middle layer

Communication

Application layer

Sun/Linux Cluster

Software Libs


DAQ Software of DOOCS

Fast

Linac

Image SlowADCADCADC

ADCADCADC

BMBM

DSDSDS MLML

FCFC SCSC

DAQ server

EVBEVB Operator GUIOperator GUI

Local GUILocal GUIRemote GUIRemote GUI

dCachedCache

DCCPDCCPDCCPDCCPDISK

DISK

RCRC

DBDB

RC GUIRC GUIRC GUIRC GUI

ROOT

storagestoragestoragestorage

Data streams

FC/SC:Fast/SlowCollector

BM:Buffer Manager

EVB:Event Builder

DS:DAQ Server


Adapting DOOCS to EUDET DAQ

Modeling hardware card via device server Existing: VME, Sedac, Profi-Bus; ODR/LDA/DIF/ASICsODR/LDA/DIF/ASICs

Equipment Name ServerEquipment Name Server (ENS): Facility(F) / Device(D) / Location(L) / Property (P) e.g. CALICE.ECAL/ODR/ODR1/Status

► F: CALICE.ECAL, CALICE.AHCAL, CALICE.DHCAL► D: ODR, LDA, DIF, ASICs;► L: ODR1,ODR2,ODRX; LDA1,LDA2,LDAX; DIF1,DIF2,DIFX;► Property: X X X ?

Build interface talking to hardware (ODR) as a starting point;

Classify the properties and functionalities of each device for our EUDET DAQ system (DIF, LDA, ODR, etc.)


Future plans

Classify the necessary properties and functions of all hardware components;

Continue their building & testing, debugging & improving meanwhile;

Continue to develop interface to hardware (ODR) talking to DOOCS;

Investigate & define the DIFLDAODR links How to deal with (a)synchronization for C&C? Putting (all) components together, test…


Summary

Off-detector Receiver has been built for receiving, event building & data storage; its performance has been testing;

Clock & Control instead of triggers, the board will be built soon;

DOOCS framework is reusable & suitable for our DAQ system.

DAQ software is in designing phase…


DAQ architecture

Link Data Aggregator(LDA)

Detector Interface(DIF)

Detector Unit ASICs

Off Detector Receiver(ODR)

P. Göttlicher, DESY

DAQ software


Current ODR PrototypeHost System

EthernetEthernetInterfaceInterface

Test DataGenerator

SourceSelector

Buffer pointerBuffer pointerManagerManager

BufferBufferMemoryMemory

UsedFIFO DMA DMA

EngineEngine

PCI Express Buses

Local StorageLocal StorageRAID ArrayRAID Array

Gigabit NICTest source

External data stream

Optical fibre

data stream

data stream

data stream

data stream

Address

da

ta stream

da

ta stream

da

ta stream

Acce

ss p

ointe

rs

Even

t e

xtracts

Rea

d p

oin

ters

Te

st D

ata

Gen

era

tor

Con

tro

l

Off-Detector-Receiver

Card driver codeCard driver codeDMA engine DMA engine ODR local softwareODR local software


RequesterRequester

Current ODR Prototype“Requester”: multithreaded application,data copied to the host memory

ODR VHDL code RequesterRequester

FIFO Dup.

Data BuffData Buff

Data Buff

DMA FIFO to

User buffer

Write pointer

Read pointer

DMA data to user buffer(s)(host memory)

DMA transfer in maximum 4 k (page size) blocks.

Supervisor

Multiple instances of the “requester” thread. Each accessing different ODR card or different channel.


Channel 0

0

100

200

300

400

0 1000 2000 3000 4000

Data size [bytes]

tran

sfer

rat

e [M

B/s

]

1 Buff

5 Buff

10 Buff

15 Buff

20 Buff

25 Buff

Channels comparison

290300310320330340350360

0 1000 2000 3000 4000

Data size [bytes]

Tra

nsf

er r

ate

[MB

/s]

Ch0

Ch1

Ch2

Ch3

Establishing optimum numberof user-defined DMA buffers.No performance improvementabove 20 allocate buffers,

Comparison of all channels.Number of data buffers: 20

Performance (Int.Data Gen)


Simultanious access

0

200

400

600

800

0 1000 2000 3000 4000

Data size [bytes]

Tra

nsf

er r

ate

[MB

/s]

Ch 0

Ch 0&1

Ch 0&1&2

Ch 0&1&2&3

Improvements in performance when accessing more then one channel. The maximum performance achieved for 3-channels access. Adding additional Channel (4-channel access) does not yield improvement in transfer rate.

Performance (continuation)


Equipment Name Server (ENS)NN N+1N+1 N+2N+2

N+2,1N+1,1N,1 N+1,2

N+2,1,1N+1,1,1 N+1,2,1 N+2,1,2

…,1 …,2 …,1 …,1 …,1 …,2 …,3

K K+1 K+2 K+3 K+4

Facilities

Devices

Locations

Properties

Entries

Select between devices of the same type in different facilities in on-line addressing of the control system.

Provide the resolution of the names of the devices used in the control system with one entry per device server.

Define user authentication parameters which are used by the control software to restrict the access to the specified control devices


ENS

Client programs

ENSFacility/Device/Location/Property

Query device name

Equipment device

Addresses of equipment server

actual data request to the

equipment server

Facility: CALICE.ECAL, CALICE.AHCAL, CALICE.DHCAL Device: ODR, LDA, DIF, ASIC1, ASIC2, ASIC3 Location: ODR1, ODR2, ODRX;

LDA1, LDA2, LDAX; DIF1, DIF2, DIFX Property: X X X ?

ENS can signal connectionssignal connections by additional properties, e.g. for device DIF:CALICE.ECAL/DIF/DIF1/ODR_CONCALICE.ECAL/DIF/DIF1/LDA_CON CALICE.ECAL/DIF/DIF1/DEBUG_MODE

off-detector & software development of daq system for the eudet calorimeters tao wu on behalf of...

Documents

data storage slide

daq pc file

data rate

modularized data

daq components

link slide

blown software daq lda

bunch train data