First group planning meeting: Introduction

• Introduce new people• A brief history of time• Goals of this meeting• Project overview• Roadmap • Guideline for the project planning

…real talks by others…

• Some starting points for discussion session(see web). Initial thoughts (or targets for you to shoot at): near term hardware needs/production plan manpower/organization/near term goals … future meetings & L2 upgrade e-log

- Ted Liu, Jan. 30th, 2003

2

New people who joined the project recently:

• Burkard Reisert (FNAL new posdoc)• Cheng-Ju Lin (FNAL posdoc)• Paul Keener (Upenn, DAQ expert)• Joe Kroll & his $$ (Upenn)• Kristian Hahn (Upenn, Nigel’s student)• Shawn Kwang (UC, Mel’s student)• Bob Blair (ANL)• John Dawson (ANL)• Jimmy Proudfoot (ANL)• Franco Spinella (INFN)

“Old” people who is eager to come back:• Sakari Pitkanen (still waiting for visa @ Finland) => youngest collaborator

This project is attracting the best people/groups @ CDF …

3

A brief history of time …

Oct. 2001: FNAL decided to build Pulsar board UofC decided to provide engineer support soon Upenn purchased obsolete interface components

Mar. 2002: INFN built SVT-TSI-PCI daughter card

Aug. 2002: ANL possibility with Atlas RoIB

Oct. 2002: Pulsar works with ANL Gigabit Ethernet mezzanine card (essentially plug&play)

ANL/FNAL/INFN/UofC/Upenn

We have a strong group, now we need a good plan

4

Goals of this meeting

• Focus on planning, not technique details• Roadmap • Near term goals and long term plan• Manpower availability (current/future)• Who is interested in working on what&when• Get a clear picture of the issues/tasks involved will have a few technique talks

5

Project Overview

may roughly divide into four “parallel” efforts:

• E1: Pulsar hardware/firmware/VME software

• E2: Design spec. for data format/algorithm/system interface

• E3: RoIB option effort (see Bob’s talk)

• E4: PCI/CPU architecture/OS/Infrastructure SF/L2D SF …

E1E2E3E4

Phase A Phase B Phase C

CDF RunIIb L2

Atlas L2

6

Roadmap may divide into three phases/stages:

• Phase A: Teststand, core firmware, HW prod/testing Algorithm firmware specification PCI/CPU/OS/infrastructure software

will show some details • Phase B: Rx algorithm firmware implementation/testing, system integration in standalone mode

• Phase C: System integration in parasitic mode, test runs with cosmic and beam

7

TRK

Control/ Merger

TS

L1

Data IOTRK

Mezz cardconnectors

Pulsar design

9U VME(VME and CDF ctrl signals are visible to all three FPGAs)

P1

P2

3 Altera APEX 20K400_652 (BGA) FPGAs 502 user IO pins each

P3

SLINKsignal lines

Data IO

spare lines

SRAM

SRAM

128K x 36 bits

3 APEX20K400 FPGAs on board = 3 Million system gates/80KB RAM per board2 128K x 36 pipelined SRAMs with No Bus Latency: 1 MB SRAM (~5ns access time)

8

Gigabit EthernetRF clock

SRAMs

Pulsar is designed to be fully self-testable: at board level as well as system level

For every input/output, there can be an output/input, All interfaces are bi-directional (except VME)

It is this feature which allows us to develop&tune an upgrade system in standalone mode Overall planning should be based on this feature

SVT/XTRP

TSI

one for all andall for one

9Main Injector

(new)

TevatronDØCDF

Chicago

p source

Booster

αSVT

XTRP

CLIST

ISO

L2 decision crate

L1

RecesHP scope

LogicalAnalyzer

HP scope

The way we have been debugging the Level 2 decision crate: very often need to use CDF+Tevatron HUGE amount of work has been done this way by a few hardworking experts

The idea is to build test stand tools to “replace” CDF and Tevatron, to make life MUCH easier.

L2 inputs

TS

Why diagnostic tools?

An old slide from an old talk on Dec. 7th, 2001: original motivation to build Pulsar: reduce impact on running exp.

10

Main Injector(new)

TevatronDØCDF

Chicago

p source

Booster

A slightly modified version for funding agency

CDF can be used asWeapons of top mass construction

Pulsar can be used as Weapons against mass distractionto reduce impact on running experiment

L2Temulation

Pulsar can be used to emulate both CDF/Tevatron and ATLAS/LHC for L2T

realdata

11

Hardware setup during Phase A:

PC

(1) Pulsar Tx => Rx =>PCI=>PC w/ 4 hotlink mezzanine cards (muon/L1/SVT/XTRP/TSI paths)Fully develop teststand capability (should test with RunIIA boards)Develop core firmware for all casesRobustness/Production test setup (see Burkard’s talk) (2) Pulsar Tx => Rx => PCI=>PC 4 Taxi mezzanine cards (Reces path)(3) Pulsar Tx => Rx => PCI=>PC 2 Hotlink (Cluster) & 2 Taxi (Isolation)

Tx

Rx

12

Core firmware example: DataIO FPGA (Rx case) => Control FPGA is similar

DAQ buffersSRAMinterface

VME responses

RAM/Spy

buffers

RAM/ Spy

buffers

MezzanineCard interface

CDF Ctrlinterface

upstreamdownstream

Blocks in green: common to all FPGA types, blocks in whiteare data path specific (core firmware only deal with thesimplest case: pack ALL data into SLINK format)

Play/recordPlay/record

•Firmware was designed this way from day one

SLINKformatter

Filteralgorithm

13

Well organized firmware effort is one of the keys to have a system:clean/easy-to-understand/robust & with minimal maintenance effort

VME responsesSLINK formatter

DAQ buffersCDFCtrl responses

SRAM ctrlMezzanine interfaces

Diagnostic RAMsSpy buffers

…

Common core firmware

Knowing one data path Pulsar knows all PulsarsData specific algorithm difference is just minor details

hotlink

muon

cluster

Taxi Isolation

Reces

LVDS+ ext. FIFO

SVT

XTRP

LVDSL1

TSI

Need dedicated people responsible for developing the core firmware

14

(1) In God we trust, everything else we test

(2) One for all and all for one

Summary of what I have said orGuideline for the project planning:

Project planning should follow Pulsar design philosophy

Anyone who has a problem with this is in the wrong project

15

Goals of this meeting

• Focus on planning, not technique details• Roadmap • Near term goals and long term plan• Manpower availability (current/future)• Who is interested in working on what&when• Get a clear picture of the issues/tasks involved

will have a few technique talks…

By now, hopefully you are all warmed up…

16

Agenda

• Introduction (TL 20mins) • Pulsar hardware status (Mircea, 5mins)• Automating Pulsar test procedures (Burkard, 10mins)• L2 data format/real data patterns (Cheng-Ju, 10mins)• Pulsar algorithm firmware spec. for Iso&Reces (Bob, 20mins)• Status/plan of S32PCI64/CPU/OS study (Paul, 20mins)• ATLAS RoIB option (Bob, 20mins)• Discussions (ALL, 60mins)• Summary (5 mins)

• dinner at 6pm

17

Material (some initial thoughts) for discussion session

• Some details on hardware needs during phase A/B/C • Manpower/organization

• Near term goals for each effort

• Future meetings & L2 upgrade e-log

• More material can be found on the meeting web pagehttp://hep.uchicago.edu/~thliu/projects/Pulsar/ L2_upgrade_meeting.html

18

Hardware setup during Phase A1:

PC

(1) Current setup (since Nov 02): Pulsar Tx => Rx =>PCI=>PC with 4 hotlink mezzanine cards (for muon/L1/SVT/TSI path)• uses CERN AUX card• uses old SLINK to PCI card &• uses ANL Gigabit Ethernet card• uses an old PC

Tx

Rx

Will need to: • test with our AUX card (next month)• need 4 more hotlink mezz cards as reference cards to test new Pulsars

Some details

19

Hardware setup during Phase A2:

PC

(2) Next setup (March 03 -- ): Pulsar Tx => Rx =>PCI=>PC with 4 Taxi mezzanine cards (for Reces path)

Tx

Rx

Will need to: • test the two new Pulsars with the 4 hotlink mezz. reference cards first

•2 more Pulsars from prototype run (ready for testing next week)• 4 Taxi mezz. cards (next month)

20

Hardware setup during Phase A3:

PC

(3) Next setup (~April 02): Pulsar Tx => Rx =>PCI=>PC with 2 hotlink/Taxi cards (for Cluster/Isolation paths)

Tx

Rx

Will need to: • test the two new Pulsars with the 4 hotlink mezz. reference cards first• new PC with new PCI card

• 2 more Pulsars from pre-production• 2 hotlink/Taxi mezz. cards

Once this setup is fully developed&tested: time for production

21

Pulsar hardware needs during Phase A (summary):(1) Current setup: Pulsar Tx => Rx =>PCI=>PC with 4 hotlink mezzanine cards (for muon/L1/SVT/TSI paths) * 4 more hotlink mezzanine cards as reference cards for new motherboards testing * one AUX card (5V) (2) Next setup: Pulsar Tx => Rx => PCI=>PC with 4 Taxi mezzanine cards (Reces path) * 2 more Pulsar boards from prototype batch (next week) * 4 Taxi mezzanine cards from prototype run * one more AUX card (3.3V)(3) Next setup: Pulsar Tx => Rx => PCI=>PC with 2 Hotlink (Cluster) & 2 Tax(Isolation) * 2 more Pulsar boards from pre-production run * 2 Hotlink/Taxi mezzanine cards

Production starts after (1) –(3) fully tested

22

MUON

RECES

ROC

TRACER

CLU/ISO

RECES

RECES

RECES

MUON

RECES

ROC

TRACER

CLU/ISO

RECES

RECES

RECES

Tx

Rx

Teststand Area

…

Hardware setup for Phase B:Does RoIB need a separate crate ?

•Initial system level integration and performance studies•Finalize data path specific firmware, reduce the data size as necessary• online monitoring software• …

TSI

23

MUON

RECES

ROC

TRACER

CLU/ISO

RECES

RECES

RECES

Rx

L2 Central Crate inTrigger Room

…

Hardware setup for Phase C:

•Initial system level integration with real system• fine tune FW/SF/performance• parasitic running if possible• special test runs with cosmic and beam• …

In addition to the teststandarea setup in standalone mode,setup the full system (Rx) in L2 central crate next to thecurrent L2 decision crate

24

Project Overview

may roughly divide into four “parallel” efforts:

• E1: Pulsar hardware/firmware/VME software

• E2: Design spec. for data format/algorithm/system interface

• E3: RoIB option effort (see Bob’s talk)

• E4: PCI/CPU architecture/OS/Infrastructure SF/L2D SF …

E1E2E3E4

Phase A Phase B Phase C

25

(E1) Pulsar hardware/firmware/VME software

Pulsar/mezz/Aux production: UC shop Production testing: Burkard/Sakari+Kristian + … Technical support: UC(Harold/Mircea)/FNAL(Bob Demaat) Infrastructure support: Crates/TSI/Testclk/Tracer: CJLin/PWittich

Core firmware spec/implementation: Burkard/Sakari + others Core VME software(testing,online): Burkard/Kristian/Sakari subsystem specific algorithm firmware/VME software: muon path: Shawn with support from Sakari/Burkard Reces path: ? … Cluster path: Kristian … Isolation: Kristian … L1/XTRP/SVT: part of the core firmware Overall coordinator: Burkard Reisert Consultant: Mel Shochet/Bob Blair/John Dawson/Peter Wilson/Jonathan Lewis...+ volunteers

26

E1: possible near term goals

(1) Feb: have the automated testing procedure fully working for 16

hotlink channels (muon path): Tx->Rx->PC;

(2) March: finish testing 2 more Pulsar boards with hotlink/Taxi mezzanine

cards and AUX cards;

(3) April: initial proof of principle test with S32PCI64 with test setup

(4) May: core firmware (Tx/Rx) fully developed and tested with

hotlink/taxi mezzanine cards;

(5) June: production/testing for Pulsar/mezz cards/AUX cards

(6) July: have Tx->Rx-PC fully developed for Reces(taxi), cluster/Isolation (hotlink/taxi);

(7) Aug: document everything above: cdfnote

27

(E2) Firmware spec./SLINK data format/data patterns

System Level(TS handshake, DAQ etc): Peter/Mel/Bob Blair/John + others Subsystem specific: muon: Mel/Cheng-Ju/Shawn Kwang Reces: Bob Blair/Cheng-Ju/? Cluster: Cheng-Ju/Mel/Kristian Isolation: Bob/Cheng-Ju/Kristian L1/XTRP/SVT: Cheng-Ju/Burkard/Mel Overall coordinator: Cheng-Ju Lin Consultant: Jonathan Lewis/Peter Wilson/Henry Frisch/Jimmy Proudfoot/... + volunteers

28

E2: possible near term goals

(1) Feb: document ALL L2 data format; (2) March: software ready to derive all real data

patterns; (3) April: initial algorithm specification for each

path; (4) May: software modeling based on algorithm

specifications online monitoring software(5) June: finalize algorithm/system firmware

specification(6) July: document everything above --> cdf note

ready

29

(E4) PCI/CPU/OS/infrastructure software/L2D software

need more thinking here for the organization…need a posdoc in this area ?

R&D setup at Upenn: Paul Keener PCI/CPU/OS/infrastructure software development Upenn has funding from the university for this R&D effort.

Testing/integration setup at ANL: Bob Blair/John Dawson +?

Testing/integration setup at FNAL: Kristian + ?

L2 algorithm software: Peter Wittich/Bob/Cheng-Ju/Kristian/Burkard...

Overall coordinator: Peter Wittich

Consultant: Jim Patrick/Rick van Berg/Joe Kroll/Franco Spinella/Bill Ashmanskas...+

30

E4: possible near term goals

(1) Feb: at the very least repeat CERN's results on S32PCI64 performance;

(2) March: pick one PC platform/OS with reasonable performance

(doesn't have to be the best)(3) April: basic infrastructure software ready early April and

transfer to FNAL teststand, along with a PC/OS. This will be used

for proof of principle test at FNAL in April and teststand

needs(4) May-June: more studies to see if there is any better choice

for PC platform/OS. study on how to send the L2 decision

back to Pulsar (5) July: final decision on PC platform/OS... and

infrastructure software... (6) Aug: final decision on how to send L2 decision back to

Pulsar. (7) Sept: document everything above --> cdfnote ready

31

Future meetings • Group meetings: Once per month is reasonable I would suggest that each groupANL(Bob)/FNAL/INFN(Luciano)/UC(Mel)/Upenn(Joe) take turn to organize/chair the meetings

• Subgroup meetings during phase A:

bi-weekly regular meetings or let each coordinator decide when to call the meetings?

• create a L2 upgrade e-log? too many meetings too much BSWe have good people who want to spend more time to do real work Based on years of observation: much less BS on e-log an effective way to improve communication

32

RunIIA and RunIIB

2B or not 2B?