Project Overview
Ted Liu
Fermilab
Sept. 27th, 2004
L2 Pulsar upgrade IRR Review
http://hep.uchicago.edu/~thliu/projects/Pulsar/L2_upgrade_meeting.html
Materials for the review can be found at:
Q&A
Are we ready for operation? No
Why are we having this review then? To help us getting ready for final operation. To review
- what we have achieved so far
- our plans for final commissioning
- is there anything missing on our to-do-list?
- is manpower enough?
- Is the schedule realistic?
- …
Will have another review early next year
Qutline
Baseline design for operation and overall status Project Organization: who is working on what General testing&commissioning methodology Firmware Overview and necessary improvements Firmware Version Control Comments on possible future system optimization Manpower issues Summary
Baseline for operation
Pulsar-based system is very flexible Should only focus on the baseline for operation
single algorithm node deliver non-SVT and SVT data on separate PCI path simple algorithm in pre-processor Pulsars simple data merging use S32PCI64 cards instead of FILAR …
Will comment on some system flexibility later further possible improvements/optimization depend on
initial operation experience, not required for operation
L1 muon
L1 XTRPL1 trigger
L2 CAL(CLIST/Iso)
PreFred
ShowMax
(RECES)
PC
SLINK
TS
SVT
Muon
Cluster
Electron
Merger
Merger
SVT
L2toTS
Commissioning strategy:(1) Use Pulsar Tx Rx in teststand(2) Split all input signals to Run IIa
system, run parasitically with detector and beam
Baseline for operationPulsar pre-processors
Main tasks for the project
Hardware: custom: Pulsar,mezzanine&AUX, LVDS & fiber active splitter commercial: fiber passive splitter, SLINK mezzanine, SLINK-PCI cards, Linux processors Firmware: transmitters and receivers for all data paths SLINK merger and L2 to trigger supervisor interfaces about 14 different types of Pulsar firmware Software: board testing, VME DAQ readout, online/offline monitoring, software for algorithm&control node, interface with rest of world …
Overall Status Hardware:
All custom hardware in hand, and all fully tested
except fiber active splitter (see Chris’s talk)
need to order two Algorithm PCs soon, one in hand Firmware:
All firmware in place and tested in beam…some needs
fine tuning for either performance optimization or robustness
more on firmware later Software:
main work left is related to algorithm and control node
see later talks for details
Who is working on what (I)
Muon/XTRP/L1 + SVT + Merger + RECES + L2TS Tx Firmware: Tomi (work done, left) Rx Firmware: Sakari Testing: Burkard, Frans (left), Cheng-Ju (L2TS) + Sakari Commissioning: Burkard + Cheng-Ju Monitoring for Muon/XTRP/L1/SVT/Merger: Shawn Monitoring for L2TS: Cheng-Ju Monitoring for RECES: Chris with Daniel’s help earlier Active Splitter for RECES: Chris LVDS Splitter Board: Wojtek
(only the main soldiers are shown here)
Who is working on what (II)
Cluster/PreFred Tx Firmware: Tomi (work done, left) Rx Firmware:
DataIO FPGA: Vadim
Control FPGA: Sakari (similar to other cases) Testing/commissioning/monitoring: Vadim and Wojtek
Algorithm node software: Kristian Control node software: Daniel
(only the main soldiers are shown here)
Who is working on what (III)
General Pulsar Hardware: Burkard VME readout software: Jane from DAQ + Burkard’s help PulsarMon design: Vadim RunCTRL/TriggerDB etc interfaces: Daniel Firmware coordination: TL
Commissioning coordination:
in the past: TL
in the future: Trigger SPLs
(See Cheng-Ju’s talk on plan and schedule)
Testing and Commissioning Methodology
Pulsar is self-testable, at both board and system level This design feature is the key to success
Pulsar Hardware testing all done with self-testing capability Pulsar firmware testing mostly done with self-testing capability Software development all done with Pulsar self-testing capability Even at system commissioning, a few Tx boards are used This has made the commissioning work much easier
System commissioning relies heavily on input signal splitting Saved us LOTs of dedicated beam time So far, only used a few hours dedicated beam time Input signal splitting has been and will be extremely helpful Will keep input signals split even after final commissioning to have a “copy” of the new system running parasitically for hot spares, for firmware/software improvements
Pulsar Firmware: BIG effort
Pulsar has three main FPGAs L2 decision upgrade:14 different type Pulsars (Tx/Rx) Key to success:
Design: common re-usable modular design Simulation:
no simulation no testing Dedicated testing support Version Control Careful firmware
update procedures …
081 L2 Pulsar Muon/XTRP Rx IIa083 L2 Pulsar SVT Road Warrior085 L2 Pulsar Muon/XTRP/L1 Tx or SVT XTRP-emu086 L2 Pulsar Muon/XTRP/L1 Rx IIb087 L2 Pulsar RECES Tx088 L2 Pulsar RECES Rx089 L2 Pulsar Cluster/PreFred Tx090 L2 Pulsar Cluster/PreFred Rx091 L2 Pulsar SVT Tx092 L2 Pulsar SVT Rx093 L2 Pulsar Merger Tx094 L2 Pulsar Merger Rx095 L2 Pulsar L2TS “Tx” (reserved spare ID)
096 L2 Pulsar L2TS097 L2 Pulsar L1 Scaler098 L2 Pulsar SVT Track Fitter
099 L2 Pulsar test Tx100 L2 Pulsar test Rx101 L2 Pulsar XFT Stereo Tx102 L2 Pulsar XFT Stereo Rx
103 L2 Pulsar SVT Hit Buffer 104 L2 Pulsar SVT AMSRW Board ID
On
e H
ard
ware
-- many
ap
plica
tion
s
14 Pulsar types/firmware for L2 decision upgradeSakari
TomiSakariTomiSakariTomiVadim/SakariTomiSakariTomiSakari
Sakari
SakariSakari
people
other types for SVT/XFTor future improvement
Interfaces (Tx/Rx):SVT/XTRP/L1/PreFred
TS & SLINK (P3)
Interfaces(Tx/Rx):
MuonRecesCluster
IsoSlink
FPGAs and data interfaces
General simplified data flow
Mezzinterface
DAQbuffers
algoslink
formatter
Same as above
VMEresponse
CDFCTRL DataIO
DataIO
DAQbuffers
merge slinkformatter
DAQbuffersVME
response
CDFCTRL
DAQbuffers
XTRP/SVT
Well organized firmware effort is the key: clean/easy-to-understand/robust with minimal
maintenance effort
Inputinterface
DAQbuffers
algoOutputFIFO
Slinkformatter
VMEresponse
CDFCTRL
DAQbuffers
• all common parts are shared by all firmware• also share many low level routines (latency timer, bunch counter etc)
DataIO or Control FPGA
VME responsesSLINK formatter
DAQ buffersCDFCtrl responses
Mezzanine interfacesLatency timerBunch counter
…
Common core firmware (library)
Knowing one data path Pulsar knows all PulsarsData specific algorithm difference is just minor details
hotlink
muon
cluster
Taxi Iso
showermax
LVDS+ ext. FIFO
SVT
track
LVDSL1
TS
core firmware developed by people dedicated to the task
Formatversion
Datasource
RegionID
ReservedBunchcount
Buffer#
8 4 2 8 28
Data elements
Latency (previous) Latency (current)
16 16
Data size Error flags
1616
Keep it simple
SLINK data format
was defined asdata size before,which means onehas to wait untilall data arrivesbefore sending outSLINK package.
will remove it,and use it aslatency stampinstead.
SLINK Merger fragments into event package
Slink inputs
AUX cardSlink output
Four Slinkmezz cards
Merger header 2
Merger trailer
Input 1 header 1
Input 1 trailer
Input 2 header 1
Input 2 trailer
Input 3 header 1
Input 3 trailer
Input 4 header 1
Input 4 trailer
Slink input 1
data
Slink input 2
data
Slink input 3
data
Slink input 4
data
Input 1 header 2
End of fragm
ent
Input 2 header 2
Slink merger output
End of fragm
ent
Input 3 header 2
End of fragm
ent
Input 4 header 2
End of fragm
ent
0xE0F00000
Merger header 1
32-bit SLINK to 64 bit PCI interface card: S32PCI64
• highly autonomous data reception• 32-bit SLINK, 64-bit PCI bus• 33MHz and 66 MHz PCI clock speed
Slinkto PCI
Slinkto PCI
mem
CPUPCI to Slink
data
data
L2 decision
General comments on Firmware
Firmware is not hard if it just has to be “sort of working” The hard part is to get it “rock solid” Firmware is not a place to show how smart one is rather to show how many mistakes one could make
Firmware testing is as important as writing/simulating Firmware testing is often much more time consuming
than the actual writing How good the firmware depends on how well it is tested
Our general strategy:
firmware developer supported by people dedicated to testing
• Files kept• VHDL source code• Files necessary for recreating this version• Readme with revision history• FPGA configuration files the FPGAs
• Source code (working version) under CVS control
• Firmware archive updated on different PCs
How to keep Firmware in order
We have so many different types of firmware Keeping them in order is non-trivial:
under CVS control: source VHDL code, pin files etc Firmware version ID VME register per FPGA
(has to update by hand) README file update Follow careful firmware update procedures
There has been no confusion about firmware versions when the procedure has been followed
te
slot 05: IDPROM: 0016 086 PULSAR MUON RX CTRL: 04408060,DataIO1: 08407020, DataIO2: 08407020 slot 13: IDPROM: 0004 092 PULSAR SVT RX CTRL: 05408060, DataIO1: 09407280, DataIO2: 09407280 slot 15: IDPROM: 0028 093 PULSAR SLINK TX CTRL: a0406300, DataIO1: 08407230, DataIO2: 08407230 slot 17: IDPROM: 0061 096 PULSAR TS INTERFACE CTRL: 0a407280, DataIO1: 09408030, DataIO2: 09406040 slot 19: IDPROM: 0012 094 PULSAR SLINK MERGER CTRL: 02408080, DataIO1: 01407220, DataIO2: 01407220 slot 21: IDPROM: 0014 094 PULSAR SLINK MERGER CTRL: 02408080, DataIO1: 01407220, DataIO2: 01407220
Example of Pulsar crate ID&version scan by Burkard on Aug 19th after Beam test:
Not all boards are shown due to limited spaceAllow us to go back to the same configuration after shutdownHow can we check these against database when in operation mode?
Necessary firmware optimization
before operation
Reces: zero-suppression + latency optimization Reces data comes into Pulsar 8 us after L1A current firmware is not optimized needed to reduce Reces path latency by a few us Status: in progress, estimated time: ~ one month
Slink formatting: sending data words out
upon arrival (not waiting for all data to arrive) improving Slink merging latency (~ a few us) estimated time: ~ one month (including testing)
More timing measurements add latency timer in a few other places (relatively easy)
Cluster path improvement (see Vadim’s talk)
Future possible system optimization after operation
Use FILAR cards instead of S32PCI64 Remove the final merger, data directly goes into FILAR Possibly also remove Reces merger with another FILAR Also allow us to run SLINK faster: 40 MHz ~60 MHz Should reduce the latency by ~ few us Need PCI software modification (no firmware change)
Move L1 scaler into firmware if needed Move to 4 PCs instead of single PC (how much do we gain?)
Additional firmware (L2TS) work and some software work
Muon and track matching in firmware Bring upgrade XFT stereo info into L2 (this will happen) …much more, depends on what we learn&need by then
FILAR can improve the system latency Higher bandwidth, less latency overhead
64-bit/66MHz PCI bus, four 2 Gbit/s S-LINK channelsReduced PCI protocol overhead (compare to S32PCI64)
L1 muon
L1 XTRPL1 trigger
L2 CAL(CLIST/Iso)
PreFred
ShowMax
(RECES)
PC
SLINK
TS
SVT
Muon
Cluster
Electron
Merger
Merger
SVT
L2toTS
System with FILARs
Pulsar pre-processors
Send all data pathsdirectly into 2 FILARs…
Manpower Issues
Why should we worry? Tomi and Frans left (two key people on firmware/testing) Only10% Burkard’s time available from now on
(key person on hardware/VME software/firmware testing/system commissioning need replacement SOON)
Kristian needs to graduate …
(key person on algorithm/PCI software need
replacement SOON …) Need new L2 software coordinator (Daniel?) … anyone else will be full time on the project?
This is very serious …need this committee to help!
Summary Review should focus on baseline for operation Keep in mind that system is flexible Hardware is in good shape
(still need to fully test the active splitters) Firmware is in reasonable shape, needs fine tuning/testing in
some cases
key is to follow our testing methodology Software needs more work, no show stoppers
clear spec is the key Manpower issues serious (in a few areas): need new people! Details are in the rest of talks …