ssd upgrade
DESCRIPTION
SSD upgrade. M. LeVine, R. Scheetz BNL Ch. Renard , S. Bouvier Subatech H. Matis , J. Thomas LBNL. HFT components. HFT. SSD IST PXL. IFC Inner Field Cage. FGT. Magnet Return Iron. TPC Volume. EAST. WEST. Solenoid. SSD. SSD characteristics. SSD ladder. - PowerPoint PPT PresentationTRANSCRIPT
M.J. LeVine 1STAR HFT meeting, Mar 10, 2010
STAR
SSD upgrade
M. LeVine, R. Scheetz BNL
Ch. Renard, S. Bouvier Subatech
H. Matis, J. Thomas LBNL
M.J. LeVine 2STAR HFT meeting, Mar 10, 2010
STAR
TPCVolume
Magnet
Return
Iron
Solenoid
IFCInner Field Cage
SSDISTPXL
EAST WEST
FGT
HFT
HFT components
M.J. LeVine 3STAR HFT meeting, Mar 10, 2010
STARSSD
M.J. LeVine 4STAR HFT meeting, Mar 10, 2010
STAR SSD characteristics
M.J. LeVine 5STAR HFT meeting, Mar 10, 2010
STAR SSD ladder
M.J. LeVine 6STAR HFT meeting, Mar 10, 2010
STAR SSD expected lifetime in RHIC
• Radiation expected during 12 weeks of 500 GeV p-p running: 3.4 krad– Extrapolated from measurements in STAR IR
• Lifetime limit of SSD silicon+associated (on ladder) electronics: >200 krad– Based on measurements of SSD module in 20 MeV
proton beam
• Radiation limit: >30 years of p-p running
M.J. LeVine 7STAR HFT meeting, Mar 10, 2010
STAR SSD upgrade scope
Improve readout performance
Change mechanical mounting to improvehermiticity and interface with IDS
Improve reliability of air cooling
M.J. LeVine 8STAR HFT meeting, Mar 10, 2010
STAR SSD readout upgrade
• SSD electronics -- designed to read out at 200 Hz
• New STAR DAQ now reads up to 1 kHz with low dead time
• SSD must conform to the new performance standard
M.J. LeVine 9STAR HFT meeting, Mar 10, 2010
STAR Readout upgrade concept
• Reading out front end:
– Replace single ADC with 16 ADCs• digitize 16 modules in parallel
– Increase sampling rate to 5.00 MHz
– All ladders processed concurrently
• Transferring data to PC– Increase link throughput to DAQ PC to 120 Mbyte/s per 5
ladders
• 1850 µs -> 450 µs– Multiple (derandomizing) buffers effectively hides this time
• Dead time: 11%@750Hz, <2%@100Hz• [cf. existing: >80%@750Hz,30% @ 100Hz]
2.5 ms -> 154 µs
M.J. LeVine 10STAR HFT meeting, Mar 10, 2010
STAR
DAQ PC
DDL
DAQ roomOuter support
coneSouth platform
VME crate
RDO (1 of 8)
SlaveFPGA
SlaveFPGA
SlaveFPGA
SlaveFPGA
SlaveFPGA
Master FPGA
DA
Q in
terfa
ceTR
G in
terfa
ce
VME FPGA
Fiber links
Ladder cards
VM
E in
terfa
ce
Readout components
M.J. LeVine 11STAR HFT meeting, Mar 10, 2010
STARData formats
non zero-suppressed– 3 10-bit ADC values to a 32-bit word– Fixed order: position in buffer/word -> geographical position
of strip
zero suppressed– Only strips with ADC value above threshold are present – ADC value (10 bits) + strip location (14 bits)– One strip per 32-bit word– Alleviates large memory access burden on DAQ PC– Doing this in real time in FPGA is simple
M.J. LeVine 12STAR HFT meeting, Mar 10, 2010
STARLadder data path
module
module
X16
5 MHz
5 MHz
adc 12bit16 bit serial output
adc 12bit16 bit serial output
80 MHz
80 MHz
FIFO
16
50 MHzpacker
2
JTAG
TDO
s
20
40 MHz
serializer to fiber
16 bit width32 words deep
Write enable:true on 10 clocks only
1 set of adc samples: 10 X 16 bits => repacked into: 2 X 4 X 20 bits
M.J. LeVine 13STAR HFT meeting, Mar 10, 2010
STAR
deserialize
demux
FIFO1
FIFO2
FIFO3
FIFO4
mux
FIFO1:10 deserialize
JTAG
TDO
s
2
20
first wd
40 M
Hz
50 MHz
word 1
word 2
word 3
word 4
20 bit
16 bit
50 MHz 10
10
5 MHz
x16
x16
word 5
unpacking 4 20-bit words to 5 16-bit words
RDO slave - unpacker
M.J. LeVine 14STAR HFT meeting, Mar 10, 2010
STAR Slave FPGA – ADC processing
Pedestal write
ADC 1
ADC 16
counter
0-767
Pedestalmemory
Pedestalmemory
read
add
ress
10
Subtract/multiplex
Subtract/multiplex
2
mod
e
Mode: ADC,pedestal, ordifference
-
+
-
+
Rea
dout
to m
aste
r FP
GA
Buffer 0..3256X32 bits
address
10
14
writ
e pe
d ad
dres
s
writ
e pe
d da
ta
Select: ADC, pedestal only, or max(difference,0)
10
Packing register
1
2
3
10
1
2
3
1
16
Buffer 0..3256X32 bits
÷ 3 8
read addressre
mai
nder
from unpacker
(no zero suppression)
M.J. LeVine 15STAR HFT meeting, Mar 10, 2010
STAR Dead time calculation - no zero suppression
0 200 400 600 800 1000 1200 14000
5
10
15
20
25
30
35
40
45
50
1 buffer2 buffers3 buffers4 buffers
random trigger rate (Hz)
Dea
d tim
e (%
)
M.J. LeVine 16STAR HFT meeting, Mar 10, 2010
STAR Dead time calculations – zero suppression
0 200 400 600 800 1000 1200 14000
5
10
15
20
25
30
35
40
45
50
1 buffer2 buffers3 buffers4 buffers
random trigger rate (Hz)
Dea
d tim
e (%
)
3 % occupancy
M.J. LeVine 17STAR HFT meeting, Mar 10, 2010
STARStatus: slave FPGA
• Components chosen– Altera Cyclone III EP3C55
• FPGA code largely finished– Functional simulations complete– Post P&R done for subsystems– Full place and route in progress
M.J. LeVine 18STAR HFT meeting, Mar 10, 2010
STAR Status: ladder board
• Analog Level shifter (mux replacement) Serial ADC (16 inputs instead of 1)
• Digital Serial ADC Event control Data packer SerDes GBIC 4 temperature monitors Slow control (JTAG) Debug (USB interface)
• Status - layout of PCB complete
M.J. LeVine 19STAR HFT meeting, Mar 10, 2010
STAR Ladder board PCB
Flex circuit layer
M.J. LeVine 20STAR HFT meeting, Mar 10, 2010
STAR Status: documentation
M.J. LeVine 21STAR HFT meeting, Mar 10, 2010
STAR Testing road map
• Ladder board standalone test – JTAG header to program FPGA for testing– Use USB interface to verify/correct FPGA behavior up to serdes
• RDO standalone test – VME interface to program, probe master, slaves– Use user-defined JTAG to verify, debug
• Integrate ladder, RDO with fiber
• Entire system can be tested w/o DAQ PC– Use VME to communicate with master FPGA
M.J. LeVine 22STAR HFT meeting, Mar 10, 2010
STAR SSD mounting
M.J. LeVine 23STAR HFT meeting, Mar 10, 2010
STAR New SSD ladder arrangement
Cooling plenum
M.J. LeVine 24STAR HFT meeting, Mar 10, 2010
STAR SSD services
Chordal GapsBetween FGTAnd WSC Shell
Air Taken from IFC Volume
OpticalData/Control
Air Coolant (suction)East End Only
ModuleConnectors(internal)
Low Voltage Power inOn Both SidesHV East Side Only
LV/HV (no HV out West)No Cooling out West
M.J. LeVine 25STAR HFT meeting, Mar 10, 2010
STAR
RDO
North Platform
South Platform
DAQ Room Cooling
(RDO & Power)
DAQ Computers
WSC ESCSSD
The SSD is mounted on the OSC
System overview
LV/HVPS
M.J. LeVine 26STAR HFT meeting, Mar 10, 2010
STAR SSD cooling
• Replace Vortex blower (76 kW)• Use maintainable vacuum• Reroute tubing• Instrument with temperature
and flow measurements
M.J. LeVine 27STAR HFT meeting, Mar 10, 2010
STAR SSD ladder power requirements
FEE (ladder modules)
Ladder cards Total
Old SSD 9.6 W 10.0 W 19.6 W
Upgraded SSD 11.5 W 13.6 W 25.1 W
M.J. LeVine 30STAR HFT meeting, Mar 10, 2010
STARSSD milestones
date milestoneQ3
FY10Review of board layout for ladder board and RDOStart board (prototype) fabrication for both
Q2 FY11
Finish testing ladder board @BNL (digital event processing only). Finish testing RDO @BNL. Integrated testing @ Subatech for both boards
Q3 FY11
Develop slow controls and DAQ software
Q4 FY11
Produce full complement of boards
Q4 FY12
Move Full System to STAR for test
Q3 FY13
Install completed SSD in STAR
M.J. LeVine 31STAR HFT meeting, Mar 10, 2010
STARSSD cost profile
Fiscal year
Base cost (K$)
Contingency (K$)
Total (K$)
FY10 120 30 150
FY11 110 20 130
FY12 220 120 340
FY13 210 120 330
FY14 0 0 0
Total 660 290 950
M.J. LeVine 35STAR HFT meeting, Mar 10, 2010
STAR New developments
• Measured analog response at ADCs• Single event upsets and remediation• Improved zero-suppression• Integration issues being addressed
– Cables, fibers, routing, grounding• Safety issues (grounding, floating PS,
fuses, HV)
M.J. LeVine 36STAR HFT meeting, Mar 10, 2010
STARAnalog response vs clock frequency
4.3 MHz
5.0 MHz
6.0 MHz
Horiz:100 ns/cm
M.J. LeVine 37STAR HFT meeting, Mar 10, 2010
STAR Single event upsets
• Ionizing radiation causes single bit errors in configuration memory (internal to FPGA)– Change FPGA behavior
• Scale from observed error frequency in TOF– Estimate 1 error per 10 minutes in SSD
• Must pro-actively detect these errors by running CRC checks while acquiring data
M.J. LeVine 38STAR HFT meeting, Mar 10, 2010
STAR Improvement to zero suppression
• Zero suppression based on preserving above-threshold strips
• Want to retain neighbors as well• Problem: strips are not read out in order
– Have to store them in intermediate buffer, in geographic order, before suppression
– RAM has been added (larger FPGA)– Implementation will come later
M.J. LeVine 39STAR HFT meeting, Mar 10, 2010
STAR Integration issues
• Power cables will be Cu-coated Al• Cables to West (through FGT) are the
most difficult due to limited area• Routing details being addressed• Grounding plan being integrated with
other HFT subdetectors
M.J. LeVine 40STAR HFT meeting, Mar 10, 2010
STAR Safety issues
• Need to ensure that details will be approved by RHIC safety committee– Floating PS with safety resistors– Fuse protection for sense wires– HV considerations
• HV is <100V, I < 0.1ma
M.J. LeVine 41STAR HFT meeting, Mar 10, 2010
STAR SSD - summary
• Design progressing well• Costs have been accurately estimated• Risks are minimal
– No technical risks– Mitigation strategies in place
M.J. LeVine 42STAR HFT meeting, Mar 10, 2010
STAR Spare slides
Spare slides
M.J. LeVine 43STAR HFT meeting, Mar 10, 2010
STAR
---- Backup material
M.J. LeVine 44STAR HFT meeting, Mar 10, 2010
STAR Existing readout configuration
• 1 ADC reads 16 modules sequentially• ADC sampling speed: 3 MHz• Connection to RDO is via copper cable
– Restricts RDO location to magnet face
• 1 RDO serves 10 ladders– Data link to DAQ computer oversubscribed
Performance of existing system:Dead time @100 Hz: 30%Dead time @ 750 Hz: >80%
M.J. LeVine 45STAR HFT meeting, Mar 10, 2010
STAR SSD mounting detail
M.J. LeVine 46STAR HFT meeting, Mar 10, 2010
STAR SSD: interior view
M.J. LeVine 47STAR HFT meeting, Mar 10, 2010
STARSlow controls
• Path from RDO to ladder defined– JTAG transported on fiber pair
• Multiple paths from VME CPU to RDO possible:– FP JTAG connector
• Compatible with existing software, hardware• Provides migration path
– VME interface• Best performance• Requires software development
M.J. LeVine 48STAR HFT meeting, Mar 10, 2010
STARSSD with 3 mounted ladders
M.J. LeVine 49STAR HFT meeting, Mar 10, 2010
STARLadder: exploded view
M.J. LeVine 50STAR HFT meeting, Mar 10, 2010
STARModule: exploded view
M.J. LeVine 52STAR HFT meeting, Mar 10, 2010
STAR Cost details for SSD readout
RDO card (qty: 10)Components $3080Fabrication $ 910Assembly $ 910Total (each card) $4900
Ladder card (qty: 50)Components $1920Fabrication $1250Assembly $ 300Total (each card) $3470
DDL links (qty: 12)SIU $ 670½ DRORC $ 840Total (per link) $1510
DAQ PC (qty: 2)Each $2900
Total production cost:$246.2K
• Includes spares• Costs are burdened• Does not include prototypes
M.J. LeVine 53STAR HFT meeting, Mar 10, 2010
STAR
Data formats: zero suppressed
• Only strips with ADC value above threshold are present – ADC value (10 bits) + strip location (14 bits)
• One strip per 32-bit word• Alleviates large memory access burden on DAQ PC• Doing this in real time in FPGA is simple
M.J. LeVine 54STAR HFT meeting, Mar 10, 2010
STAR
Data flow: Ladder card
• Each yellow box (x5) represents one ladder– Connection card + ADC card– Dual input ADC (x8) with bit-serial outputs– 16 outputs feed 1 gigabit serializer
• Green boxes: readout card• Connection to readout card
– gigabit fiber pair per ladder
M.J. LeVine 55STAR HFT meeting, Mar 10, 2010
STAR
Dataflow: Readout card
• Per ladder (5X):• Optical link • Deserializer• Slave FPGA
• 1 master FPGA– Manages data from the ladder FPGAs– Manages the DDL connection to the DAQ PC– Manages connection to TRG
M.J. LeVine 56STAR HFT meeting, Mar 10, 2010
STARExisting readout configuration
M.J. LeVine 57STAR HFT meeting, Mar 10, 2010
STAR Data flow organization
M.J. LeVine 58STAR HFT meeting, Mar 10, 2010
STARReadout components
RDO
DAQ PC
DD
L
Dup
lex
fiber
pai
r
Outer support cone
South platform VME crate DAQ room
Ladder card(1 of 5)
Ladder card (1 of 5)
M.J. LeVine 59STAR HFT meeting, Mar 10, 2010
STAR Funding profile
• FY 2010 $150K prototyping• FY 2011 $25K shipping, tools, …• FY 2012 $250K production• FY 2013 $15K tools
M.J. LeVine 60STAR HFT meeting, Mar 10, 2010
STAR SSD mechanical development
• Replace Vortex blower (76 kW)• Use maintainable vacuum• Reroute tubing – avoid kinks• Instrument with temperature
and flow measurements
M.J. LeVine 61STAR HFT meeting, Mar 10, 2010
STAR Status: ladder board (analog)
16 x
Hybrid(Alice128)No change
Flex(<=1m long)No change
Power switch(+2V & agnd)No change
Power measure(-2V)
No change
Level shift(4V->2.5V)
Mux replacmt.
Serial ADC(12b,5MS/s)
16 instead of 1
connector(30p)
No change
M.J. LeVine 62STAR HFT meeting, Mar 10, 2010
STARStatus: ladder board (digital)
x16 x1Power switch(+2V & agnd)No change
Power measure(-2V)
No change
Level shift(4V->2.5V)
Serial ADC(12b,5MS/s)
SerDes(24b,40MHz)
Data packer(5*16->4*20)
Bias & latchup()
No change slow control(JTAG)
Event ctrl(hold,test,clk,token)
temperature(x4)
gbic(1Gb/s)
debug(8b,usb)
Flex(<=1m long)No change
M.J. LeVine 63STAR HFT meeting, Mar 10, 2010
STARSSD cost profile
2009 2010 2011 2012 2013 2014$0
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
$350,000
$400,000
$450,000
$500,000
M.J. LeVine 64STAR HFT meeting, Mar 10, 2010
STAR Ladder board PCB
Flex circuit layer
M.J. LeVine 65STAR HFT meeting, Mar 10, 2010
STAR SSD original layout
M.J. LeVine 66STAR HFT meeting, Mar 10, 2010
STAR Risk assessment
• Digital design fully simulated• Oversights in layout
– Schedule and budget allow for extra iteration in prototypes
• Schedule impact– None (slack in schedule)
• Cost impact– No impact foreseen on production costs
M.J. LeVine 67STAR HFT meeting, Mar 10, 2010
STAR
Upgrade overview
• Ladder-modular organization– 5 ladders send data to a readout card– 4 readout cards @ each end of SSD
• Each readout card feeds a DDL link to a DAQ PC– A PC-based DRORC card hosts 2 DDL fibers