flare workshop november 4-6, 2004 1 data acquisition mark bowden, margaret votava
TRANSCRIPT
FLARE Workshop November 4-6, 2004 1
Data AcquisitionData Acquisition
Mark Bowden, Margaret VotavaMark Bowden, Margaret Votava
2FLARE Workshop November 4-6, 2004
ParametersParameters
250,000 channels
4 Bytes/chan
2 MHz sample rate
Front-end data rate = 2 TBytes/sec
3FLARE Workshop November 4-6, 2004
Front-end (ICARUS)Front-end (ICARUS)
Caen V789 (Digital Board) + V791 (32 channel A/D Board) Rack 18 X 32 channels
4FLARE Workshop November 4-6, 2004
FLARE Front-end AssumptionsFLARE Front-end Assumptions
• Newer components allow greater packaging density
128 or 256 channels (if new amplifier/shaper ASIC)
Digital board can be reduced to single FPGA andcombined with Analog board
FE Board count drops from 16,000 to 2,000
• Eliminate VME
VME is expensive and provides no added value
Direct Ethernet connections to FE boards
6U Eurocard or 1U (pizza box) packaging
5FLARE Workshop November 4-6, 2004
FLARE Front-endFLARE Front-end
40 X 128 channels (6U packaging) 20 X 256 channels (1U packaging)
48V Power
6FLARE Workshop November 4-6, 2004
Data RatesData Rates
1 MByte/sec
2 GByte/sec
1 KByte/sec
2 MByte/sec
1 MByte/sec
2 GByte/sec
1 GByte/sec
2 TByte/sec
Without FE Data Compression
With FE Data Compression
Spill Only*
Continuous
* 2 msec per 2 sec cycle ** assuming 128 channels/FEB
- per FEB**
- system total
Note: BTeV is 0.5 TByte/sec
7FLARE Workshop November 4-6, 2004
Data NetworkData Network
…
FE switches (100) $15KGBE switches (50) $25KCables $10KTotal $50K
… … … … …
……
……
……
……
……
……
5 GByte/sec
8FLARE Workshop November 4-6, 2004
Switch-based DAQSwitch-based DAQ
• Any size/bandwidth system can be built from low cost switch components.
• Cost scales as NlogN.
• Up to 100m (copper) between switches.
9FLARE Workshop November 4-6, 2004
Processing & Data StorageProcessing & Data Storage
1U dual processor $2K
DVD writer $7K
(Delay purchase of computers to maximize performance/cost.)
10FLARE Workshop November 4-6, 2004
““Event” ProcessingEvent” Processing
• An “Event” is an arbitrary time slice of data from all channels
2 msec 4 MBytes
20 msec 40 MBytes
200 msec 400 MBytes
2 sec 4 GBytes
• Choose largest time slice that fits in processor memory
• For this application, ~100 msec
11FLARE Workshop November 4-6, 2004
““Event” ProcessingEvent” Processing
Front-end (2000 data sources)
Ethernet Network
Processors
1 2 2000
1 2 200
Source 1
Source 2
Source 3
1 2 3 200
1 2 3 200
1 2 3 200
1 2 3
1 2 3
1 2 3
Destination
12FLARE Workshop November 4-6, 2004
““Event” ProcessingEvent” Processing
Front-end
Ethernet Network
Processors
1 2 2000
1 2 200
Data transfer time = ~40 MBytes/sec * time_slice [4 sec]
Processing time = Nprocessors * time_slice [20 sec]
13FLARE Workshop November 4-6, 2004
Data StorageData Storage
Tape$300K/PByte
Hard Disk$800K/PByte
Blu Ray$900K/PByte
<$100K/PByte
DVD-R$100K/PByte
DVD-RW$300K/PByte
14FLARE Workshop November 4-6, 2004
Data StorageData Storage
• continuous readout, without online tracking
- 2 GBytes/sec X 107 sec = 20 PBytes/yr
- $2-6M/yr media cost
• continuous readout, with online tracking
- 2 MBytes/sec X 107 sec = 20 TBytes/yr
- $10K/yr media cost
Note: BTeV includes 2 PBytes of hard disk (@ $600K/PByte) in baseDAQ construction cost. Offline storage is part of operating cost.
15FLARE Workshop November 4-6, 2004
Ethernet Network(~$50K)
Processing Farm(~$500K)
Data Storage(~$50K @ 2MB/sec,~$1M @ 2GB/sec+ media cost)
DAQ CostsDAQ Costs
Note: DAQ based on all commercial hardware
16FLARE Workshop November 4-6, 2004
Additional hardware costs
Slow Controls ~$200KTiming System ~$50KDevelopment System ~$100KInfrastructure ~$50K
DAQ CostsDAQ Costs
17FLARE Workshop November 4-6, 2004
• with online tracking
DAQ hardware ~$1.0MDAQ software ~$2.0MDAQ management ~$0.5M
DAQ operating cost ~$100K/yr
• without online tracking
(fewer online processors, but more data recorders)
DAQ hardware ~$1.7MDAQ software ~$1.5MDAQ management ~$0.5M
DAQ operating cost ~$4M/yr
DAQ Cost SummaryDAQ Cost Summary
18FLARE Workshop November 4-6, 2004
• data compression at front-end is preferred(“necessary” for continuous readout)
• online tracking will significantly reduce operating cost by reducing storage media expense
• DAQ base cost estimate is ~$3.5M
• DAQ contingency and overhead estimate is ~$1M
ConclusionsConclusions