high throughput...

High throughput DAQ systems Niko Neufeld, CERN/PH

Outline

•  History & tradiBonal DAQ –  Buses –  LEP / Tevatron

•  LHC DAQ –  IntroducBon –  Network based DAQ –  Ethernet –  Scaling Challenges

•  Switches •  Nodes •  Challenges (buffer occupancy) Packet-‐sizes

•  Push & Pull

–  Real LHC DAQ architectures (bandwidth vs complexity)

–  EvenSilterfarms •  “UlBmate” DAQ

–  Trigger-‐free / Sampling, ILC, Clic

–  Almost there: CMB –  Longterm: LHC upgrade –  Ethernet (again)? / InfiniBand

High throughput DAQ, Niko Neufeld, CERN 2

Disclaimer

•  I have been working in this field since 11 years and admit readily to a biased view

•  I have selected DAQ systems mostly to illustrate throughput / performance à not menBoned does not mean that it is not an interesBng system

•  “High throughput” brings a focus on large experiments: the greatest heroism in DAQ is found in small experiments, where the DAQ is done by one or two people part-‐Bme and in test-‐beams. I pay my respects to them!


4

Tycho Brahe and the Orbit of Mars

•  First measurement campaign •  SystemaBc data acquisiBon

–  Controlled condiBons (same locaBon, same day and month)

–  Careful observaBon of boundary condiBons (weather, light condiBons etc…) -‐ important for data quality / systemaBc uncertainBes

I've studied all available charts of the planets and stars and none of them match the others. There are just as many measurements and methods as there are astronomers and all of them disagree. What's needed is a long term project with the aim of mapping the heavens conducted from a single location over a period of several years.��Tycho Brahe, 1563 (age 17).

High throughput DAQ, Niko Neufeld, CERN

5

The First SystemaBc Data AcquisiBon

•  Data acquired over 18 years, normally e every month •  Each measurement lasted at least 1 hr with the naked eye •  Red line (only in the animated version) shows comparison with modern theory


6

Tycho’s DAQ in today’s Terminology •  Trigger = in general something which tells you when is the

“right” moment to take your data –  In Tycho’s case the posiBon of the sun, respecBvely the moon was

the trigger –  the trigger rate ~ 3.85 x 10-‐6 Hz (one measurement / month)

compare with LHCb 1.0 x 106 Hz •  Event-‐data (“event”) = the summary of all sensor data, which are

recorded from an individual physical event –  In Tycho’s case the entry in his logbook (about 100 characters /

entry) –  In a modern detector the Bme a parBcle passed through a specific

piece of the detector and the signal (charge, light) it leh there •  Band-‐width (bw) (“throughput”) = Amount of data transferred /

per unit of Bme –  “Transferred” = wriien to his logbook –  “unit of Bme” = duraBon of measurement –  bwTycho = ~ 100 Bytes / h = 0.0003 kB/s bwLHCb = 55.000 Bytes / us = 55000000 kB/s


7

Lessons from Tycho

•  Tycho did not do the correct analysis (he believed that the earth was at the center of the solar system) of the Mars data. This was done by Johannes Kepler (1571-‐1630), eventually paving the way for Newton’s laws à good data will always be useful, even if you yourself don’t understand them!

•  The size & speed of a DAQ system are not correlated with the importance of the discovery!


Physics, Detectors, Trigger & DAQ


rare, need many collisions

High rate collider

Fast electronics

Data acquisiBon Trigger

Event Filter

Mass Storage

data

decisions

signals

9

Before the DAQ -‐ a detector channel

Amplifier Filter

Shaper

Range compression clock (TTC) Sampling

Digital filter

Zero suppression

Buffer

Format & Readout Buffer

Feature extracBon

Detector / Sensor

to Data AcquisiBon System High throughput DAQ, Niko Neufeld, CERN

10

Crate-‐based DAQ •  Many detector channels are read-‐

out on a dedicated PCB (“board”) •  Many of these boards are put in a

common chassis or crate •  These boards need

–  Mechanical support –  Power –  A standardized way to access their

data (our measurement values) •  All this (and more J) is provided

by standards for (readout) electronics such as VME (IEEE 1014), Fastbus, Camac , ATCA, uTCA

Backplane Connectors (for power and data)

VME Board Plugs into Backplane

6/9U VME Crate (a.k.a. “Subrack”)

19”

9U


11

Example: VME

•  Readout boards in a VME-‐crate –  mechanical standard

for –  electrical standard for

power on the backplane

–  signal and protocol standard for communicaBon on a bus


12

CommunicaBon in a Crate: Buses •  A bus connects two or more devices and allows the to communicate •  The bus is shared between all devices on the bus à arbitraBon is required •  Devices can be masters or slaves and can be uniquely idenBfied

("addressed") on the bus •  Number of devices and physical bus-‐length is limited (scalability!)

–  For synchronous high-‐speed buses, physical length is correlated with the number of devices (e.g. PCI)

–  Typical buses have a lot of control, data and address lines •  Buses are typically useful for systems << 1 GB/s

Device 1

Master

Data Lines

Slave

Select Line

Device 2 Device 4 Device 3 Device 2 Device 4

Master

Data Lines

Slave

Select Line

Device 1 Device 3


13

Crate-‐based DAQ at LEP (DELPHI)

•  FE data

•  Full Event

•  200 Fastbus crates, 75 processors

•  total event-‐size ~ 100 kB •  rate of accepted events O(10

Hz)

adapted from C. Gaspar (CERN) High throughput DAQ, Niko Neufeld, CERN

Combining crates and LANs: the D0 DAQ (L3)


ROC

ROC

ROC

ROC

ROC

Farm Node

Farm Node

Farm Node

Read Out Crates are VME crates that receive data from the detector. Event-‐size 300 kB, at ~ 1 kHz

"  Most data is digiBzed on the detector and sent to the Movable CounBng House "  Detector specific cards in the ROC

"  DAQ HW reads out the cards and makes the data format uniform

"  Event is built in the Farm Node "  There is no dedicated event builder

"  Level 3 Trigger Decision is rendered in the node in sohware

Farm Nodes are located about 20 m away (electrically isolated)

Between the two is a very large CISCO switch… (2002)

The DO DAQ in 2002 -‐ 2011


"  ROC’s contain a Single Board Computer to control the readout. "  VMIC 7750’s, PIII, 933 MHz "  128 MB RAM "  VME via a PCI Universe II chip "  Dual 100 Mb ethernet "  4 have been upgraded to Gb ethernet due to increased data size

"  Farm Nodes: 288 total, 2 and 4 cores per pizza box "  AMD and Xeon’s of differing classes and speeds "  Single 100 Mb Ethernet "  Less than last CHEP!

"  CISCO 6590 switch "  16 Gb/s backplane "  9 module slots, all full "  8 port GB "  112 MB shared output buffer per 48 ports

LHC DAQ

DAQ for mulB-‐Gigabyte/s experiments

17

Moving on to Bigger Things…

The CMS Detector High throughput DAQ, Niko Neufeld, CERN

18

Moving on to Bigger Things…

?

•  15 million detector channels •  @ 40 MHz •  = ~15 * 1,000,000 * 40 * 1,000,000 bytes

•  = ~ 600 TB/sec


19

Know Your Enemy: pp Collisions at 14 TeV at 1034 cm-‐2s-‐1

•  σ(pp) = 70 mb -‐-‐> >7 x 108 /s (!)

•  In ATLAS and CMS* 20 min bias events will overlap

•  H→ZZ Z →µµ H→ 4 muons: the cleanest (“golden”) signature

Reconstructed tracks with pt > 25 GeV

And this (not the H though…)

repeats every 25 ns…

*)LHCb @2x1033 cm-2-1 isn’t much nicer and in Alice (PbPb) it will be even worse High throughput DAQ, Niko Neufeld, CERN

LHC Trigger/DAQ parameters # Level-‐0,1,2 Event Network Storage Trigger Rate (Hz) Size (Byte) Bandw.(GB/s) MB/s (Event/s) 4 Pb-‐Pb 500 5x107 25 1250 (102) p-‐p 103 2x106 200 (102) 3 LV-‐1 105 1.5x106 4.5 300 (2x102)

LV-‐2 3x103 2 LV-‐1 105 106 100 ~1000 (102) 2 LV-‐0 106 5.5x104 55 150 (2x103)

ALICE

ATLAS

CMS

LHCb

20 High throughput DAQ, Niko Neufeld, CERN

DAQ implemented on a LAN


Powerful Core routers

“Readout Units” for protocol adaptaBon

Custom links from the detector

Edge switches

Servers for event filtering

Typical number of pieces Detector 1

1000

100 to 1000

2 to 8

50 to 100

> 1000

22

Event Building over a LAN

Data Acquisition Switch

To Trigger Algorithms

1 Event fragments are received from detector front-end 2 Event fragments are

read out over a network to an event builder 3 Event builder

assembles fragments into a complete event 4 Complete events are

processed by trigger algorithms

Event Builder 3

Event Builder 2

Event Builder 1


One network to rule the all •  Ethernet, IEEE 802.3xx, has almost

become synonymous with Local Area Networking

•  Ethernet has many nice features: cheap, simple, cheap, etc…

•  Ethernet does not: –  guarantee delivery of messages –  allow mulBple network paths –  provide quality of service or bandwidth

assignment (albeit to a varying degree this is provided by many switches)

•  Because of this raw Ethernet is rarely used, usually it serves as a transport medium for IP, UDP, TCP etc…


•  Flow-‐control in standard Ethernet is only defined between immediate neighbors

•  Sending staBon is free to throw away x-‐offed frames (and ohen does L)

Xoff data

CMS Data AcquisiBon


2-‐Stage Event Builder

25

12.5 kHz 12.5 kHz

100 kHz

12.5 kHz …

1st stage “FED-builder” Assemble data from 8 front-ends into one super-fragment at 100 kHz

8 independent “DAQ slices” Assemble super-fragments into full events


Super-‐Fragment Builder (1st stage)

•  based on Myrinet technology (NICs and wormhole routed crossbar switches)

•  NIC hosted by FRL module at sources •  NIC hosted by PC (“RU”) at destination •  ~1500 Myrinet fibre pairs (2.5 Gbps signal rate) from

underground to surface •  Typically 64 times 8x8 EVB configuration •  Packets routed through independent switches

(conceptually 64) •  Working point ~2 kB fragments at 100 kHz •  Destination assignment: round-robin •  loss-less and backpressure when congested


Scaling in LAN based DAQ

28

CongesBon

2 •  "Bang" translates into

random, uncontrolled packet-‐loss

•  In Ethernet this is perfectly valid behavior and implemented by many low-‐latency devices

•  Higher Level protocols are supposed to handle the packet loss due to lack of buffering

•  This problem comes from synchronized sources sending to the same desBnaBon at the same 1me


2

2

Bang

29

Switch Buffer

Push-‐Based Event Building


1 Event Manager tells readout boards where events must be sent (round-robin) 2 Readout boards do

not buffer, so switch must 3 No feedback from

Event Builders to Readout system

“Send next event

to EB1” “Send

next event to EB2”

Event Builder 2

Event Builder 1

Event Manager


30

Cut-‐through switching Head of Line Blocking

2 2 4

Packet to node 4 must wait even though port to node 4 is free

•  The reason for this is the First in First Out (FIFO) structure of the input buffer

•  Queuing theory tells us* that for random traffic (and infinitely many switch ports) the throughput of the switch will go down to 58.6% à that means on 100 MBit/s network the nodes will "see" effecBvely only ~ 58 MBit/s

*) "Input Versus Output Queueing on a Space-Division Packet Switch"; Karol, M. et al. ; IEEE Trans. Comm., 35/12


4 2

1 3

Using more of that bandwidth

•  Cut-‐through switching is excellent for low-‐latency (no buffering) and reduces cost (no buffer memories), but “wastes” bandwidth –  It’s like building more roads than required just so that everybody can go whenever they want immediately

•  For opBmal usage of installed bandwidth there are in general two strategies: –  Use store-‐and-‐forward switching (next slide) –  Use traffic-‐shaping / traffic-‐control

•  Different protocols (“pull-‐based event-‐building”), mulB-‐level readout •  end-‐to-‐end flow control •  virtual circuits (with credit-‐scheme) (InfiniBand) •  Barrel-‐shiher


32

Output Queuing •  In pracBce virtual output

queueing is used: at each input there is a queue à for n ports O(n2) queues must be managed

•  Assuming the buffers are large enough(!) such a switch will sustain random traffic at 100% nominal link load

2 2

Packet to node 2 waits at output to port 2. Way to node 4 is free


2

1 3

4

4

Store-‐and-‐Forward in the LHC DAQs

•  256 MB shared memory / 48 ports

•  Up to 1260 ports (1000 BaseT)

•  Price / port ~ 500 -‐ 1000 USD

•  Used by all LHC experiments

•  6 kW power, 21 U high •  Loads of features (most of them unused J in the experiments)


Buffer Usage in F10 E1200i


0

500

1000

1500

2000

2500

3000

3500

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Buffe

r occup

ancy in kB

Port (connected to edge-‐router)

Buffer usage in core-‐router with a test using 270 sources @ 350 kHz event-‐rate

•  256 MB shared between 48 ports

•  17 ports used as “output”

•  This measurement for small LHCb events (50 kB) at 1/3 of nominal capacity

35

Event Builder 1

Push-‐Based Event Building with store& forward switching and load-‐balancing


1 Event Builders notify Event Manager available capacity 2 Event Manager

ensures that data are sent only to nodes with available capacity 3 Readout system

relies on feedback from Event Builders

“Send next event

to EB1” “Send


EB1: EB2: EB3:

0 0 0

“Send me an event!”


1 1 1



0 1 1

0 0 1 “Send


1 0 1

Event Builder 1

Event Builder 2

Event Builder 3


Event Manager

DAQ networks beyond push and store & forward

• Reducing network load • Pull-‐based event-‐building • Advanced flow-‐control

Progressive refinement: ATLAS

•  3-‐level trigger •  ParBal read-‐out (possible because of the nature of ATLAS physics) at high rate

•  Full read-‐out at relaBvely low rate

• à smaller network


ATLAS DAQ (network view)


Farm interface processors collect full data of accepted events from buffers and distribute them though backend core chassis to third level trigger processor farms over a TCP routed network

OpBonal connecBon to use Level 2 farms for Level 3 use

2 * 1G trunks Collect accepted events and transfer them over 10G fiber to storage and Grid

39

Event Builder 1

Pull-‐Based Event Building


1 Event Builders notify Event Manager of available capacity 2 Event Manager elects

event-builder node 3 Readout traffic is driven by Event Builders

“EB1, get next

event” “EB2, get

next event”

EB1: EB2: EB3:

0 0 0



1 1 1



0 1 1

0 0 1

1 0 1

Event Builder 1

Event Builder 2

Event Builder 3


“Send event to EB 1!” “Send event

to EB 1!” “Send event to EB 1!” “Send event

to EB 1!”

Advanced Flow-‐control DCB, QCN


Why is DCB interesBng?

•  Data Center Bridging tries to brings together the advantages of Fibrechannel, Ethernet and InfiniBand on a single medium

•  It has been triggered by the storage community (the people selling Fibrechannel over Ethernet and iSCSI)

•  It achieves low latency and reliable transport at constant throughput through flow-‐control and buffering in the sources

•  802.1Qau, 802.1Qbb, 802.3bd could allow us to go away from store&forward in DAQ LANs (à extensive R&D required)


Moving the data in and through a PC

Sending and receiving data

•  MulBple network protocols result in mulBple sohware layers

•  Data moving can be expensive –  Passing data through the

layers someBmes cannot be done without copying

–  Header informaBon needs to be stripped off à splicing, alignment etc…

•  In Ethernet receiving is quite a bit more expensive than sending

•  Holy grail is zero-‐copy event-‐building (difficult with classical Ethernet, easier with InfiniBand)


The cost of event-‐building in the server


0

20

40

60

80

100

120

140

0

10

20

30

40

50

60

0 500 1000 1500 2000 2500 3000 3500

MB (RAM

shared

)

% single 5420 core

Event-‐rate [Hz]

CPU %

MEM

% CPU of one Intel 5420 core (a 4 core processor running at 2.5 GHz MEM is resident memory (i.e. pages locked in RAM) Precision of measurements is about 10% for CPU and 1% for RAM

Frame-‐size / payload & frame-‐rate Or: why don’t they increase the MTU?


0

20

40

60

80

100

120

1

10

100

1000

10000

64 128 512 1500 4096 9000

frame-‐rate [kHz]

max payload [MB/s]

1500 bytes leads to good link-‐usage Network guys are happy But frame-‐rate could sBll be lowered

Overheads calculated only for Ethernet: 38 bytes / frame IP adds another 20 TCP another 20(!)

Gigabit Ethernet = 109 bit/s

Tuning for bursty traffic

•  In general provide for lots of buffers in the kernel, big socket buffers for the applicaBon and tune the IRQ moderaBon

•  Examples here are for Linux, 2.6.18 kernel, Intel 82554 NICs (typical tuning in the LHCb DAQ cluster)


/sbin/ethtool -G eth1 rx 1020 # set number of RX descriptors in NIC to max # the following are set with sysctl -w net.core.netdev_max_backlog = 4000 net.core.rmem_max = 67108864 # the application is tuned with setsockopt()

Interrupt ModeraBon

•  Careful tuning necessary – mulB-‐core machines can take more IRQs (but: spin-‐locking…)

•  Can afford to ignore at 1 Gbit/s but will need to come back to this for 10 Gbit/s and 40 Gbit/s


High Level Trigger Farms


And that, in simple terms, is what�� we do in the High Level Trigger

Event-‐filtering

•  Enormous amount of CPU power needed to filter events •  AlternaBve is not to filter and store everything (ALICE)

•  OperaBng System: Linux SLC5 32-‐bit and 64-‐bits: standard kernels, no (hard) real-‐Bme

•  Hardware: •  PC-‐server (Intel and AMD): rack-‐mount and blades

•  All CPU-‐power local: no grid, no clouds (yet?)


Online Trigger Farms 2011 ALICE ATLAS CMS LHCb

# cores 2700 17000 10000 15500

total available power (kW)

~ 2000(1) ~ 1000 550

currently used power (kW)

~ 250 450(2) ~ 145

total available cooling power

~ 500 ~ 820 800 (currently)

525

total available rack-‐space (Us)

~ 2000 2400 ~ 3600 2200

CPU type(s) AMD Opteron, Intel 54xx, Intel 56xx

Intel 54xx, Intel 56xx




��(1) Available from transformer (2) PSU rating

Faster, Larger – the future A bit of markeBng for upcomig DAQ systems

SuperB DAQ •  CollecBon in

Readout-‐Crates

•  Ethernet read-‐out

•  60 Gbit/s •  à if you

look for a challenge in DAQ, you must join LHCb J -‐ or work on the SLHC


Compressed Baryonic Maier (CBM)


•  Heavy Ion experiment planned at future FAIR facility at GSI (Darmstadt)

•  Timescale: ~2014 Detector Elements • Si for Tracking • RICH and TRDs for Particle

identification • RPCs for ToF measurement • ECal for Electromagnetic

Calorimetry

AverageMultiplicities: 160 p 400 π- 400 π+ 44 K+ 13 K 800 γ 1817 total at 10 MHz

High MulBpliciBes


Quite Messy Events… (cf. Alice) ❏  Hardware triggering

problemaBc ➢ Complex ReconstrucBon ➢ ‘ConBnuous’ beam

❏  Trigger-‐Free Readout ➢ ‘ConBnuous’ beam ➢ Self-‐Triggered channels

with precise Bme-‐stamps ➢ CorrelaBon and

associaBon later in CPU farm

CBM DAQ Architecture


data dispatcher

FEEdeliver time stamped data

CNetcollect data into buffers

Detector Detector Detector

collect

FEEdeliver time stamped data

CNetcollect data into buffers

Detector Detector Detector

collect

~50000 FEE chips

event dispatcher

1000x1000 switchingBNetsort time stamped data

~1000 links a 1 GB/sec

event dispatcher






HNethigh level selection

to high level computingand archiving

~1 GB/sec Output



~1 GB/sec Output



~1 GB/sec Output

processing

PNetprocess eventslevel 1&2 selection

subfarm subfarm subfarm ~100 subfarms

~100 nodes per subfarm

~10 dispatchers/subfarm

processing


subfarm subfarm subfarm ~100 subfarms




subfarmsubfarm subfarmsubfarm subfarmsubfarm ~100 subfarms



~1000 collectors

~1000 active buffers

TNettime distributionTNettime distributionTNettime distribution

CBM CharacterisBcs/Challenges

Very much network based à 5 different networks –  Very low-‐jiaer (10 ps) Bming distribuBon network –  Data collecBon network to link detector elements with front-‐end electronics (link speed O(GB/s))

–  High-‐performance (~O(TB/s)) event building switching network connecBng O(1000) Data Collectors to O(100) Subfarms

–  Processing network within a subfarm interconnecBng O(100) processing elements for triggering/data compression

–  Output Network for collecBng data ready for archiving aher selecBon.



LHCb DAQ Architecture from 2018

HLT

Readout Supervisor

L0 Hardware

Trigger

Tell1 1 MHZ

1 Gb Ethernet

Readout Supervisor

InteracBon

Trigger

Tell40

40 MHZ 1…40 MHZ

10 Gb Ethernet HLT++

Upgrade

Current

-‐ All data will be readout @ collision rate 40 MHz by all frontend electronics (FEE) à a trigger-‐free read-‐out! -‐ Zero-‐suppression will be done in FEEs to reduce the number of the GigaBit Transceiver (GBT) links

GBT (4.8Gb/s) data rate 3.2 Gb/s

Requirements on LHCb DAQ Network •  Design:

–  100 kB@30 MHz (10 MHz out of 40 MHz are empty) è 24 Tbit/s network required

–  Keep average link-‐load at 80% of wire-‐speed –  ~5000 Servers needed to filter the data (depends

on Moore’s Law) •  We need:

–  (input) Ports for Readout boards (ROB) from the detector: 3500x10 Gb/s

–  (output) Ports for Event Filter Farm (EFF): 5000x10 Gb/s

–  Bandwidth: 34 Tb/s (unidirecBonal, including load-‐factor)

•  Scaling: build several (8) sub-‐networks or slices. Each Readout Board is connect to each slice.


Fat-‐Tree Topology for One Slice •  48-‐port 10 GbE switches •  Mix readout-‐boards (ROB) and filter-‐farm-‐servers in one

switch –  15 x readout-‐boards –  18 x servers –  15 x uplinks

Non-‐block switching use 65% of installed bandwidth (classical DAQ only 50%) •  Each slice accomodates

–  690 x inputs (ROBS) –  828 x outputs servers

RaBo (server/ROB) is adjustable


InfiniBand

•  High bandwidth (32 Gbit/s, 56 Gbit/s …) – always a step ahead of Ethernet J

•  Low price / switch-‐port •  Very low latency


SDR DDR QDR FDR EDR HDR NDR

1X 2 Gbit/s

4 Gbit/s

8 Gbit/s

14 Gbit/s

25 Gbit/s

125 Gbit/s 750 Gbit/s

4X 8 Gbit/s

16 Gbit/s

32 Gbit/s

56 Gbit/s

100 Gbit/s


12X 24 Gbit/s

48 Gbit/s

96 Gbit/s

168 Gbit/s

300 Gbit/s


InfiniBand: the good, the bad and the ugly

•  Cheap, but higher-‐speed grades will require opfcs •  Powerful flow-‐control / traffic management •  Nafve support for RDMA (low CPU overhead) •  Cabling made for clusters (short distances) •  Very small vendor base •  Complex sojware stack •  Some of the advanced possibilifes (RDMA, advanced flow-‐control) become available on “converged” NICs (e.g. Mellanox, Chelsio)


Is the future of DAQ in the OFED?


Future DAQ systems (choices)

•  Certainly LAN based –  InfiniBand deserves a serious evaluaBon for high-‐bandwidth (> 100 GB/s)

–  In Ethernet if DCB works, might be able to build networks from smaller units, otherwise we will stay with large store&forward boxes

•  Trend to “trigger-‐free” à do everything in sohware à bigger DAQ will conBnue –  Physics data-‐handling in commodity CPUs

•  Will there be a place for mulB-‐core / coprocessor cards (Intel MIC / CUDA)? –  IMHO this will depend on if we can establish a development framework which allows for longterm maintenance of the sohware by non-‐”geek” users, much more than on the actual technology


Summary and Future

•  Large modern DAQ systems are based enBrely on Ethernet and big PC-‐server farms

•  Bursty, uni-‐direcBonal traffic is a challenge in the network and the receivers, and requires substanBal buffering in the switches

•  The future: –  It seems that buffering in switches is being reduced (latency vs. buffering)

–  Advanced flow-‐control is coming, but it will need to be tested if it is sufficient for DAQ

–  Ethernet is sBll strongest, but InfiniBand looks like a very interesBng alternaBve

–  Integrated protocols (RDMA) can offload servers, but will be more complex


Publicita / Publicité / Commercial

Thanks

•  I acknowledge gratefully the help of many colleagues who provided both material and suggesBons: Guoming Liu, Beat Jost, David Francis, Frans Meijers, Gordon Wais, Clara Gaspar


high throughput...

Documents