Real-Time Ethernet and PCI ExpressDo they fit together?
Michal Sojka, Rostislav Lisovy, Zdenek Hanzalek
Czech Technical University in Prague,Faculty of Electrical Engineering
Technicka 2, 121 35 Prague 6, Czech RepublicEmail: {sojkam1,lisovros,hanzalek}@fel.cvut.cz
RATE 2013 WorkshopDecember 3, Vancouver, Canada
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 1 / 14
Introduction
Real-Time Ethernet
I Real-Time = Deterministic
I TDMA – Deterministic medium access algorithm
I Also called “Time-triggered communication”
Slot 0 Slot 1 Slot 2
TDMA cycle
Time
Slot 0
Node A Node B
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 2 / 14
Introduction
Real-Time Ethernet
I Real-Time = Deterministic
I TDMA – Deterministic medium access algorithm
I Also called “Time-triggered communication”
Slot 0 Slot 1 Slot 2
TDMA cycle
Time
Slot 0
Node A Node B
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 2 / 14
Introduction
Time Synchronization
I TDMA requires synchronized time between nodes
Slot 0 Slot 2
Slot 0 Slot 2
Slot 1
Time
Node A
Node B
I Synchronization accuracy influences protocol efficiency
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 3 / 14
Introduction
Time Synchronization
I TDMA requires synchronized time between nodes
Slot 0 Slot 2
Slot 0 Slot 2
Slot 1
Time
Node A
Node B
I Synchronization accuracy influences protocol efficiency
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 3 / 14
Introduction
Precision Time Protocol (PTP)
I Standardized as IEEE 1588
I Can take advantage of hardware timestamping of transmittedor received frames
I Hardware timestamping is available on many modern NICs1
I Synchronization accuracy: < 1µs
I Sounds good!
1Network Interface CardM. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 4 / 14
Introduction
PC Hardware
I Cheap, powerful, extendable ⇒ attractive
I Popular in academic papers about real-timeEthernet
I IEEE 1588 NIC + Real-Time OS = Perfectsolution for networked real-time systems
Is that true?
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 5 / 14
Introduction
PC Hardware
I Cheap, powerful, extendable ⇒ attractive
I Popular in academic papers about real-timeEthernet
I IEEE 1588 NIC + Real-Time OS = Perfectsolution for networked real-time systems
Is that true?
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 5 / 14
Modern PC
Modern PC Architecture = PCI Express Architecture
Core 0
Root Complex RAM DDR3
PCIe GraphicsAdapter Slot
(GFX)
PCIex16
CPU
DMI / PCIePCH (Chipset)
SATA
USB
PCIe
LPC
Core 1 Core 2 Core 3
Serial / Parallelport; Keyboard,Mouse PS/2
HPET
PCIeSwitch
PCIe to PCIBridge
PCIe
Slo
tsty
pic
ally
x4
(IO
)
PCIeSwitch
PCISlots
NIC
?
PCI Express (PCIe)
I Communication betweenCPU, memory andperipherals
I Packet-based serialnetwork with switches
I Shared resource
I Differences between PCIeslots (bandwidth, latency,jitter)
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 6 / 14
Modern PC
Modern PC Architecture = PCI Express Architecture
Core 0
Root Complex RAM DDR3
PCIe GraphicsAdapter Slot
(GFX)
PCIex16
CPU
DMI / PCIePCH (Chipset)
SATA
USB
PCIe
LPC
Core 1 Core 2 Core 3
Serial / Parallelport; Keyboard,Mouse PS/2
HPET
PCIeSwitch
PCIe to PCIBridge
PCIe
Slo
tsty
pic
ally
x4
(IO
)
PCIeSwitch
PCISlots
NIC
?
PCI Express (PCIe)
I Communication betweenCPU, memory andperipherals
I Packet-based serialnetwork with switches
I Shared resource
I Differences between PCIeslots (bandwidth, latency,jitter)
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 6 / 14
PCI Express Latencies
Experiments with PCI Express Latencies
I Intel 82576 NIC + Ivy Bridge CPU + bare-metal microhypervisorI Unloaded and loaded system
Reading a 32 bit register from the NIC
I Worst-case latency: 3.5µsI Jitter: 3µs
I NIC clock register: 64 bit (⇒ ×2)
Timing comparison
I CPU reads the NIC clock: worst-case latency 7µs, jitter 6µsI NIC clock has accuracy of < 1µs
I Compare: Parallel PCI, 10 years ago ⇒ average latency and jitterbelow 1µs.
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 7 / 14
PCI Express Latencies
Experiments with PCI Express Latencies
I Intel 82576 NIC + Ivy Bridge CPU + bare-metal microhypervisorI Unloaded and loaded system
Reading a 32 bit register from the NIC
I Worst-case latency: 3.5µsI Jitter: 3µs
I NIC clock register: 64 bit (⇒ ×2)
Timing comparison
I CPU reads the NIC clock: worst-case latency 7µs, jitter 6µsI NIC clock has accuracy of < 1µs
I Compare: Parallel PCI, 10 years ago ⇒ average latency and jitterbelow 1µs.
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 7 / 14
PCI Express Latencies
Experiments with PCI Express Latencies
I Intel 82576 NIC + Ivy Bridge CPU + bare-metal microhypervisorI Unloaded and loaded system
Reading a 32 bit register from the NIC
I Worst-case latency: 3.5µsI Jitter: 3µs
I NIC clock register: 64 bit (⇒ ×2)
Timing comparison
I CPU reads the NIC clock: worst-case latency 7µs, jitter 6µsI NIC clock has accuracy of < 1µs
I Compare: Parallel PCI, 10 years ago ⇒ average latency and jitterbelow 1µs.
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 7 / 14
PCI Express Latencies
Sources of PCI Express Latencies
PCIe topology – Slots connected directly to the CPU have lowerlatency than slots provided by the “chipset”
Active State Power Management (ASPM) – Technology thatsaves power by switching off idle PCIe links
Unrelated traffic – Load initiated by other CPUs or PCIe devices inthe system
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 8 / 14
PCI Express Latencies
Sending an Ethernet Frame
Core 0
Root Complex RAM DDR3
PCIe GraphicsAdapter Slot
(GFX)
PCIex16
CPU
DMI / PCIePCH (Chipset)
SATA
USB
PCIe
LPC
Core 1 Core 2 Core 3
Serial / Parallelport; Keyboard,Mouse PS/2
HPET
PCIeSwitch
PCIe to PCIBridge
PCIe
Slo
tsty
pic
ally
x4
(IO
)
PCIeSwitch
PCISlots
NIC
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 9 / 14
PCI Express Latencies
Sending an Ethernet Frame
Core 0
Root Complex RAM DDR3
PCIe GraphicsAdapter Slot
(GFX)
PCIex16
CPU
DMI / PCIePCH (Chipset)
SATA
USB
PCIe
LPC
Core 1 Core 2 Core 3
Serial / Parallelport; Keyboard,Mouse PS/2
HPET
PCIeSwitch
PCIe to PCIBridge
PCIe
Slo
tsty
pic
ally
x4
(IO
)
PCIeSwitch
PCISlots
NIC
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 9 / 14
PCI Express Latencies
Sending an Ethernet Frame
Core 0
Root Complex RAM DDR3
PCIe GraphicsAdapter Slot
(GFX)
PCIex16
CPU
DMI / PCIePCH (Chipset)
SATA
USB
PCIe
LPC
Core 1 Core 2 Core 3
Serial / Parallelport; Keyboard,Mouse PS/2
HPET
PCIeSwitch
PCIe to PCIBridge
PCIe
Slo
tsty
pic
ally
x4
(IO
)
PCIeSwitch
PCISlots
NIC
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 9 / 14
PCI Express Latencies
Sending an Ethernet Frame
Core 0
Root Complex RAM DDR3
PCIe GraphicsAdapter Slot
(GFX)
PCIex16
CPU
DMI / PCIePCH (Chipset)
SATA
USB
PCIe
LPC
Core 1 Core 2 Core 3
Serial / Parallelport; Keyboard,Mouse PS/2
HPET
PCIeSwitch
PCIe to PCIBridge
PCIe
Slo
tsty
pic
ally
x4
(IO
)
PCIeSwitch
PCISlots
NIC
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 9 / 14
PCI Express Latencies
Sending an Ethernet Frame
Core 0
Root Complex RAM DDR3
PCIe GraphicsAdapter Slot
(GFX)
PCIex16
CPU
DMI / PCIePCH (Chipset)
SATA
USB
PCIe
LPC
Core 1 Core 2 Core 3
Serial / Parallelport; Keyboard,Mouse PS/2
HPET
PCIeSwitch
PCIe to PCIBridge
PCIe
Slo
tsty
pic
ally
x4
(IO
)
PCIeSwitch
PCISlots
NIC
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 9 / 14
PCI Express Latencies
Sending an Ethernet Frame
Core 0
Root Complex RAM DDR3
PCIe GraphicsAdapter Slot
(GFX)
PCIex16
CPU
DMI / PCIePCH (Chipset)
SATA
USB
PCIe
LPC
Core 1 Core 2 Core 3
Serial / Parallelport; Keyboard,Mouse PS/2
HPET
PCIeSwitch
PCIe to PCIBridge
PCIe
Slo
tsty
pic
ally
x4
(IO
)
PCIeSwitch
PCISlots
NIC
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 9 / 14
PCI Express Latencies
Frame Transmission Latency
I 166 bytes long frame: worst-case latency 20µs, jitter 10µs
I 1200 bytes long frame: worst-case latency 22µs, jitter 10µs
I ≈ 3 PCIe transactions per frame
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 10 / 14
PCI Express Latencies
Time Comparison
PCIe-caused TX jitter
Frame TX time @ 100Mbps
Frame TX time @ 1Gbps
0 5 10 15 20 25
166 bytes
1200 bytes
Time [µs]
Packet size
Worst-case TX latency GFX slotIO slot
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 11 / 14
Conclusions
Consequences
1. Software implementation of time-triggered protocols cannotachieve network utilization above about 30% (@ 100 Mbps).
2. Precise measurement applications running on a CPU cannotuse the clock precision offered by PTP. Even unloaded systemexhibits jitter of about 5µs in certain setups.
3. Some applications might benefit from having synchronizedexecution of CPU tasks with network time. This is possiblebut one needs to account for high PCIe jitters.
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 12 / 14
Conclusions
Consequences
1. Software implementation of time-triggered protocols cannotachieve network utilization above about 30% (@ 100 Mbps).
2. Precise measurement applications running on a CPU cannotuse the clock precision offered by PTP. Even unloaded systemexhibits jitter of about 5µs in certain setups.
3. Some applications might benefit from having synchronizedexecution of CPU tasks with network time. This is possiblebut one needs to account for high PCIe jitters.
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 12 / 14
Conclusions
Consequences
1. Software implementation of time-triggered protocols cannotachieve network utilization above about 30% (@ 100 Mbps).
2. Precise measurement applications running on a CPU cannotuse the clock precision offered by PTP. Even unloaded systemexhibits jitter of about 5µs in certain setups.
3. Some applications might benefit from having synchronizedexecution of CPU tasks with network time. This is possiblebut one needs to account for high PCIe jitters.
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 12 / 14
Conclusions
What Can Be Done About It?
I Benchmark PCIe latencies and jitter of different slotsI We work on a Linux tool that can do that without special
experimental setup
I PCIe specification: Virtual Channels
I Different QoS parameters for each channelI Isochronous traffic – bandwidth reservationI Not available in common PC hardware (?)
I PTP over PCI Express?
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 13 / 14
Conclusions
Thank you!
R. Lisovy, M. Sojka, and Z. Hanzalek, “PCI Express as a killer ofsoftware-based real-time Ethernet,” in Real-Time Networks, The
12th International Workshop on, July 2013.
M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 14 / 14