nvme over fabrics- · 2018. 5. 29. · 15 storage platform latency data path ddr4 ddr4 pcie switch...
TRANSCRIPT
![Page 1: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/1.jpg)
![Page 2: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/2.jpg)
NVMe ove r Fab r i cs -H i g h p e r f o r m a n c e S S D s n e t w o r k e d f o r c o m p o s a b l e i n f r a s t r u c t u r e
Rob Davis, VP Storage Technology, Mellanox
![Page 3: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/3.jpg)
3
OCP Evolution…
Physical
Rack
Server Density Compute Storage
DisaggregationLately called “Composable
Infrastructure”
![Page 4: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/4.jpg)
4
Why NVMe over Fabrics?
![Page 5: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/5.jpg)
5
NVMe Technology Background
Optimized for flash Traditional SCSI designed for diskNVMe bypasses unneeded layersDramatically reducing latency and
increasing bandwidth
![Page 6: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/6.jpg)
6
NVMe over Fabrics Enables Storage Networking of NVMe SSDsSharing NVMe-based storage
with multiple serversBetter utilization: capacity, rack
space, and powerBetter scalabilityManagement Fault isolation
While maintaining NVMe Performance
Gb/s
![Page 7: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/7.jpg)
7
NVMe over Fabrics Technology
Extends NVMe efficiency over a fabricNVMe commands and data structures are
transferred end to endRelies on RDMA for performanceBypassing TCP/IP
![Page 8: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/8.jpg)
8
Faster Storage Needs a Faster Network
10GbE
![Page 9: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/9.jpg)
9
Faster Network Wires Solves Some of the Problem…
Ethernet & InfiniBand
End-to-End 25, 40, 50, 56, 100Gb
Going to 200 and 400Gb
![Page 10: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/10.jpg)
10
Faster Protocols Solves More…
![Page 11: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/11.jpg)
11
Faster Protocols Solves More…
![Page 12: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/12.jpg)
12
NVMe, NVMe-oF, and RoCE Protocols
![Page 13: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/13.jpg)
13
Network Based Offload Engines Complete the PictureProtocol offloads NVMe over Fabrics RoCE
Security offloads Encryption Platform isolation
Storage offloads Data Integrity Compression
![Page 14: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/14.jpg)
14
Importance of Latency
Common Switch & NIC
Lo
gari
thm
ic s
cale
600ns Network hops multiply latency
Ethernet Storage Fabric - ESF
![Page 15: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/15.jpg)
15
Storage Platform Latency
Data Path
DD
R4
DD
R4
PCIe Switch
NVME
SSD
NVME
SSD
Initiator x86
ConnectX-5
Initiator x86
ConnectX-5
Control Path
6M IOPs, 512B block size 2M IOPs, 4K block side 50% CPU utilization ~15usec latency (not including SSD)
ConnectX no Offload
Target2 100Gb Initiators
DD
R4
DD
R4
16 ARM cores
PCIe Switch
NVME
SSD
NVME
SSD
Initiator x86
ConnectX-5
Initiator x86
ConnectX-5
SOC
8M IOPs, 512B block size 5M IOPs, 4K block side 0.01% CPU utilization ~5usec latency (not including SSD)
BlueField Offload
Target2 100Gb Initiators
![Page 16: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/16.jpg)
16
Composable Infrastructure
Switch
Compute
Compute
Compute
Compute
Compute
Compute
Compute
Compute
Compute
Compute
Compute
Switch
Compute
Compute
Compute
Compute
Compute
Compute
Compute
Compute
Compute
NVMe over Fabrics enables Composable
Infrastructure
• Low latency
• High bandwidth
• Nearly local disk performance
High performance network components
are required
• ESF
• 1 usec or less latency
• Protocol offloads
Further offloads needed for storage
features
• Security
• Compression
• Data integrity
![Page 17: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/17.jpg)
17
Offloads in the Compute Node Adapter
Right place for some applications Data in flight encryption Applications vs. storage node decision
In-line processing is efficient Data must flow through network adapter regardless Minimize need for special software flow Lower latency data path
Natural place for security boundary Isolated execution – separate domain from host
![Page 18: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/18.jpg)
18
Advanced NICs Accelerate Storage Apps
Video Big Data SecurityEfficient Data TransportStorage
X86 SW Processing
NetworkQSFP/
SFP28
QSFP/
SFP28
PCIe Gen3/4
Application Acceleration Delivers Infrastructure Efficiency
X86 Security Challenges & Vulnerabilities No isolation = Vulnerable Infrastructure
Application & security domain are identical
Poor performance Poor scalability Vulnerable to DDOS attacks
![Page 19: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/19.jpg)
19
SmartNICs Accelerate Storage Apps & Security & …
SOC Security Benefits Security & workload isolation & offload Logically separated security domains Secure boot & firmware update Workload control & visibility Fully programmable
QSFP/
SFP28
QSFP/
SFP28
BlueField SOC
Accelerators
64-bit ARM Cores
DR
AM
Video Big Data SecurityStorage
Application Acceleration Delivers Infrastructure Efficiency X86 Processing
PCIe Gen3/4
PCIe Switch
Efficient Data Transport
Network
![Page 20: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/20.jpg)
20
Network Accelerates Workloads
ConnectX-5 100G
ConnectX-4 Lx 25G
ConnectX-4 50GSpectrum 25, 50, 100G Switches
Smart NIC
BlueField SoC
Breakout Cables
Active Optical Cables
Optical Transceivers
Multimode & Single Mode
Copper Cables
Security Big Data OffloadVideo Acceleration Storage
![Page 21: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/21.jpg)
21
NVMe over Fabrics Enables Composable Infrastructure and Much More…
NVMe over Fabrics Local storage performance across a network
High performance network components - ESF RDMA Low latencyHigh Bandwidth
Hardware offload accelerators Protocols Security Storage features
Composable Infrastructure, Security, Video, Big Data…
![Page 23: NVMe over Fabrics- · 2018. 5. 29. · 15 Storage Platform Latency Data Path DDR4 DDR4 PCIe Switch NVME SSD NVME SSD Initiator x86 ConnectX-5 Initiator x86 ConnectX-5 Control Path](https://reader034.vdocument.in/reader034/viewer/2022051915/600644e6d213ec73d24bdba4/html5/thumbnails/23.jpg)