ofed storage protocols...reliable datagram service rds iscsi rdma protocol (initiator) iser scsi...
Post on 22-Aug-2020
21 Views
Preview:
TRANSCRIPT
www.openfabrics.org
OFED Storage Protocols
R. PearsonSystem Fabric Works, Inc.
2www.openfabrics.org
Agenda
Why OFED StorageIntroduction to OFED Storage ProtocolsOFED Storage Protocol Update
3www.openfabrics.org
Why OFED Storage
4www.openfabrics.org
Goals of I/O Consolidation
Cluster of ServersMultiple Fabrics
High CapEx and High TCOSingle InfiniBand Fabric
Low CapEx and Optimal TCO
“One Wire”
5www.openfabrics.org
Unified InfiniBand Fabric
PerformanceUltimate scalabilityCost of ownershipDedicated I/O channels enable consolidation
6www.openfabrics.org
Industry Support for Native InfiniBand Storage
7www.openfabrics.org
Storage Interconnects
Type Max length Peak data rate Max devices
PATA (IDE) 46cm 133MB/s 2
SCSI 12m 320MB/s 16
iSCSI (Ethernet) ? ? ?
IB Cu (DDR) 15m/5m 1gbps/2gbps 16M(48K?)
IB optical (mm/WAN) ~300m/? 1GB/s 16M
SATA/eSATA 1m/2m 300MB/s 1
SAS 6m 300MB/s 16K
FC 50km 400MB/s 16M
8www.openfabrics.org
Introduction to OFED Storage Protocols
9www.openfabrics.org
Linux Storage Stack (das/san)
ApplicationUser
KernelVFS Layer
File Systems
Block Drivers
SCSIMid Layer
HBA Drivers
HBA
Storage
(ext2, ext3, reiser, xfs, …)
(sd, md, lvm, ..)
(scsi, fc, sata, ide, …)
(read, write, open, close, …)
10www.openfabrics.org
Linux Storage Stack (nas)
ApplicationUser
KernelVFS Layer
NFSD(cache)
RPC
TCP/IP
NIC Driver
NIC
Filer
(read, write, open, close, …)
11www.openfabrics.org
Linux Storage Stack (iSCSI)
ApplicationUser
KernelVFS Layer
File Systems
Block Drivers
SCSIMid Layer
iSCSI
NIC
Storage
TCP/IP
NIC Driver
12www.openfabrics.org
OpenFabrics Software Stack
RDMA NICR-NIC
Host Channel Adapter
HCA
User Direct Access Programming Lib
UDAPL
Reliable Datagram Service
RDS
iSCSI RDMA Protocol (Initiator)
iSER
SCSI RDMA Protocol (Initiator)
SRP
Sockets Direct Protocol
SDP
IP over InfiniBandIPoIB
Performance Manager Agent
PMA
Subnet Manager Agent
SMA
Management Datagram
MAD
Subnet Administrator
SA
Common
InfiniBand
iWARP
Key
InfiniBand HCAInfiniBand HCA iWARP RiWARP R--NICNIC
HardwareSpecific Driver
Hardware SpecificDriver
ConnectionManagerMAD
InfiniBand Verbs / API
SA Client
ConnectionManager
Connection ManagerAbstraction (CMA)
User Level Verbs / API
SDPIPoIB SRP iSER RDS
UDAPL
SDP Library
User Level MAD API
Open SM
DiagTools
Hardware
Provider
Mid-Layer
Upper Layer Protocol
User APIs
Kernel SpaceUser Space
NFS-RDMARPC
ClusterFile Sys
Application Level
SMA
R-NIC Driver API
ClusteredDB Access
(Oracle10g RAC)
SocketsBasedAccess
(IBM DB2)
VariousMPIs
Access toFile
Systems
BlockStorageAccess
IP BasedApp
Access
Apps & Access
Methodsfor usingOF Stack
13www.openfabrics.org
Linux Storage Stack (das/san)
ApplicationUser
KernelVFS Layer
File Systems
Block Drivers
SCSIMid Layer
SRP/iSER
OFED
Storage
(ext2, ext3, reiser, xfs, …)
(sd, md, lvm, ..)
(read, write, open, close, …)
HCA
14www.openfabrics.org
Linux Storage Stack (nas)
ApplicationUser
KernelVFS Layer
NFSD(cache)
RPC
TCP/IP
IPoIB
OFED
Filer
(read, write, open, close, …)
HCA
15www.openfabrics.org
Linux Storage Stack (iSCSI)
ApplicationUser
KernelVFS Layer
File Systems
Block Drivers
SCSIMid Layer
iSCSI
NIC
StorageTCP/IP
IPoIB
OFED
16www.openfabrics.org
Common I/O Scheme
Initiator Fabric Target
LogicalView
PhysicalView
RDMA read/write
System Memory CPU CPUcommand
response
Buffers
Storage hardware
17www.openfabrics.org
Common I/O Scheme
Initiator Fabric Target
LogicalView
PhysicalView
RDMA read/write
System Memory CPU CPUcommand
response
Buffers
Storage hardware
rkey
lkey
MR
MR
18www.openfabrics.org
Common I/O Scheme
Initiator Fabric Target
LogicalView
PhysicalView
RDMA read/write
System Memory CPU CPUcommand
response
Buffers
Storage hardware
rkey
lkey
19www.openfabrics.org
Common I/O Scheme
Initiator Fabric Target
LogicalView
PhysicalView
RDMA read/write
System Memory CPU CPUcommand
response
Buffers
Storage hardware
rkey
lkey
FMR
MR
20www.openfabrics.org
OFED Storage Protocol Update
21www.openfabrics.org
SRP Update
SCSI RDMA Protocol (SCSI over IB)T10.org specification, similar to FCP (SCSI over FC)All data transfers use InifiniBand RDMA
InitiatorLinux proprietary and open source “Gen1”-based in productionOFED 1.0 SRP Initiator in Beta (still in development)Windows Beta release in July
TargetLinux open source “Gen1”-based target on OFAVendors in production or in final qualification
Linux and other RTOSs
22www.openfabrics.org
SRP Next Steps
InitiatorHost failover/multipathingBoot support for Linux and Windows
TargetCall for interest to port target to OFED
Mixed file system / block storage Linux targets require this capability
23www.openfabrics.org
iSER Update
iSCSI Extensions for RDMAIETF IP storage specification, Leverages iSCSI management and discovery protocolsiSER eliminates iSCSI/TCP bottlenecks
RDMA, hardware transportInitiator
Production supported in OFED 1.0Tested against Voltaire iSER target running in IB/FC router
TargetLinux open source “Gen1”-based target on OFAVendors porting to OFED
24www.openfabrics.org
NFS RDMA Status
Release 5 NFS/RDMA client and server for Linux 2.6.16.16 on SourceForge
Supports OFA CM and RDMA verbs APIClient maintainer is Tom Talpey, NetappServer maintainer is Tom Tucker, Open Grid Computing
Linux NFS RDMA client and server testingPasses the Connectathon test suite, IOzone and network stress testsTested with several Mellanox-based IB HCAs and standard Linux x86 serversInitial test data shows 800MB/s+ read to cache
Release 5 ready for evaluation by users, system integrators and OEMs
25www.openfabrics.org
OFED Storage Summary
Benefits of consolidated InfiniBand I/O are clear and being realized by real-world deployment today
Vendors in production or in final qualification stages of block storage solutions
SRP and iSER will both continue development in OF
NFSoRDMA ready for community trialsStay in lock step with OFED
Vendors porting clustered file systems to OFED
26www.openfabrics.org
System Fabric Works
Software/architecture/consulting servicesVendor neutralExperienced teamCustomers include:
DDN/LSI/Chelsio/QLogic/Cisco/Silver Storm/Trilabs/Agilent/USDoD/GE/Panta/Themis…
27www.openfabrics.org
Embedded Products
Vendor Part Number DescriptionDiversified Technology
SBS/GE Fanuc IB4X-CPCI-2A 2 port HCA cPCI 3U/6U
TDB VITA41 IB SBC
VxWorks 5.5 IB stack
IB4X-PMC-2A 2 port HCA PMC moduleTDB VITA41 IB switch
Sky Computers
ATC5232 IB ATCA SBC
ATC2148 ATCA IB Switch
AMS0508 5 slot ATCA chassis IB compatible
AMS1412 14 slot ATCA chassis IB compatible
Various IB based systems
top related