investigating serial attached scsi (sas) over tcp ( tsas )
Post on 23-Feb-2016
42 Views
Preview:
DESCRIPTION
TRANSCRIPT
1
Investigating Serial Attached SCSI (SAS) over TCP (tSAS)
UCCS
Master’s Project
Deepti Reddy
2
SCSI vs SAS
SCSI (Small Computer System Interface)
• The SCSI protocol is an application layer storage protocol.
• It's a standard for connecting peripherals to your computer via a standard hardware interface, which uses standard SCSI commands.
SAS (Serial Attached SCSI)
• SAS is the successor of SCSI technology• Initially introduced at 3Gb/s in 2004• Currently, supports 6Gb/s• 12Gbps by 2012• Significantly increased the available bandwidth offered by
legacy SCSI storage systems.• Use of expanders increases the scalability over legacy
SCSI
3
SCSI vs SAS
SCSI SAS
Topology Parallel Bus Serial BusSpeed 3.2 Gbps 3 Gbps, 6Gbps,
12Gbps
Distance 1 to 12 meters 8 metersNumber of Targets 14 devices 128 expanders.
> 16,000 with cascaded expanders
Devices SCSI only SAS & SATAConnectivity Single Port Dual PortDrive Form Factor 3.5” 2.5”Cost Low Medium
Comparing SCSI and SAS
4
iSCSI iSCSI protocol describes a means of transporting of the SCSI
packets over TCP/IP The iSCSI specification is revised and updated by the Internet
Engineering task Force (IEFT). Work by S. Aiken, D. Grunwald, A. Pleszkun and J. Willeke shows
the performance of a commercial iSCSI software implementation compared quite favorably with fibrechannel [7]
iSCSI protocol layering model
5
Typical SAS Topology
X X X
X
- DISK DRIVES
- EXPANDERS
DriverSAS CONTROLLER(HOST BUS ADAPTER)
~8m
6
Motivation & Challenges Overcome the distance and scalability
limitations of traditional Serial Attached SCSI (SAS) by using tSAS.
Like iSCSI, tSAS takes advantage of existing internet infrastructure, internet management facilities as well as addresses distance limitations
Provide research results for future industry specification for tSAS and iSCSI.
7
tSAS Topology
X
X
- DISK DRIVES
- EXPANDERS
SCSI DrivertSAS CONTROLLER/HBA
- TCP link- SAS link
SCSI DrivertSAS
HBA
8
X
X
- DISK DRIVES
- EXPANDERS
SCSI DrivertSAS CONTROLLER/HBA
- TCP link- SAS link
tSASHBA
SCSI Driver
tSAS Topology
9
Goals of the project Investigate how tSAS can be implemented
Design and develop a mock tSAS application− Write a client program using C/C++ that sends a set
of tSAS commands to a server.− Write a server program using C/C++ that receives a
set of tSAS commands, processes them and responds to the client with tSAS responses.
Evaluate the tSAS solution and compare with a mock iSCSI client/server application.
10
Software and Hardware solutions for tSAS implementations
Software based tSAS solution− Cheaper than a hardware based solution− All tSAS processing is done by the processor and
TCP/IP operations are also executed by the CPU− NIC is merely an interface to the network
TCP offload engine solution− NICs with integrated TOEs have hardware built into
the card that allows the TCP/IP processing to be done at the interface
Hardware based tSAS solution− In a hardware-based tSAS environment, the initiator
and target machines contain a host bus adapter (HBA) that is responsible for both TCP/IP and tSAS processing
11
tSAS Approach The tSAS Request is initially sent by the tSAS
Initiator to the tSAS Target over TCP.
The tSAS Target strips off the TCP header and sends the SAS frames using the SAS Initiator block on the tSAS Target to the SAS expander/drive.
The SAS expander/drive sends SAS frames to the tSAS Target.
Finally, the tSAS Target embeds the SAS frames received from the expander/drive over TCP and sends it to the tSAS Initiator.
12
tSAS Message Format
The above shows how a legacy SAS header and data are embedded in an Ethernet frame.
iSCSI uses the same approach where the iSCSI header and data are encapsulated in an Ethernet frame.
Ethernet Header
IP Header
TCP Header
eSAS Header
eSASData
Ethernet Trailer
TCP SegmentIP Datagram
Ethernet Frame
13
tSAS Approach
The above diagram shows a typical tSAS (SSP Read ) Request & Response sequence diagram.
iSCSI uses a similar approach where the back-end between the iSCSI target and SCSI drives uses the legacy SCSI protocol.
tSAS Request
tSAS Initiator
tSAS Target
SAS Expander
Open Address Frame
Open Accept
SAS Request Frame
Data Frame..
Data Frame
tSAS Response
Response Frame
Data Frame..
Data Frame
14
Primitives− Primitives are special 8b/10b encoded characters
that are used as frame delimiters, for out of band signaling, control sequencing in SAS
− Most primitives can be conveniently ignored on the tSAS side except a few such as Broadcast primitives
− SAS primitive can be encapsulated in an Ethernet frame
Discovery− Discovery in tSAS will be similar to SAS and will be
accomplished by sending Serial management protocol (SMP) commands over TCP to the initiators and expanders downstream to learn the topology.
tSAS Approach
15
SMP Request
ClientSMP Initiator
Port on Expander
SMP Target Port on Target Expander
Open Address Frame
Open AcceptSMP Request
SMP Response
CloseSMP Response
tSAS Approach
SMP Request & Response Sequence Diagram
16
Test benches used for performance calculations
tSAS and iSCSI Mock Application− The client application and server application run on two different
Windows machines/servers connected using a NetGear ProSafe Gigabit Switch
− Client makes Read/Write requests and the server processes and responds to requests made by the Client
Legacy SAS− A 6 Gbps Sas Host Bus Adapter in a PCIe slot of a Super Micro
server− A 6 Gbps SAS target connected to the Host Bus Adapter− IOMeter in Windows and VDBench in RHEL used to generate
Read/Write IOs and measure performance− A SAS analyzer placed between the HBA and the SAS Target
Legacy iSCSI− Two windows machines/servers were used− On one machine the StarWind iSCSi Initiator was used− On another machine, the KernSafe iSCSi target software was
used to create a iSCSI target− IOMeter was used to send Read/Write requests from the iSCSI
Initiator to the iSCSI Target and measure performance− Wireshark used to capture Network Traffic
17
tSAS vs iSCSI Mock Application at 10 Mbps
0 200 400 600 800 1000 12000
5001000150020002500
tSAS vs iSCSI READ 10 MbpsX-axis – Transfer Size (Bytes)Y-axis – Time in milliseconds
tSAS READ Completion timeiSCSI READ Completion Time
0 500 1000 1500 2000 25000
10002000300040005000600070008000
tSAS vs iSCSI Write 10MbpsX-axis – Transfer Size (Bytes)Y-axis – Time in milliseconds
tSAS Write 10MBpsiSCSI Write 10MBps
18
tSAS vs iSCSI Mock Application at 100 Mbps
0 500 1000 1500 2000 25000
100200300400500600700
tSAS vs iSCSI READ 100 MbpsX-axis – Transfer Size (Bytes)Y-axis – Time in milliseconds
tSAS Read 100 MbpsiSCSI Read 100 Mbps
0 500 1000 1500 2000 25000
200400600800
1000
tSAS vs iSCSI READ 100 MbpsX-axis – Transfer Size (Bytes)Y-axis – Time in milliseconds
tSAS Write 100 MbpsiSCSI Write 100 Mbps
19
tSAS vs iSCSI Mock Application at 1 Gbps
0 500 1000 1500 2000 25000
1020304050
tSAS vs iSCSI READ 1 GbpsX-axis – Transfer Size (Bytes)Y-axis – Time in milliseconds
tSAS Read 1 GbpsiSCSi Read 1 Gbps
0 500 1000 1500 2000 25000
20406080
tSAS vs iSCSI READ 1 GbpsX-axis – Transfer Size (Bytes)Y-axis – Time in milliseconds
tSAS Write 1 GbpsiSCSI Write 1 Gbps
20
tSAS 512K Read at 10 Mbps, 100 Mbps and 1 Gbps
0 200 400 600 800 1000 12000
200
400
600
800
1000
1200
Time for READ Completion with transfer size of 512K at 10 Mbps, 100Mbps and 1Gbps
X-axis – Transfer Size (Bytes)Y-axis – Time in milliseconds
Time for READ Compeltion
21
Comparing tSAS mock application to legacy iSCSI and legacy SAS
tSAS results are extrapolated to 100 Gbps and so are the legacy iSCSI results to compare tSAS with legacy iSCSI
− Assuming 100 Gbps connection rate performs 43.348 times better than 1 Gbps connection rates based on the 1 Gbps and 10 Gbps server scalability paper by NetApp where it is concluded that 10 Gbps performs 4.3348 times better than 1 Gbps.
− These comparison results can’t be considered to be a fair comparison as the tSAS mock application is not a full fledged tSAS implementation
tSAS results are compared to legacy SAS− Legacy SAS performance results between the HBA and the SAS
drive observed using the SAS analyzer for various transfer sizes under 2 MB show that Read/Write completions are < 1 ms. tSAS performs much slower than this at 100 Gbps
− tSAS is compared to legacy SAS without including the delay at drive
− These comparison results can’t be considered to be a fair comparison as the legacy SAS application is HBA based while the tSAS application is S/W based and is not a full fledged application
22
Comparing tSAS mock application to legacy iSCSI
0 500 1000 1500 2000 25000
5
10
15
20
Write performance tSAS at 100 Gbps vs legacy iSCSI at 100 Gbps using IOMeter without delay at
the driveTransfer Sizes (Bytes) vs Time (Milliseconds)
Write performance extrapo-lated tSAS 100 GbpsLegacy iSCSI Write timings using IOMeter without de-lay at the drive extrapo-lated 100 Gbps
0 500 1000 1500 2000 250002468
1012141618
Read performance tSAS at 100 Gbps vs legacy iSCSI at 100 Gbps using IOMeter without delay at the drive
Transfer Sizes (Bytes) vs Time (Milliseconds)
Read performance extrapo-lated tSAS 100 GbpsLegacy iSCSI Read timings using IOMeter without delay at the drive extrapolated 100 Gbps
23
Comparing tSAS Mock application to legacy SASX-Axis – Transfer Length (Bytes), Y-Axis – Time in
Milliseconds
0 500 1000 1500 2000 250002468
10121416
tSAS 100 Gbps Performance vs Legacy SSP Read tim-ings without delay at the drive
Read extrapolated tSAS 100 GbpsLegacy SAS(SSP) Read Tim-ings using IOMeter without Delay at the driveLegacy SAS(SSP) Read Tim-ings using VDBench without Delay at the Drive
0 500 1000 1500 2000 25000
2
4
6
8
10
12
tSAS 100 Gbps Performance vs Legacy SSP Write timings without delay at the drive
Write extrapolated tSAS 100 GbpsLegacy SAS(SSP) Write Timings using IOMeter without Delay at the driveLegacy SAS(SSP) Write Timings using VDBench without Delay at the Drive
24
Similar Work
Michael Ko’s patent on Serial Attached SCSI over Ethernet proposes a very similar solution to the tSAS solution provided in this project.
iSCSI specification (SCSI over TCP) itself is similar to a tSAS solution (SAS over TCP). The iSCSI solution can be heavily leveraged for a tSAS solution.
The Fibre Channel over TCP/IP specification also can be leveraged to design and implement a tSAS solution.
25
Conclusion tSAS is a viable solution tSAS will be faster than a similar iSCSI implementation
− Frame sizes in tSAS smaller than in iSCSI− Back-end of tSAS will be legacy SAS− tSAS should be visibly faster at larger transfer sizes− tSAS hardware solution would be the fastest vs a
software solution− tSAS would perform better at smaller transfer sizes
tSAS will overcome the distance limitation of legacy SAS
26
Future Work More data points
− The tSAS mock application can be run using a faster switch with connection rates greater than 1 Gbps to get more data points
Piggybacking− Response frames are piggybacked with the last DATA
frame or a DATA frame is piggy backed with a request frame
Jumbo frames − Can be used to increase the amount of DATA that is
passed from the initiator and target per Ethernet packet improving the performance results.
The Storage Associations can be motivated with similar work to fuel the conception of a tSAS specification
tSAS code can be implemented in a SAS HBA and performance can be measured using this direct implementation
27
References [1] T10/1760-D Information Technology – Serial Attached SCSI – 2 (SAS-2), T10, 18 April 2009, Available from http://www.t10.org/drafts.htm#SCSI3_SAS [2] Harry Mason, Serial attached SCSI Establishes its Position in the Enterprise, LSI Corporation, available from http://www.scsita.org/aboutscsi/sas/6GbpsSAS.pdf [3] J Satran, K Meth, C. Sapuntzakis, M. Chadalapka, E. Zeidner, RFC 3720 Internet Small Computer Systems Interface (iSCSI) Specification IETF, April 2004, available from http://www.ietf.org/rfc/rfc3720.txt [4] Cai, Y.; Fang, L.; Ratemo, R.; Liu, J.; Gross, K.; Kozma, M.; A test case for 3Gbps serial attached SCSI (SAS) Test Conference, 2005. Proceedings. ITC 2005. IEEE International, February 2006, available from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1584027
28
References [6] Kalmath Meth, Julian Satran, Design of the iSCSI Protocol,Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings. 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2003, available from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1194848&tag=1 [7] Stephen Aiken, Dirk Grunwald, Andrew R. Pleszkun, Jesse Willeke, A Performance Analysis of the iSCSI Protocol 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003, available from http://www.storageconference.org/2003/papers/20-Aikens-Performance.pdf [8] M. Rajagopal, E. Rodriguez, R. Weber, RFC 3821 Fibre Channel over TCP/IP (FCIP) standard, IETF, July 2004, available from http://tools.ietf.org/html/rfc3821
[9] BLi Bigang, Shu Jiwu, Zheng Weimin, SCSI Target Simulator Based on FC and IP Protocols in TH-MSNS *
Department of Computer Science and Technology, Tsinghua University, Beijing China, 2005
29
References [10] S. Chaitanya, K. Butler, A. Sivasubramaniam, P. McDaniel, M. Vilayannur, Design, Implementation and Evaluation of Security in iSCSI-based Network Storage Systems, StorageSS '06 Proceedings of the second ACM workshop on Storage security and survivability, October 2006, available from http://portal.acm.org/citation.cfm?id=1179564
[11] 1Gbps and 10Gbps Ethernet Server Scalability, NetApp, available fromhttp://partners.netapp.com/go/techontap/matl/downloads/redhat- neterion_10g.pdf
[12] Michael A. Ko, LAYERING SERIAL ATTACHED SMALL COMPUTER SYSTEM INTERFACE (SAS) vOVER ETHERNET, United States Patent Application 20080228897, 09/18/2008 available from http://www.faqs.org/patents/app/20080228897
[13] SCSI Primary Commands - 4 (SPC-4), Revision 31, T10, 2011/06/13, available from http://www.t10.org/members/w_spc4.htm
top related