best practice and deployment of the network for … practice and deployment of the network for...

Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center

Samir Sharma, Juniper Networks

Author: Samir Sharma, Juniper Networks

Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center © 2012 Storage Networking Industry Association. All Rights Reserved. 2 2

SNIA Legal Notice

The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations and literature under the following conditions:

Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.

This presentation is a project of the SNIA Education Committee. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.


Abstract

Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center

Low Latency, high bandwidth, lossless, spanning tree, IO Convergence, Layer 2, Layer 3, Storage Resource Management ,Two Tier, Three Tier, and Flat… when deploying iSCSI, NAS and DAS what should the network topology be? How can network topology affect Large, Medium and Small Scale network deployments? In this presentation we will discuss best practice and deployments of the network for iSCSI , NAS and DAS in the Data Center.

Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center © 2012 Storage Networking Industry Association. All Rights Reserved.

Agenda

4

1 Scope of this presentation?

2 Definitions – The Storage Puzzle

3 Benefits and where are they being deployed?

4 Lossless or Lossy … That is the Question…

5 Ideal and Non Ideal Ethernet Storage Network Deployments

6 Mapping the Deployment to what is Ideal?


Agenda

5

1 Scope of this presentation?


Background The pieces of the puzzle that make up Ethernet network storage deployments. Topologies Ethernet Network topologies that are conducive to storage deployments. Summary and final thoughts.

Scope of this presentation

6

?


Agenda

7

2 Definitions – The Storage Puzzle

3

6


Definitions – The Storage Puzzle

8


File Level Storage File Level storage : is the most common storage system that we find with our hard-drives, NAS systems, etc. In this type of storage, the storage disk is configured with a particular protocol (Like NFS, etc) and files are stored and accessed from it as such, in bulk.

Advantages of File Level Storage System: • File level storage system is simple to implement and simple to use. • It stores files and folders and is visible as such, to both the systems storing the files and

the systems accessing it. • File level storage systems are generally inexpensive, when compared to block level

storage systems. • File level storage systems are more popular with NAS based storage systems –

Network Attached Storage. • They can be configured with common file level protocols like NTFS (Windows), NFS

(Linux), etc. • File level storage systems are well suited for bulk file storage. • The file level storage device itself can generally handle operations like access control,

integration with corporate directories, etc.

9


Block Level Storage Block level storage : is raw blocks (storage volumes) are created and each block can be controlled like an individual hard drive. Generally, these blocks are controlled by the Server based Operating Systems. Each block/ storage volume can be individually formatted with the required file system.

Advantages of Block level storage systems: • Block level storage systems offer a better performance/ speed than file level storage

systems. • Each block / storage volume can be treated as an independent disk drive and are

controlled by external Server OS. • Each block / storage volume can be formatted with the file system required by the

application (NFS / NTFS / SMB , etc). • Block level storage systems are very popular with SAN – Storage Area Networks. • Block level storage systems are more reliable, and their transport systems are very

efficient. • Block level storage can be used to store files and also provide the storage required for

special applications like Databases, VMFS (Virtual Machine File Systems), etc. • They can support external boot-up of the systems connected to them.

10


Direct-attached Storage (DAS) Direct-attached storage (DAS) refers to a digital storage system directly attached to a server or workstation, without a storage network in between. Protocols used in DAS: ATA, SATA, eSATA, SCSI, SAS, and Fibre Channel.

Pros: Cheep, Compact great for very small business. Con: It is hard to share data or unused resources

with other servers.

11

http://en.wikipedia.org/wiki/Data_storage_device�

http://en.wikipedia.org/wiki/Data_storage_device�

http://en.wikipedia.org/wiki/Server_(computing)�

http://en.wikipedia.org/wiki/Workstation�

http://en.wikipedia.org/wiki/Storage_area_network�

http://en.wikipedia.org/wiki/Storage_area_network�

http://en.wikipedia.org/wiki/AT_Attachment�

http://en.wikipedia.org/wiki/SATA�

http://en.wikipedia.org/wiki/ESATA�

http://en.wikipedia.org/wiki/SCSI�

http://en.wikipedia.org/wiki/Serial_Attached_SCSI�

http://en.wikipedia.org/wiki/Fibre_Channel�

http://en.wikipedia.org/wiki/Fibre_Channel�


Network-attached storage (NAS)

Network-attached storage (NAS) is file-level computer data storage connected to a computer network providing data access to heterogeneous clients. It provides file-based storage. Protocols used in NAS: uses file-based protocols NFS, SMB/CIFS, or AFP. NAS units rarely limit clients to a single protocol.

Pros: Often a striped down OS and hardware is needed. Cons: Typically not conducive for large scale deployments.

12

http://en.wikipedia.org/wiki/Computer_data_storage�

http://en.wikipedia.org/wiki/Computer_data_storage�

http://en.wikipedia.org/wiki/Computer_network�

http://en.wikipedia.org/wiki/Heterogeneous_computing�

http://en.wikipedia.org/wiki/Heterogeneous_computing�

http://en.wikipedia.org/wiki/Network_File_System_(protocol)�

http://en.wikipedia.org/wiki/Server_Message_Block�

http://en.wikipedia.org/wiki/CIFS�

http://en.wikipedia.org/wiki/Apple_Filing_Protocol�


Internet Small Computer System Interface (iSCSI)

Internet Small Computer System Interface (iSCSI)- is a transport for iSCSI using TCP/IP (typically native 1GbE, 10GbE, 40GbE and 100GbE)

Pros: • Cost effect components, hard drives, servers, network cards, cables, switches, optics.. • Leverages TCP/IP to guarantee lossless traffic and in order frame delivery. • Built in security, authentication with RADIUS servers and can leverage IPSec, MACSec.

Cons: • Rely on overlay protocols to get lossless and in order frame delivery. Causing

complexity and overhead in the network.

13


Fibre Channel over Ethernet (FCoE)

Fibre Channel over Ethernet (FCoE) is an encapsulation of Fiber Channel frames over Ethernet networks. Why is this important? FCoE will carry FC frames (eg SCSI commands) directly over Ethernet.

This is one of the key drivers that enables SAN LAN convergence.

14

Ethernet HD FC Frame Ethernet

FCS

FCoE Frame = Ethernet Frame that has an FC frame inside it.

http://en.wikipedia.org/wiki/Fiber_Channel�

http://en.wikipedia.org/wiki/Fiber_Channel�

http://en.wikipedia.org/wiki/Ethernet�


Agenda

15

3 Benefits and where are they being deployed?


Benefits of Ethernet Storage

16

COST: Transport ultimately more economical than other storage transports.

SPEED: Higher bandwidth

• 10GbE • 40GbE • 100GbE

EQUIPMENT More cost effective

• Storage equipment • Servers • Networking equipment

In-order/ Lossless: TCP/IP = In-order, guaranteed delivery DCB = Lossless, traffic seperation


Where is it being deployed?

17

Large

Medium Business

Small Business

FCoE Converged Access

End to end FCoE & iSCSI

iSCSI


Agenda

18

4 Lossless or Lossy … That is the Question…


Lossless or Lossy.. That is the Question….

Very Important Rule: Thy shall never loose or drop a storage frame. Lossless Network Topology: No Frame drop between initiator and target. Lossy Network Topology: The network allows for frame drop within the network and rely on overlay protocols to handle frame drop (eg TCP/IP).

19


Realities of Traditional Ethernet in the Data Center

• Traditionally not meant to transport lossless traffic • Forced to rely on protocols like TCP/IP to achieve a lossless topology

-> Resulting in protocol overhead/congestion due to retransmission. (iSCSI used TCP/IP) Frame loss

• Multi-tier • Multiple device to manage • Spanning Tree adds inefficiencies 50% of links blocked • Heavily Oversubscribed 10:1, 5:1, 4:1 take your pick……

Network Topologies

• Mix Traffic Types • Video Running on Ethernet • Voice Running on Ethernet • Traditional Applications Running on Ethernet • Now add Storage and Server Virtualization to the mix……

Network Congestion

Possible Problem Areas


• Need: No frame drops

between initiator and target

Lossless Network Topology

• Need: Frames must be in order between initiator to target

In order frame delivery

Storage over Ethernet What do you need?

• Solution : If a frame is dropped TCP/IP will retransmit

• Result: This adds latency and can exacerbate congestion

Relying TCP/IP for the Solution….

• Solution : TCP/IP guarantees in-order delivery per session

Fibre Channel’s Solution….

• Need: No frame drops

between initiator and target Lossless Network Topology/ In order

frame delivery

• Uses a buffer credit mechanism to create a lossless network.

• Eliminates frame drop and frame retransmission


Solution : Map storage flows to PFC

Creates a lossless network PFC and DCB (dcbx allows the creation of arbitrary application tlv’s based on higher level protocol to easily capture and configure for cifs,nfs etc …)

No need for retransmission (Closest way to emulate FC Credit

Mechanism)

Lossless Ethernet Storage Fabric

Mapping Different Flows on Separate Priority… Result True Traffic Separation


Agenda

23

5 Ideal and Non Ideal Ethernet Storage Network Deployments


Multi-tier legacy network

Too slow Too expensive Too complex

The challenges

Realities of Ethernet in the Data Center

Up to 75% of traffic E W

S

N

Scale

Com

plex

ity

Spanning Tree disables up to 50%

of bandwidth

Unnecessary layers add hops

and latency

Up to 50% of the ports interconnect

switches, not servers or

storage Network Complexity TCP/IP can’t scale


Non-Ideal Network Topology

25

Ethernet L2/L3 Aggregation Switch

Ethernet L2/L3 Access Switch

Ethernet L3 Core Switch/Router

POD 1 POD 2 POD 3 POD 4


Non-Ideal Network Topology

Ethernet L2/L3 Aggregation Switch

Ethernet L2/L3 Access Switch

Ethernet L3 Core Switch/Router

POD 1 POD 2 POD 3 POD 4

• 7 switches • 10 chances to drop the frame • 7 devices to manage….

The Network gets in the way…


Summary of issues..

Three Tier design (access, aggregation and core) design flaws:

To many hops add latency. Spanning Tree removes ~50% of the links thus increasing congestion. Lossy fabric rely on overlay protocols like TCP/IP this adding to congestion.

27


Simple solutions..

Three Tier design (access, aggregation and core) design flaws:

To many hops add latency. Spanning Tree removes ~50% of the links thus increasing congestion. Lossy fabric rely on overlay protocols like TCP/IP this adding to congestion.

28

Solution Flatten the Network Topology

Use overlay protocols like MC-LAG

Create a lossless fabric by mapping all storage flows to a lossless queue (use DCB)


Traditional Network Topology

29

Access Layer DCB Enabled

Aggregation Layer DCB Enabled

Storage Array

Servers


Ideal Network Topology

30

Spine

Leaf

MC-LAG

One way……..

40GbE/100GbE

10GbE/40GbE

Use MC-LAG to remove the ills of Spanning Tree….

Servers Connected to Leaf/Access Layer

Storage Array


Ideal Network Topology

31

Flat Fabric

Desired way..

• Everything is connected to everything (Pooling all resources together) • Access is always 1 hop away. • Removing the ills of Spanning Tree….


Agenda

32

6 Mapping the Deployment to what is Ideal?


Hadoop Network Topology

33

Spine

Leaf

MC-LAG

One way

40GbE/100GbE

10GbE/40GbE


Hadoop Network Topology

34

Flat Fabric

Desired Way….

Note • No need for STP everything is connect to everything else one hop away. • Best Latency, least amounts of hops.


Mixing DAS, NAS, iSCSI and FCoE

35

FC SAN DCB Enabled Fabric

FC SAN


Example: DCB Enabled Fabric/Network Mixing DAS, NAS, iSCSI in a DCB Enabled Fabric

36

Note • Keep Traffic separated on different priorities on all network devices. Eg) Priority 1 = Tape Traffic (on all switches) Priority 2 = iSCSI Traffic (on all switches) Priority 3 = FCoE Traffic (on all switches) Priority 4 = NAS Traffic (on all switches)

MC-LAG


Example: DCB Enabled Fabric/Network Mixing DAS, NAS, iSCSI in a DCB Enabled Fabric

37

DCB Enabled Fabric

Note • Keep Traffic separated on different priorities on network device. Eg) Priority 1 = Tape Traffic (on all switches) Priority 2 = iSCSI Traffic (on all switches) Priority 3 = FCoE Traffic (on all switches) Priority 4 = NAS Traffic (on all switches) Flat Topology: Easy to do operationally when there is only one device to manage……


LOSSLESS

Remote office Data Replication

38

Data Center Backup Data Center

LOSSLESS LOSSLESS

Campus

Things to think about… • Does your WAN devices have enough buffer to handle frame that are in-flight after pause is sent. • Distance correlates directly to the amount of buffer needed.

PAUSE

Frames in-fight



39

TCP/IP


Note • WAN Routers should be able buffer several flows to account for frame drop and ensure retransmission.

LOSSLESS LOSSLESS LOSSY with Protocol Overlay

Campus



40

MPLS/VPLS


Note • WAN links are dedicated to just storage traffic and no other traffic. • Speed is pre-negotiated to ensure no packet drop.

LOSSLESS LOSSLESS LOSSY with Protocol Overlay

Campus


Final Thoughts

Loop in the Network (Spanning Tree) Flatten your fabric to remove loops, this allows better pooling of Ethernet storage. Or Use protocols like MC-LAG (Spin-Leaf) to remove loop.

Lossless Fabric/Networks

Map flows on the same priority on all network devices. For long distances be mindful of the amount of buffer on the network device. Else frame loss will be introduced.

Lossy Fabric/Networks

Use a protocol like TCP/IP to ensure proper frame delivery. If you are brave and just want to rely on Ethernet. (Not suggested)

Dedicated to just storage traffic and no other traffic. Pre-negotiated speed to ensure no packet drop.

41


Q&A / Feedback

Many thanks to the following individuals for their contributions to this tutorial.

- SNIA Education Committee

Joseph White Kishore Inampudi Simon Gordon Haruki Sonehara Andy Ingram

Send any questions or comments on this presentation to SNIA: [email protected]

mailto:[email protected]�

best practice and deployment of the network for … practice and deployment of the network for...

Documents