fcoe deep dive

DCB and FCoE Deep dive

Jaromír Pilař (jpilar@cisco.com)Consulting Systems Engineer, CCIE 2910

What Is I/O Consolidation� IT organizations operate multiple parallel networks

IP and other LAN protocols over an Ethernet networkSAN over a Fibre Channel networkHPC/IPC over an InfiniBand network

� I/O consolidation supports all three types of traffic onto a single network

� Servers have a common interface adapter that supports all three types of traffic

IPC: Inter-Process Communication

Consolidation - one of major trends in datacenterBut where is the main consolidation potential ?

� Majority of ports in fabric is in access layer regardless of fabric type => access layer has the highest potential for consolidation

� Different fabrics (network, SAN, HPC) have different requirements => do we have the technology which can serve them all at once?

� If we have it => is the technology mature enough and affordable to be massively deployed?

ProcessorMemory

I/O Consolidation in the Network

Storag

IPCProcessorMemory

I/O Subsystem

Storag

I/O I/O I/O

IPC: Inter-Process Communication

FC TrafficFC HBA

I/O Consolidation in the Host� Fewer CNAs (Converged Network Adapters) instead of NICs, HBAs, and HCAs

� Limited number of interfaces for Blade Servers

All Traffic Goes over 10 GE

FC TrafficFC HBA

NIC Enet TrafficNIC Enet TrafficNIC Enet TrafficHCA IPC Traffic

IPC TrafficHCA

Cabling and I/O Consolidation

Merging the Requirements

LAN/IP

� Must be EthernetToo much investmentToo many applications that assume Ethernet

� Must follow the Fibre Channel model

� Losing frames is not an option

Storage IPC(Inter-Process

Communication)

� Doesn’t care of the underlying network, provided that:

It is cheapIt is low latencyIt supports APIs like OFED, RDS, MPI, sockets

Why Consolidation Attempts Have Not Succeeded Yet?� Previous attempts

Fibre ChannelNever credible as data network infrastructure

InfiniBandNot Ethernet

iSCSINot Fibre Channel

� Before PCI-Express there was not enough I/O bandwidth in the servers

� It needs to be Ethernet, but…1 GE didn’t have enough bandwidth

Multicore CPU Architectures Allowing Bigger and Multiple Workloads on the Same Machine

Server Virtualization Driving the Need for More Bandwidth per Server Due to Server Consolidation

Growing Need for Network Storage Driving the Demand for Higher Network Bandwidth to the Server

Drivers for 10GE to the Servers

Multicore CPUs and Server Virtualization Driving the Demand for Higher Bandwidth Network Connections

Enabling Technologies

Three Challenges + One

Why Are Frames Lost?Collision

� No longer present in full duplex Ethernet

� Very rare in the data center

Transmission Error

� Most common cause

� Congestion is a switch issue, not a link issue

A full duplex IEEE 802.3 link does not lose frames

� It must be dealt with in the bridge/switch

By IEEE 802.1

Congestion

Can Ethernet Be Lossless?� Yes, with Ethernet PAUSE Frame

PAUSESTOP

Ethernet Link

Switch A Switch B

Queue Full

� Defined in IEEE 802.3—Annex 31BThe PAUSE operation is used to inhibit transmission of data frames for a specified period of time

� Ethernet PAUSE transforms Ethernet into a lossless fabric

How PAUSE WorksThreshold

PAUSEFrame

Stop SendingFrames for ThisInterval of Time

PAUSEFrame

Start SendingFrames Again

Let’s Compare PAUSE with FC Buffer to Buffer Credit� Eight credits preagreed

R_RDYR_RDY

PAUSE Frame Format� A standard Ethernet frame, not tagged

� EtherType = 0x8808 means MAC Control Frame

� Opcode = 0x0101 means PAUSE� Pause_Time is the time the link needs to remain paused in Pause Quanta (512-bits time)

� There is a single Pause_Time for the whole link

Pad42 Bytes

01:80:C2:00:00:01

Source Station MAC

EtherType = 0x8808

PAUSE Frame

Pause_TimeOpcode = 0x0001

Why Is PAUSE Not Widely Deployed?� Inconsistent implementations

Standard allows for asymmetric implementations Easy to fix

� PAUSE applies to the whole linksSingle mechanism for all traffic classes

� This may cause “traffic interference”e.g., Storage traffic paused due to a congestion on IP traffic

Priority Flow Control (PFC)� a.k.a. PPP (Per Priority Pause)� PFC enables PAUSE functionality per Ethernet priority

IEEE 802.1Q defines eight prioritiesTraffic classes are mapped to different priorities:

No traffic interferenceIP traffic may be paused while storage traffic is being forwardedOr, vice versa

Requires independent resources per priority (buffers)� High level of industry support

Cisco distributed proposalStandard track in IEEE 802.1Qbb

EtherType = IEEE 802.1Q Priority CFI VLAN IDIEEE 802.1Q Tag

16 3 1 12 Bits

Priority Flow Control in Action

EightPriorities

Switch A Switch B

Transmit QueuesEthernet Link

Receive Queues

OneTwoThreeFourFive

SevenEight

OneTwoThreeFourFive

SevenEight

SixSTOP PAUSE

PFC Frame Format� Similar to the PAUSE frame� Opcode = 0x0101 is used to distinguish PFC from PAUSE

� Class vector indicates for which priorities the frame carries valid Pause information

� There are eight Time fields, one per priority

Class Enable VectorTime (Class 0)

Pad28 Bytes

01:80:C2:00:00:01

Source Station MAC

EtherType = 0x8808Opcode = 0x0101

Time (Class 1)Time (Class 2)Time (Class 3)Time (Class 4)Time (Class 5)Time (Class 6)Time (class 7)

Priority Flow Control

In Order to Build a Deployable I/O Consolidation Solution, the Following Additional Components Are Required:

Is Anything Else Required?

� Discovery protocol (DCBX)� Bandwidth manager

DCBX� Hop-by-hop negotiation for:

Priority Flow Control (PFC)Bandwidth managementApplicationsLogical link-down

� Based on LLDP (Link Level Discovery Protocol)Added reliable transport

� Allows either full configuration or configuration checkingLink partners can choose supported features and willingness to accept configuration from peer

Bandwidth Management� IEEE 802.1Q defines priorities, but not a simple, effective, and consistent scheduling mechanism

� Products typically implement some form of Deficit Weighted Round Robin (DWRR)

Configuration and interworking is problematic� Proposal for HW-efficient, two-level DWRR with strict priority support

Consistent behavior and configuration across network elements� Standard track in IEEE 802.1Qaz

Priority Groups

Priorities Are Assigned to Individual

Traffic Classes

PriorityGroups

Priority Groups Are Then Scheduled

First Level of Scheduling Inside Each Group

Final Link Behavior

Example of Link Bandwidth Allocation10 GE Link Realized Traffic Utilization

T1 T2 T3

LAN Traffic(40%)

Storage Traffic(30%)

(30%) HPC Traffic(30%)

HPC Traffic—Priority Class “High”—20% Guaranteed BandwidthLAN Traffic—Priority Class “Medium”—50% Guaranteed BandwidthStorage Traffic—Priority Class “Medium-High”—30% Default Bandwidth

Offered Traffic

3 Gbs 4 Gbs 6 Gbs

3 Gbs 3 Gbs

3 Gbs 3 Gbs 3 Gbs

T1 T2 T3

FCoE: Fibre Channel over Ethernet

What is Fibre Channel over Ethernet?� From a Fibre Channel standpoint it’s

FC connectivity over a new type of cable called… an Ethernet cloud

� From an Ethernet standpoints it’sYet another ULP (Upper Layer Protocol) to be transported, but… a challenging one!

� And technically…

FCoE is an extension of Fibre Channelonto a Lossless Ethernet fabric

FCoE Enablers and EncapsulationEthe

ader FC Payload CRC

Same as a physical FC frame

Control information: version, ordered sets (SOF, EOF)

Normal ethernet frame, ethertype = FCoE

� 10Gbps Ethernet� Lossless Ethernet

Matches the lossless behavior guaranteed in FC by B2B credits� Ethernet jumbo frames

Max FC frame payload = 2112 bytesTotal max frame size = 2180 bytes

FCoE Is Fibre ChannelFCoE Is Fibre Channel at the Host and Switch Level

Same Operational Model Same Techniques ofTraffic Management

Same Managementand Security Models

Easy to Understand

Completely Based on the FC ModelSame Host-to-Switch and Switch-to-Switch Behavior of FCe.g., in Order Delivery or FSPF Load BalancingWWNs, FC-IDs, Hard/Soft Zoning, DNS, RSCN

The two protocols have:• Two different Ethertypes• Two different frame formats• Both are defined in FC-BB-5

Protocol Organization

FCoE itself � Is the data plane protocol� It is used to carry most of the

FC frames and all the SCSI traffic

� Uses Fabric Assigned MAC address (dynamic)

FCoE itself � Is the data plane protocol� It is used to carry most of the

FC frames and all the SCSI traffic

� Uses Fabric Assigned MAC address (dynamic)

FIP (FCoE Initialization Protocol)

� It is the control plane protocol� It is used to discover the FC

entities connected to an Ethernet cloud

� It is also used to login to and logout from the FC fabric

� Uses unique BIA on CNA for MAC

FIP (FCoE Initialization Protocol)

� It is the control plane protocol� It is used to discover the FC

entities connected to an Ethernet cloud

� It is also used to login to and logout from the FC fabric

� Uses unique BIA on CNA for MAC

http://www.cisco.biz/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html

All Standards for FCoE Are Technically StableStatus of the Standards

Inv Dev Appr Pub

Technically Stable

FC-BB-5

Inv Dev Appr Pub

Technically stable in October, 2008Completed in June 2009Published in May, 2010

Completed in July 2010, awaiting publication

Completed in July 2010 (completing Approval Phase 3)

Myths and Misunderstandings

Myth: “You Can’t Do End-to-End FCoE.”

StorageNewsletter.com., “Exclusive Interview with Darren Thomas, Head of Dell Storage.” June 29, 2010

The FC-BB-5 standard fully supports end-to-end FCoE

If someone says it does not, it means he did not read the standard

However, current implementations may be behind the standard and do not fully support it yet

Myth: "FC-BB-6 Means the FCoE Standard Isn't Done Yet."

� Standards are like operating systems - they add features to previous versions

� Different versions (e.g., FC-BB-4, FC-BB-5, FC-BB-6) have different features

� FC-BB-5 fully defined the way to transport Fibre Channel over Ethernet

�FC-BB-6 is working on adding features and functionality

QCN Operates At a Different Level Than FCoE

Myth: “You Need QCN (802.1Qau) for End-To-End FCoE.”

� QCN is a core-to-edge protocol to deal with persistent congestion situations in a Layer 2 network

CongestionTraffic

QCN message

QCN messageDA: H3SA: H1

DA: H3SA: H2

DA: H1SA: H3

DA: H2SA: H3

When congestion is detectedthe core switch samples some frames, swaps their MAC addresses, and sends notifications backward

FCoE Crosses Layer 2 DomainsTherefore, QCN is Useless for FCoE

DA: FCF-MAC(A)SA: FPMA(H2)

Encaps. FC frameD_ID = FC-ID(T2)S_ID = FC-ID(H2)

DA: FPMA(T2)SA: FCF-MAC(C)

DA: FCF-MAC(B)SA: FCF-MAC(A)

DA: FCF-MAC(C)SA: FCF-MAC(B)

FCF A FCF B FCF C

What Does This Mean?

Source: FC-BB-5 rev 2.00, June 4, 2009

FCoE = Layer 3 From Ethernet Perspective

Layer 2 of the Fibre Channel model maps to Layer 3 of the Ethernet model

Ethernet (OSI) ModelFC Model

FC-1FC-2PFC-2MFC-2VFC-3FC-4

Layer 1 - PHYLayer 2 - MAC

“Layer 3”FCoE Entity

FC-2VFC-3FC-4 FC Levels

(Unchanged)

IEEE 802.3 Layers

QCN happens here

FCoE happens here (including Multihop)

ResponseMyth: "You Need TRILL To Run FCoE"

TRILL defines an alternative way to Spanning Tree to forward Ethernet frames in an Ethernet network

Also supports multipathingHas nothing to do with congestion

Source: Mellor, Chris. ”DCB is Not Enough.” The Register August 3, 2010

What Does This Mean?

Source: FC-BB-5 rev 2.00, June 4, 2009

FCoE = Layer 3 From Ethernet Perspective

� Layer 2 of the Fibre Channel model maps to Layer 3 of the Ethernet model

Ethernet (OSI) ModelFC Model

FC-1FC-2PFC-2MFC-2VFC-3FC-4

Layer 1 - PHYLayer 2 - MAC

“Layer 3”FCoE Entity

FC-2VFC-3FC-4 FC Levels

(Unchanged)

IEEE 802.3 Layers

TRILL happens here

FCoE happens here (including Multihop)

Nexus 5000 and 2000 family products for Unified Fabric

Cisco Nexus 5010/50201st generation of Nexus 5xxx familyIndustry’s First I/O Consolidation Virtualization Fabric for

Enterprise Data CenterIndustry’s First I/O Consolidation Virtualization Fabric for

Enterprise Data Center

Nexus 5000SwitchFamily Nexus 5020 - 56-Port L2 Switch

• 40 Ports 10GE/FCoE/DCE, fixed• 2 Expansion Modules

Nexus 5010 - 28-Port L2 Switch• 20 Ports 10GE/FCoE/DCE, fixed

• 1 Expansion Module

FC + Ethernet • 4 Ports 10GbE/FCoE/DCE

• 4 Ports 1/2/4G FC

Fibre Channel • 8 Ports 1/2/4G FC

ExpansionModules Ethernet

• 6 Ports 10GE/FCoE/DCE

OSCisco Fabric Manager and Cisco Data Center Network Manager

Cisco NX-OSMgmt

PartnersSW FCoE/DCE + 2x10GE2x10GE/DCE/FCoE 2x10GE

Fibre Channel • 6 Ports 2/4/8G FC

Mgmt 0, Console, USB

Mgmt 0, Console, USB Redundant Fan ModulesRedundant Fan Modules Redundant 750W AC Power SuppliesRedundant 750W AC Power Supplies

• 32 Fixed SFP+ Ports• Line Rate• Hardware Capable of 1/10 Gigabit Ethernet• Traditional Ethernet or Fibre Channel over Ethernet• L3 capable (post FCS)• FabricPath and TRILL capable (post FCS)• 40 GE ready

Expansion Modules (GEM2)• 16p SFP+ Ethernet Ports• 8p Eth + 8p Native FCExpansion Modules (GEM2)• 16p SFP+ Ethernet Ports• 8p Eth + 8p Native FC

Cisco Nexus 55482nd generation of Nexus 5xxx family

� 32x 1/10GE host interfaces; 8x 10GE on network interfaces� 10GE interfaces support FCoE� HW supports 1G but SW support in a post-FCS release

� Can mix-and-match with existing GE and next-gen GE FEX in network topologies� Host port-channel support� ACL classification� SPAN source/destination support� Only FIP enabled CNAs supported (Gen 2)

Nexus 2000 family extension10GE FCOE capable Fabric Extender - Nexus 2232

32 10GE/FCoE SFP+ Downlinks 8 10GE/FCoE SFP+ Uplinks

Unified FabricInitial Deployments

• Servers and FCoE targets are directly connected to the Nexus 5000 over 10Gig FCoE

• Nexus 5000 operates as the FCF• Native Ethernet LAN network and Native Fibre Channel network break off at the Nexus 5000 access layer

Direct Attached Topology

Enhanced Ethernet and FCoEEthernet LANNative Fibre Channel

SAN ASAN A SAN BSAN B

FIP enabled CNAsFIP enabled CNAs

vPCvPC

FIP or Pre-FIP enabled CNAsFIP or Pre-FIP enabled CNAs

FCoE TargetsFCoE Targets

Ethernet/LANEthernet/LAN

Nexus 5000FCF

Nexus 5000FCF Nexus 5000

FCFNexus 5000

Unified FabricMultihop FCoE Deployment with Nexus 4000I

• Blade servers connect to Nexus 4000 over 10Gig FCoE

� Nexus 4000 is a FIP-Snooping Bridge

• Nexus 4000 connects to Nexus 5000 over 10Gig FCoE � Nexus 5000 operates as the FCF

• Native Ethernet LAN network and Native Fibre Channel network break off at the Nexus 5000

SAN ASAN A SAN BSAN B

Nexus 5000FCF

FCoE TargetsFCoE Targets

Blade Chassis

Nexus 4000: FIP Snooping BridgeCNA mezzanine

Nexus 5000FCF

Ethernet/LANEthernet/LAN

• Servers connect to FEX 2232 over 10Gig FCoE� Server connections to the FEX can be Active/Standy or over a vPC

• FEX 2232 is single homed to upstream Nexus 5000� FEX 2232 can be connected with individual links or a port-channel

Nexus 5000FCF

vPCvPC

FEX-2232FEX-2232 FEX-2232FEX-2232

Ethernet/LAN CoreEthernet/LAN Core SAN ASAN A SAN BSAN B

FIP enabled CNAs

Unified FabricMultihop FCoE Deployment with Nexus 2232PP

� With NX-OS 5.0(2)N2, VE_Ports are supported on/between the Nexus 5000 and Nexus 5500 Series Switches

� VE_Ports are run between switches acting as Fibre Channel Forwarders (FCFs)

� VE_Ports are bound to the underlying 10G infrastructure� VE_Ports can be bound to a single 10GE port

� VE_Ports can be bound to a port-channel interface consisting of multiple 10GE links

FCoEFC

All above switches are Nexus 5X00 Series acting as an FCF

Unified FabricMultihop FCoE Deployment using VE ports

FCoEAdapters

Ethernet10G

Fibre ChannelEthernet

Fibre Channel Drivers

Ethernet Drivers

Operating System

Fibre Channel Drivers

Ethernet Drivers

Operating System

Fibre ChannelEthernet

10GbEE

CNA: Converged Network AdapterLAN CNAHBA

View from Operating System � Standard drivers� Same management

� Operating system independent interfaces

Qlogic 8042 Gen1 CNA

� PCI Express Gen1 x8� Dual port 10GbE

Passive copperOptical SR

� Multi-chip solution� Full height, full length� Power = 27W� QLogic 4Gb FC controller and drivers� Intel Ethernet controller and drivers� Windows, Linux, & Vmware (ESX 3.5U4 & 4.0) support

Qlogic 8100 Series Gen 2 CNA� PCI Express x8 slot� Single and dual port 10GbE

Active & passive copperOptical SR & LROperates with QLogic optics only

� Fully Integrated ASIC� Power ~7.4W (Dual Port with Optical SR)

No heat sink required� Low Profile Form Factor

FCOE vs. FC performance test

FCOE vs. FC test - results

Open-FCoE Software

SCSI Layer

HBA Driver

Linux Kernel

HBA Mgmt Plane

File System layers

SCSI Layer

FCoE Layer

Linux Kernel

Net Device

FCoE Mgmt Plane

File System layers

Ethernet

Ethernet Driver

OpenFC Layer

Ethernet

Server Server

More information

More Information� Standards Sites

�http://ieee.org�http://t11.org/fcoe�http://fcoe.com

� Case Studies�http://www.cisco.com/en/US/products/ps9670/prod_case_studies_list.html

� Book�I/O Consolidation in the Data Center- By Silvano Gai & Claudio DeSanti

Thank You

fcoe deep dive

Documents

deep dive xamarin.ios

deep sea fishing: a swordfish deep -dive · deep sea...

rails deep dive

nielsen deep dive

business deep dive

nehalem deep dive

wcf- basichttprelay deep dive - microsoft adapter deep dive...

wpf deep dive

deep dive + empower

deep dive - api.jasmineactive.com

powerapps deep dive

mts deep dive data center and csd deep dive optiq

hdfs deep dive

deep dive #1: reports - amazon s3discover more opportunities...

netwrix auditor deep dive deep€¦ · deep dive netwrix...

vsl deep dive

openstack deep dive

deep dive – coordination

starling deep dive

recoverpoint deep dive