cisco_san.pdf

Cisco Storage Design Fundamentals

Version 3.0

Student Guide

Table of Contents Course Introduction

Overview .................................................................................................................................................1 Recommended Prerequisites..................................................................................................................1 Course Outline ........................................................................................................................................2 Cisco Certifications .................................................................................................................................3 Administrative Information.......................................................................................................................5 About Firefly ............................................................................................................................................7

Lesson 1: SCSI and Fibre Channel Primer Overview .................................................................................................................................................9 SCSI Protocol Overview........................................................................................................................10 SCSI Operations ...................................................................................................................................17 Fibre Channel Overview........................................................................................................................22 Fibre Channel Flow Control ..................................................................................................................32 Fibre Channel Addressing.....................................................................................................................41 Fabric Login ..........................................................................................................................................45 Standard Fabric Services......................................................................................................................49

Lesson 2: Cisco MDS 9000 Introduction Overview ...............................................................................................................................................57 Cisco Storage Solutions Overview........................................................................................................58 Airflow and Power .................................................................................................................................65 Software Packages and Licensing........................................................................................................70

Lesson 3: Architecture and System Components Overview ...............................................................................................................................................77 System Architecture ..............................................................................................................................78 Oversubscription and Bandwidth Reservation ......................................................................................89 Credits and Buffers .............................................................................................................................101

Lesson 4: The Multilayer SAN Overview .............................................................................................................................................105 Virtual SANs........................................................................................................................................106 How VSANs Work ...............................................................................................................................111 Inter-VSAN Routing (IVR) ...................................................................................................................116 PortChannels ......................................................................................................................................124 Intelligent Addressing..........................................................................................................................136 Cisco Fabric ServicesUnifying the Fabric........................................................................................143 Switch Interoperability .........................................................................................................................145

ii Cisco Storage Design Fundamentals (CSDF) v3.0 Copyright 2006, Cisco Systems, Inc.

Lesson 5: Remote Lab Overview Overview .............................................................................................................................................151 System Memory Areas........................................................................................................................152 CLI Overview.......................................................................................................................................154 Fabric Manager and Device Manager ................................................................................................161 System Setup and Configuration ........................................................................................................164 Using the MDS 9000 Remote Storage Labs.......................................................................................166

Lesson 6: Network-Based Storage Applications Overview .............................................................................................................................................171 Storage Virtualization Overview..........................................................................................................172 Network-Based Storage Virtualization ................................................................................................177 Network-Hosted Applications..............................................................................................................181 Network-Assisted Applications............................................................................................................185 Network-Accelerated Applications ......................................................................................................190 Fibre Channel Write Acceleration .......................................................................................................193

Lesson 7: Optimizing Performance Overview .............................................................................................................................................199 Oversubscription and Blocking ...........................................................................................................200 Virtual Output Queues ........................................................................................................................203 Fibre Channel Congestion Control .....................................................................................................207 Quality of Service................................................................................................................................209 Port Tracking.......................................................................................................................................220 Load Balancing ...................................................................................................................................222 SAN Performance Management.........................................................................................................224

Lesson 8: Securing the SAN Fabric Overview .............................................................................................................................................237 SAN Security Issues ...........................................................................................................................238 Zoning .................................................................................................................................................244 Port and Fabric Binding ......................................................................................................................252 Authentication and Encryption ............................................................................................................256 Management Security .........................................................................................................................259 End-to-End Security Design................................................................................................................266

Lesson 9: Designing SAN Extension Solutions Overview .............................................................................................................................................269 SAN Extension Applications ...............................................................................................................270 SAN Extension Transports..................................................................................................................273 Extending SANs with WDM ................................................................................................................278 Fibre Channel over IP .........................................................................................................................287 Extending SANs with FCIP .................................................................................................................291 Cisco MDS 9000 IP Services Modules ...............................................................................................296 High Availability FCIP Configurations .................................................................................................302 Using IVR for SAN Extension .............................................................................................................306 SAN Extension Security......................................................................................................................309 FCIP Performance Enhancements .....................................................................................................311

Lesson 10: Building iSCSI Solutions Overview .............................................................................................................................................325 Whats the Problem?...........................................................................................................................326 iSCSI Overview...................................................................................................................................329 MDS 9000 IP Services Modules .........................................................................................................336 When to Deploy iSCSI ........................................................................................................................342 High-Availability iSCSI Configurations................................................................................................346 iSCSI Security .....................................................................................................................................353 iSCSI Target Discovery.......................................................................................................................360 Wide Area File Services......................................................................................................................362

CSDF

Cisco Storage Design Fundamentals

Overview CSDF is an intensive 2-day instructor-led training (ILT) lecture/lab course that provides learners with basic skills in designing Cisco storage networks. You will learn about and implement a broad range of features on the Cisco MDS 9000 platform, including Virtual SANs (VSANs), PortChannels, advanced security features, SAN extension with FCIP, and iSCSI solutions.

In the lab, you will configure the switch from an out-of-the-box state and install the Cisco Fabric Manager GUI management application. You will configure VSANs, zones, PortChannels, and FCIP to implement a high-availability extended SAN design.

This course provides an introduction to the MDS 9000 family for pre-sales engineers, system engineers, network engineers, and technical decision makers who need to design and implement SAN fabrics using MDS 9000 Family switches. Enrollment is open to Cisco SEs, Cisco channel partners, and customers.

Recommended Prerequisites You will gain the most from this course if you have experience working with storage and storage networking technologies.

2 Cisco Storage Design Fundamentals (CSDF) v3.0 Copyright 2006, Cisco Systems, Inc.

Course Outline This slide shows the lessons in this course.

222 2006 Cisco Systems, Inc. All rights reserved.

Course Overview

SCSI and Fibre Channel Primer Introduction to the MDS 9000 Platform Architecture and System Components The Multilayer SAN System Areas and Lab Overview Network-Based Storage Applications Optimizing Performance Securing the SAN Fabric Designing SAN Extension Solutions Building iSCSI Solutions

Copyright 2006, Cisco Systems, Inc. Cisco Storage Design Fundamentals 3

Cisco Certifications


Recommended Training Through Cisco Learning Partners

Required Exam

Prerequisite: Valid CCNA CertificationMDS Configuration and Troubleshooting (MDSCT)

Cisco Storage Networking Certification Path

Enhance Your Cisco Certifications and Validate Your Areas of Expertise

Cisco Storage Networking Specialists

Cisco Storage Networking Support Specialist

Cisco Storage Networking Design Specialist

642-354 Cisco Multiprotocol Storage Essentials (CMSE)

Cisco Advanced Storage Implementation and Troubleshooting (CASI)

Recommended Training Through Cisco Learning Partners

Required Exam

Prerequisite: Valid CCNA CertificationCisco MDS Storage Networking Fundamentals (CMSNF or CSDF)

642-353Cisco Storage Design Essentials (CSDE)

The Cisco Storage Networking Certification Program is part of the Cisco Career Certifications program. The title of Cisco Qualified Specialist (CQS) is awarded to individuals who have demonstrated significant competency in a specific technology, solution area, or job role, which is demonstrated through the successful completion of one or more proctored exams. The CQS Storage Networking program consists of two parallel tracks:

The Cisco Storage Networking Support Specialist (CSNSS) track is for systems engineers, network engineers, and field engineers who install, configure, and troubleshoot Cisco storage networks.

The Cisco Storage Networking Design Specialist (CSNDS) track is for pre-sales systems and network engineers who design Cisco storage networks. IT managers and project managers will also benefit from this certification.



Cisco CertificationsCCIE Storage Networking

Cisco provides three levels of general certifications for IT professionals with several different tracks to meet individual needs. Cisco also provides focused certifications for designated areas such as cable communications and security. There are many paths to Cisco certification, but only one requirementpassing one or more exams demonstrating knowledge and skill. For details, go to www.cisco.com/go/certifications.

CCIE certification in Storage Networking indicates expert level knowledge of intelligent storage solutions over extended network infrastructure using multiple transport options such as Fibre Channel, iSCSI, FCIP and FICON. Storage Networking extensions allow companies to improve disaster recovery, optimize performance and take advantage of network services such as volume management, data replication, and enhanced integration with blade servers and storage appliances.

There are no formal prerequisites for CCIE certification. Other professional certifications and/or specific training courses are not required. Instead, candidates are expected to have an in-depth understanding of the subtleties, intricacies and challenges of end-to-end storage area networking. You are strongly encouraged to have 3-5 years of job experience before attempting certification. To obtain your CCIE, you must first pass a written qualification exam and then a corresponding hands-on lab exam.


Administrative Information Please silence your cell phones.


Learner Introductions

Your name Your

company Skills and

knowledge Brief history Objective

Please introduce yourself.



Course Evaluations

www.fireflycom.net/evals

Please take time to complete the course evaluations after the class ends. Your feedback helps us continually improve the quality of our courses.


About Firefly


About Firefly

Technology FocusDatacenter IP and SecurityContent Networking and WAN OptimizationStorage NetworkingBusiness ContinuanceOptical Networking

ServicesGlobal DeliveryCurriculum DevelopmentState-of-the-Art Remote Labs and E-LearningNeeds AssessmentConsultative Education

Solutions FocusIntegrated Data Center SolutionsCore IP Services ProvisioningMultiprotocol SANsBusiness ContinuanceAll Application TiersApplication OptimizationApplication Security



Firefly MDS 9000 Training

Support TrackCisco Storage Networking Support Specialist (CSNSS)Systems Engineers, Technical Consultants, and Field Engineers

MDS Configuration and TroubleshootingExtended Edition (MDSCT + FCIP) Cisco Multiprotocol Storage Essentials (CMSE) Cisco Advance Storage Implementation and Troubleshooting (CASI)

Cisco Mainframe Storage Solutions (CMSS)

Design TrackCisco Storage Networking Design Specialist (CSNDS)Systems Engineers, Technical Consultants, Storage Architects and SAN Designers

Cisco Storage Design Fundamentals (CSDF) Cisco Storage Design Essentials (CSDE) Cisco Storage Design BootCamp (CSDF + CSDE)


Firefly CCIE-SAN Training

Firefly CCIE Storage KickStartDeveloped by FireflyIntensive 2-week training program designed for SAN Systems Engineers, Architects, and Support EngineersAlso prepares students for Cisco Storage Networking Support Specialist (CSNSS) certification examIncludes the contents of the MDSCT, CMSE, CASI and CMSS coursesusually 14 daysTaught only by senior professionals who have passed the CCIE Storage written exam

Firefly CCIE Storage Lab BootCampDeveloped by Firefly5-day intensive hands-on experience designed for students who have already passed the CCIE Written ExamTaught only by senior Firefly instructors who have achieved CCIE SAN

Lesson 1

SCSI and Fibre Channel Primer

Overview This lesson provides a brief overview of the SCSI and Fibre Channel protocols.

Objectives Upon completing this lesson, you will be able to explain the fundamentals of SCSI and Fibre Channel. This includes being able to meet these objectives:

Describe SCSI technology Describe the operations of the SCSI protocol Explain why FC is a data transport technology that is well-suited to storage networks Explain the fundamental design of FC flow control Describe the two addressing schemes used on Fibre Channel networks Describe the session establishment protocols that are performed by N_Ports and F_Ports in

a fabric topology

List the standard services provided by fabric switches as defined by the FC specification


SCSI Protocol Overview


SCSI Protocol Overview

The SCSI protocol defines how commands, status, and data blocks are exchanged between initiators and targets. SCSI is a Block I/O protocol. The Initiator always sends the command, reads or writes data blocks to the Target and receives a final Response.

Delivery Subsystem

[ Parallel SCSI or FCP or IP ]

LUNs

DeviceServer

ApplicationClient

Requests

Responses

Tasks

Initiator Target

SCSI Protocol Overview The Small Computer System Interface (SCSI) is a standard that evolved from a proprietary design by Shugart Associates in the 70s called the SASI bus.

SCSI performs the heavy lifting of passing commands, status, and block data between platforms and storage devices. One function of operating systems is to hide the complexity of the computing environment from the end user. Management of system resources including , memory, peripheral devices, display, context switching between concurrent applications, and so on, are generally concealed behind the user interface. The internal operations of the OS must be robust, closely monitor changes of state, ensure that transactions are completed within the allowable time frames, and automatically initiate recovery or retires in the event of incomplete or failed procedures. For I/O operations for peripheral devices such as disk, tape, optical storage, printers, and scanners, these functions are provided by the SCSI protocol, typically embedded in a device driver or logic onboard a host adapter.

Because the SCSI protocol layer sits between the operating system and the peripheral resources, it has different functional components. Applications typically access data as files or records. Although these may be ultimately stored on disk or tape media in the form of data blocks, retrieval of the file requires a hierarchy of functions to assemble raw data blocks into a coherent file that can be manipulated by an application.

SCSI architecture defines the relationship between initiators (hosts) and targets (for example disks or tape) as a client/server exchange. The SCSI-3 application client resides in the host and represents the upper layer application, file system, and operating system I/O requests. The SCSI-3 device server sits in the target device, responding to requests.

Copyright 2006, Cisco Systems, Inc. SCSI and Fibre Channel Primer 11


SCSI Parallel Technology

SCSI uses a parallel architecture in which data is sent simultaneously over multiple wires.

SCSI is half-duplex - data travels in one direction at a time. On a parallel SCSI bus, a device must assume exclusive control over the bus in order to communicate.

The SCSI Initiator then selects the SCSI Target and sends a Command to initiate a data transfer.

At the end of the transfer, the device is de-selected and the bus is free.

Parallel Half-duplex Shared bus Limited distance

SCSI Parallel Technology The bus/target/LUN triad is defined from parallel SCSI technology. The bus represents one of several potential SCSI interfaces installed in the host, each supporting a separate string of disks. The target represents a single disk controller on the string. And the LUN designation allows for additional disks governed by a controller for example, a RAID device.

The following are characteristics of parallel SCSI technology:

SCSI uses a parallel architecture in which data is sent simultaneously over multiple wires. SCSI is half-duplexdata travels in one direction at a time. On a SCSI bus, a device must assume exclusive control over the bus in order to

communicate. (SCSI is sometimes referred to as a simplex channel because only one device can transmit at a time).



Multidrop Topology and Addressing

Data / Address Bus

Clock / Control Signals

Interface Interface InterfaceTerm

inator

IDID

77IDID

66IDID

55IDID

00

LUN 0LUN 1

LUN 2LUN 3

LUN 0LUN 1

LUN 2LUN 3

LUN 0LUN 1

Address = BUS : Target ID : LUN

SCSI Targets

SCSI ID

Terminator

SCSI Initiator(I/O Adapter)

A SCSI Initiator addresses its SCSI Target using the SCSI Nexus Bus : Target ID : LUN

FC

HBA

Multidrop Topology and Addressing All of the devices on a SCSI bus are connected to a single cable. This is called a multidrop topology:

Data bits are sent in parallel on separate wires. Control signals are sent on a separate set of wires.

Only one device at a time can transmita transmitting device has exclusive use of the bus. A special circuit called a terminator must be installed at the end of the cable. The cable

must be terminated to prevent unwanted electrical effects from corrupting the signal.

A multidrop topology has inherent limitations:

Parallel transmission of data bits allows more data to be sent in a given time period but data bits may arrive early or late (skew) and lead to data errors.

The fact that control signals, such as clock signals, are sent on a separate set of wires also makes synchronization more difficult.

It is an inefficient way to use the available bandwidth, because only one communication session can exist at a time.

Termination circuits are built into most SCSI devices, but the administrator often has to set a jumper on the device to enable termination.

Incorrect cable termination can cause either a severe failure or intermittent, difficult-to-trace errors.

To achieve faster data transfer rates, vendors doubled the number of data lines on the cable from 8 (narrow SCSI) to 16 (wide SCSI).

Vendors have increased the clock rate, which increased the transfer rates, but this also increased the possibility of data errors due to skew or electrical interference.


Parallel SCSI is limited to a maximum cable length of 25m.

SCSI was designed to support a few devices at most, so its device addressing scheme is fairly simpleand not very flexible. SCSI devices use hard addressing:

Each device has a series of jumpers that determine the devices physical address, or SCSI ID. The ID is software-configurable on some devices.

Each device must have a unique SCSI ID. Before adding a device to the cable, the administrator must know the ID of every other device connected to the cable and choose a unique ID for this new device.

The ID of each device determines its priority on the bus. For example, the SCSI Initiator with ID 7 always has a higher priority than the SCSI Target with ID 6. Because each device must have exclusive use of the bus while it is transmitting, ID 6 must wait until ID 7 has finished transmitting. Fixed priority makes it more difficult for administrators to control performance and quality-of-service.



SCSI-3 Architecture Model

SCSI-FCPPort DriverSCSI-FCPPort Driver

SCSI ParallelPort Driver

SCSI ParallelPort Driver

SASPort Driver

SASPort Driver

iSCSI IPPort DriveriSCSI IP

Port DriverSBP-2

Port DriverSBP-2

Port Driver

Fibre ChannelPort

Fibre ChannelPort

SCSI Parallel Port

SCSI Parallel Port

SAS Serial Port

SAS Serial Port

EthernetPort

EthernetPort

IEEE-1394(Firewire)

Port

IEEE-1394(Firewire)

Port

FC

FC HBASCSI Adapter NIC

*SCSI-3 Separation of physical interface, transport protocols, and SCSI Command Set

SCSI Block Commands

SCSI Streaming Commands

SCSI Enclosure Services

SCSI Media Commands

SCSI Primary Commands

SAS Interface

The SCSI-3 Architecture Model The SCSI-3 family of standards introduced several new variations of SCSI commands and a protocol, including serial SCSI-3 and special command sets for streaming and media handling required for tape. As shown in the diagram, the command layer is independent of the protocol layer, which is required to carry SCSI-3 commands between devices. This enables more flexibility in substituting different transports beneath the SCSI-3 command interface to the operating system.

The SCSI Architecture Model (SAM) consists of four layers of functionality:

1. The physical interconnect layer specifies the characteristics of the physical SCSI link:

FC-PH is the physical interconnect specification for Fibre Channel.

Serial Storage Architecture (SSA) is a storage bus aimed primarily at the server market.

IEEE1394 is the FireWire specification.

SCSI Parallel Interface (SPI) is the specification used for parallel SCSI buses.

2. The transport protocol layer defines the protocols used for session management:

SCSI-FCP is the transport protocol specification for Fibre Channel.

Serial Storage Protocol (SSP) is the transport protocol used by SSA devices.

Serial Bus Protocol (SBP) is the transport protocol used by IEEE1394 devices.


3. The shared command set layer consists of command sets for accessing storage resources:

SCSI Primary Commands (SPC) are common to all SCSI devices.

SCSI Block Commands (SBC) are used with block-oriented devices, such as disks.

SCSI Stream Commands (SSC) are used with stream-oriented devices, such as tapes.

SCSI Media Changer Commands (SMC) are used to implement media changers, such as robotic tape libraries and CD-ROM carousels.

SCSI Enclosure Services (SES) defines commands used to monitor and manage SCSI device enclosures like RAID Arrays, including fans, power and temperature monitoring.

4. The SCSI Common Access Method (CAM) defines the SCSI device driver application programming interface (API).



Serial SCSI-3 over Fibre ChannelSCSI-FCP

Interface Interface Interface

LUN 0

LUN 1LUN 2

LUN 3

LUN 0LUN 1

LUN 2LUN 3

LUN 0LUN 1

SCSI TargetspWWN

SCSI Initiator

(FC HBA)

SCSI Commands, Data and Responses are carried in the payload of a frame from source to destination.

In SCSI-FCP, the SCSI IDs are mapped to the unique worldwide name in each FC Port.

Fibre ChannelFabricI

nter

face

pWWN

Payload

FC Link

FC Frame

FC

HBA

Serial SCSI-3 over Fibre ChannelSCSI-FCP All of the devices are attached to the same fabric and connected via Fibre Channel links to one or more interconnected switches.

The SCSI Initiator and SCSI Target ports are zoned together within the Fibre Channel switch.

Each device logs in to the fabric (FLOGI) and registers itself with the Name Server in the switch.

The FC-HBA queries the Name Server and discovers other FC ports in the same zone as itself.

The FC HBA then logs in to each Target port (PLOGI) and they exchange Fibre Channel parameters.

The SCSI Initiator (SCSI-FCP driver) then logs in to the SCSI Target behind the FC Target port (PRLI) and establishes a communication channel between SCSI Initiator and SCSI Target.

The SCSI Initiator commences a SCSI operation by sending a SCSI Command Descriptor Block (CDB) down to the FC HBA with instructions to send it to a specific LUN behind a Target FC port (SCSI Target). The command is carried in the payload of the FC Frame to the target FC port.

The SCSI Target receives the CDB and acts upon it. Usually this would be a Read or Write command. Data is then carried in the payload of the FC Frame between SCSI Initiator and SCSI Target.

Finally, when the operation is complete, the SCSI Target will send a Response back to the SCSI Initiator in the payload of a FC Frame.


SCSI Operations


SCSI Operation

SCSI specifies three phases of operation Command send the required command and parameters

via a Command Descriptor Block (CDB)

Data Transfer data in accordance with the command Response Receive confirmation of command execution

WriteCommand

Data

Response

Xfer-Rdy

ReadCommand

Data

ResponseTargetInitiator

FC

HBA

FC

Phases of Operation Every communication between SCSI Initiator and SCSI Target is formed by sequences of events called bus phases. Each phase has a purpose and is linked to other phases to execute SCSI commands and transfer data and messages back and forth.

The majority of the SCSI protocol is controlled by the SCSI Initiator. The SCSI Target is usually passive and waits for a command.

Only the SCSI initiator can initiate a SCSI operation, by selecting a SCSI Target and sending a CDB (Command Descriptor Block) to it.

If the CDB contains a Read command, the SCSI Target moves its heads into position and retrieves the data from its disk sectors. This data is returned to the SCSI Initiator.

If the CDB contains a Write Command, the SCSI Target prepares its buffers and returns a Xfer-Rdy. When the SCSI Initiator receives Xfer-Rdy it can commence writing data.

Finally, when the operation is complete, the SCSI Target returns a Response to indicate a successful (or unsuccessful) data transfer.



SCSI Command Descriptor Blocks

A Command is executed by the Initiator sending a CDB to a Target.

In serial SCSI-3, the CDB is carried in the payload of the Command Frame

First Byte Operation CodeFirst Byte Operation Code

Transfer Data starting at this LBATransfer Data starting at this LBA

Number of SCSI Blocks to be transferredNumber of SCSI Blocks to be transferred

Last Byte Control ByteLast Byte Control Byte

Byte 7 6 5 4 3 2 1 0Command Code0

123456789

Reserved

Transfer LengthControl

Service ActionLogical Block AddressLogical Block AddressLogical Block AddressLogical Block Address

ReservedTransfer Length

LSB

LSB

MSB

MSB

Group Code

SCSI Command Descriptor Block (CDB)

SCSI Command Descriptor Block (CDB)

SCSI Command Descriptor Blocks SCSI commands are built from a common structure:

Operation Code Byte N: Bytes of parameters Control Byte

The Operation Code consists of a Group Code and a command Code

Group Code establishes the total command length. Command Code establishes the command function.

The number of Bytes of parameters (N) can be determined from the Operation Code byte which is located in byte 0 of the Command Descriptor Block (CDB).

The Control Byte, which is located in the last byte of a Command Descriptor Block, contains control bits that define the behavior of the command.

The Logical Block Address is an absolute address of where the first block should be written (or read) on the disk.

LBA 00 is the first sector on the disk volume or LUN, LBA 01 is the second sector and so on, until we reach the last sector of the disk volume or LUN.

When the CDB is sent to a block device (Disk), blocks are always 512 Bytes long. The Transfer Length contains the number of 512 Byte blocks to be transferred.

When the CDB is sent to a streaming device (Tape), the block length is negotiated. The Transfer Length contains the number of blocks to be transferred.


CDBs can be different sizes 6Byte CDB, 10 Byte CDB, 12 Byte CDB, 16 Byte CDB etc. to accommodate larger disk volumes, or transfer lengths. 10 Byte CDBs are common



SCSI Commands

SCSI supports several specific commands for eachmedia type, and primary commands that all devicesunderstand.

The following commands are of particular interest:- REPORT LUNS How many LUNs do you have?

- INQUIRY What device are you?

- TEST UNIT READY Is the LUN available?

- REPORT CAPACITY What size is each LUN?

SCSI Commands Report LUNs are used by Operating Systems to discover LUNs attached to a particular hardware address. They are typically sent by the Initiator to LUN 0.

Inquiries are used by the Operating System to determine the capabilities of each LUN that was discovered with REPORT LUNS.

Test Unit Ready is used to check the condition of a particular LUN.

Report Capacity is sent to each LUN in turn to obtain the size of each LUN.



Building an I/O Request1. Application makes File I/O request to Volume Manager2. Volume Manager maps volume to SCSI ID & Target LUN3. File System maps Files to Blocks, makes Block I/O request4. Command, LBA, Block count and LUN sent to SCSI driver5. SCSI driver creates CDB (Command Descriptor Block)6. FC driver creates command frame with CDB in payload7. FC driver sends command frame to Target LUN and awaits

response

WriteCommand

Data

Response

Xfer-Rdy

ReadCommand

Data

ResponseEOFPayload CRCSOF Header

Command CodeReserved

Transfer LengthControl

Service ActionLogical Block AddressLogical Block AddressLogical Block AddressLogical Block Address

ReservedTransfer Length

LSB

LSB

MSB

MSB

Group Code

Volume Mgr

File System

SCSI Driver

FC Driver

Application

CDB

FC Frame

Server

Building an I/O Request This slide explains the process of the Initiator talking to the Target

1. Application makes File I/O request to Volume Manager

2. Volume Manager maps volume to SCSI ID & Target LUN

3. File System maps Files to Blocks, makes Block I/O request

4. Command, LBA, Block count and LUN sent to SCSI driver

5. SCSI driver creates CDB (Command Descriptor Block)

6. FC driver creates command frame with CDB in payload

7. FC driver sends command frame to Target LUN and awaits response


Fibre Channel Overview


Fibre Channel Overview

Fibre Channel is a protocol used for efficiently transporting data between devices connected to the same fabric.

Fibre Channel provides reliable and efficient data delivery with high throughput and low latency.

Fibre Channel is the transport technology most commonly used for SANs today.

IPNetwork

Fibre ChannelFabric

FC

HBA

FC

HBA

FC

FCFC

HBA

FC

Fibre Channel Overview FC is a protocol used for efficiently transporting data between devices in the same fabric. It is the network interconnect technology that is most commonly used for SANs today.

Traditional storage technologies, such as SCSI, are designed for controlled, local environments. They support few devices and only short distances, but they deliver data quickly and reliably. Traditional data network technologies, such as Ethernet, are designed for chaotic, distributed environments. They support many devices and long distances, but delivery of data can be delayed (Latency)

FC combines the best of both worlds. It supports many devices and longer distances, and it provides reliable and efficient data delivery with high throughput and low latency.

Like SCSI, Fibre Channel is a Block I/O protocol, delivering data blocks (usually 512 Bytes long) between devices in the same Fabric.

In the diagram, the network on the right, consisting of servers and storage devices, is an FC SAN. The SAN consists of servers and storage devices connected by an FC network.



Fibre Channel Topologies

Arbitrated Loop provides shared bandwidth at low cost

Switched Fabric provides aggregate bandwidth and scalability but requires complex FC switches, which increase the cost

Most SANs today use the Switched Fabric topology

Point-to-Point

FC

FCHBA

Arbitrated Loop

FC

FC

FC

FCHBA

FCHBA

FCHBA

Switched Fabric

FCHBA

FCHBA

FCHBA

FCHBA

FC

FC

FC

FCFC Fabric

Fibre Channel Topologies Fibre Channel Protocol includes three basic SAN topologies.

Point-to-Point Exactly two FC ports connected together. Both devices have exclusive access to the full

link bandwidth

Arbitrated Loop Up to 126 FC ports connected together on a Private Loop (not connected to a FC Switch) Up to 127 FC ports connected together on a Public Loop (connected via a FL port on a FC

Switch)

All devices share the available bandwidth around the loop, therefore a practical limit might only be 20 or so devices.

A device that wishes to communicate with another device must do the following operations.

1. Arbitrate to gain control of the Loop

2. Open the port it wishes to communicate with

3. Send or Receive Data frames

4. Close the port

5. Release the loop, ready for the next transfer.

Usually only two devices communicate at a time, the other FC ports in the loop are passive


When the loop is broken or a device is added or removed, the downstream FC port sends thousands of LIP primitive sequences to inform the other loop devices that the loop has been broken.

The LIP (Loop Initialization Procedure) is used to assign (or re-assign) Arbitrated Loop Physical Addresses (ALPAs) to each FC Port on the loop. This operation is disruptive and frames may be lost during this phase. Nowadays, most users would connected FCAL devices via a FC Hub to minimize disruption.

Switched Fabric The topology of choice for FC SANs. Each connected device has access to full bandwidth

on its link through the switch port it is connected to.

The FC SAN can be expanded by adding more switches and increasing the number of ports for connected devices.

The FC 24 bit addressing scheme allows for potentially 16,500,000 devices to be connected. A realistic number is a few thousand. This is because there can only be a maximum 239 switches in a single fabric and most switches today have a small number of ports each.

Each FC switch must provide services for management of the SAN. These services include a Name Server, Domain Manager, FSPF Topology Database, Zoning Server, Time Server etc.



Fibre Channel Switched Fabric Topology

A Fabric contains one or more switches, connected together via Inter Switch Links (ISLs)

FC Fabric

FCHBA

FCFC

HBAFCHBA

FC

FC

ISL

FC

FCHBA

FCHBA

FCHBA

FC

FCFC

FC

FCHBA

FCHBA

FCHBA

FC

FCFC

Fibre Channel Switched Fabric Topology The Switched Fabric topology incorporates one or more high-bandwidth FC switches, to handle data traffic among host and storage devices.

Each switch is assigned a unique ID called a Domain. There can be a maximum 239 switch domains in a fabric, however, McData impose a 32 Domain limit in their designs.

FC Switches are connected together via Inter-Switch Links (ISLs). Each device is exclusively connected to its FC port on the switch via a bi-directional Full

Duplex link.

All connected devices share the same addressing space within the fabric and can potentially communicate with each other.

Frames flow from device to device via one or more FC Switches. As a frame moves from switch to switch, this is called a hop. McData impose a 3 hop limit in their designs. Brocade impose a 7 hop limit and Cisco impose a 10 hop limit.



FC

HBA

FC

FC

Fibre Channel Ports

Ports are intelligent interface points on the Fibre Channel network: Embedded in a FC Host Bus Adapter (HBA) Embedded in a fabric switch Embedded in a storage array or tape controller A Link connects exactly two Ports together FC Ports are assigned a dynamic FCID

FC

Ports

ServerStorage

SwitchFC HBA Arraycontroller

Tape deviceFC

FC

Link

Node

Node

Fibre Channel Ports In data networking terminology, ports are often thought of as just physical interfaces where you plug in the cable. In FC however, ports are intelligent interfaces, responsible for actively performing critical network functions.

The preceding graphic contains several ports. There are ports in the host I/O adapter (host bus adapter [HBA]), ports in the switch, and ports in the storage devices.

FC terminology differentiates between several different types of ports, each of which performs a specific role on the SAN. You will encounter these terms often as you continue to learn about FC, so it is important that you learn to recognize the different port types. In addition to the common ports defined for FC, Cisco has developed some proprietary port types.

Fibre Channel Ports are assigned a unique address, a Fibre Channel ID (FCID) at login time.



Standard Fibre Channel Ports

FCAL Hub

StorageArray

StorageArray

Host

Host

WAN Bridge

Link Link

Inter Switch Link

Valid combinations NL NL NL FL

N N N F

E E E B

NL NL

FL

E

E

E

B

NN F F

FC

FC

FCHBA

FCHBA

Standard Fibre Channel Ports An N_Port (Node Port) is a port on a node that connects to a fabric. I/O adapters and array

controllers contain one or more N_Ports. N_Ports can also directly connect two nodes in a point-to-point topology

An F_Port (Fabric Port) is a port on a switch that connects to an N_Port. An E_Port (Extension Port) is a port on a switch that connects to an E_Port on another

switch.

An FL_Port (Fabric Loop Port) is a port on a switch that connects to an arbitrated loop. Logically, an FL_Port is considered part of both the fabric and the loop. FL_Ports are always physically located on the switch.

Note that FC hubs, although they obviously have physical interfaces, do not contain FC ports. Hubs are basically just passive signal splitters and amplifiers. They do not actively participate in the operation of the network. On an arbitrated loop, the node ports manage all FC operations. Not all switches support FL_Port operation. For example, some McDATA switches do not support FL_Port operation.

An NL_Port (Node Loop Port) is a port on a node that connects to another port in an arbitrated loop topology. There are two types of NL_Ports: Private NL_Ports can communicate only with other loop ports; public NL_Ports can communicate with other loop ports and with N_Ports on an attached fabric.

Note that the term L_Port (Loop Port) is sometimes used to refer to any port on an arbitrated loop topology. L_Port can mean either FL_Port or NL_Port. In reality, there is no such thing as an L_Port.


Nowadays, most ports are universal. They automatically sense the port they are connected to and adopt the correct valid port type automatically. However, it is good practice to lock down the port type to its correct function.



Fibre Channel Frame

02112 Byte Data FieldCRC EOFSOF Header

OptionalHeaders 02048 Byte Payload

Fill Bytes

4 24 02048 0-3 4 4 Bytes0-64

4 Bytes wide x 6 Words = 24 Bytes

R_CTL

CS_CTL

TYPE

SEQ_ID

OX_ID

Parameter Field

Destination Address D_ID

Source Address S_ID

Frame Control F_CTL

DF_CTL SEQ_CNT

RX_ID

D_ID = Where the frame is going to

S_ID = Where the frame is coming from

TYPE = Payload Protocol

SEQ_CNT and SEQ_ID = Sequence IDs

OX_ID and RX_ID = Exchange IDs

Fibre Channel Frames The maximum total length of an FC frame is 2148 bytes, or 537 words. ( A Word = 4 Bytes).

A 4-byte SOF (Start of Frame) delimiter A 24-byte header A data payload that can vary from 0 to 2112 bytes. Typically 2048 Bytes for SCSI-FCP. A 4-byte CRC (Cyclic Redundancy Check) that is used to detect bit-level errors in the

header or payload

A 4-byte EOF (End of Frame) delimiter

The Header contains fields used for identifying and routing the frame across the fabric.

R_CTL: Routing Control field defines the frames function. D_ID: Destination Address. The FCID of the FC Port where the frame is being sent to. CS_CTL: Class Specific Control field. Only used for Class 1 and 4. S_ID: Source Address. The FCID of the FC Port where the frame has come from. TYPE: The Upper Layer Protocol Data type contained in the payload. This is hex 08 for

SCSI-FCP.

F_CTL: Frame Control field contains miscellaneous control information regarding the frame, including how many fill bytes there are (0-3).

SEQ_ID: Sequence ID. The unique identifying number of the Sequence within the Exchange.

DF_CTL: Data Field Control. This field defines the use of the Optional Headers. SCSI-FCP doesnt use Optional Headers.


SEQ_CNT: Sequence Count. The number of the frame within a sequence. The first frame is hex 00

OX_ID: Originating Exchange ID. A Unique identifying number provided by the source FC Port.

RX_ID: Responding Exchange ID. A Unique identifying number provided by the destination FC Port. OX_ID and RX_ID together define the Exchange ID.

PARMS: Parameter Field. Usually provides a relative offset into the ULP data buffer.

The frame payload consists of 3 elements:

The payload itself, containing data or commands, is variable and can be up to 2112 bytes. The first 64 bytes of the payload can be used to incorporate optional headers. This would

reduce the data payload size to 2048 bytes (2KB). SCSI-FCP usually carries multiples of 512 Byte blocks.

The payload ends with 0-3 fill bytes. This is necessary because the smallest unit of data recognized by FC is a 4-byte word. However, the ULP is not aware of this FC requirement, and the data payload for a frame might not end on a word boundary. FC therefore adds up to 3 fill bytes to the end of the payloadas many as are needed to ensure that the payload ends on a word boundary.



Fibre Channel Data Constructs

Sequence

Word

Frame

Exchange

Initiator Target

Words are 4 Bytes long. The smallest unit of transfer in Fibre Channel.Frames are made up from several Words. Sequences are uni-directional and contain one or more framesExchanges are bi-directional and contain three or more sequences

FC

HBA

FC

ReadCommand

Data

ResponseExchange

Sequence 1 Sequence 2

Sequence 3

Fibre Channel Data Constructs The preceding graphic shows a transaction between a host (Initiator) and a storage device (Target):

The smallest unit of data is a word. Words consist of 32 bits (4 bytes) of data that are encoded into a 40-bit form by the 8b/10b encoding process.

Words are packaged into frames. An FC frame is equivalent to an IP packet. A sequence is a series of frames sent from one node to another node. Sequences are

unidirectionalin other words, a sequence is a set of frames that are issued by one node.

An exchange is a series of sequences sent between tow nodes. The exchange is the mechanism used by two ports to identify and manage a discrete transaction. The exchange defines an entire transaction, such as a SCSI read or write request. An exchange is opened whenever a transaction is started between two ports and is closed when the transaction ends. An FC exchange is equivalent to a TCP session.


Fibre Channel Flow Control


PAUSE

Fibre Channel Flow Control

How data interchange is controlled in a network:

The flow control strategy used by Ethernet and other data networks can degrade performance:

Transmitter does not stop transmitting packets until after the receivers buffers overflow and packets are already lost

Lost packets must be retransmitted

Degradation can be severe under heavy traffic loads

Rx

Flow Control in Ethernet

DataDataData DataData Data

Lost packets

Tx

Flow control is a mechanism for ensuring that frames are sent only when there is somewhere for them to go. Just as traffic lights are used to control the flow of traffic in cities, flow control manages the data flow in an FC fabric.

Some data networks, such as Ethernet, use a flow-control strategy that can result in degraded performance:

A transmitting port (Tx) can begin sending data packets at any time. When the receiving ports (Rx) buffers are completely filled and cannot accept any more

packets, Rx tells Tx to stop or slow the flow of data.

After Rx has processed some data and has some buffers available to accept more packets, it tells Tx to resume sending data.

This strategy results in lost packets when the receiving port is overloaded, because the receiving port tells the transmitting port to stop sending data after it has already overflowed. Lost packets must be retransmitted, which degrades performance. Performance degradation can become severe under heavy traffic loads.



DATA

READYREADY

What is Credit-Based Flow Control?

Fibre Channel uses a credit-based strategy:When receiver is ready to accept a frame it sends a credit to the transmitter, giving permission for the transmitter to send a frame.

The receiver is always in control.

Benefits:Prevents loss of frames due to buffer overflow

Maximizes link throughput and performance under high loads

Disadvantages:Long distance links require lots more credits

Tx Rx

Port Rx has0101

free buffers

Flow Control in Fibre Channel

What is Credit-Based Flow Control? To improve performance under high traffic loads, FC uses a credit-based flow control strategy in which the receiver must issue a credit for each frame that is sent by the transmitter before that frame can be sent.

A credit-based strategy ensures that the receiving port is always in control. The receiving port must issue a credit for each frame that is sent by the transmitter. This strategy prevents frames from being lost when the receiving port runs out of free buffers. Preventing lost frames maximizes performance under high traffic load conditions because the transmitting port does not have to resend frames.

The preceding diagram illustrates a credit-based flow control process:

The transmitting port (Tx) counts the number of free buffers at the receiving port (Rx). Before Tx can send a frame, Rx must notify Tx that Rx has a free buffer and is ready to

accept a frame. When Tx receives the notification (called a credit), it increments its count of the number of free buffers at Rx.

Tx only sends frames when it knows that Rx can accept them. When Tx sends a frame it decrements the credit count When the credit count falls to zero, Tx must stop sending frames and wait for another credit



Types of Flow Control

Fibre Channel defines two types of flow control:Buffer-to-buffer R_RDY Port-to-Port across every Link

End-to-end ACK Between Source and Destination Ports

Buffer-to-buffer flow control

N_Port F_Port E_Port E_Port F_Port N_Port

End-to-end flow controlSource Destination

Types of Flow Control FC defines two types of flow control:

Buffer-to-buffer flow control takes place between two ports that are connected by a FC link, such as an N_Port and an F_Port, or two E_Ports, or two NL_Ports.

The receiving port at the other end of the link sends a primitive signal (4 Bytes) called a R_RDY (Receiver Ready) to the transmitting port.

End-to-end flow control takes place between the source port and the destination port. Whenever the receiving port receives a frame it acknowledges that frame with an ACK frame (36 Bytes).

Note that buffer-to-buffer flow control is performed between E_Ports in the fabric, but it is not performed between the incoming and outgoing ports in a given switch. In other words, FC buffer-to-buffer flow control is not used between two F_Ports or between an F_Port and an E_Port within a switch. FC standards do not define how switches route frames across the switch.

Buffer-to-buffer flow control is used in the following situations:

Class 1 connection request frames use buffer-to-buffer flow control, but Class 1 data traffic uses only end-to-end flow control.

Class 2 and Class 3 frames always use buffer-to-buffer flow control. Class F service uses buffer-to-buffer flow control. In an Arbitrated Loop, every communication session is a virtual dedicated point-to-point

circuit between a source port and destination port. Therefore, there is little difference between buffer-to-buffer and end-to-end flow control. Buffer-to-buffer flow control alone is generally sufficient for arbitrated loop topologies.


End-to-end flow control is used in the following situations:

Classes 1, 2, 4, and 6 use end-to-end flow control. Class 2 service uses both buffer-to-buffer and end-to-end flow control.



R_RDY

Buffer to Buffer and End-to-End Flow Control

End-to-end flow control

N_PortA

N_PortB

R_RDY

Fabric

F_Port F_Port

Buffer-to-bufferflow control

Buffer-to-bufferflow control

ACK

R_RDYData

R_RDY

12

34

5

Buffer-to-Buffer Flow Control and End to End Flow Control The preceding preceding diagram illustrates buffer-to-buffer flow control in Class 3:

1. Before N_Port A can transmit a frame, it must receive the primitive signal R_RDY from its attached F_Port. The R_RDY signal tells N_Port A that its F_Port has a free buffer.

2. When it receives the R_RDY signal, N_Port A transmits a frame.

3. The frame is passed through the fabric. Buffer-to-buffer flow control is performed between every pair of E_Ports, although this is not shown here.

4. At the other side of the fabric, the destination F_Port must wait for an R_RDY signal from N_Port B.

5. When N_Port B sends an R_RDY, the F_Port transmits the data frame.

End-to-end flow control is designed to overcome the limitations of buffer-to-buffer flow control. The preceding preceding diagram illustrates end-to-end flow control in Class 2:

1. Standard buffer-to-buffer flow control is performed for each data frame.

2. After the destination N_Port B receives a frame, it waits for an R_RDY from the F_Port.

3. When N_Port B receives an R_RDY, it sends an acknowledgement (ACK) frame back to N_Port A.

4. At the other side of the fabric, the initiator F_Port must wait for an R_RDY signal from N_Port A.

5. When N_Port A sends an R_RDY, the F_Port transmits the ACK frame.


End-to-end flow control involves only the port at which a frame originates and the ultimate destination port, regardless of how many FC switches are in the data path.

When end-to-end flow control is used, the transmitting port is responsible for ensuring that all frames are delivered. Only when the transmitting N_Port receives the last ACK frame in response to a sequence of frames sent does it know that all frames have been delivered correctly, and only then will it empty its ULP data buffers. If a returning ACK indicates that the receiving port has detected an error, the transmitting N_Port has access to the ULP data buffers and can resend all of the frames in the sequence.



Allocating Buffer Credits

Frame serialization time:2Gb Link rate of 2.125 Gbps = 4.7ns/byte

Frame size = 2048 data + 36 header + 24 IDLE = 2108 bytes

Frame serialization time = 2108 x 4.7 = 9.9s 10s per frame

Initiator N_Port

10Km

Target N_Port

Frame

10s

(Round_Trip_Time + Processing_Time)Serialization_Time

Credits =

Allocating Buffer Credits You can calculate the number of credits required on a link to maintain optimal performance using the following formula:

Credits = (Round_Trip_Time + Processing_Time) / Serialization_Time

Example This diagram and the following two diagrams illustrate how the required number of BB_Credits are calculated for a 10km, 2Gbps FC link:

At a link rate of 2.125 Gbps, the time required to serialize (transmit) each byte is 4.7ns. (Note that each byte is 10 bits due to 8b/10b encoding.)

The maximum SCSI-FCP Fibre Channel payload size is 2048 bytes, because SCSI usually transfers multiple SCSI blocks of 512 Bytes each.

The payload size used in an actual customer environment would be based on the I/O characteristics of the customers applications.

You also need to account for the frame overheads. These are: SOF (Start of Frame) 4 Bytes FC Header, which is 24 Bytes CRC which is 4 Bytes EOF (End of Frame) which is also 4 Bytes Also, the number of IDLEs between frames, which is usually 6 IDLEs, or 24 bytes. This

gives a total of 2108 bytes.

The total serialization time at 2Gbps for a 2108-byte frame (including idles) is 9.9s, or approximately 10s.



Frame

20s

Allocating Buffer Credits

Propagation delay:Speed of light in fiber 5s/KmTime to transmit frame across 10Km 50s

Processing time:Assume same as de-serialization time 10s

Response time:Time to transmit R_RDY across 10Km 50s

Total latency 50s + 10s + 50s = 110s

Initiator N_Port

10Km

Target N_Port

R_RDY 50s

Frame

10s50s

The speed of light in a fiber optic cable is approximately 5ns per metre or 5s per kilometer, so each frame will take about 50s to travel across the 10Km link.

The receiving port must then process the frame, free a buffer, and generate an R_RDY. This processing time can varyfor example, if the receiver ULP driver is busy, the frame

might not be processed immediately.

In this case, we can assume that the receiving port will process the frame immediately, so the processing time is equal to the time it takes to de-serialize the frame.

Assume that the de-serialization time is equal to the serialization time: 10s The receiving port then transmits a credit (R_RDY) back across the link. This response

takes another 50s to reach the transmitter.

The total latency on the link is equal to the frame serialization time plus the round-trip time across the link, or about 110s.



Frame

Allocating Buffer Credits (cont.)

Given frame serialization time 10s and total latency 110s, There could be up to 5 frames on the link at one time, 1 being processed,

and 5 credits being returned.

Therefore, at 2Gbps we require approx 10 credits to maintain full bandwidth

A good rule of thumb is..

At 2Gbps with a 2KB payload, you need approx 1 credit per Km

Initiator N_Port

10Km

Target N_Port

R_RDY R_RDY R_RDY R_RDY R_RDY

Frame Frame Frame Frame Frame

Given a frame serialization time of 10s, and a total round-trip latency of 110s, there could be up to 5 frames on the link at one time plus one being received and processed by the receiving port. In addition, 5 credits are being returned to the transmitting port down the other side of the link.

In other words, ignoring the de-serialization time, approximately 10 buffer-to-buffer credits are required to make full use of the bandwidth of the 10km link at 2Gbps with 2KB frames.


Fibre Channel Addressing


World-Wide Names

Every Fibre Channel port and node has a hard-coded address called a World Wide Name (WWN):

Allocated to manufacturer by IEEECoded into each device when manufactured64 or 128 bits (128 bits most common today)

Switch Name Server maps WWNs to FC addresses (FCID)World-Wide Node Names (nWWNs) uniquely identify devices (Nodes)World-Wide Port Names (pWWNs) uniquely identify each port in a device

200000456801EF25WWNNExample WWNExample WWN

20:00:00:45:68:01:EF:25WWN 21:00:00:45:68:01:EF:25pWWN A22:00:00:45:68:01:EF:25pWWN B

20:00:00:45:68:01:EF:25nWWNExample WWNs from a Dual-Ported DeviceExample WWNs from a Dual-Ported Device

World-Wide Names WWNs are unique identifiers that are hard-coded into FC devices. Every FC port has at least one WWN. Vendors buy blocks of WWNs from the IEEE and allocate them to devices in the factory.

WWNs are important for enabling fabric services because they are:

Guaranteed to be globally unique Permanently associated with devices

These characteristics ensure that the fabric can reliably identify and locate devices, which is an important consideration for fabric services. When a management service or application needs to quickly locate a specific device:

1. The service or application queries the switch Name Server service with the WWN of the target device

2. The Name Server looks up and returns the current port address (FCID) that is associated with the target WWN

3. The service or application communicates with the target device using the port address


There are two types of WWNs:

nWWNs uniquely identify devices (Nodes). Every host bus adaptor (HBA), array controller, switch, gateway, and FC disk drive has a single unique WWNN.

pWWNs uniquely identify each port in a device. A dual-ported HBA has three WWNs: one nWWN and a pWWN for each port.

nWWNs and pWWNs are both needed because devices can have multiple ports. On single-ported devices, the nWWN and pWWN are usually the same. On multi-ported devices, however, the pWWN is used to uniquely identify each port. Ports must be uniquely identifiable because each port participates in a unique data path. nWWNs are required because the node itself must sometimes be uniquely identified. For example, path failover and multiplexing software can detect redundant paths to a device by observing that the same WWNN is associated with multiple pWWNs.

Cisco MDS switches use the following acronyms:

pWWN (Port WWN) nWWN (Node WWN)



Dynamic FCID Addressing

00000000 00000000 AL_PAPrivate Loop

AL_PADomainDomain AreaAreaPublic Loop

DomainDomain AreaArea PortFabric000708151623Bit

FCIDs are dynamically assigned to each device FC Port by the switch during the Fabric Login process.

Each Switch in a fabric is assigned a Domain ID (1-239) Port is usually 00 for a N_Port or AL_PA for a NL_Port Area is usually tied to the physical switch port that the device is

connected to, but this is restrictive. MDS switches instead, logically assign the Area and Port to a FCID.

Dynamic FCID Addressing FCIDs are dynamically assigned to each FC port by the switch when it receives a Fabric Login (FLOGI) from the device:

Each FC switch in the fabric is assigned a unique Domain ID from 1 to 239 (except McDATA switches, which assign only domains 97 to 127).

Traditional FC switches will assign the Area ID based upon the physical port on the switch that the device is connected to. For example, a device connected to port 3 on the switch will receive an Area ID of hex 03. Therefore the FCID is tied to the physical port on the switch.

The Port ID is usually hex 00 for a N_Port or the AL_PA (Arbitrated Loop Physical Address) for a NL_Port. This means that every N_Port connected to the switch is reserved an entire area of 256 addresses, although it will only use 00. This is a wasteful use of addresses and one of the reasons why Fibre Channel cannot support the full 16.5 million addresses.

The Cisco MDS does not tie the Area to the physical port on the switch, but will assign the FCID logically in sequence starting with an area of 00.

The latest HBAs support Flat Addressing and the Cisco MDS will combine the Area and Port fields together as a 16 bit Port ID field. Each device is assigned an FCID in sequential order starting at 0000, 0001 etc. Legacy devices will be assigned a fixed Port ID of 00 per Area as defined above.


The FC-AL Address Space In a public (fabric-attached) loop:

Public NL_Ports are assigned a full 24-bit fabric address when they log into the fabric. There are 126 AL_PA addresses available to NL_Ports in an arbitrated loop; the AL_PA

0x00 is reserved for the FL_Port (which is logically part of both the fabric and the loop).

The Domain and Area fields are identical to those of the FL_Port to which the loop is connected.

In a private (isolated) loop :

Private NL_Ports can communicate with each other based upon the AL_PA, which is assigned to each port during loop initialization.

Private NL_ports are not assigned a 24-bit fabric address, and the Domain and Area segments are not used.


Fabric Login


Fabric Login

TargetNode

InitiatorNode

N_Port A N_Port BF_Port A F_Port B

Process A Process B

FLOGIFLOGI FLOGIFLOGI

PLOGIPLOGI

PRLIPRLI

Fabric

FCIDs are dynamically assigned to each device FC Port by the switch during the Fabric Login (FLOGI) process.

Each device will register (PLOGI) with the switchs Name Server Initiators will query the Name Server for available targets, then send

PLOGI to the target to exchange FC parameters Initiator will login to each Target using Process Login (PRLI) to

establish a channel of communication between them (Image Pair)

PLOGIPLOGI PLOGIPLOGI

Before an N_Port can begin exchanging data with other N_Ports, three processes must occur:

The N_Port must log in to its attached F_Port. This process is known as Fabric Login (FLOGI). During PLOGI, both ports exchange Fibre Channel common parameters. ie. Buffer credits, buffer size, classes of service supported etc.

The Initiator N_Port must log in to its target N_Port. This process is also known as Port Login (PLOGI). This time the initiator and target ports exchange Fibre Channel common parameters like before. If one port supports 2KB buffers but the other only supports 1KB buffers, they will negotiate down to the lowest common factor ie 1KB buffers.

The Initiator N_Port must exchange information about ULP support with its target N_Port to ensure that the initiator and target process can communicate. This process is known as Process Login (PRLI). Parameters exchanged are specific to the Upper Layer Protocol (ULP). For instance, one port will state that it is an Initiator, the other must say that it is a Target. If both ports are Initiators, then the PRLI is rejected.



Fabric Login Analyzer Screenshot

Analyzer screenshot showing the contents of a FLOGI frame sent to the Fabric Login Server in the FC switch.

Fabric Login Analyzer Screenshot The preceding image shows an analyzer trace that displays part of a fabric login sequence. The top of the trace shows the NOS-OLS-LR-LRR sequence that occurred while the link was being initialized. (NOS is off the screen)

The right-hand panel shows the contents of the FLOGI frame from the N_Port to the FLOGI Server on the switch F_Port (FFFFFE).

Useful information can be obtained by studying these analyzer traces:

Notice that at this time the N_Port does not yet have an address. 00.00.00 Notice also that the World Wide Port Name is the same as the World Wide Node Name.

This is common in single ported nodes.

The N_Port does not support Class 1, but it does support Classes 2 and 3. The N_Port supports Alternate Buffer Credit Management Method and can guarantee 2

BB_Credits at its receiver port.

You can see that this is a single-frame Class 3 sequence because the Start of Frame is SOFi3 and End of Frame is EOFt, meaning that this initial first frame is also the last one in the sequence.



Registered State Change Notification (RSCN)

Link failure

Host Storage

FabricController

SCRSCRSCRSCRRSCNRSCNRSCNRSCN

LS_ACCLS_ACCLS_ACCLS_ACC

How can a host keep track of its targets? It can poll each device regularly, but this would be a high overhead, or It can register for State Change Notification (SCN) with the switch The switch will send a RSCN whenever targets go offline/online The host will then query the Name Server to find out what has changed

FC

HBA

FC

FC

The Registered State Change Notification (RSCN) Process Changes to the state of the fabric can affect the operation of ports. Examples of fabric state changes include:

A node port is added or removed from the fabric Inter-switch links (ISLs) are added or removed from the fabric A membership change occurs in a zone

Ports must be notified when these changes occur.

The RSCN Process The FC-SW standard provides a mechanism through which switches can automatically notify ports that changes to the fabric have occurred. This mechanism, known as the RSCN process, is implemented by a fabric service called the Fabric Controller. The RSCN process works as follows:

Nodes register for notification by sending a State Change Registration (SCR) frame to the Fabric Controller.

The Fabric Controller transmits RSCN commands to registered nodes when a fabric state change event occurs. RSCNs are transmitted as unicast frames because multicast is an optional service and is not supported by many switches.

Only nodes that might be affected by the state change are notified. For example, if the state change occurs within Zone A, and Port X is not part of Zone A, then Port X will not receive an RSCN.

Nodes respond to the RSCN with an LS_ACC frame.


The RSCN message identifies the ports that were affected by the state change event, and it identifies the general nature of the event. After receiving an RSCN, the node can then use additional Link Services commands to obtain more information about the event. For example, if the RSCN specifies that the status of Port Y has changed, the nodes that receive the RSCN can attempt to verify the current (new) state of Port Y by querying the Name Server.

The Fabric Controller will generate RSCNs in the following circumstances:

A fabric login (FLOGI) from an Nx_Port. The path between two Nx_Ports has changed (e.g., a change to the fabric routing tables that

affects the ability of the fabric to deliver frames in order, or an E_Port initialization or failure)

An implicit fabric logout of an Nx_Port, including implicit logout resulting from loss-of-signal, link failure, or when the fabric receives a FLOGI from a port that had already completed FLOGI.

Any other fabric-detected state change of an Nx_Port. Loop initialization of an L_Port, and the L_bit was set in the LISA Sequence. An Nx_Port can also issue a request to the Fabric Controller to generate an RSCN. For

example, if one port in a multi-ported node fails, another port in that node can send an RSCN to notify the fabric about the failure.


Standard Fabric Services


Standard Fabric Services

Common Transport

Link Services

FC-0 Physical interface

FC-1 Encoding

FC-2 Framing & flow control

FC-3 Generic Services

FC-4 ULP Mapping

Generic ServicesGeneric ServicesFabric ControllerAlias Server

Con

fi gur

a tio

nS

e rve

r

Management ServerManagement Server

Zon e

S

erv e

r

Unz

o ned

Na m

e S

e rve

r

Name Server

Domain Manager

Key Server

Time Server

The FC-SW-2 specification defines several services that are required for fabric management. These services include:

Name Server Login Server Address Manager Alias Server Fabric Controller Management Server Key Distribution Server Time Server

The FC-SW-2 specification does not require that switches implement all of these services; some services can be implemented as an external server function. However, the services discussed in this lesson are typically implemented in the switch, as in Cisco MDS 9000 Family Switches.



Domain Manager

Principal SwitchSelection

FabricConfiguration

Domain IDAllocation

FCID Databaseand Cache

FC_ID Allocation

The Domain Manager

Port ManagerPort Manager Login ServerLogin Server

ManagementServices

ManagementServices

WWN ManagerWWN ManagerVSAN ManagerVSAN Manager

The Domain Manager The Domain Manager is the logical function of a switch that is responsible for the assignment of addresses in a fabric. The Domain Manager is responsible for:

Allocating domain IDs (requesting a domain ID, and assigning domain IDs to other switches if this switch is the Principal Switch)

Allocating port addresses (FC_IDs) Participating in the Principal Switch selection process Performing the Fabric Build and Reconfiguration processes when the topology changes

The Domain Manager supports the Fabric Port Login Server, which is the service that N_Ports use when logging in to the fabric. When an N_Port logs into the fabric, it sends a FLOGI command to the Login Server. The Login Server then requests an FC_ID from the Domain Manager and assigns the FC_ID the N_Port in its ACC reply to the FLOGI request.

The preceding diagram shows how the Domain Manager interacts with other fabric services:

The VSAN Manager provides the Domain Manager with VSAN configuration and status information.

The WWN Manager tells the Domain Manager what WWN is assigned to the VSAN. The Port Manager provides the Domain Manager with information about the fabric

topology (a list of E_Ports) and notifies the Domain Manager about E_Port state changes.

The Login Server receives N_Port requests for FC_IDs during FLOGI. The Domain Manager interacts with management services to allow administrators to view

and modify Domain Manager parameters.



The Name Server

Name Server stores data about nodes, such as:FC_IDsnWWNs and pWWNsFibre Channel operating parametersSupported protocolsSupported Classes of Service

Supports soft zoning Provides information only about nodes in the requestors zone Distributed Name Server (dNS) resides in each switch Responsible for entries associated with that switchs domain Maintains local data copies and updates via RSCNs Sends RSCNs to the fabric when a local change occurs

The Name Server FC Name Server is a database implemented by the switch that stores information about each node, including:

FC_IDs WWPN and WWNNs FC operating parameters, such as supported ULPs and Classes of Service

The Name Server:

Supports soft zoning by performing WWN lookups to verify zone membership Enforces zoning by only providing information about nodes in the requestors zone Is used by management applications that need to obtain information about the fabric

Each switch in a fabric contains its own resident name server, called a distributed Name Server (dNS). Each dNS within a switch is responsible for the name entries associated with the domain assigned to the switch. The dNS instances synchronize their databases using the RSCN process.

When a client Nx_Port wants to query the Name Service, it submits a request to its local via the Well Known Address for the Name Server. If the required information is not available locally, the dNS within the local switch responds to the request by making any necessary requests of other dNS instances contained in the other switches. The communication between switches that is performed to acquire the requested information is transparent to the original requesting client.

Partial responses to dNS queries are allowed. If an entry switch sends a partial response back to an Nx_Port, it must set the partial response bit in the CT header.


Name Server Operations When ports and nodes register with the Name Server, their characteristics are stored as objects in the Name Server database.

The Port Identifier is the Fibre Channel port address identifier (FC_ID) assigned to an N_Port or NL_Port during fabric login (FLOGI). The Port Identifier is the primary key for all objects in a Name Server record. All objects are ultimately related back to this object. Because a node may have more than one port, the Node Name is a secondary key for some objects.

There are three types of Name Server requests:

Get Object: This request is used to query the Name Server Register Object: Only one object at a time can be registered with the Name Server. A

Client registers information in the Name Server database by sending a registration request containing a Port Identifier or Node Name.

Deregister Object: Only one global deregistration request is defined for the Name Server.

Name Server information is available, upon request, to other nodes, subject to zoning restrictions. If zones exist within the fabric, the Name Server restricts access to information in the Name Server database based on the zone configuration.

When a port logs out of a fabric, the Name Server deregisters all objects associated with that port.



The Management Server

Information is provided without regard to zone -single access point for information about the fabric topology

Read-only accessServices provided:

Fabric Configuration Service (FCS)Zone ServiceUnzoned Name Service

The Management Server The FC Management Server provides a single access point for obtaining information about the fabric topology. Whereas the Name Server only provides information about ports configured within the zone of the port requesting information, the Management Server provides information about the entire fabric, without regard to zone. The Management Server allows SAN management applications to discover and monitor SAN components, but it does not allow applications to configure the fabricthe Management Server provides read-only access to its data.

The Management Server provides the following services:

The Fabric Configuration Service (FCS) supports configuration management of the fabric. This service allows applications to discover the topology and attributes of the fabric.

The Zone Service provides zone information for the fabric to either management applications or directly to clients.

The Unzoned Name Service provides access to provide information about the fabric without regard to zones. This service allows management applications to see all the devices on the entire fabric.

54 Cisco Storage

cisco_san.pdf

Documents