01 san overview

8/3/2019 01 SAN Overview

1/64

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

SAN Overview - 1

The objectives for this module are shown here. Please take a moment to read them.


2/64


SAN Overview - 2

The objectives for this lesson are shown here. Please take a moment to read them.


3/64


SAN Overview - 3

Fibre Channel (FCP) is a serial data transfer interface that operates over copper wire and/or optical

fiber at data rates up to 8 Gbps (gigabits per second) and up to 10 Gbps when used as ISL (E_Ports) on

supported switches.

Networking and I/O protocols (such as SCSI commands) are mapped to Fibre Channel constructs, and

then encapsulated and transported within Fibre Channel frames. This process allows high-speed

transfer of multiple protocols over the same physical interface.

Fibre Channel systems are assembled from familiar types of components: adapters, hubs, switches and

storage devices.

Host bus adapters (HBAs) are installed in computers and servers in the same manner as a SCSI host

bus adapter or a network interface card (NIC).

Hubs link individual elements together to form a shared bandwidth loop.

Fibre Channel switches provide full bandwidth connections for highly scalable systems without a

practical limit to the number of connections supported (16 million addresses are possible).

The word fiber indicates the physical media. The word fibre indicates the Fibre Channel protocol and

standards.


4/64


SAN Overview - 4

The Fibre Channel standards define a layered protocol:

ULP (Upper level Protocol) is not actually part of Fibre Channel

FC-4 level of Fibre Channel is designed to hand off to another protocol such as SCSI.

Fundamentally, the commands at FC-4 for SCSI allow SCSI initiators and targets to communicate

over Fibre Channel

FC-3 Defines a set services to support advanced functions

FC-2 level serves as the transport mechanism of Fibre Channel. The transported data is transparent

to FC-2 and visible to FC-3 and above. Includes data framing, frame sequencing, flow control, and

class of service

FC-1 defines the transmission protocol including serial encoding and decoding, special characters

and error control. Information is encoded 8 bits at a time into a 10 bit transmission character and

transmitted over the cable

FC-0 is the lowest level and defines the physical link in the system, including the fiber, connectors,optical and electrical components. This level covers a variety of media and the associated drivers

and receivers capable of operating at a wide range of speeds


5/64


SAN Overview - 5

Frames are the basic building blocks of a Fibre Channel connection. The frames contain the

information to transmit, the address of the source and destination ports, and link control information.

All information in Fibre Channel is passed in frames. The maximum amount of data carried in a frameis 2112 bytes; the total frame size is 2148 bytes.

The header contains the Source and Destination Addresses, which allow the frame to be routed to the

correct port. The Type field interpretation is dependent on whether the frame is a link control or Fibre

Channel data frame. For example, if the frame is a data frame, a 08 in the Type field indicates SCSI

FCP information in the Data field.


6/64


SAN Overview - 6

Fibre Channel addresses are used to designate the source and destination of frames in the Fibre

Channel network. The Fibre Channel address field is 24 bits/3 bytes in length. Unlike Ethernet, these

addresses are not burned in, but are assigned when the node enters the loop or is connected to the

switch.

The Fibre Channel Address identifiers are three bytes in length. The Frame Header contains two three-

byte fields for address identifiers, the D_ID and S_ID fields. Each N_Port has a fabric-unique

identifier, the N_Port Identifier, by which it is known. The source and destination N_Port Identifiers

and alias address identifiers are used to route frames within the fabric.

The Physical Address is switch-specific and dynamically generated during the Fabric Login (FLOGI).


7/64

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

SAN Overview - 7

A Fabric is a virtual space in which all storage nodes communicate with each other over

distances. It can be created with a single switch or a group of switches connected together.

The primary function of the fabric is to receive data frames from a source N_Port and route them to thedestination N_Port whose address identifier is specified in the frames. Each N_Port is physically

attached through a link to the fabric.

Many switches, such as the Cisco MDS, support multiple fabrics within the same switch and also

within the same physical switch topology. When a device logs into a fabric, its information is

maintained in a database. At the same time, information required for it to access other devices or

changes to the topology is provided by another database.


8/64


SAN Overview - 8

The following are some of the common services found in a fabric:

Login Service: This Service (at address FFFFFE) is used by all nodes when they perform a Fabric

Login (FLOGI). For a node to communicate in a fabric, it has to register itself with this service.

When it does so, it sends a Source Identifier (S_ID) set to zero, that is, having the following hex

value 0x000000. The login service then returns a D_ID to the node with the with the Domain, Area,

and Port fields qualified. This gives the node (N_Port) a 24-bit address that identifies it uniquely in

the entire fabric. The fabric uses this unique address for routing services.

Name Service: This Service (at address FFFFFC) is used to store information about all devices

attached to the fabric. When a node performs a PLOGI, it registers itself with the name server. The

name server stores all these entries in a locally resident database on each switch. Each switch in the

fabric topology exchanges its Name Service information with other switches in the fabric to

maintain a synchronized, distributed view of the fabric.

Fabric Controller: This Service (at address FFFFFD) provides state change notification to all

registered nodes in the fabric. Devices that require a fabric controller are those devices that need tokeep track of changes in a fabric, for example scanning storage targets in a fabric. This is requested

by submitting a SCN (State Change Notification) frame to the controller which then responds back

with a RSCN (Registered State Change Notification) to the frame.


9/64


SAN Overview - 9

There are three types of login supported in Fibre Channel; Fabric, Port and Process. All node ports

must attempt to log in with the Fabric. This is typically done right after the link or the Loop has been

initialized.

When a device is physically connected to a fabric switch port, the Fibre Channel protocol establishes a

logical connection between the node and the fabric switch. This is called Link Initialization. Ordered

Sets are sent between the node and the switch to establish the link.


10/64


SAN Overview - 10

Once the physical link is established, the node sends a special frame (FLOGI) to the port to allow it to

communicate with the rest of the fabric through the Fabric Login or FLOGI. The FLOGI frame

contains the S_ID field filled in with its ALPA value (00 for a switched fabric). This frame is received

by the login server, which is located at address FFFFFE. The login server responds back with the D_IDfield filled with the domain ID and area location. In other words, the device now gets a 24-bit address

by which it is identified in the fabric. All ports except private NL_Ports have to go through this

sequence for them to be able to communicate with each other in the fabric.

Next, each node performs a Name Server registration. The Name Server is located at address FFFFFC

and obtains information from the node through the port login frame (PLOGI) and through subsequent

registration frames. Information in the Name Server is stored in the form of database objects. The node

may register values for all or some database objects depending on the requirement. The most

commonly registered objects are: 24-bit fabric address, 64-bit Port Name (WWPN), 64-bit Node Name

(WWNN), ULPs (SCSI, IP etc.), Classes of service supported and Port type. The Node also requests a

list of nodes that support the same FC-4 Upper Layer Protocols as itself. This list usually depends onwhether there are restrictions placed on which devices the node can talk to i.e. zoning.


11/64


SAN Overview - 11

The node then attempts a Port Login (PLOGI) to all nodes from the list it receives from the switchs

Name Server. It provides a specific set of operating characteristics associated with the destination

N_Port, Classes of Service are supported. It also initializes the destination end-to-end credit. The

process is repeated as other nodes are attached to other ports on the switch.

Finally, the node then sets up the environment between itself (originating N_Port) and the device its

communicating with (responding N_Port) by performing a Process Login. This environment is then

used to determine if there is a LUN present. This is the point at which storage connectivity is

established. A group of related processes is collectively known as an image pair. The processes

involved can be system processes, system images, control unit images or ULP processes. The use of

process login (PRLI) is required by a specific upper-level protocol such as SCSI-FCP mapping.


12/64


SAN Overview - 12

This table documents the various processes that occur when a Fibre Channel device is connected to an

FC-based SAN.

When a link light on the switch or on an HBA does not come on, it means that the link initializationcomponent in the FLOGI process is not complete. To troubleshoot this, one usually starts with the

hardware and then moves up to the HBA driver level.


13/64


SAN Overview - 13

The Internet Small Computer Systems Interface (iSCSI) protocol provides a means of transporting

SCSI packets over TCP/IP. iSCSI works by wrapping SCSI commands into TCP and transporting them

over an IP network. Since iSCSI is IP based traffic, it can be routed or switched on standard Ethernet

equipment.


14/64


SAN Overview - 14

Traditional Ethernet adapters (NIC) are designed to transfer packetized file level data among PCs,

servers, and storage devices such as NAS appliances. However, NICs do not traditionally transfer

block level data, which is handled by a storage host bus adapter, such as Fibre Channel or parallel

SCSI. In order for a NIC to process block level data, the data needs to be placed into a TCP/IP packetbefore being sent over the IP network. Through the use of iSCSI drivers on the host or server, a NIC

can transmit packets of block level data over an IP network. When using a NIC, the server handles the

packet creation of block level data and performs all of the TCP/IP processing. This is extremely CPU

intensive and lowers the overall server performance.

The TCP/IP processing performance bottleneck has been the driving force behind the development of

TCP/IP offload engines (TOE) on adapter cards. A TOE moves the TCP/IP processing from the host

CPU to the TOE card. Thus, a TCP/IP offload storage NIC operates more like a storage HBA rather

than a standard NIC.

An iSCSI HBA is similar to a Fibre Channel HBA where all operations occur on the card. The iSCSI

HBA is required for EMC Boot from SAN environment.


15/64


SAN Overview - 15

Because a message is divided into a number of packets, each packet can, if necessary, be sent by a

different route across the network. Packets can arrive in a different order than the order they were sent

in. The Internet Protocol just delivers them. It's up to another protocol, the Transmission Control

Protocol (TCP) to put them back in the right order.

An iSCSI packet contains SCSI data and the iSCSI header, which is created by the iSCSI initiator, and

is then wrapped in other protocol layers to facilitate its transport.

The Ethernet Header is used to provide addressing for the physical layer while the IP Header provides

packet routing information used for moving the information across the network, and the TCP Header

contains the information needed to guarantee delivery to the target destination. The iSCSI Header

defines how to extract SCSI commands and data.


16/64


SAN Overview - 16

Native iSCSI allows for all communications using Ethernet. Initiators may be directly attached to

iSCSI Targets, or may be connected using standard Ethernet routers and switches. Bridging

architectures allow for the Initiators to exist in an Ethernet environment, while the storage remains in a

Fibre Channel SAN.


17/64


SAN Overview - 17

To ensure that data reaches all the users who need it, organizations are now looking for new ways to

transport data throughout the enterprise locally over the SAN as well as over much longer distances.

One of the best ways to achieve this goal is to interconnect geographically dispersed SANs throughreliable, high-speed links. This approach involves transporting Fibre Channel block data over existing

IP infrastructures currently used throughout the enterprise. The new FCIP protocol standard has rapidly

gained acceptance as a manageable, cost-effective way to blend the best of both worlds: Fibre Channel

block data storage and proven, widely deployed IP infrastructure. As a result, organizations now have

an excellent way to protect, store, and move their data while leveraging existing technology

investments.


18/64


SAN Overview - 18

The top layer of the FCIP protocol stack has SCSI Applications, which include the SCSI driver

program that executes read and write commands. These block, stream, and other command types are

grouped into a layer which also contains SCSI data and status information. Below the SCSI layer is

the Fibre Channel Protocol (FCP) layer, which is simply a fibre channel frame whose payload is SCSI.The FCP layer rides on top of the Fibre Channel transport layer. This layer can then run natively within

a SAN fabric environment or be encapsulated into IP at the Fibre Channel over IP (FCIP) layer. TCP

and IP are then used to transport the encapsulated information across wired or wireless Ethernet, or

another transport that supports TCP/IP traffic.

Notice how FCIP creates a logical tunnel between two Fibre Channel protocol stacks. In reality, the

information flows through the IP network represented by the cloud.


19/64


SAN Overview - 19

FCIP entities are switches or other network adapters used during FCIP. The primary purpose of an

FCIP entity is to forward FC frames. Primitive signals, sequences and class 1 FC frames are not

transmitted through FCIP because they can't encode using FC Frame encapsulation. An IP network

sees the FCIP entities as peers, therefore require TCP/IP communication. FCIP entities contain 1+ TCPendpoints in IP-based network. From a Fibre Channel perspective, the pairs of FCIP entities and their

FC entities, forward FC frames between FC fabric elements. The End nodes dont know an IP link

exists, therefore the path taken by the FC frames follow the normal routing procedure established by IP

network. FCIP doesnt participate in the FC frame routing. FCIP control and service may use QoS

features.

FCIP entities don't participate in the discovery of FC source and destination identifiers. Before a link is

established between two FCIP entities the IP address and TCP port of its pair must be configured with

the initiating entity.


20/64


SAN Overview - 20

FCIP can transport existing Fibre Channel services across the IP network such that two or more

interconnected SANs can appear as a single large SAN and can be managed by traditional SAN

management applications. In addition, FCIP enables SAN applications to support additional protocols

without modification. These applications might include disk mirroring between buildings in a campusnetwork or remote replication over the WAN. The type of applications utilized are based on the

distance the data must travel, the network bandwidth, and the QoS requirements and/or abilities of the

network connection.

While some implementations of FCIP are point-to-point tunnels, the protocol does not require that

the gateways support only point-to-point tunneling. The FCIP standard supports all Fibre Channel

services, including FSPF routing algorithms, such that multiple logical links created from a single

gateway can route Fibre Channel packets over the IP infrastructure. Not only is FCIP routable, but IP

networks do not need to know anything about the packets being routed.

The Fibre Channel services handle all routing between logical links, while the TCP protocol handlesthe delivery of packets to the specific gateway device.


21/64


SAN Overview - 21

A Fibre Channel switch, acting as a FCIP Gateway, encapsulates FC frames into IP packets. Once

encapsulated into IP packets, they can be sent across the IP network through the FCIP link. The IP

network could be made up of Gigabit Ethernet, Fast Ethernet, IP switches and routers, etc. At the other

end of the IP network, the original Fibre Channel frames are recovered and sent through the SAN bythe receiving Fibre Channel Switch.

The FCIP connected FC SANs essentially form into a new unified fabric. The IP network is transparent

to the Fibre Channel fabric. Only edge devices need to be aware of FCIP encapsulation. This solution

can take advantage of existing Fiber Channel networks and tie them together using existing IP

networks.


22/64


SAN Overview - 22



23/64


SAN Overview - 23

The Port initialization takes place first. In this step the port determines the Port Type. For this

discussion, we will assume the Port has configured itself as an E_Port (Expansion Port). Once this port

is a recognized E_Port, Exchange Link Parameters (ELP) are sent. The ELP exchanges information

about the switch such as its WWN and Class of Service.

Next, the Fabric must elect a Principal Switch. This is accomplished by the Exchange Fabric

Parameters (EFP) and Principal Switch Selection (DIA) operations. Once the Principal Switch is

elected, Domain_IDs are assigned to the other member switches of the Fabric. Switches will attempt to

keep their previous Domain_ID.

The Final steps in configuring the Fabric are to build the Topology Database using Fibre Shortest Path

First (FSPF) and then create the Routing Tables.

Although not specified, the switches attempt to combine the existing Zone information into a single

Zone Set to be distributed throughout the new Fabric.


24/64


SAN Overview - 24

An E_Port is the point at which a Switch is connected to another Switch to create a multi-Switch

Fabric. An E_Port is the point at which a Switch is connected to a Bridge device. It normally functions

as a conduit between the Switches for frames destined for remote N_Ports and NL_Ports. An E_Port is

also used to carry frames between Switches for purposes of configuring and maintaining the Fabric.

An E_Port shall support the Class F service. An E_Port shall also be capable of routing one or more of

the following classes of service: Class 1 service, Class 2 service, Class 3 service. An E_Port shall not

admit to the Fabric any Primitive Sequences, or any Primitive Signals other than Idle, that the E_Port

receives on its inbound fibre.

An E_Port contains an FC-PH Transport element through which all frames are passed, and Primitives

are transferred across the Link to and from the other E_Port. Frames received from the other E_Port

are either directed to the Switch Construct via the Switch Transport element, or directed to the

Link_Control_Facility. The Link_Control_Facility receives frames related to Switch Fabric Internal

Link Services such as ELP, and transmits responses to those Link Service frames.

Frames received from the FC-PH Transport element that are destined for other ports are directed by the

Switch Transport to the Switch Construct for further routing. Frames received from the Switch

Construct by the Switch Transport are directed either to the FC-PH Transport for transmission to the

other E_Port, or to the Internal_Control_Facility. The Internal_Control_Facility receives frames related

to Switch Fabric Internal Link Services, and transmits responses to those Internal Link Services

frames.

Information is passed between the Internal_Control_Facility and the Link_Control_Facility to effect

the control and configuration of the Transport elements.


25/64


SAN Overview - 25

Switches in the fabric communicate through EFP (exchange fabric parameter) frames to determine the

principal switch as follows:

If one (and only one) director/switch is set to Principal, it becomes the principal switch. This

switch remains the principal switch as long as it is on line in the fabric, even if someone changes

another switchs Switch Priority to Principal.

If more than one director/switch is set to Principal, the director/switch with the lowest WWN

number among those set to Principal becomes the Principal switch. The Switch Priority setting for

every other switch/director (including any others that were previously set to Principal)

automatically changes to Default.

If no director/switch is set to Principal but one or more are set to Default, the director/switch with

the lowest WWN number among those set to Default becomes the principal switch. The Switch

Priority setting for every other switch/director remains unchanged.

If all directors/switches are set to Never Principal, no director/switch can be principal. In this case,

all of the ISLs will segment, with reason code 05 (no switch is capable of becoming a Principal

switch).


26/64


SAN Overview - 26

Each switch in a multiswitch fabric is identified by a unique Domain ID, in the range 1 through 239.

Domain IDs are used in 24-bit Fibre Channel addresses to identify source and destination ports in a

fabric.

When adding a new switch, assign it a Preferred Domain ID that is not in use. When the switch comes

on line, it requests that it be assigned this ID. If the ID is already in use, the principal switch assigns an

unused one.

To change a Domain ID, a switch must be taken off line. Some HBAs can handle a Domain ID change

without requiring device driver administration changes. In some operating environments, however

(currently HP-UX and AIX), drivers require device driver-level administrative changes when the

Domain ID is changed.

If two operational fabrics join, they determine if any Domain ID conflicts exist between the fabrics. If

a conflict exists, the interswitch link (ISL) E_Ports segment themselves from each other, preventing

the fabrics from joining. To avoid this problem, assign a unique Preferred Domain ID to each switch ina data center. If there are plans to go data center to data center, apply this uniqueness across both data

centers. This will facilitate future fabric mergers, and is particularly important if zoning is

implemented through port number (and by default Domain ID) rather than WWN.


27/64


SAN Overview - 27

EMC recommends that you always attempt to give each switch a unique preferred Domain ID, whether

it is targeted for an existing fabric or a new fabric. (The exception to this rule would be if you were

swapping a new switch for a failed switch.) This provides the best opportunity to merge a switch or

fabric into another fabric with the least amount of manual intervention.

Since Domain IDs become part of an HBA or storage array port's fabric address, and data is transferred

to and from devices based on their address, Domain IDs cannot be changed without first bringing the

switch to an offline state. Every switch that is currently online will have been assigned a Domain ID

from the principal switch. If the fabric contains only one switch, that switch is the principal switch for

that fabric. Domain ID negotiation done during a fabric merge will not change the current Domain ID

when the switch is connected to the fabric while online.

During the exchange of parameters following a merger event, if the switches identify a domain

conflict, they will not merge and the ISLs connecting them will remain segmented. Taking a switch

that has segmented from the fabric due to a Domain ID conflict offline and then bringing it back online

allows the switch to renegotiate for a new unique Domain ID. It should be noted that taking a switch

offline terminates communication between the devices that are on that switch. EMC recommends that

if a conflict does exist that you change the preferred Domain ID of the switch yourself so that you can

ensure that it is unique in your entire environment and that it always attempts to use the same ID, even

after future merges and fabric changes


28/64


SAN Overview - 28

One switch in the fabric is responsible for the distribution of Domain IDs, and plays a role in the route creationfor fabric management traffic. This switch is known as the principal switch. Since this switch must communicatewith all other switches and is the basis for fabric traffic routing, this switch should be centrally located. This willassist in the uniform delivery of information in the fabric, and provide a consistent response to fabric buildevents.

Principal switches are selected during both the creation of the fabric and during fabric reconstruction events.Two pieces of information located on the switch will determine the selection of the principal switch in the fabric.The Domain ID negotiation is governed by the entity in the fabric called the principal switch. Principal switchselection is made based on the switch with combination of the highest principal switch priority and then thelowest World Wide Name.

Proper placement of the principal switch in the fabric can lead to short negotiation times, resulting in the fabricreturning to a normal state in an shorter amount of time. For these reasons, EMC recommends that the switch atthe logical center of the fabric be made the principal switch. This would normally mean the switch with both theleast amount of hops to the farthest extent of the fabric and/or the switch that has connections to the most otherswitches in the fabric. These two placement strategies will help to ensure that your principal switch access times

are as quick as they can be.The basic rules of Principle Switch selection are:

A standalone switch will declare itself the Principal Switch.

Each Fabric can have only one Principal Switch.

If both switches are ONLINE, a principal switch election will occur.

Switch Priority determines who becomes Principal Switch.

Domain IDs DO NOT determine who becomes Principal Switch.


29/64


SAN Overview - 29

The fabric-capable device drivers on Symmetrix-supported host bus adapters (HBAs) discover devices

by using a Name Server, which is a fabric-based function accessible to all nodes on the fabric via a

well-known fabric address.

When an N_Port logs onto the fabric, it registers its World Wide Port Name and FC-4 type (SCSI) with

the Name Server. A driver performing device discovery queries the Name Server for a list of all SCSI

devices that have registered with it. The driver uses this list to attempt a login to each of the registered

devices. In this manner, a fabric device driver discovers all logged-in Symmetrix ports attached to the

switch.


30/64


SAN Overview - 30

Connectrix Directors and switches use a number of algorithms to distribute the load over a set of ISLs.

These algorithms include:

Fibre Shortest Path First

Multiple Equal-Cost Paths

Load Assignment on World Wide Node Names

The ANSI standard for routing in a Fibre Channel fabric is Fibre Channel Shortest Path First (FSPF).

This standard measures routing cost in terms of the number of hops a frame must take from its source

to its destination.

A hop is a direct path between two switches with no other switches in between. A link between an end

device and a switch does not count as a hop.

The FSPF algorithm creates routes of minimum cost throughout the fabric. Every entry port into a

switch has a designated exit for each of the possible destinations. These exit assignments change onlyif there is a reconfiguration event in the fabric. This single-static exit ensures in-order delivery of the

frames within an exchange. If an ISL or a new switch is added or removed, the assignments may

change. These assignments are done with logged-in N_Ports. No routing assignments are done for

F_Ports that have no logged-in ports.

When there are multiple equal-cost paths between a source and a destination, the routing algorithm

spreads the load over the multiple paths in a round robin fashion, sequentially assigning a set of entry

ports to a set of exit ports so that each exit port has the same number of assigned entry ports (or a

difference of only one).


31/64


SAN Overview - 31

In this standard, the term path selection indicates the discovery of the best path from source to

destination, and the term routing indicates the actual forwarding of frames to a specific destination.

FSPF performs hop-by-hop routing which means that a Switch only needs to know the next hop on the

best path to the destination. The replicated topology database insures that every Switch in the Fabrichas the same view of the Fabric itself, and therefore all Switches will make consistent routing

decisions.

Typically a Switch needs to know, for each destination domain in the Fabric, which path should be

used to route a frame to that domain. A routing table entry minimally consists of a destination

Domain_ID, and an E_Port to which frames are forwarded to the destination Switch.


32/64


SAN Overview - 32

The fewest hop path is always selected. If a new path with the least cost becomes available, frames

will immediately be switched to the newer path. If multiple minimum hop paths between switches

exist, switches will do load sharing

In this example, the cost for each link is equal:

There are two ISLs between switches Green and Yellow

Three HBAs login to switch Green with data destined to Switch Yellow

HBS 1 uses Path C (ISL 3)

HBA 2 uses Path D (ISL 4)

HBS 3 uses Path C (ISL 3)

Each link has a specific cost associated with it. The higher the throughput, the lower the cost. For

example a 1Gbps link has a cost of 1000 while a 2Gbps link costs 500. In this case, the 2Gbps link is

always used. You also can configure link costs for switches.


33/64


SAN Overview - 33

Zoning is a switch function that allows devices within the fabric to be logically segmented into groups

that can communicate with each other. When a device logs into a fabric, it is registered by the name

server. When a port logs into the fabric, it goes through a device discovery process with other devices

registered as SCSI FCP in the name server. The zoning function controls this process by only lettingports in the same zone establish these link level services.

A collection of zones is called a zone set. The zone set can be active or inactive. An active zone set is

the collection of zones currently being used by the switched fabric to manage data traffic.

Single HBA zoning consists of a single HBA port and one or more storage ports. A port can reside in

multiple zones. This provides the ability to map a single storage port to multiple host ports. For

example, a Symmetrix FA port or a CLARiiON SP port can be mapped to multiple single HBA zones.

This allows multiple hosts to share a single storage port.


34/64


SAN Overview - 34

Switch Domain IDs must be unique

If two enabled switches are cabled together, they must have unique Domain Ids, or the Fabric is

segmented.

Ensuring unique Domain ID

If a new switch is cabled to the Fabric and then enabled, the master switch assigns a unique

Domain ID to the new switch.

Domain IDs can be configured by the administrator to ensure that uniqueness.

If you power cycle, the switch will receive a unique Domain ID.

Other configuration items that may affect cascade

Non-SCSI Tachyon Mode

Disable Device Probing

VC Encoded Address Mode Per-frame Route Priority


35/64


SAN Overview - 35



36/64


SAN Overview - 36

There are several ways to monitor and manage Fibre Channel switches in a fabric:

If the switches in the fabric are contained in a cabinet with a Service Processor (SP), console software loaded

on the SP can be used to manage them.

Some switches also offer a console port, which is used for serial connection to the switch for initialconfiguration using a Command Line Interface (CLI). This is typically used to set the management IP

address on the switch. Subsequently, all configuration and monitoring can be done via IP. Telnet or ssh may

be used to log into the switch over IP, and issue CLI commands to it. The primary purpose of the CLI is to

automate management of a large number of switches/directors with the use of scripts, although the CLI may

be used interactively, too. In addition, almost all models of switches support a browser-based graphical

interface for management.

There are vendor-specific tools and management suites that can be used to configure and monitor the entire

fabric. They include:

B-Series: Web Tools and Connectrix Manager

M-Models include Connectrix Manager Basic

MDS-Series: Cisco Fabric Manager

SAN Manager, an integral part of EMC ControlCenter, provides some management and monitoring capabilities

for devices from both vendors.

A final option is to deploy a third-party management framework such as Tivoli. Such frameworks can use SNMP

(Simple Network Management Protocol) to monitor all fabric elements.


37/64


SAN Overview - 37

Web Tools is an easy-to-use, browser-based application for switch management and is included with

all Connectrix B-Series products. Web Tools simplifies switch management by enabling administrators

to configure, monitor, and manage switch and fabric parameters from a single online access point. Web

Tools supports the use of aliases for easy identification of zone members. With Web Tools, firmwareupgrade is a one-step process. The Switch View allows you to check the status of a switch in the fabric.

The LED icon for the port reporting an issue changes color.


38/64


SAN Overview - 38

Connectrix Manager is a licensed software product widely used for the management of Connectrix B

switches and directors. It can be run locally on the Connectrix Service Processor, or remotely on any

network-attached workstation. Since this application is Java-based, IT administrators can run it from

virtually any type of client device.


39/64


SAN Overview - 39

Fabric Manager and Device Manager must be installed in a server and can contain several clients. This

Java-based tool simplifies management of the MDS Series through an integrated approach to fabric

administration, device discovery, topology mapping, and configuration functions for the switch, fabric,

and port.

Features of Fabric Manager include:

Fabric visualization: Automatic discovery, zone and path highlighting

Comprehensive configuration across multiple switches

Powerful configuration analysis including real-time monitoring, alerts, zone merge analysis, and

configuration checking

Network diagnostics: Probes network and switch health, enabling administrators to pinpoint

connectivity and performance issues

Comprehensive security: Storage Media Encryption Protection against unauthorized management

access with Simple Network Management Protocol Version 3 (SNMPV3), Secure Shell Protocol(SSH), and role-based access control (RBAC)


40/64


SAN Overview - 40

SAN Manager provides a single interface to manage LUN Masking, switch zoning, device monitoring

and management. The integration of SAN Manager into ControlCenter provides a distributed

infrastructure allowing for remote management of a SAN. It offers reporting and monitoring features

such as threshold alarms, state change alerts and component failure notifications for devices in theSAN.

SAN Manager has capabilities to automatically discover, map, and display the entire SAN topology at

a level of detail desired by the administrator. It can also display specific physical and logical

information about each object in the fabric. Administrators can view details on physical components

such as host bus adapters, Fibre Channel switches and storage arrays, as well as logical components

such as zones and LUN masking policies. SAN Manager offers support for non-EMC arrays such as

HDS Lightning, HP StorageWorks, and IBM Shark.


41/64


SAN Overview - 41

Command Line Interface is available for any Connectrix Switch. Each switch vendor implements its

own syntax for common switch and fabric functions.


42/64


SAN Overview - 42



43/64


SAN Overview - 43

You can install a switch as a stand-alone unit on a flat surface, in an Electronic Industries Association

(EIA) cabinet or the EC-1700 Cabinet using the Slide Rack Mount Kit included with the switch.

For this example, we review the configuration of a B-Series switch, The following modules cover thedetails for each Connectrix series.


44/64


SAN Overview - 44

Floor loading is a badly understood issue in Data Centers. Not just in the place where the equipment eventually ends up but

also en-route from the loading bay to the final resting place.

Most data centers have a raised floor made of specialist tiles held up on struts and a tile mounting framework pushing down

on the underlying slab and fabric of the building. Each of these elements must be strong enough to withstand the weight ofthe equipment to be installed. In addition, the floor must be strong enough to deal with carrying its design weight with a

number of the tiles removed for maintenance.

Typically a data center floor will have a loading capability measured in Kilo Pascals (kPa) aka Kilograms per square meter

(Kg/m2) or Pounds per square foot (lbs/ft2) depending on your preferred units of measure . For most EMC scenarios, the

floor tiles must be concrete filled or Heavy-duty tiles. Perforated and vented tiles as well as damaged tiles and tiles with

holes for cables (particularly on the edge of the tile rather than in the middle) are significantly weaker than complete tiles.

There are three kinds of load that we need to take into account:

Static Load - the load a stationary object places on the floor (kg/sq m)

Concentrated Load - The point load transferred directly to the floor from a wheel or foot support (kg/sq cm)

Rolling or Dynamic Load - The load placed on the floor as the object is pushed through the data center into its final

location (kg/Tile)

The underlying slab needs to be strong enough to support the gross load of the entire floor, including the struts, tiles,

equipment and M&E plant. If in doubt instruct a Structural Engineer or consult the initial design documents for your

building.

As an example, the installation of the MDS 9513 Director in a rack requires a mechanical lift to place the chassis in the

rack. A fully loaded 9513 can weigh about 300 pounds. A full rack with all components will be over 1000 pounds of static

load.


45/64


SAN Overview - 45

Verify the following requirements are met prior to product and management interface installation.

Ensure:

A site plan is prepared, configuration planning tasks are complete, planning considerations are

evaluated, and related planning checklists are complete.

Fibre Channel SAN design is complete.

Support is available for one of the following product management methods:

A browser-capable PC and Internet connectivity to support the product-resident Web Tools, or

A rack-mount service processor or browser-capable PC and LAN segment connectivity to

support operation of Connectrix Manager and Element Manager applications.

Support equipment and technical personnel are available for the installation.

The required number and type of fiber-optic jumper cables are delivered and available. Ensure

cables are the correct length and have the required connectors.

An EC-1700 cabinet or customer-supplied 19-inch equipment rack and associated hardware areavailable (optional).

Remote workstations or simple network management protocol (SNMP) workstations are available

(optional). Workstations are customer-supplied and connected through a public or dedicated LAN

segment.


46/64


SAN Overview - 46

Take the following measures to ensure a successful installation and operation of the switch. In this

example the switch requires specific environmental conditions.

Install the switch with the non-port side facing the cool-air aisle to ensure proper cooling. Thisprevents the fan assemblies from pulling in heated exhaust air.

Verify that the supply circuit, line fusing and wire size conform to the electrical rating of the switch.

Verify that the ambient air temperature at the non-port side of the switch does not exceed the listed

rating of the switch.

Verify air flow is available to the nonport vents.


47/64


SAN Overview - 47

The exact installation procedures vary among the different cabinet systems. Always review the

product documentation prior to installation.

Assemble the two side slide assemblies.

Install the inner slide rails on the switch chassis:

Position one of the inner slide rails with the flat side against the switch, oriented as needed for

slide removal.

Attach the rail using the three Phillips head 8-32 x 3/16 inch screws provided with the

installation kit.

Tighten the screws to a torque of 15 inch-pounds.

Repeat with the second inner slide rail on the other side of the switch.

Verify that two of the slide mount L-brackets are attached to each outer slide rail. These screws

should be at a torque of 15 inch-pounds.

Install the power cord clips onto the outer slide rails. The clips must be installed on the outersurface of the slide rails so that they do not interfere with rail movement.

Insert the power cords into the power cord clips with the pronged end of the power cords toward

the cool air aisle for access to the power source.

Install the outer slide rails in the cabinet.

Install the switch in the cabinet.

WarningWarning: Using screws longer than the specified length can damage the switch.


48/64


SAN Overview - 48

Installation of the SFP transceivers is the final step:

Remove the shipping plugs from the ports to be used.

The transceiver is keyed so that it can be inserted into the port only with the correct orientation. If

it does not slide in easily, ensure it is correctly oriented.

Position a transceiver so that it is oriented correctly and insert into the port until it is firmly seated

and the latching mechanism clicks. For instructions specific to the type of transceiver, refer to the

transceiver manufacturers documentation.

Repeat for the remaining ports as required.


49/64


SAN Overview - 49

Connect the power cord to one or both of the power supplies and to a power outlet. Ensure that the

power cord has a minimum service loop of 6 inches available at the connection to the switch, and is

routed so that it is not exposed to stress. Turn on the power supply by flipping the AC switch to 1.

The switch is powered on and begins running the POST. Repeat these steps for the second powersupply.

For optimal redundancy, ensure that separate circuits are provided for each power supply.

When the switch is powered on, it automatically runs the POST to guarantee switch stability. Errors

that occur during POST are written to the system error log. Verify that the POST completes

successfully.


50/64


SAN Overview - 50

Connect the serial cable to the serial port on the switch and to an RS-232 serial port on the

workstation. If the serial port on the workstation is RJ-45 instead of RS-232, you can remove the

adapter on the end of the serial cable and insert the exposed RJ-45 connector into the RJ-45 serial port

on the workstation.

The serial cable must be a pass-through serial Cable. Null Modem cables establish a connection

but do not transfer data or commands. It is recommended that you use your laptop to configure

these switches through HyperTerminal.


51/64


SAN Overview - 51

Change the IP address, subnet mask, and gateway address as directed by the customers network

administrator.

Verify the address was correctly set by entering the ipAddrShow command at the prompt. Record theIP address on the label provided for this purpose on the port side of the switch. Once the IP address is

verified as correct, log off of the serial console, remove the serial cable, and replace the shipping plug

in the serial port.


52/64


SAN Overview - 52

When configuring the switch using the telnet, Web Tools or other out-of-band management systems,

you must be aware of LAN configuration. Devices and applications such as routers, firewalls, proxy

servers, DNS, DHCP could interfere with network connectivity between the managements station and

the switch.

In the event of a failure to communicate, some simple tests should be performed. To determine if

connectivity is possible, begin with the basic TCP/IP test of PING . If this is

successful you have an IP connection. For higher level applications to function you may next need to

test for Port availability and LAN timeouts. Commands such as TRACERT, NSLOOKUP and

NETSTAT are useful to determine the network configuration between station and switch.


53/64


SAN Overview - 53

Before you install Web Tools on your workstation, verify that your switches and workstation meet the

Web Tools requirements listed in the current product documents. Web Tools requires any browser that

conforms to HTML version 4.0, JavaScript version 1.0, and Java Plug-in 1.6.0 or higher.

If your fabric includes a switch with a limited switch license and you are opening Web Tools using that

switch, if the fabric exceeds the switch limit indicated in the license, Web Tools allows a 30-day grace

period in which you can still monitor the switch through Web Tools. However, Web Tools displays

warning messages periodically.

These messages warn you that your fabric size exceeds the supported switch configuration limit and

tells you how long you have before Web Tools will be disabled. After the 30-day grace period, you are

no longer able to open Web Tools from the switch with the limited switch license if that switch is still

exceeding the switch limit.

Web Tools is part of the Fabric OS of a switch. When you open Web Tools on a switch, you can

manage other switches in the fabric that have lower or higher firmware versions. It is important to notethat when accessing these switches you are opening the switchs version of Web Tools, and the

functionality available for those switches might vary.


54/64


SAN Overview - 54

To perform the initial login into a switch:

Verify that the switch is connected to your IP network through the RJ-45 Ethernet port to enable

connection through telnet.

Open a telnet connection to the switch. The login prompt is displayed if the telnet connection

successfully found the switch in the network.

Enter the user ID (usually user or admin) at the login prompt.

login: admin

Enter the default admin password. The default password is password. You are prompted to change

the password.

Enter the new password.

Verify that the login was successful.

A prompt is displayed showing the switch name and user ID to which you are logged.


55/64


SAN Overview - 55

It is important to have consistent system configuration settings since inconsistent parameters

amongst switches in the same fabric can cause fabric segmentation. To display and check system

configuration settings, perform the following procedure.

Fabric parameters include all the items listed in the configure command. Fabric parameters

must be identical for each switch across a fabric.

To save time when configuring the fabric parameters:

Configure one switch first (using the configure command.

Use the configUpload command to save the configuration information.

Use the configdownload command to download it onto each of the remaining switches.

The default Domain ID is 1. If the default domain ID is already in use when the switch is connected to

the fabric, the Domain ID for the new switch is automatically reset to a unique value. The Domain IDs

that are currently in use can be determined using thefabricshow

command.Domain IDs are assigned dynamically when a switch is enabled. However, the Domain ID can be set

manually in order to control the number, or to resolve a Domain ID conflict when merging fabrics.

Valid Domain IDs are 1 through 239.


56/64


SAN Overview - 56

Switch Date and Time - All switches maintain current date and time in non-volatile memory. Date and

time are used for logging events. Switch operation does not depend on the date and time; a switch with

an incorrect date and time value still functions properly.

Switch Status Policies - The policy parameter determines the number of failed or non-operational

units for each contributor that will trigger a status change in the switch.

Each parameter can be adjusted so that a specific threshold must be reached before that parameter

changes the overall status of a switch to MARGINAL or DOWN. For example, if the FaultyPorts

DOWN parameter is set to 3, the status of the switch will change if 3 ports fail.

Tracking Switch Changes - The Track Change feature allows you to keep record of specific changes

that may not be considered switch events, but may be useful. The output from the track changes feature

is dumped to the error log for the switch. Use the errdump command or errshow command to view the

error log. Items in the error log created from the Track changes feature are labeled TRACK. An

SNMP-TRAP mode can also be enabled.

Routing - In a stable fabric, frames are always delivered in order, even when the traffic between

switches is shared among multiple paths. However, when topology changes occur in the fabric (for

instance, a link goes down), traffic is rerouted around the failure. When topology changes occur, some

frames could be delivered out of order.


57/64


SAN Overview - 57

Switch Name - You can customize the switch names for the logical switches. If you chose to change

the default switch name, use a switch name that is unique and meaningful. Changing the switch name

causes a domain address format RSCN to be issued.

Switch names:

Can be up to 15-characters long

Must begin with an alpha character

Can consist of any combination of alphanumeric and underscore characters


58/64


SAN Overview - 58

Licensed features such as Fabric Watch and Performance Monitoring are already loaded onto

the switch firmware, but must be enabled with a license key. Once you have purchased these

features, you are provided with a key to unlock the features in the firmware.

You must log on to PowerLink or contact EMC Customer Support to convert the transaction key to a

license key.

Verifying License Activation

To verify that the required licenses are activated on the switch, perform the following steps:

Log into the command line interface as the Admin user.

Enter the licenseshow command at the command line prompt.

A list displays all of the licenses currently activated on the switch.


59/64


SAN Overview - 59

Configure the software features (such as Fabric Watch, Zoning, and Secure Fabric OS) for each switch.

To save time you may configure the software features on one switch, then save the configuration file,

and download it to the each of the remaining switches.


60/64


SAN Overview - 60

Power off all devices (to minimize PLOGIs) and connect them to the switch, according to your

topology. For devices that cannot be powered off, connect the devices, but use the portdisable

command to disable the port on the switch. When powering the devices back on, wait for each

device to complete the fabric login before powering on the next one.

EMC recommends that you create a blocking zone config and load that to the fabric before connecting

devices. More information on this is presented in the B-Series Zoning module.

To verify that you have fabric wide device connectivity, display the fabric wide device count. To

display the fabric-wide device count from a switch:

Log in to the switch as the admin user.

Enter the nsallshow command at the command line.

This command displays 24-bit fibre channel addresses of all devices in the fabric.


61/64


SAN Overview - 61

To verify that your switch is operating correctly, display information about the switch and port status.

To display information about the switch and port status:

Log in to the switch as the admin user.

Enter the switchshow command at the command line. This command displays a switch summary

and a port summary.

Verify that the device is connected to the switch by entering the nsshow [-r] command.

Use the -r option to replace the TTL attribute output with SCR (state change registration) information

in the display. SCR is the state change registration of a device. This value indicates what type of RSCN

a device registers to receive.


62/64


SAN Overview - 62

To verify that you have fabric-wide switch connectivity, display a summary of information about the

fabric.

To display a summary of information about the fabric:1. Log in to the switch as the admin user.

2. Enter the fabricshow command at the command line. This command displays a summary of

all the switches in the fabric.


63/64


SAN Overview - 63

It is recommended that you make a hard copy backup of all key configuration data, including license key

information for every switch, and store it in a safe and secure place for emergency reference. Refer to the

following procedures.

Print out the information from the licenseshow command and store in a secure location. Print out the information from the configUpload command (contains license and configuration information)

and store in a secure location.

Print out the information from the ipaddrshow command (select option 4 to display all configured

addresses) and store in a secure location.

Depending on the security procedures of your company, you may want to keep a record of the user levels and

passwords for all switches in the fabric. This is sensitive information and access to such information should be

limited. Save all key configuration data, including license key information for every switch and upload it to a

host for emergency reference.

The configuration file is written as three sections, and is broken up as follows:

The first section contains the switch boot parameters. It has variables such as the switch's name and IPaddress. This section corresponds to the first few lines of output of the configshow command.

The second section contains general switch configuration variables, such as diagnostic settings, fabric

configuration settings, and SNMP settings. This section corresponds to the output of the configshow

command (after the first few lines), although there are more lines uploaded than shown by the command.

The third section contains zoning configuration parameters.


64/64


These are the key points covered in this module. Please take a moment to review them.

01 san overview

Documents