troubleshooting cisco storage networks - net130.com storage networks session opt-3050. ... switch...

Copyright © 2001, Cisco Systems, Inc. All rights reserved. Printed in USA.8021_05_2003_c2

111© 2003, Cisco Systems, Inc. All rights reserved.OPT-30508021_05_2003_c2


TroubleshootingCisco Storage Networks

Session OPT-3050



Session Agenda

• Fibre Channel Ports—Physical Level and Type

• Focus on Device Discovery and Fibre Channel Services

• E-Port ISL Troubleshooting, Including FCIP

• SN5428 Specific Troubleshooting

• PA-FC-1G Specific Troubleshooting

• MDS9000 Specific Troubleshooting

• iSCSI Driver Troubleshooting


Fibre Channel Port Fundamentals



Fibre Channel Port Identification

FC Switch

FC Switch

FC-AL Hub

NL

NL

FL

E

E

N

F

NN

B_Port and TE_Ports B_Port and TE_Ports Will Be Covered in Will Be Covered in Later SectionLater Section

Can Be a G, E, F, FLCan Be a G, E, F, FLU_PortU_Port

G_Port With Loop CapabilityG_Port With Loop CapabilityGL_PortGL_Port

F_Port with Loop CapabilityF_Port with Loop CapabilityFL_PortFL_Port

N_Port with Loop CapabilityN_Port with Loop CapabilityNL_PortNL_Port

Port with Both E and F Port CapabilityPort with Both E and F Port CapabilityG_PortG_Port

Ports for Interconnect of SwitchesPorts for Interconnect of SwitchesE_PortE_Port

Port on the FabricPort on the FabricF_PortF_Port

Any Port on a Node DeviceAny Port on a Node DeviceN_PortN_Port


Fibre Channel Transmission Words

• Ordered sets that are transmitted continuously to indicate that specific conditions within the link are encountered

• Transmitted while the condition exist

• Four primitive sequences

Not Operational Sequence (NOS)

Offline Sequence (OLS)

Link Reset Sequence (LR)

Link Reset Response Sequence (LRR)



Link Initialization Flow

These Are All Special Ordered Sets of 8B/10B Coding:

NOS = Not Operational Sequence

OLS = Offline Sequence

LR = Link Reset

LRR =Link Reset Response

AC = Active State

LR = Link Recovery State

LF = Link Failure State

OL = Offline State

Link Failure ConditionLink Failure ConditionAC ACAC

LFLF

LR

OL

LR

AC

AC

NOS

OLS

LR

Idle

LRR

Idle

Port A Port B


Port Type Connection Protocol

• What port types use what sequences

N_Port—Uses NOS/OLS to connect

F_Port—Uses NOS/OLS to attach to single N_Port

FL_Port—Uses LIP to attach 1 to 126 NL_Ports

E_Port—Uses NOS/OLS to interconnect switches (Inter-Link Switch ISL)

B_Port—Same as E_Port but only bridges (FCIP)

G_Port—Uses NOS/OLS can be a F or E port

U_Port—G, E, F,FL; it basically is a transition state

MDS9000 Also Has TE Port and SD Port, Covered in MDS Section



Where You Find Link Level Issues

MDS9000 show int fc PA-FC-1G show fc fc-stats


Where You Find Link Level Issues

SN5428 show interface fc# stats



Other Common Link Level Issues

• GBIC or SFP issues

MDS9000# show interface fc x/x transceiver (SFP)

SN5428-2# show interface brief

• MM optical cable issues

Check for light on TX of FC port and end of cable

• Differences in optics

Install qualified GBIC or SPF; most SFP’s will work for both Fibre Channel and GiGE, MM vs. SM, LW vs. SW


Agenda—Next Subject





Device Discovery and Fibre Channel Services


World Wide Names

• Each switch element is assigned a 64 bit WWN at time of manufacture

• Each switch port is assigned a 64 bit WWPN at the time manufacture

• During Fabric Logon (FLOGI) the switch identifies the WWN in the service parameters of the accept frame

These Address Assignments Can then Correlate Each Fabric Port with Switch Routing and the

Fiber Channel ID (FCID)



FC World Wide Name Addressing

2 Port HBA “Node WWN”20 00 00 01 64 40 da 20

Disk Drive

“Port 1 WWPN” 21 00 00 01 64 40 da 20“Port 2 WWPN” 22 00 00 01 64 40 da 20

40:da:2000:01:640:002

Single Hex Digit Selected by Vendor1= ieee Standard2= ieee Extended

3 HEX Digits Used by Vendor to Identify Port

IEEE Assigned Vendor Assigned

“Port 1 WWPN” 21 00 00 20 37 a7 0d d8 “Port 2 WWPN” 22 00 00 20 37 a7 0d d8

“Disk WWN” 20 00 00 20 37 a7 0d d8


Fabric Addressing—FCID

• The 24 bit address is partitioned into 3 fieldsDeviceArea (depends on vendor)Domain

• This partitioning is used in FSPF routing as source and destination ID’s

• The Fabric Login Server at address ‘FF FF FE’ assigns the address to the connecting N_Port

DeviceAreaSwitch Domain

8 Bits 8 Bits 8 Bits



Name Server

• Repository of information regarding the components that make up the Fibre Channel network

• Located at address ‘FF FF FC’

• Components can register their characteristics with the Directory Server

• An N_Port Host can query the Directory Server for specific information


Name Server Command Requests

• Get objects from directoryGA_NXT—Get all next

GID_FT—Get FCID

• Register objects in directoryRFT_ID—Register FC-4 types

• Deregister objectsDA_ID—Deregister all

These Are Some of the More Commonly Used Commands Used to Talk to the Name Server:



Fabric Controller

• Each switch has a fabric controller

• Assigned address ‘FF FF FD’Every fabric controller in the fabric has the same addressResponsible for managing fabric, initialization, FSPF routing, responsible for RSCN’s and listens for SCN’s

• Responsible to receive request and generate responses for the switch fabric

• The MDS9000 also has the Fabric Domain Controller at address FFFCxx where xx=vsan domain number; this is used for LUN discovery


Fabric Login

Fabric F_PortX’FF FF FE’

64-bit Node WWN64 Bit Port WWN

FLOGI

Accept

Registration Information

PLOGI

Accept

Registration for State Changes

SCR Fabric ControllerX’FF FF FD’Accept

Name Server Query

Accept

Address Discovery for SCSI Storage

Devices

Host System N_Node

By Registering for State Change Notifications the Port or Domain Controller Can Then Notify the Connected Ports with RSCN’s when a Port Changes

Distributed Name Server Database

Name ServerX’FF FF FC’


24-bit port ID



Host

24-Bit Address AssignmentHost Connected to Port 3 on Switch

Switch Domain ID = 5

FLOGI

Accept

24 Bit Port ID Assigned Is x”051300’ Host Connected

to Port 4 on Switch

Switch Domain ID = 1 Storage

Array

Accept

FLOGI

24 Bit Port ID Assigned Is x”011400’


Port to Port


Host on Port 3 of Switch

Storage on Port 4 of Switch

ID 5

ID 1

DBName Server Query

Get FC-4 DevicesX’011400’

D_ID x’011400 S_ID x’051300 Accept

D_ID x’051300 S_ID x’011400

D_ID x’011400 S_ID x’051300

SCSI Inquiry Report LUNS

Presents LUN’s to Host

FSPFRouting

PLOGI

PRLI

Accept








E-Port/B-Port ISL Troubleshooting



E-Port Troubleshooting

• Differences between E-Port and B-Port

• Understanding the E-Port inter-switch Link Connection (ISL)

• Troubleshooting domain

• E-Port and B-Port fabric connection issues


VB-Port

Exchange FCIP-Link Parameters

Exchange FCIP-Link Parameters

IP NetworkFC SAN

FCIP E-Port and B-Port Relationships

FC Switch

FCIP E-Port FCIP E-Port

Exchange Link Parameters


B PortOperation

B PortOperation

FCIP B-Port FCIP B-Port 7200 w/

PA-FC-1Gor SN5428-2

7200 w/ PA-FC-1G

or SN5428-2

Exchange Fabric Parameters

ESC (Exchange Switch Capabilities)

FC SAN

FC Switch

FC SAN

FC Switch

FCIP E-Port

Exchange FabricParameters


ESC

E PortOperation

E PortOperation

FC SAN

FC Switch

Exchange LinkParameters


Exchange FCIP-Link ParametersExchange FCIP-Link Parameters


IP Network

VE-Port

ESC ESC

FCIP E-Port



Understanding the ISL Connection

• E_Port and Cisco MDS TE_Port

Link initialization

Exchange link parameters

Link Reset

Exchange switch capabilities

Exchange port parameters—MDS9000 only to determine if connected to another MDS

Exchange fabric Parameters

Assign domain IDs

Establish routes

Merge zones if required

• B_Port

Link initialization

Exchange link parameters

Link Reset


E/TE Port Link Exchange Explained

Exchange Switch CapabilitiesExchange Switch Capabilities

Exchange Port ParametersExchange Port Parameters

Exchange Port ParametersExchange Port Parameters

LR to Initialize Flow Control Buffer Credit Parameters

At This Point a B_Port Connection Is Up and E_Port to E_Port Exchange Continues

Exchange Link ParametersExchange Link Parameters

Primitive Sequences




Exchange Fabric ParametersExchange Fabric Parameters


Domain Identifier AssignedDomain Identifier Assigned

Request Domain IDRequest Domain ID



Merge RequestMerge Request

Merge Request—ZoningMerge Request—Zoning



At This Point FSPF Continues to Update Link State Information Between the Switches

FSPF HelloFSPF Hello




E_Port, TE_Port Connection Problems

1. Trunk E-Port configuration mismatchFabric parameters and timers mismatch switch

2. Zoning parameters mismatch per VSAN or switch

3. VSAN configuration mismatch (MDS9000)

4. Domain IDs must not conflict on either VSAN or switch


E-Port, TE_Port Troubleshooting

1. Isolation due to ELP (Exchange Link Parameters) failure

ED_TOV Timer

RA_TOV Timer

Frame size

FS_TOV timer

2. Isolation due to zone merge failure

Same name used to describe 2 different types or content mismatch in the merging switches

If the same zones on 2 different switches contain different members

3. Isolation due to port VSAN mismatch

E_ports not in same VSAN

MDS9000# sh vsan membership

Output of Good TE Port ISL

Error Indication Will be Displayed HereError Indication Will be Displayed Here



E-Port TroubleshootingDomain Assignment Failure

• Each VSAN or FC Switch has its own domain ID, what happens when they are the same?

• If both switches are in production network, there are two ways to resolve the issue but both are disruptive to the FC network

E_Port becomes isolated because the domain ID’s overlap on the 2 switches

New switch being added to the network will be assigned an ID if not managed with one

1. A Cisco MDS9000 can be configured to force a RCF (Reconfigure Fabric); this will force a reselection of Domain ID’s and resolve the issue

2. Take the switch offline and manually configure Domain ID and re-enable switch


Specifications that Define Operations

• ANSI: http://www.t11.org/index.htmFC-SW-2 describes the operation and interaction of Fibre Channel switches, including E_Port, B_Port and Fabric operation

FC-BB-2 is a mapping that pertains to the extension of Fibre Channel switched networks across a TCP/IP network backbone and defines reference models that support E_Port and B_Port

• IETF IPS Working Group:Fibre Channel over TCP/IP covers the TCP/IP requirements for transporting Fibre Channel frames over an IP network

FC frame encapsulation defines the common Fibre Channel encapsulation format









SN5428 Specific Troubleshooting



SN5428 Troubleshooting

• Debug uses

• iSCSI troubleshooting

• SN5428 Fibre channel switch troubleshooting—E-Port

• FCIP B-Port troubleshooting


SN 5428 Functional Blocks

GE1 GE2 FC7 FC8FC6FC5FC4FC3FC2FC1ConsoleMgmt10/100

HA10/100

FC Switch

SCSI Routing Engine

Control Plane and Management

• Two internal FC ports allow access between the routing engine and FC switch ports

Interfaces fci1 and fci2 will look like Initiators logged into the nameserver

fci1 fci2

fc15fc0



SN5428 Debugs

• Depends on which support mode the SN5428 is in

iSCSI router mode—debug scsirouter

FCIP mode—debug FCIP

• Both modes have debug for GigE interface

• Debugging different from typical Cisco IOS

• Debugs are enabled and captured to a circular file by default

• Captured debug information is then displayed via show commands


SN5428 Data Flow Debugs

• [SN5428A]# debug scsirouter scsirouternameiscsitrace iscsi trace facility (debugs iSCSI PDU on IP side)

scsitrace Enable SCSI trace (debugs scsi CDB, FC side)

target Set variables in a SCSI Router (singles out target)

• [SN5428A]# debug scsirouter scsiroutername iscsitrace Enable Enable iscsi trace facility (always typed at end of

debug string)

fromto Trace PDUs from/to specified host/network

pducnt Set max number PDUs to trace

pdusize Set PDU trace size



SN 5428 Functional Blocks

GE1 GE2 FC7 FC8FC6FC5FC4FC3FC2FC1ConsoleMgmt10/100

HA10/100

FC Switch O/S

SN5428 O/S

Control Plane and Management

Show debug rawlundatabase

Show debug mailboxtrace

Show devicesShow devices rediscover (clears this table and forces a a GA_NXT)

Port Database

Fibre Channel Device

Discovery Table

Show interface FCx

Show interface GEx

T2 Targets with LUN’s

Show debug scsirouter all scsitrace

300 Line Circular Buffer; Latest

Entry at Bottom

fci1 fci2

fc15fc0

T1iSCSI PDU and CDB Global Trace


show debug rawlundatabase allEntry Address = 0x7595b6c

fabricLoginFailureCode=0x0,fabricLoginExtendedCode=0x0,fabricLoginTimeoutCode=0x0

ReportLunsLLDStatus=0x0,ReportLunsLLDStatusModifier=0x0,ReportLunsSCSIStatus=0x0,ReportLunsASCASCQ=0x0,ReportLunsLunCount=1

InquiryLLDStatus=0x0,InquiryLLDStatusModifier=0x0,InquiryLastLunWithLLDError=0x0,

InquirySCSIStatus=0x0,InquiryASCASCQ=0x0,InquiryLastLunWithSCSIStatusError=0x0

boolLunsNotSupported=0x0,InquiryLastLunNotSupported=0x0

loopId=0x1,masterState=0x6,slaveState=0x7,loggedIn=0,roles=1,valid=1,portId=0xe8,scanLuns=0x0

numberLuns=0x1,reportAsyncEvent=0x0,node_wwn=0x20000020 0x37a70dd8, port_wwn=0x21000020 0x37a70dd8

lun=0, wwnn=0x20000020 0x37a70dd8, reportAsyncEvent=0x0

stdInquiry data for lun=0x0

bytes0-7=0x00000332 0x8b00500a

vendorId=SEAGATE , product=ST318451FC , revision=0001 device Type=0x0

DeviceIdPage:bytes0-3= 0x0083000c,bytes4-7= 0x01030008,bytes8-11= 0x20000020

:bytes12-15= 0x37a70dd8,bytes16-19=0x00800014,bytes20-23=0x33434330

S/N Page:bytes0-3= 0x00800014 s/n=3CC02AVP00007107EPTS

FC Device Debugging

• This is a complete output of all LUN discovery that is done by the Fibre Channel firmware in the SN5428, the report LUNs is done to all WWPN’s returned to the SN542X after a query to the directory server

WWPN that WWPN that Inquiry Is Being Inquiry Is Being

Done ToDone To

LUN Count LUN Count ReturnedReturned

LUN Is LUN 0, and This Is the LUN Is LUN 0, and This Is the Data Return from the Report Data Return from the Report

LUN CommandLUN Command

WWN on the WWN on the LUN PortLUN Port



GigE Packet Tracing

Debugging A Little Different from IOS Type Outputs

Debug Is Turned On, But Output Is Directed to a File

Debug Monitoring Is Done with Showing of Stats

Debug Is Halted With “No”

Show Command Is Used to Display the Captured Data

The Output From the Captured File Is Saved and Can Be Shown Until Another Debug Over Writes the File

Ascii Output Can Be Selected for Readable Form


iSCSI PDU Debugs

• Details of iSCSI PDU’s

• ascii readable

• Use pdusize 256 to read ascii

• Use stats to get packet capture status

• Trick is to use keyboard repeat keys to turn debugs off

Readable Exchange

Data



Troubleshooting SN5428 FC Switch

• [SN5428-2]# show fcswitch Eport Display of eport FSPF information

fabric Display of domains in the SAN Fabric

global-nameserver Complete SAN wide nameserver entries including host devices

linkstate Display linkstate transitions

nameserver Displays this FC switch target devices


SN5428 Outputs

Show interface fci2 stats sh debug fcswitch all—Will Give Log Output

…List of Logged Messages Are Output Here…List of Logged Messages Are Output Here

Frame Counts and Errors on the Internal FC Interface



SN5428 Fibre Channel Switch E_Port to E_port Troubleshooting

SN5428 Domain 1

MDS9216 Domain 101 (0x65)

Port FC 3

Internal SN5428 WWPN’s

Port fc 1/1E-Port ISL


SN5428 E_Port to E_port

SN5428 Domain 1


Port FC3

Port FC2

Loop Targets on JBOD

HBA Attached Host

E-Port ISL



Interrogating E_Port Connection Via FCIP B_Port

SN5428 Domain 1


FCIP Link

FCIP B_Port on SN5428 is Sourced from Port 255 Internally; the Remote E_Port Connection is Sourced Via Port 0 Internally

SN5428 internal WWPN’s Will Not be Shown in FCIP Mode

FCIP Port 0


Interrogating E_Port ConnectionVia FCIP B_Port

Devices e4 and e5

SN5428 Domain 1


FCIP Link

FC Port 2 FCIP Link Info










C7200 and PA-FC-1G Specific Troubleshooting



PA-FC-1G FCIP Troubleshooting

• Port adaptor configuration troubleshooting

• Debugging of PA-FC-1G

• Debugging the E-Port connection to the Fibre Channel Switch


PA-FC-1G

• Example of FCIP connectivity between McData and Brocade using the PA-FC-1G Port Adapter

Local C7200

Remote C7200



FCPA Configuration

Local Side Tunnel Remote Side Tunnel

interface Fcpa1/0

ip address 4.4.4.1 255.255.255.0

fc-tunnel Local

src-ip 4.4.4.2

dest-ip 3.3.3.2

src-port 3000

dest-port 2000

tcp sack

tcp mws 256

tcp kad 7200

ip tos 0

inservice

interface Fcpa2/0

ip address 3.3.3.1 255.255.255.0

fc-tunnel Remote

src-ip 3.3.3.2

dest-ip 4.4.4.2

src-port 2000

dest-port 3000

tcp sack

tcp mws 256

tcp kad 7200

ip tos 0

inservice


PA-FC-1G B_Port Initiation

• B_Port initialization can be divided into roughly three stages:

1. Link initialization

2. Exchange Link Parameters (ELP frames)

3. Link reset

• Most typical problems are issues with ELP exchange• PA-FC-1G only runs at 1 GIG, speed, connected Fibre Channel

Switches need to be fixed at 1 GiG (no auto speed)• Connecting Fibre Channel switches may need to run in SW-2

compatibility mode if different vendor types• Best practice is to connect Fibre Channel Switch E_Ports together

first prior to connection to PA-FC-1G FCIP connection



PWWN of B_Port Interface on PA-FC-1G

PWWN of E_Port Interface on Connected FC Switch; This

Will be Blank Until Successful ELP Acknowledgment

WWN of on Connected FC Switc; This Will be Blank Until

Successful ELP Acknowledgment

BB Credits the PA-FC-1G Has

Available to Send Frames to Connected FC Switch E_Port

Up = FC B_Port Had Successful ELP with

E_Port

Session Manager State

Timeout Values Set on B_Port

Interface

Set TCP Values Per Config

Configuration is Correct Between

Local FCPA Interface and

Local FC Tunnel

Highest Support MTU

MSS Value Negotiated After Tunnel Is

Inservice

Show fc-tunnel Detail—First Page

This Is Basically the Fibre Channel Interface and B_Port

Connection

This Is Basically the Fibre Channel Interface and B_Port

Connection


Sh fc-tunnel Detail—Second Page

Credits and Timers Settings on E-Port of FC Switch (Learned at ELP)

Buffers Credits for Class-F

Frames; Ack’ed from Remote End

Successful Exchange Link Parameters Completed,

1=Yes, 0=No, 1 Needed for FC Interface to Come Up

Echo Frames Enabled, This Is an Extended Link Service Echo Command Sent by the Port to

the Remote FCIP Link End Point (B-Port); This is Monitored by

TCP for a Connection Keepalive

Forced at 1 GiG Today

TOV Parameters That Are Hard Configured within the

B-Port Interface

BB Credits Set Based on Switch Vendor ID



show fc tcpconn

• Used to monitor performance issue related to WAN transport

7200#sh fc tcpconn

Interface: Fcpa6/0

TCP variables for Connection Fcpa6/0:

=======================================

snd_una: 2861918, snd_nxt: 2862482, snd_up: 2715310, snd_wnd: 65535

snd_max: 3124058, snd_cwnd: 92224, snd_retx_max: 2861918, snd_ssthresh: 2176

t_unack_datasz: 564, t_flags: 612, t_state: TCPS_ESTABLISHED,

rcv_wnd: 262140, rcv_nxt: 4152717, rcv_adv: 4414857

TCP Timers: REXMT=0 PERSIST=0 KEEP=7189 2MSL=0TCP State

Window Sizes


Debugs for FCPA

• Many different debugs to run• All are pretty safe except cordova-driver has a

chance to cause the 7200 to choke• Most issues are at install with connection to Fibre

Channel Switch7400#debug fc ?all All Modules

cli cli Module

cordova-driver Cordova Driver Module

fcap FCAP Module

fd FD Module

northstar-driver Northstar Driver Module

sm Session Manager Module

tcp TCP Module



A Look at Debug FCPA FD all

Inservice (same as no shut)Fcpa1/0: TCP Connection Established!

*Sep 5 02:08:09.658: GG_FCPA_FD_EV: Fcpa1/0 fd_process_tcm_message(): SM2FD_CONN_OPENED_EV - TCP connection success

*Sep 5 02:08:09.658: GG_FCPA_FD_EV: Fcpa1/0 fd_process_tcm_message(): Initializing CPU credits to 32

Link Reset Rx state +++ LR2 +++

Link Active state - +++ AC +++

Link Initilization Completed

*Sep 5 02:08:10.494: GG_FCPA_FD_EV: 9.782: Fcpa1/0 fd_process_bprt_fc_frame (): ELP frame !

*Sep 5 02:08:10.526: GG_FCPA_FD_EV: Fcpa1/0 fd_process_ns_pkt_msg (): ID_FCINIT - 0xBC49BF49 – LR received

Link Reset Rx state +++ LR2 +++

Link Active state - +++ AC +++

Link Reset completed

Link offline Rx wait state ++ OL3 +++

Link failure Rx state +++ LF1 +++

Fcpa1/0: B-port is UP!

SW_ACC frame from GiGE

SW_ACC frame from FC

CPU_CREDITS: 24 adding 8

Indication ELP Exchange Was

Good

F-Class Frames Established Across the

FCIP Tunnel

FC Link Initialization for Buffers

This Link Reset to Establish FC Credit

Buffers for Data

Primitive Sequences for Physical Link Connection

IP Tunnel Is Good Configuration Good


Issues with Fibre Channel Connected Switch

• B-Port connections require Fibre Channel switches to be configured in SW-2 Interop mode if not the same vendor type

• Fibre Channel Switch options specific to that vendor may not be used i.e. proprietary vendor interworking settings

• All E-Port issues discussed in the E-Port troubleshooting section apply when working with B-Port connections







• SN5428 Specific Troubleshooting• PA-FC-1G Specific Troubleshooting



MDS9000 Specific Troubleshooting



Section AgendaMDS 9000 Troubleshooting

• Using Debug Serviceability Tools

• Switch to Switch Fabric Connectivity

• Storage and Host Discovery, Distribution across the SAN

• Troubleshooting MDS9000


Gathering Traces for Analysis

• Using built-in FCAnalyzer

• Using Ethereal

• Using the MDS Port Analyzer Adapter

• Using Span

• Using an external FC Analyzer



MDS FCAnalyzer

• Output is displayed to the console in readable format

• Is only used to monitor Fibre Channel Traffic to and from supervisor on the MDS9000

• Span is used for FC port to FC port monitoring


FCAnalyzer Detail Trace

• This is the command entered at the MDS CLI (while in config mode) to start the FCAnalyzer trace; here we show how to filter for VSAN 1 traffic only

MDS-9509(config)# FCanalyzer local display-filter (mdshdr.vsan==0x01)

• Other MDS built in filters are found in Chapter 23 on MDS9000 configuration guide

• FCAnalyzer is turned off after output has been displayed to your screen and user exits configuration mode



FCAnalyzer—Brief

eth2 Is an Internal Port Associated with the Analyzer

Fiber Channel Traffic Is Seen Going to and from Fabric Login Controller at Address FE (FLOGI)

Connecting Device 7e0000 Registers for State Changes and Gets Response

PLOGI’s and Name Query’s to the MDS Nameserver Address FC

Device Logs into MDS Platform Management Server

Process Login’s to Device on Switch 7e Port 00 Address 00 Are Captured and MDS Logs Out

eth2 Is an Internal Port Associated with the Analyzer

Fiber Channel Traffic Is Seen Going to and from Fabric Login Controller at Address FE (FLOGI)

Connecting Device 7e0000 Registers for State Changes and Gets Response

PLOGI’s and Name Query’s to the MDS Nameserver Address FC

Device Logs into MDS Platform Management Server

Process Login’s to Device on Switch 7e Port 00 Address 00 Are Captured and MDS Logs Out

MDS Fabric Processes Login from the Device and then Will Also Log into Attached

Device and Query for Device Type and LUN’s


Fibre Channel Capture Breakdown

Destination is Switch Fabric Login Server

From Local Port

Fabric Login

Class 3 Device Parameters

WWN’s of MDS 00:10:73 = Vendor ID of Attached Device a JNI HBA



Ethereal—Protocol Analyzer

• Rich functionality:

GUI front-end—runs on Linux, Windows, Solaris and many other OSs

Filter-based capture and display

Minimal protocol statistics

Colorized display

Find frame, mark frame, read captured traces

Customizable columns

• Supported in Ethereal version 0.9.9 and higher


What Ethereal Can Do

• Decodes: NIC Adapter’s encapsulation header

All FC protocols

SCSI (SPC-2, SBC-2, SSC-2)

VSAN-aware

Matches response with request and so can decode responses completely



What Ethereal Cannot Do

• Industry-strength wire-speed captureEthereal’s speed is limited by machine/OS limitations

• No decoding of FC-1 including:No 10bit/8bit encoding

No ordered sets

• Works with MDS’s SPAN functionalityCannot act as a bump-in-the-wire like other FC protocol analyzers


MDS FCAnalyzer Remote with Ethereal

• For capture of internal MDS9000 switch traffic, FC traffic to and from supervisor not FC port to FC port traffic

• Make sure MDS can ping workstationIPS-TEST(config)# fcanalyzer remote 10.1.10.78 active

IPS-TEST# sh fcanalyzer

ActiveClient = 10.1.10.78, DEFAULTIPS-TEST#clear fcanalyzer turns off analyzer

At DOS Prompt:ethereal -i rpcap://10.1.10.76/eth2Or at Ethereal Interface Drop-Down:rpcap://10.1.10.76/eth2

Workstation Running EtherealIP Address 10.1.10.78

FCFC

Management 10/100 Port on MDS Address 10.1.10.76SCSI

HBAHBA



Use of Span Feature

• Used for FC Port to FC Port analyzing

• Same type of tool as used on Cisco Catalyst products

• Ingress and egress ports are sent to a FC port set up as a Span Destination (SD port type)

• No limits to where the ports are located on the MDS switch

• Used to output to third party test equipment or to Cisco Port Analyzer Adapter


Source Types Supported by SPAN

• Physical ports—F, FL, TE, E

• Port channelsAll ports in a port channel are configured for SPAN

Same filters are applied to all the ports

• VSAN—traffic is ‘spanned’ only in ingress directionAll FC ports in a VSAN are included as a source

• Two sessions can share a source port but in different directions (ingress/egress)



MDS 9000 Port Analyzer Adapter

• MDS Port Analyzer Adapter is a protocol converter

Converts FC frames into Ethernet frames

FC-specific information is carried in encapsulation header

Enables use of ethernet protocol analyzers to decode FC protocols

• Cisco has released decodes for all FC protocols, the encapsulation header and SCSI decoder to public domain

Part of the popular Ethereal protocol analyzer


Span Configurationwith Single Port Analyzer

Ingress to the Supervisor

Egress from the Supervisor

Span Destination Port (SD)

FC1/1FC 2/16

FC 6/8

FC Analyzer

TX TXRX RX

Dropped



Port Analyzer Connection Using Port Analyzer with Ethereal

• For capture of device port to device port Fibre Channel traffic to a IP attached workstation running Ethereal

1. Configure Port Analyzer Adapter (dip switch settings, mostly plug and play)2. Configure SPAN port on MDS90003. Set Ethereal to capture on local Ethernet interface

Cisco PortAnalyzer Adaptor

Workstation Anywhere on IP Network

SCSI

HBAHBA

FCFC


Fibre Channel Capture without Port Analyzer with Third Party Analyzer

• For capture of device port to device port Fibre Channel traffic, to a Fibre Channel attached tester

TX

TX

RX

RX

FC 2/1

FC 5/1

FC 1/4

FC 2/13 Dropped DroppedSCSI

HBAHBA

2 Port FC Analyzer

TX TXRX RX

MDS9000#config t

span session 10

destination interface fc1/4

source interface fc 2/1 tx

MDS9000#config t

span session 11

destination interface fc2/13

source interface fc5/1 rx

FCFC



Fibre Channel Port Issues

• In order for a F_Port to come up on a MDS switch

The switch port must first acquire bit and word synchronization with the N_Port

N_Port must issue a FLOGI to the MDS

FCOT = Fibre Channel Optical Transport


Using FCPing and FCtrace

• Need FCID or pwwn• Ping uses PRLI

Extended Link Service • Checks connectivity

and latency on devices attached to MDS Switches directly or across TE ports

• FCtrace command is two path descriptors that identify the path taken on a hop-by-hop basis including a timestamp at each hop in both directions

IPS-TEST# show flogi database vsan 7

---------------------------------------------------------------------------

INTERFACE VSAN FCID PORT NAME NODE NAME

---------------------------------------------------------------------------

fc1/8 7 0x650001 10:00:00:00:c9:30:ba:06 20:00:00:00:c9:30:ba:06

Total number of flogi = 1.

IPS-TEST# fcping fcid 0x650001 vsan 7

28 bytes from 0x650001 time = 1896 usec





5 frames sent, 5 frames received, 0 timeouts

Round-trip min/avg/max = 305/891/1896 usec

IPS-TEST# fctrace fcid 0x4c0007 vsan 1

Route present for : 0x4c0007

20:00:00:05:30:00:37:de(0xfffc4a)

Timestamp Invalid.

20:00:00:05:30:00:2f:de(0xfffc4b)

Timestamp Invalid.

20:00:00:05:30:00:44:5e(0xfffc4c)

Timestamp Invalid.

20:00:00:05:30:00:2f:de(0xfffc4b)



Debugging Host Login Issues

1. Show interface fc 1/8

2. Show port internal info int fc1/8

3. debug port event int fc1/8

4. Show flogi database vsan 7

Int fc1/8

SCSI

HBAHBA

FCFC


Interrogating Fibre Channel Name Server Database

• Show fcns database

• Show fcs database



Debugging Name Server Issues

• If a host is not receiving a proper response, or if not able to discover a target, enable the following debug

1. debug fcns events (this will give the request and response to a host’s request)

2. Check if the host and the disk are in the same zone

3. Check if the host and the disk are in the same vsan









• iSCSI Driver Troubleshooting



iSCSI Driver Troubleshooting


Understanding the iSCSI Connection

• iSCSI Protocol Host Login and flows

• A look at W2K tools for troubleshooting iSCSI

• A look at Linux driver utilities and status logs



Response

Target list: com.cisco.Target-disk1Com.cisco.Target-Disk2

Target list: com.cisco.Target-disk1Com.cisco.Target-Disk2

Session Type: Discovery-SessionSession Type: Discovery-Session

TCP Port 3260TCP Port 3260

TargetInitiator iSCSI Initiator Login (Discovery Session)iSCSI Initiator Login (Discovery Session)

Establish TCP SessionTCPTCP

iSCSI DriveriSCSI Driver

iSCSI Login Sequence

Command—SendTargets

Command—Login

Optional Authentication PhaseOptional Authentication Phase


TCP Port 3260TCP Port 3260

TargetInitiator iSCSI Initiator Login to TargetiSCSI Initiator Login to Target

Establish Normal TCP Session

TCPTCP

iSCSI DriveriSCSI Driver

iSCSI Login Sequence

Command—Login

Session-Type: Normal -Session, Target: com.cisco.Target-disk1Session-Type: Normal -Session, Target: com.cisco.Target-disk1

Optional Authentication PhaseOptional Authentication Phase

Operational Parameter Negotiation

After This Completes We Are in Full Feature ModeAfter This Completes We Are in Full Feature Mode

Command—SCSI CommandsNormal SCSI Commands… Normal SCSI Commands…



Windows iSCSI Driver Status

CHAP Password—If Passwords Are Set then You Are Using CHAP, Make Sure Your Target Device Is Set to Work with CHAP

Relogin Will Drop TCP Session and Cause Relogin and Discovery with All Targets


W2K TCP Registry Tweaks

• 2 ways to tweak directory1. Use regedit and adjust values

manually

2. Use a freeware tool like DR TCP

Interface MTU Adjusted on Per NIC Interface All Other Options Done on TCP Stack in Windows

http://www.dslreports.com/front/drtcp.html



Windows iSCSI Debug


Linux Support Commands

To check version of Linux driver loaded:

[Linux]# cd/usr/src

[Linux]# ls -l iscsi

lrwxrwxrwx 1 root root 17 Feb 28 14:31 iscsi -> linux-iscsi -3.1.1

Useful Commands:/var/log/iscsi.log—This log contains iSCSI diagnostic information if there were problems during the iSCSI kernel module load

/usr/bin/iscsi-ls l—Utility to view current available iSCSI devices on the system (new in 3.2 drivers)

/proc/scsi/iscsi—Identifies the controller number assigned to the iSCSI HBA

/etc/rc.d/init.d/iscsi reload—Rediscover command for new iSCSI devices that have been added to iscsi.conf (does not effect connected targets )

Read and follow release notes that come with the Cisco Linux Driver



Linux Log Detail


Linux iSCSI Display Information

[root@nt5-ge log]# cd /proc/scsi/iscsi

[root@nt5-ge iscsi]# ls

1

[root@nt5-ge iscsi]# cat 1

# iSCSI driver version: 3.1.1 (28-Feb-2003)#

# SCSI: iSCSI:

# Bus Tgt LUN IP address Port TargetName

0 8 0 10.1.2.91 3260 iqn.1987-05.com.cisco.00.ec5bed7e2793a5402188312c585bbc09.llb5a1

0 9 0 10.1.2.91 3260 iqn.1987-05.com.cisco.00.b318983517beee6ea95f130f98c1bcd5.llb5a2

0 10 0 10.1.2.91 3260 iqn.1987-05.com.cisco.00.ebd3b01484d3db4ddfaede782ab8a4f6.llb5a30 11 0 10.1.2.91 3260 iqn.1987-05.com.cisco.00.e9829d85e3637c164016535d6fd7967f.llb5a4

0 12 0 10.1.2.91 3260 iqn.1987-05.com.cisco.00.18e8e7f6afc9bc871265c383375252a7.llb5a58




Detail Information About Each iSCSI Device on Each Controller

/proc/scsi/iscsi



Please Complete Your Evaluation Form

Session OPT-3050




EXTRAS on Troubleshooting MDS IPS-8


MDS IPS-8

• iSCSI and FCIP configuration troubleshooting

• A look at important output displays

• Best practices for debugging



• Manually configured targets

• Show iscsi local-node shows if target is manually configured or is auto-created

Configured and Auto Import Targets in the iSCSI Domain


Displays:iSCSI Session

iSCSI Discovery Session

Auto iSCSI Targets

Manual iSCSI Targets



Displays:Auto Configured iSCSI Initiators

• iSCSI configured-initiators are iSCSI hosts that are automatically configured by the MDS

• The IQN node name will be unique for all iSCSI hosts but the assigned pWWN can change

TIP: This Unique iqn Number Can Be Used to Statically Configure the Initiator

TIP: This Unique iqn Number Can Be Used to Statically Configure the Initiator


Displays: Zoning Options

Target Zone

Members

iSCSI Initiator

Zone Member

Tip: Use symbolic iqn Name Because This Will Not Change but wwn CouldTip: Use symbolic iqn Name Because This Will Not Change but wwn Could



Displays: Details on iSCSI Initiator in the iSCSI Domain

nWWN Auto Assigned and Can

Be Used to Configure Static

Entry; This Would be Required if WWN Access Is Used on

Storage Array

pWWN Auto Assigned and Can be

Used for Static Config

Assigned Targets


Displays:Show iSCSI Interface

iSCSI Sessions with Targets on This Interface; TCP Sessions Also

Includes iSCSI Discovery Sessions

MTU Path Discovery, if Using Large Frames

Like Jumbo on the Host This Needs to be Set

iSCSI PDU

Counts

Tx Window Size as

Determined by Host Setting

Selective Acknowledgement Feature—Disabled by Default



Interface Displays

Transfer Rates

Active TCP Connections; Port 3260 is

iSCSI

Discovery Session and 1 iSCSI Target

Session


Displays:Specific iSCSI Show Commands

• Configured-initiator Show iscsi configured initiators

• Global information Show iscsi global config

• Local-node Show iscsi local nodes

• Port Show iscsi FV port information

• Remote-node Show iscsi node information

• Session Show iscsi session information

• Stats Show iscsi statistics



show iscsi remote-node detail

iSCSI Host Created in MDS Configuration and Was Placed in VSAN 4

iSCSI Host Assigned pwwn; Info on What

GiGE Port Connection Is Coming in on


iSCSI Debugs

all Turn on all ISCSI debug

api Configure ISCSI API debugging

buf Configure ISCSI buffer debugging

error Configure ISCSI error debugging

event Configure ISCSI event debugging

fcpevent Configure ISCSI fcp event debugging

fcpframe Configure ISCSI fcp frame debugging

fcpfsm Configure ISCSI FCP FSM debugging

fsm Configure ISCSI FSM debugging

login Configure ISCSI login debugging

parse Configure ISCSI parsing debugging

pkt Configure ISCSI packet debugging

task Configure ISCSI transaction debugging

transaction Configure ISCSI transaction debugging

warn Configure ISCSI warning debugging

MDS9200A# Attach Module 2

Attaching to Module 2 ...

To Exit Type ‘Exit', to Abort Type '$.'

Module-2#

• Most debugs require a port or session for information you are wanting to debug

• Requires attachment to module debug ips iscsi



FCIP Interface MDS-MDS

TE Port Connected to Another

MDS

MDS to MDS Uses 2 TCP

Connections

One Connection Is F-Class

Flow Second Connection Is the Data

Timestamp Information and

Acceptable Difference Would be

Displayed Here

Special Frame Verification and Will Display Peer

Switch wwn

TCP MTU, Can Be

Adjusted Up to 8K


Show TCP stats

MDS9509# show ips stats tcp interface gigabitethernet 2/8TCP Statistics for port GigabitEthernet2/8

Connection Stats6 active openings, 8 accepts6 failed attempts, 0 reset received, 8 established

Segment stats295930 received, 1131824 sent, 109 retransmitted0 bad segments received, 0 reset sent

TCP Active Connections Local Address Remote Address State Send-Q Recv-Q10.10.10.2:3225 10.10.11.2:65519 ESTABLISH 0 010.10.10.2:3225 10.10.11.2:65521 ESTABLISH 87568 010.10.10.2:3225 0.0.0.0:0 LISTEN 0 0

Excessive Retransmits Indicates Possible Core

Drops and/or tcpWindow Size Should

Be Adjusted

Used for Class F Control Traffic Only; FCIP Uses Port 3225

TCP Listen Port Ready for New

tcp Connections

Data Traffic Connection—Only Connection You Would See for Both Class F and Data for a B-Port Connection



Show counters

• Frames averaged over 5 minutes and total count of frames since last clear counters or since tunnel up

MDS9509# show interface fcip 1 counters bri

-------------------------------------------------------------------------------

Interface Input (rate is 5 min avg) Output (rate is 5 min avg)

----------------------------- -----------------------------

Rate Total Rate Total

Mbits/s Frames Mbits/s Frames

-------------------------------------------------------------------------------fcip1 18 0 18 0


debug ips ipstack

• IP debugging for iSCSI and FCIP

all Configure ALL IP Stack debugging

arp Configure IP Stack ARP debugging

arp-detail Configure verbose IP Stack ARP debugging

error Configure IP Stack Error debugging

eth Configure IP Stack Ethernet debugging

eth-cfg Configure IP Stack Ethernet Configuration debugging

eth-chan Configure IP Stack Ether Channel debugging

eth-detail Configure verbose debugging of Ethernet layer

icmp Configure IP Stack ICMP debugging

if Configure IP Stack interface debugging

in Configure IP Stack IN debugging

ip Configure IP Stack IP debugging

ip-detail Configure verbose IP Stack IP debugging

route Configure IP Stack route debugging

sdip Configure IP Stack SDIP debugging

tcp Configure IP Stack TCP debugging

udp Configure IP Stack UDP debugging

udp-detail Configure verbose IP Stack UDP debugging

vrrp Configure IP Stack VRRP debugging

vrrp-detail Configure verbose IP Stack VRRP debugging

troubleshooting cisco storage networks - net130.com storage networks session opt-3050. ... switch...

Documents