brocade fabric technology with the nimble storage …...about nimble storage nimble storage, inc....

53-1004093-01December 2015

Brocade Fabric Technologywith the Nimble StorageCS300 Flash ArrayValidation Test Report

Supporting Fabric OS 7.4.0

© 2015, Brocade Communications Systems, Inc. All Rights Reserved.

ADX, Brocade, Brocade Assurance, the B-wing symbol, DCX, Fabric OS, HyperEdge, ICX, MLX, MyBrocade, OpenScript, The EffortlessNetwork, VCS, VDX, Vplane, and Vyatta are registered trademarks, and Fabric Vision and vADX are trademarks of BrocadeCommunications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned may betrademarks of others.

Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning anyequipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this documentat any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not becurrently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained inthis document may require an export license from the United States government.

The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to theaccuracy of this document or any loss, cost, liability, or damages arising from the information contained herein or the computer programs thataccompany it.

The product described by this document may contain open source software covered by the GNU General Public License or other opensource license agreements. To find out which open source software is included in Brocade products, view the licensing terms applicable tothe open source software, and obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.

http://www.brocade.com/support/oscd

Contents

Preface..................................................................................................................................... 5Overview........................................................................................................... 5Purpose of This Document................................................................................5Audience........................................................................................................... 5Objectives......................................................................................................... 6Test Conclusions...............................................................................................6Related Documents...........................................................................................6Document History..............................................................................................7About Brocade.................................................................................................. 7About Nimble Storage....................................................................................... 7

Test Plan...................................................................................................................................9Scope................................................................................................................ 9What’s New in This Report................................................................................9Test Configuration...........................................................................................10DUT Descriptions............................................................................................ 10DUT Specifications..........................................................................................11Test Equipment............................................................................................... 11

Configure Equipment.............................................................................................................. 13Task 1: Brocade FC Fabric Configuration.......................................................13Task 2: Nimble Storage Array Configuration...................................................15Task 3: Host Setup..........................................................................................17

Test Cases.............................................................................................................................. 211.0 Fabric Initialization—Base Functionality................................................... 22

1.0.1 Storage Device—Physical and Logical Login with SpeedNegotiation.................................................................................... 22

1.0.2 Zoning and LUN Mapping.......................................................... 231.0.3 Storage Device Fabric I/O Integrity............................................ 231.0.4 Storage Device Multipath Configuration—Path integrity............ 24

1.1 Fabric—Advanced Functionality............................................................... 251.1.1 Storage Device Bottleneck Detection Using MAPS FPI—

With Congested Host.....................................................................251.1.2 Storage Device Bottleneck Detection Using MAPS—With

Congested Fabric.......................................................................... 261.1.3 Storage Device—QoS Integrity with QoS Zone Based

Traffic Prioritization........................................................................271.1.4 Storage Device—QoS Integrity with CS_CTL-Based Traffic

Prioritization...................................................................................271.1.5 Storage Device—FC Protocol Jammer Test Suite.....................301.1.6 Clear Link Diagnostics (D_Port) Test.........................................31

1.2 Stress and Error Recovery with Device Multipath.....................................321.2.1 Storage Device Fabric I/O Integrity—Congested Fabric............ 331.2.2 Storage Device Nameserver Integrity—Device Recovery

with Port Toggle.............................................................................33

Brocade Fabric Technology with the Nimble Storage CS300 Flash Array Validation Test Report 353-1004093-01

1.2.3 Storage Device Nameserver Integrity—Device Recoverywith Device Relocation................................................................ 34

1.2.4 Storage Device—Device Recovery with Device PortToggle—Extended Run............................................................... 35

1.2.5 Storage Device Recovery—ISL Port Toggle ...........................361.2.6 Storage Device Recovery—ISL Port Toggle (Entire Switch)... 371.2.7 Storage Device Recovery—Director Blade Maintenance........ 381.2.8 Storage Device Recovery—Switch Offline...............................391.2.9 Storage Device Recovery—Switch Firmware Download......... 40

1.3 Storage Device—Fibre Channel Routing (FCR) InternetworkingTests........................................................................................................41

1.3.1 Storage Device Internetworking Validation with the FCHost............................................................................................. 41

1.3.2 Storage Device Internetworking Validation with FCoE TestUsing VDX FlexPort.....................................................................42

1.3.3 Storage Device Edge Recovery After FCR Disruptions...........431.3.4 Storage Device Backbone Recovery After FCR Disruptions... 44

1.4 Optional/Additional Tests........................................................................ 451.4.1 Storage Device Firmware Update............................................451.4.2 Performance I/O Testing with Varying Workload

Dimensions..................................................................................461.4.3 VMware I/O Analyzer Workload Test.......................................47

4 Brocade Fabric Technology with the Nimble Storage CS300 Flash Array Validation Test Report53-1004093-01

Preface

● Overview........................................................................................................................... 5● Purpose of This Document................................................................................................5● Audience........................................................................................................................... 5● Objectives......................................................................................................................... 6● Test Conclusions...............................................................................................................6● Related Documents...........................................................................................................6● Document History..............................................................................................................7● About Brocade.................................................................................................................. 7● About Nimble Storage....................................................................................................... 7

OverviewThe Solid State Ready (SSR) program is a comprehensive testing and configuration initiative to provideFibre Channel SAN and IP interoperability with flash storage. This program provides testing of multiplefabrics, heterogeneous servers, NICs, and HBAs in large port-count Brocade environments. Testingcovers future Brocade OS versions and vendor software releases. The SSR qualification program helpsverify seamless interoperability and optimum performance with solid state storage in Brocade SANfabrics.

Purpose of This DocumentThis document provides the validation of Brocade fabric technology with the Nimble Storage CS300Adaptive Flash 16-GB Fibre Channel storage array, using multiple switch platforms, HBAs, and serveroperating systems. This validation shows that the CS300 interoperates properly within a Brocade Gen 5Fibre Channel fabric, while supporting the performance and low latency associated with solid statestorage.

AudienceThis document is written for a technical audience, including solution architects, system engineers, andtechnical development representatives.


Objectives1. Test the Nimble CS300 array with Brocade FC fabrics, in single and routed configurations for

different stress and error recovery scenarios, and validate the interoperability and integration of thearray with the Brocade FC fabric.

2. Validate the performance of the FC fabric in a solid state storage environment for high-throughputand low-latency applications.

Test Conclusions1. Achieved a 100-percent pass rate on all test cases in the SSR qualification test plan. The network

and the storage were able to handle the various stress and error recovery scenarios without issues.2. Different I/O workload scenarios were simulated using the Medusa and VMware I/O Analyzer tools,

and sustained performance levels were demonstrated across all workload types.3. The results confirm that the Nimble Storage CS300 array interoperates seamlessly with Brocade

Fibre Channel fabrics, and together they demonstrate high availability, performance, and lowlatency.

4. The Brocade Gen5 16-Gb FC switches were able to efficiently handle the sustained throughput andlatency performance requirements with fewer ISL trunks. Multiple ISLs to different switches in thefabric should be set up to provide path redundancy through the fabric.

5. The Monitoring and Alerting Policy Suite (MAPS) heath monitor should be enabled on all switchesin the FC fabric to report fabric-wide events and traffic performance metrics. The additional MAPSfeature of Fabric Performance Impact monitoring should also be enabled to detect bottlenecks inthe form of timeouts and latency. Using MAPS is recommended to maximize high-performance,low-latency storage.

6. Peer zoning helps reduce the zone database size and the zoning complexity, while providing theRSCN and hardware resource efficiencies of single-initiator zoning.

7. QoS zoning should be used to classify host-target traffic into high, medium, or low priority zonesand to prioritize traffic through the FC fabric for the desired host-target pair by allocating moreresources to the traffic in the higher priority zone.

8. Enabling Emulex ExpressLane on a LUN provides prioritized queuing on the HBA for traffic to thatLUN and also sets the CS_CTL tag on the frame, which allows the traffic to be prioritized throughthe FC fabric based on the value of the CS_CTL tag and the corresponding priority level.

9. For optimal availability and performance, consideration should be given to the multipathconfiguration on the host side. While Windows 2008 and 2012 provide Round Robin behavior bydefault, Linux systems benefit from adding a custom entry to /etc/multipath.conf, and VMWare hostssystems should be changed from the default “Most Recently Used (VMWare)” setting to “Round-Robin (VMWare).” Actively using all available paths provides a significant improvement inperformance throughput.

10.The ClearLink D_Port diagnostic test is a valuable tool for validating link quality before adding newdevices or ISLs to the fabric and for troubleshooting existing links.

Related Documents• Fabric OS Administrator's Guide, v7.4.0• Brocade SAN Design and Best Practices• Monitoring and Alerting Policy Suite Administrator’s Guide, v7.4.0

Objectives


http://www.brocade.com/content/dam/common/documents/content-types/administration-guide/fos-740-adminguide.pdf

http://www.brocade.com/content/brocade/en/backend-content/pdf-page.html?/content/dam/common/documents/content-types/product-design-guide/san-design-best-practices.pdf

http://www.brocade.com/content/dam/common/documents/content-types/administration-guide/fos-740-maps.pdf

• Network OS Layer 2 Switching Configuration Guide, v5.0.1• Emulex ExpressLane Configuration• Emulex OneCommand Manager User Manual• QLogic QLE2672 Adapter User’s Guide• QLogic BR-1860 Adapter Administrator’s Guide

Document History

Date Version Description

December 2015 1.0 Initial draft.

About BrocadeBrocade networking solutions help the world's leading organizations transition smoothly to a worldwhere applications and information reside anywhere. This vision is realized through the Brocade One™strategy, which is designed to deliver key business benefits such as unmatched simplicity, non-stopnetworking, application optimization, and investment protection.

Innovative Ethernet and storage networking solutions for data center, campus, and service providernetworks help reduce complexity and cost while enabling virtualization and cloud computing to increasebusiness agility.

To help ensure a complete solution, Brocade partners with world-class IT companies and providescomprehensive education, support, and professional services offerings.

To learn more, visit www.brocade.com.

About Nimble StorageNimble Storage, Inc. provides a flash-optimized hybrid storage platform. Nimble Storage CS-Seriesarrays are the building blocks of Adaptive Flash, a storage platform that dynamically and intelligentlyallocates storage resources to satisfy the changing needs of business-critical applications.

Our Adaptive Flash platform comprises two core innovations: CASL™, our flash-optimized file system,and InfoSight™, our cloud-based management software. CASL is industry leading in its ability toleverage flash and disk for the broadest range of workloads, delivering the performance of flash-onlyarrays while simultaneously scaling to petabyte scale deployments cost-effectively. In addition, CASLdelivers the best-in-class resilience and application-integrated data protection. InfoSight monitors ourcustomers’ infrastructure on a near real-time basis and averts potential problems before they occur,radically simplifying our customers’ day-to-day operations and improving the health of theirinfrastructure.

Document History


http://www.brocade.com/downloads/documents/product_manuals/B_VDX/NOS_L2Guide_v501.pdf

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&cad=rja&uact=8&ved=0CCMQFjAB&url=http%3A%2F%2Fwww.emulex.com%2Fartifacts%2FBBE3C568-802C-4CD0-8471-C7B1680232CC%2Felx_wp_all_expresslane.pdf&ei=K-BLVcDZIpfpoAT41oC4DQ&usg=AFQjCNEpdBY_8qiiBUcmP9Puh2o4EC0elQ&sig2=W_7n3oUL3TtegJYFUYbEPA&bvm=bv.92765956,d.aWw

http://www-dl.emulex.com/support/elx/rt10.4.0/ga/Docs/final/apps/ocm_gui_manual_elx.pdf

http://filedownloads.qlogic.com/Files/TempDownlods/90481/UsersGuide_2600_FCAdapter_FC0054609-00G.pdf

http://filedownloads.qlogic.com/Files/TempDownlods/90546/AdminGuide_BRSeriesAdapters_BR0054501-00C.pdf

http://www.brocade.com

About Nimble Storage


Test Plan

● Scope................................................................................................................................ 9● What’s New in This Report................................................................................................9● Test Configuration...........................................................................................................10● DUT Descriptions............................................................................................................ 10● DUT Specifications..........................................................................................................11● Test Equipment............................................................................................................... 11

The storage array is connected to two SAN fabrics and multiple server hosts to drive I/O in a multipathconfiguration. Error injection is introduced, and failover and recovery behaviors are observed. I/Operformance is observed across different workload configurations.

ScopeTesting is performed with a mix of GA and development versions of Brocade Fabric OS (FOS) in aheterogeneous environment. Test beds include Brocade directors and switches configured in routedand non-routed fabric configurations.

Testing is centered on interoperability and optimal configuration. Performance is observed within thecontext of best-practice fabric configuration; however, absolute maximum benchmark reporting ofstorage performance is beyond the scope of this test.

Details of the test steps are covered under the “Test Cases” section. Standard test-bed setup includesIBM/HP/Dell chassis server hosts with Brocade/QLogic/Emulex HBAs with two uplinks from every hostto a Brocade FC fabric. I/O generators include Medusa Labs Test Tools and VMware I/O Analyzer.

What’s New in This Report1. The array firmware version under test is 2.2.5.0-197583-opt.2. The Brocade Fabric OS (FOS) version under test is 7.4.0.3. Emulex and QLogic adapters have updated firmware and drivers.4. Two new test cases have been added:

• 1.1.4 Storage Device—QoS Integrity with CS_CTL-Based Traffic Prioritization• 1.1.6 Clear Link Diagnostics (D_Port) Test

For detailed information, see the “DUT Descriptions” section.

The previous version of the test report covering Brocade FOS 7.3.1 and Nimble array firmware version2.2.2.0-143348-opt is available at Brocade Fabric Technology with the Nimble Storage CS300Validation Test Report.


http://www.brocade.com/content/html/en/validation-test-report/brocade-fabric-nimble-storage-cs300-vt/index.html

http://www.brocade.com/content/html/en/validation-test-report/brocade-fabric-nimble-storage-cs300-vt/index.html

Test Configuration

DUT DescriptionsThe following tables provide details about the devices under test (DUTs).

Storage ArrayTABLE 1

DUT ID Model Vendor Description

Nimble StorageCS300

CS300 Nimble Storage The Nimble Storage CS300 Adaptive Flash storagearray is a hybrid flash array. Each controller supportsdual 16-Gb Fibre Channel connections in an active/passive configuration.

SwitchTABLE 2

DUT ID Model Vendor Description

6501-1...9 BR-6510 Brocade 48-port 16-Gb FC switch

DCX-1 DCX 8510-8 Brocade 8-slot 16-Gb FC chassis

DCX-2 DCX 8510-4 Brocade 4-slot 16-Gb FC chassis

VDX-1, 2 VDX 6740 Brocade 48-port 10-Gb switch (48x10Gb/4x40Gb)

Test Configuration


DUT Specifications

Storage Version

Nimble Storage CS300 Adaptive Flash array 2.2.5.0-197583-opt

Brocade Switches Version

DCX 8510-8 Fabric OS 7.4.0

DCX 8510-4 Fabric OS 7.4.0

6510 + Integrated Routing, Fabric Vision Licenses Fabric OS 7.4.0

VDX 6740 Network OS 5.0.1

Adapters Version

Brocade 1860 2-port 16-Gb FC HBA Driver and firmware 3.2.5.1

QLogic QLE2672 2-port 16-GB FC HBA Driver 8.07.00.16, firmware 7.04.01

QLogic QLE2562 2-port 8-GB FC HBA Driver 9.1.11.28, firmware 7.04.00

Emulex LPe 16202-X 2-port 16-Gb FC HBA Driver 10.4.246.0, firmware 10.4.255.23

Brocade 1020 2-port CNA adapter Driver and firmware 3.2.5.1

DUT ID Servers RAM Processor OS

SRV-1 HP ProLiant DL380P G8 160 GB Intel Xeon E5-2640 VMware 5.5 [cluster]

SRV-2 HP ProLiant DL380P G8 160 GB Intel Xeon E5-2640 VMware 5.5 [cluster]

SRV-3 IBM System x3630 M4 24 GB Intel Xeon E5-2420 Windows Server 2012 R2

SRV-4 Dell PowerEdge R720 160 GB Intel Xeon E5-2640 RHEL 7.0 x86_64

SRV-5 HP ProLiant DL385p G8 16 GB AMD Opteron 6212 Windows Server 2012

SRV-6 Dell PowerEdge R720 16 GB Intel Xeon E5-2620 RHEL 6.5 x86_64

SRV-7 HP ProLiant DL380P G8 32 GB Intel Xeon E5-2690v2 Windows Server 2012 R2

SRV-8 IBM System x3630 M4 16 GB Intel Xeon E5-2620 RHEL 6.5 x86_64

Test Equipment

Device/Software Tools Version

Finisar 16-Gb Analyzer/Jammer XGIG5K2001153

Medusa Labs Test Tools 6.0.1.156759

VMware I/O Analyzer 1.6.0

DUT Specifications


Test Equipment


Configure Equipment

● Task 1: Brocade FC Fabric Configuration.......................................................................13● Task 2: Nimble Storage Array Configuration...................................................................15● Task 3: Host Setup..........................................................................................................17

Task 1: Brocade FC Fabric ConfigurationConfiguration settings of the Brocade switches in the test bed are covered here.

1. Zoning is configured using the “Peer Zoning” feature in Brocade Fabric OS 7.4.0. Peer zoning allowsa “principal” device to communicate with the other devices in the zone. The principal device managesa peer zone. Other “nonprincipal” devices in the zone can communicate with the principal deviceonly; they cannot communicate with each other.

root> zonecreate –peerzone nimble_peer –principal “ 56:c9:ce:90:f5:ed:f8:01; 56:c9:ce:90:f5:ed:f8:02;56:c9:ce:90:f5:ed:f8:03; 56:c9:ce:90:f5:ed:f8:04” –members “10:00:8c:7c:ff:24:4c:00; 10:00:8c:7c:ff:24:4c:01 …”root> zoneshow --peerzone allzone: nimble_peer Property Member: 00:02:00:00:00:03:00:08 Created by: User Principal Member(s): 56:c9:ce:90:f5:ed:f8:01; 56:c9:ce:90:f5:ed:f8:02;56:c9:ce:90:f5:ed:f8:03; 56:c9:ce:90:f5:ed:f8:04 Peer Member(s): 10:00:8c:7c:ff:24:4c:00; 10:00:8c:7c:ff:24:4c:01 10:00:00:90:fa:61:92:3b; 10:00:00:90:fa:61:92:3c; 10:00:8c:7c:ff:05:60:01; 10:00:8c:7c:ff:05:60:00; 10:00:8c:7c:ff:14:e0:01; 10:00:8c:7c:ff:14:e0:00; 10:00:8c:7c:ff:03:bc:01; 10:00:8c:7c:ff:03:bc:00; 10:00:8c:7c:ff:03:9b:00; 10:00:8c:7c:ff:03:9b:01; 10:00:8c:7c:ff:05:72:02; 10:00:8c:7c:ff:05:72:03

2. Configure MAPS on the switches (requires a Fabric Vision License). MAPS enables reporting oflatency and congestion alerts on each switch, as well as a number of switch and fabric healthmetrics. For more information on configuring and using MAPS, see the Monitoring and Alerting PolicySuite Administrator’s Guide, v7.4.0.

Enable the desired MAPS policy using any of the available default policies or by creating a custompolicy. For this test, the default Aggressive Policy is used. This policy provides the most sensitivethreshold levels for detection of latency and congestion.

> mapsconfig --enablemaps -policy dflt_aggressive_policyDefine the reporting actions that MAPS will take. In this test, errors discovered by MAPS will generate a RASLog entry and send e-mail to the configured recipients.> mapsconfig --actions raslog,emailThe Fabric Performance Impact (FPI) category within MAPS monitors the current condition of thelatency seen on F_Ports over different time windows and uses that to determine the performanceimpact to the fabric and network. In order to use FPI monitoring, legacy (bottelneckmon) must bedisabled:

> bottleneckmon –disableEnable FPI monitoring:




> mapsConfig –enableFPImonTo examine the MAPS configuration in effect, use the following command:

> mapsconfig –showConfigured Notifications: RASLOG,EMAILMail Recipient: [email protected],[email protected] Monitoring: EnabledPaused members :===============PORT :CIRCUIT :SFP :Use the following command to view a summary of the events or rules triggered and to view theobjects on which the rules were triggered over a specified period of time.

> mapsdb –show1 Dashboard Information:=======================DB start time: Wed Dec 17 20:59:09 2014Active policy: dflt_aggressive_policyConfigured Notifications: RASLOG,EMAILFenced Ports : NoneDecommissioned Ports : None

3. Configure FlexPorts on VDX switches for Fibre Channel support (an FCoE Base License isrequired). These ports are used as uplinks from the VCS fabric to the Fibre Channel fabric in thistest. For detailed information on configuring Brocade VDX 6740 switches for Fibre Channel support,see the Network OS Layer 2 Switching Configuration Guide, v5.0.

Example of configuring a Fibre Channel port on a Brocade VDX 6740 switch:

# conf t(config)# hardware(config-hardware)# flexport 112/0/38(config-flexport-112/0/38)# type fibre-channelFlexPort speeds are configured in connector groups. In this test-bed example, the speed HighMixedis chosen. This speed allows the Fibre Channel FlexPort to operate at 16-Gb speed, which isoptimal because the uplink is attaching to a 16-Gb Fibre Chanel SAN. Also, not all ports in thisconnector group are configured as Fibre Channel ports; some are still configured as Ethernet ports,and the HighMixed setting supports Ethernet speeds on these ports as well. For detailedinformation on configuring VDX 6740 connector-group speeds, see the Network OS Layer 2Switching Configuration Guide.

• LowMixed: 2/4/8G Fibre Channel and Ethernet speeds (default)• HighMixed: 16G Fibre Channel and Ethernet speeds• FibreChannel: 2/4/8/16G Fibre Channel

Use the following command to set the connector-group speed to HighMixed:

(config-hardware)# connector-group 111/0/3(config-connector-group-111/0/3)# speed HighMixed

4. Configure zones for FCoE initiators on the VDX switches.

Example zone on a Brocade VDX 6740 switch:

# show zoning enabled-configurationzoning enabled-configuration cfg-name NOS_SSRzoning enabled-configuration enabled-zone lsan_hb067166_nimble member-entry 10:00:8c:7c:ff:1f:7b:00 member-entry 10:00:8c:7c:ff:1f:7b:01 member-entry 56:c9:ce:90:f5:ed:f8:01 member-entry 56:c9:ce:90:f5:ed:f8:02 member-entry 56:c9:ce:90:f5:ed:f8:03 member-entry 56:c9:ce:90:f5:ed:f8:04

5. Configure Fibre Channel Routing (an Integrated Routing license is required). Detailed informationon FCR setup can be found in the Fabric OS Administrators Guide, 7.3.0.

Configure Equipment





http://www.brocade.com/downloads/documents/product_manuals/B_SAN/FOS_AdminGd_v730.pdf

Example FCR configuration:

> fcrconfigure –bbfid 100> fosconfig --enable fcr> portcfgexport [port#] -a1 –m[0/5] -f 10 {0=Brocade FC fabric; 5=Brocade NOS fabric}The prefix “lsan” is used when configuring zones for use in Fibre Channel Routing.

Example of zone prefixed with “lsan”:

> zoneshow lsan_hb067166_nimble zone: lsan_hb067166_nimble 10:00:8c:7c:ff:23:b7:00; 10:00:8c:7c:ff:23:b7:01; 52:4a:93:7d:f3:5f:61:00; 52:4a:93:7d:f3:5f:61:11Example output of exported devices:

> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric---------------------------------------------------------------------------- 10 21:00:00:24:ff:48:b9:6a 02f001 20 551a00 Imported 10 21:00:00:24:ff:48:b9:6b 02f101 20 541e00 Imported 10 52:4a:93:7d:f3:5f:61:00 02f201 20 550e00 Imported 10 52:4a:93:7d:f3:5f:61:01 02f401 20 540400 Imported

Task 2: Nimble Storage Array Configuration1. Create a host initiator group for each host, and add WWPNs for the host.

Task 2: Nimble Storage Array Configuration


FIGURE 1 Initiator Configuration on Nimble Storage Array

2. Create new volumes, and assign them to the host initiator group. For this test configuration, 8 x10GB volumes are created. Configuration can be accomplished through the array GUI or the CLIinterface; for the purpose of easily creating multiple similar LUNs, the Nimble array CLI interface isused here.

Nimble OS $ vol --create ssd067178-1 --size 10240 --quota 100 --warn_level 100 --initiatorgrp ssd067178Nimble OS $ vol --create ssd067178-2 --size 10240 --quota 100 --warn_level 100 --initiatorgrp ssd067178Nimble OS $ vol --create ssd067178-3 --size 10240 --quota 100 --warn_level 100 --initiatorgrp ssd067178Nimble OS $ vol --create ssd067178-4 --size 10240 --quota 100 --warn_level 100 --initiatorgrp ssd067178Nimble OS $ vol --create ssd067178-5 --size 10240 --quota 100 --warn_level 100 --initiatorgrp ssd067178Nimble OS $ vol --create ssd067178-6 --size 10240 --quota 100 --warn_level 100 --initiatorgrp ssd067178Nimble OS $ vol --create ssd067178-7 --size 10240 --quota 100 --warn_level 100 --initiatorgrp ssd067178Nimble OS $ vol --create ssd067178-8 --size 10240 --quota 100 --warn_level 100 --initiatorgrp ssd067178

Configure Equipment


FIGURE 2 Finished LUN Configuration and Presentation on Nimble Storage Array

Task 3: Host SetupConfiguration settings of servers in the test bed are covered here.

1. Provision a minimum of two uplinks from the host to the FC fabric for redundancy, and use nativemultipath tools to manage the available paths and load-balance across them.

2. Multipath configuration on Linux hosts. This configuration allows all paths to be used in a round-robinfashion. This provides superior performance to the default Linux settings which would only use asingle active path per LUN.

Recommended /etc/multipath.conf entry on Linux systems:

device { vendor "Nimble" product "Server" path_selector "round-robin 0" features "1 queue_if_no_path" no_path_retry 20 path_grouping_policy group_by_serial path_checker tur rr_min_io 20 failback immediate rr_weight priorities }Example multipath configuration on Linux:

# multipath -llmpathe (2e3a65de830b8a6d06c9ce9000629663f) dm-10 Nimble ,Server size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=50 status=active| |- 1:0:9:5 sdbz 68:208 active ready running| |- 1:0:8:5 sdbr 68:80 active ready running| |- 10:0:0:5 sdcx 70:80 active ready running| `- 10:0:1:5 sddf 70:208 active ready running`-+- policy='round-robin 0' prio=1 status=enabled |- 1:0:0:5 sdf 8:80 active ghost running |- 1:0:11:5 sdcp 69:208 active ghost running |- 10:0:10:5 sdfz 131:80 active ghost running `- 10:0:9:5 sdfr 130:208 active ghost runningmpathd (2702dbeb7f1c3e8326c9ce9000629663f) dm-4 Nimble ,Server size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=50 status=active| |- 1:0:9:4 sdby 68:192 active ready running| |- 1:0:8:4 sdbq 68:64 active ready running| |- 10:0:1:4 sdde 70:192 active ready running

Task 3: Host Setup


| `- 10:0:0:4 sdcw 70:64 active ready running`-+- policy='round-robin 0' prio=1 status=enabled |- 1:0:0:4 sde 8:64 active ghost running |- 1:0:11:4 sdco 69:192 active ghost running |- 10:0:10:4 sdfy 131:64 active ghost running `- 10:0:9:4 sdfq 130:192 active ghost runningmpathc (241a6f5dfd37f15156c9ce9000629663f) dm-2 Nimble ,Server size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=50 status=active| |- 1:0:9:3 sdbx 68:176 active ready running| |- 1:0:8:3 sdbp 68:48 active ready running| |- 10:0:0:3 sdcv 70:48 active ready running| `- 10:0:1:3 sddd 70:176 active ready running`-+- policy='round-robin 0' prio=1 status=enabled |- 1:0:11:3 sdcn 69:176 active ghost running |- 1:0:0:3 sdd 8:48 active ghost running |- 10:0:10:3 sdfx 131:48 active ghost running `- 10:0:9:3 sdfp 130:176 active ghost runningmpathb (2ce2c56af948b57776c9ce9000629663f) dm-12 Nimble ,Server size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=50 status=active| |- 1:0:9:2 sdbw 68:160 active ready running| |- 1:0:8:2 sdbo 68:32 active ready running| |- 10:0:0:2 sdcu 70:32 active ready running| `- 10:0:1:2 sddc 70:160 active ready running`-+- policy='round-robin 0' prio=1 status=enabled |- 1:0:0:2 sdc 8:32 active ghost running |- 1:0:11:2 sdcm 69:160 active ghost running |- 10:0:10:2 sdfw 131:32 active ghost running `- 10:0:9:2 sdfo 130:160 active ghost runningmpatha (29fa58dd7764784596c9ce9000629663f) dm-5 Nimble ,Server size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=50 status=active| |- 1:0:9:1 sdbv 68:144 active ready running| |- 1:0:8:1 sdbn 68:16 active ready running| |- 10:0:0:1 sdct 70:16 active ready running| `- 10:0:1:1 sddb 70:144 active ready running`-+- policy='round-robin 0' prio=1 status=enabled |- 1:0:0:1 sdb 8:16 active ghost running |- 1:0:11:1 sdcl 69:144 active ghost running |- 10:0:10:1 sdfv 131:16 active ghost running `- 10:0:9:1 sdfn 130:144 active ghost runningmpathh (2a4a1164589763f876c9ce9000629663f) dm-11 Nimble ,Server size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=50 status=active| |- 1:0:8:8 sdbu 68:128 active ready running| |- 1:0:9:8 sdcc 69:0 active ready running| |- 10:0:0:8 sdda 70:128 active ready running| `- 10:0:1:8 sddi 71:0 active ready running`-+- policy='round-robin 0' prio=1 status=enabled |- 1:0:0:8 sdi 8:128 active ghost running |- 1:0:11:8 sdcs 70:0 active ghost running |- 10:0:10:8 sdgc 131:128 active ghost running `- 10:0:9:8 sdfu 131:0 active ghost runningmpathg (2751a4e6d6b326c696c9ce9000629663f) dm-13 Nimble ,Server size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=50 status=active| |- 1:0:8:7 sdbt 68:112 active ready running| |- 1:0:9:7 sdcb 68:240 active ready running| |- 10:0:0:7 sdcz 70:112 active ready running| `- 10:0:1:7 sddh 70:240 active ready running`-+- policy='round-robin 0' prio=1 status=enabled |- 1:0:0:7 sdh 8:112 active ghost running |- 1:0:11:7 sdcr 69:240 active ghost running |- 10:0:10:7 sdgb 131:112 active ghost running `- 10:0:9:7 sdft 130:240 active ghost runningmpathf (2122e26c04efc79ee6c9ce9000629663f) dm-14 Nimble ,Server size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=50 status=active| |- 1:0:8:6 sdbs 68:96 active ready running| |- 1:0:9:6 sdca 68:224 active ready running| |- 10:0:0:6 sdcy 70:96 active ready running| `- 10:0:1:6 sddg 70:224 active ready running`-+- policy='round-robin 0' prio=1 status=enabled |- 1:0:11:6 sdcq 69:224 active ghost running |- 1:0:0:6 sdg 8:96 active ghost running |- 10:0:10:6 sdga 131:96 active ghost running `- 10:0:9:6 sdfs 130:224 active ghost running

Configure Equipment


3. Configure multipathing on Windows hosts.

• Enable Windows MPIO on the host.• Install the Nimble Windows Toolkit utility for Windows systems from the Nimble support portal.

This utility provides the Nimble Device Specific Module (DSM) for MPIO and adds an entry for theNimble devices to the MPIO Devices list.

FIGURE 3 Windows MPIO Multipathing with Nimble Storage Array

4. Configure multipathing on VMware hosts. Change the path selection to Round-Robin. Round-Robinprovides superior performance to the Most Recently Used setting, which uses only a single activepath per LUN.

Configure Equipment


FIGURE 4 Round Robin Multipath Configuration on VMware

5. Set up workload generators. On Windows and Linux systems, Medusa Labs Test Tools is installed.On VMware systems, VMware I/O Analyzer is installed.

Configure Equipment


Test Cases

● 1.0 Fabric Initialization—Base Functionality................................................................... 22● 1.1 Fabric—Advanced Functionality............................................................................... 25● 1.2 Stress and Error Recovery with Device Multipath.....................................................32● 1.3 Storage Device—Fibre Channel Routing (FCR) Internetworking Tests....................41● 1.4 Optional/Additional Tests.......................................................................................... 45

1.0 Fabric Initialization—Base Functionality

Confirm basic Fibre Channel functionality of the storage array.

1.0.1 Storage Device—Physical and Logical Login with Speed Negotiation

1.0.2 Zoning and LUN Mapping

1.0.3 Storage Device Fabric I/O Integrity

1.0.4 Storage Device Multipath Configuration—Path Integrity

1.1 Fabric—Advanced Functionality

Examine the storage behavior related to more advanced fabric features suchas QoS, bottleneck detection, and advanced frame recovery.

1.1.1 Storage Device Bottleneck Detection Using MAPS FPI—With Congested Host

1.1.2 Storage Device Bottleneck Detection Using MAPS FPI—With CongestedFabric

1.1.3 Storage Device—QoS Integrity with QoS Zone Based Traffic Prioritization

1.1.4 Storage Device—QoS Integrity with CS_CTL-Based Traffic Prioritization

1.1.5 Storage Device—FC Protocol Jammer Test Suite

1.1.6 Clear Link Diagnostics (D_Port) Test

1.2 Stress and Error Recovery with Device Multipath

Confirm proper HA/failover behavior of storage in a multipath environment.

1.2.1 Storage Device Fabric I/O Integrity—Congested Fabric

1.2.2 Storage Device Nameserver Integrity—Device Recovery with Port Toggle

1.2.3 Storage Device Nameserver Integrity—Device Recovery with DeviceRelocation

1.2.4 Storage Device—Device Recovery with Device Port Toggle—Extended Run

1.2.5 Storage Device Recovery—ISL Port Toggle


1.2.6 Storage Device Recovery—ISL Port Toggle (Entire Switch)

1.2.7 Storage Device Recovery—Director Blade Maintenance

1.2.8 Storage Device Recovery—Switch Offline

1.2.9 Storage Device Recovery—Switch Firmware Download

1.3 Storage Device—Fibre Channel Routing (FCR) Internetworking Tests

Confirm proper storage functioning within routed fabrics.

1.3.1 Storage Device Internetworking Validation with the FC Host

1.3.2 Storage Device Internetworking Validation with FCoE Test Using VDX FlexPort

1.3.3 Storage Device Edge Recovery After FCR Disruptions

1.3.4 Storage Device Backbone Recovery After FCR Disruptions

1.4 Optional/Additional Tests

1.4.1 Storage Device Firmware Update

1.4.2 Performance I/O Testing with Varying Workload Dimensions

1.4.3 VMware I/O Analyzer Workload Suite


1.0.1 Storage Device—Physical and Logical Login with SpeedNegotiation

Test ObjectiveVerify device login to the switch and nameserver with all supported speed settings.

ProcedureTest Execution

1. Set switch ports to 4/8/16/Auto_Negotiate speed settings.

portcfgspeed <port> [4/8/16/0]Result Validation



1. Validate link states on the array, and verify speed negotiation and device login at different speeds.Use the portshow command on the switch to check the link state and speed, and use thenscamshow command to verify device login to the fabric.

2. Check the switch port status, and verify the “actual” and “configured” link speed. Check the nameserver for device login.

# nscamshow# portshow Xroot> portshow 19portIndex: 19portName: port19portHealth: HEALTHYportState: 1 OnlineProtocol: FCportWwn of device(s) connected: 10:00:8c:7c:ff:22:f7:81Distance: normalportSpeed: N16Gbps <-

ResultPASS. Storage logs into the fabric and is link up at 4Gb/8Gb/16Gb. I/O is run for verification.


Test ObjectiveVerify that host-to-LUN access exists with valid zoning.


1. Create an FC zone on the fabric with the initiator and target WWNs.2. Create host groups and LUNs on the array with access to the initiator WWN.

Result Validation

Verify that LUNs are discovered on the hosts with host-specific tools.

- Linux: Check output of lsscsi- Windows: Check output of Computer Management > Storage > Disk Management- VMware: Check output at Configuration > Storage > Devices

ResultPASS. The host has read/write access to the presented LUNs.

1.0.3 Storage Device Fabric I/O Integrity

Test ObjectiveValidate single-path host-to-LUN I/O with read/write/verify testing.




1. Set up read/write I/O to the LUN using Medusa.2. Perform link disruptions with port toggles/cable pulls.

Result Validation

Check Medusa I/O logs and verify that I/O resumes after a short downtime. Medusa I/O may pause,but should recover without errors.

ResultPASS. I/O resumes without errors.

1.0.4 Storage Device Multipath Configuration—Path integrity

Test ObjectiveVerify that multipath configures successfully, and verify the integrity of each path.


1. Set up the host with at least two initiator ports zoned with two target ports on the array.2. Set up multipath on the host.3. Start I/O.4. Isolate paths individually, and run I/O on the separate paths.

Result Validation

1. Check the host multipath properties to verify that the toggled path recovers.

- Windows: mpclaim –s –d - Linux: multipath –ll- VMware: Check the paths at Configuration > Storage > Devices > Manage Paths

2. Check the host and storage logs for any failures.3. Check the switch error logs and switch port status after toggling.

# errdumpall# portstatsshow X# portshow Xroot> portshow 19portHealth: HEALTHY <-. . .portState: 1 Online <-. . .portWwn of device(s) connected: 10:00:8c:7c:ff:22:f7:81 <-Distance: normalportSpeed: N16Gbps

4. Check I/O logs, and verify that I/O continues without errors.

ResultPASS. Each individual path is valid, and the paths combine successfully in a multipath configuration.

1.0.4 Storage Device Multipath Configuration—Path integrity



1.1.1 Storage Device Bottleneck Detection Using MAPS FPI—WithCongested Host

Test ObjectiveVerify that congestion on host ports is detected. Verify storage device and fabric behavior duringcongestion.


1. Configure MAPS and FPI on all switches. (For more information, see the “Configure Equipment”section.)

2. Start I/O from a single host initiator to multiple targets.3. Monitor switch logs for IO_PERF_IMPACT/ IO_FRAME_LOSS warnings.

Result Validation

Check switch error logs and the MAPS dashboard for bottleneck warnings.

# errdumpall | grep IO_root> errdumpall | grep IO_2014/12/17-11:56:00:672622, [MAPS-1003], 117148/115018, FID 128, WARNING, B6510_066_088, Port 16, Condition=ALL_F_PORTS(DEV_LATENCY_IMPACT==IO_PERF_IMPACT), Current Value:[DEV_LATENCY_IMPACT,IO_PERF_IMPACT, 30.0% in 10 secs], RuleName=defALL_F_PORTS_IO_PERF_IMPACT, Dashboard Category=Fabric Performance Impact., actionHndlr.c, line: 755, comp:md, ltime:2014/12/17-11:56:00:671909# mapsdb --show allroot> mapsdb --show1 Dashboard Information:=======================Active policy: dflt_aggressive_policy. . .2 Switch Health Report:=======================Current Switch Policy Status: HEALTHY3.1 Summary Report:===================Category |Today |Last 7 days |--------------------------------------------------------------------------------Port Health |No Errors |Out of operating range |Fru Health |In operating range |In operating range |Security Violations |No Errors |In operating range |Fabric State Changes |No Errors |In operating range |Switch Resource |In operating range |In operating range |Traffic Performance |In operating range |In operating range |FCIP Health |Not applicable |Not applicable |Fabric Performance Impact|Out of operating range |Out of operating range |



ResultPASS. With MAPS reporting configured, performance warnings are reported as expected. Thebottlenecked ports are displayed on the MAPS dashboard, and a RASLog warning is created.

1.1.2 Storage Device Bottleneck Detection Using MAPS—WithCongested Fabric

Test ObjectiveValidate bottleneck detection with congested fabric.


1. Configure MAPS on all switches. (For more information, see the “Configure Equipment” section.)2. Isolate a single ISL in the fabric.3. Start I/O from multiple host initiators to multiple targets.4. Monitor switch logs for traffic performance warnings.

Result Validation

Check switch error logs and the MAPS dashboard for bottleneck warnings.

# errdumpall | grep Traffic2014/12/18-08:39:03:497512, [MAPS-1003], 9637/5496, FID 128, WARNING, B6510_066_082, Port 33, Condition=ALL_TARGET_PORTS(RX/hour>60.00), Current Value:[RX,65.16 %], RuleName=defALL_TARGET_PORTSRX_60, Dashboard Category=Traffic Performance., actionHndlr.c, line: 755, comp:md, ltime:2014/12/18-08:39:03:496754# mapsdb --show allroot> mapsdb --show1 Dashboard Information:========================Active policy: dflt_aggressive_policy. . .2 Switch Health Report:=======================Current Switch Policy Status: HEALTHY3.1 Summary Report:===================Category |Today |Last 7 days |--------------------------------------------------------------------------------Port Health |No Errors |Out of operating range |Fru Health |In operating range |In operating range |Security Violations |No Errors |In operating range |Fabric State Changes |No Errors |In operating range |Switch Resource |In operating range |In operating range |Traffic Performance |In operating range |In operating range |FCIP Health |Not applicable |Not applicable |Fabric Performance Impact|Out of operating range |Out of operating range |

ResultsPASS. With MAPS reporting configured, performance warnings are reported as expected.

1.1.2 Storage Device Bottleneck Detection Using MAPS—With Congested Fabric


1.1.3 Storage Device—QoS Integrity with QoS Zone Based TrafficPrioritization

Test ObjectiveValidate QoS functionality.


1. Set up initiator-target pairs with Low/Medium/High QoS zones in the fabric.2. Start I/O across all pairs, and validate traffic priority.

Result Validation

1. Check I/O logs, and verify that I/O continues without errors.2. Check the switch error logs and the switch port status for errors.

# porterrshowroot> porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err 0: 1.7g 2.7g 0 0 0 0 0 0 0 946 1 0 1 0 0 0 0 0 1: 431.1m 431.4m 0 0 0 0 0 0 0 30 1 0 1 0 0 0 0 0 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

3. Verify traffic prioritization by examining performance and VC utilization.

ResultPASS. I/O completes without errors in low, medium, and high priority zones.

1.1.4 Storage Device—QoS Integrity with CS_CTL-Based TrafficPrioritization

Test ObjectiveValidate CS_CTL QoS functionality.


1. Enable ExpressLane on the Emulex adapter driver by setting the value to 1 and setting the priority to3 for high.

1.1.3 Storage Device—QoS Integrity with QoS Zone Based Traffic Prioritization


FIGURE 5 Configuring Driver Parameters in Emulex OneCommand Interface

2. Enable CS_CTL QoS Auto-Mode at the chassis level on all switches in the Brocade fabric.

> configurechassisConfigure...cfgload attributes (yes, y, no, n): [no]Custom attributes (yes, y, no, n): [no] system attributes (yes, y, no, n): [no] fos attributes (yes, y, no, n): [no] y Reboot needed to effect new CSCTL Mode CSCTL QoS Mode (0 = default; 1 = auto mode): (0..1) [0] 1

To verify:root> configshow -all | grep csctlfos.csctlMode:1

3. Enable CS_CTL QoS on initiator and target ports.

root> portcfgqos --enable 7 csctl_modeEnabling CSCTL mode flows causes QoS zone flows to lose priority on such ports.Do you want to proceed?(y/n):y

To verify:

Test Cases


root> portcfgshow 7 | grep -i csctlCSCTL mode: ON

4. Enable ExpressLane QoS on selected LUNs. For multipath LUNs, enable ExpressLane on eachdevice path:

FIGURE 6 Enabling LUNs for ExpressLane in the Emulex OneCommand Interface

5. Start high-throughput I/O to all LUNs.

Result Validation

• Check I/O logs, and verify that I/O continues without errors for all LUNs.• Verify that I/O performance is improved on ExpressLane-enabled LUNs.• Verify CS_CTL prioritization in the fabric by monitoring the high VC buffer credits on the ISLs.• Check the switch error logs and the switch port status for errors. # porterrshow

Test Cases


root> porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err 0: 1.7g 2.7g 0 0 0 0 0 0 0 946 1 0 1 0 0 0 0 0 1: 431.1m 431.4m 0 0 0 0 0 0 0 30 1 0 1 0 0 0 0 0 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

ResultPASS. Prioritized LUNs display higher performance, the traffic receives prioritization through thefabric, and I/O completes without errors.


Test ObjectivePerform FC Jammer tests including areas such as CRC corruption, packet corruption, missing frames,host error recovery, and target error recovery.


1. Insert the Jammer device in the I/O path on the storage link.2. Execute the following Jammer scenarios:

• Delete one frame.• Delete R_RDY.• Replace CRC of data frame.• Replace EOF of data frame.• Replace “good status” with “check condition.”• Replace IDLE with LR.• Truncate frame.• Create S_ID/D_ID error of data frame.

3. Verify Jammer operations and recovery with Analyzer.

Result Validation

Check the Medusa log, and verify that I/O recovers and completes without errors.

ResultPASS. Introduce packet anomalies, and verify proper recovery.




Test ObjectiveValidate the Clear Link Diagnostics feature on supported HBAs. The feature is tested on the EmulexLPe16202, QLogic 2672, and QLogic/Brocade 1860.


1. The Brocade switch can be configured in static, dynamic, and on-demand D_Port modes. The switchsupports dynamic D_Port by default at the chassis level.

root> configureConfigure... Fabric parameters (yes, y, no, n): [no] D-Port Parameters (yes, y, no, n): [no] y Dynamic D-Port (on, off): [on]On Demand D-Port (on, off): [off]

2. With the Emulex LPe16202 adapter, the switch port is in dynamic mode and the test is manuallyinitiated on the HBA port from the Emulex OneCommand Manager.

root> switchshow0 0 030000 id N16 In_Sync FC D-Port 10:00:00:90:fa:61:92:3b Dynamic

3. With the QLogic 2672 adapter, the HBA port is in dynamic D_Port mode and the test is initiated byconfiguring the switch port as a static D_Port. The test starts automatically after enabling the D_Portat the switch.

> portdisable <port#>> portcfgdport ––enable <port#>Caution: D_Port functionality is only available on 16Gb-capable platforms with



16Gb FC SFPs, 10Gb FC SFPs, 8Gb LWL/ELWL FC SFPs, QSFPs, or QSFP+.> portenable <port#>(reverse the above steps to disable D_Port mode)

To verify:root> portcfgshow <port#> | grep -i d-portD-Port mode: ONroot> switchshow7 7 581300 id N16 Online FC D-Port Loopback->Port 7

4. With the QLogic/Brocade 1860 adapter, the HBA port can be in static or dynamic D_Port mode.

Enabling static D_Port on the HBA:# bcu port --disable 2/0port disabled# bcu diag --dportenable 2/0D-port mode for port 2/0 enabled.

5. Execute the D_Port test, and verify that all tests pass and that no port and link problems arereported.

Result Validation

Check the D_Port test results on the switch ports and host HBA diagnostic utilities.

root> portdporttest --show 14D-Port Information:===================Port: 14Remote WWPN: 10:00:8c:7c:ff:23:b7:00Mode: AutomaticNo. of test frames: 1 MillionTest frame size: 1024 BytesFEC (enabled/option/active): Yes/No/NoCR (enabled/option/active): No/No/NoStart time: Wed Apr 15 12:02:53 2015End time: Wed Apr 15 12:03:52 2015Status: PASSED================================================================================Test Start time Result EST(HH:MM:SS) Comments================================================================================Electrical loopback 12:03:01 PASSED -------- --------Optical loopback 12:03:31 PASSED -------- --------Link traffic test 12:03:39 PASSED -------- --------================================================================================Roundtrip link latency: 157 nano-secondsEstimated cable distance: 3 metersBuffers required: 1 (for 2112 byte frames at 16Gbps speed)Egress power: Tx: -3.4 dBm, Rx: Not Avail.Ingress power: Rx: -3.3 dBm, Tx: Not Avail.

ResultPASS. The D_Port feature validates on all tested HBAs.





Test ObjectiveValidate I/O integrity in a congested fabric environment.


From all initiators, start a mixture of read, read/write, and write traffic continuously to all their targets fora 24-hour period.

Result Validation

1. Check the host and storage logs for errors.2. Verify link congestion, and check the switch logs for errors.

# errdumpall# portperfshow# porterrshowroot> porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err 0: 1.7g 2.7g 0 0 0 0 0 0 0 946 1 0 1 0 0 0 0 0 1: 431.1m 431.4m 0 0 0 0 0 0 0 30 1 0 1 0 0 0 0 0 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

3. Check I/O generator tool logs to verify that I/O runs without errors.

ResultPASS. In a congested fabric, I/O runs successfully without errors or failover.

1.2.2 Storage Device Nameserver Integrity—Device Recovery with PortToggle

Test ObjectiveValidate path recovery behavior on storage and host ports.


1. Set up multipath on the host, and start I/O.2. Perform multiple iterations of sequential port toggles across initiator and target switch ports.

Result Validation



1. Check switch port status after toggling, and check for errors in the switch logs.


2. Check the host multipath status on hosts to verify that the toggled path recovers.


3. Check host and storage error logs, and verify that I/O continues without errors.

ResultPASS. Failover between 8 logical paths (2 host x 4 storage) tests successfully. Paths recover, and I/Ocompletes without errors.

1.2.3 Storage Device Nameserver Integrity—Device Recovery withDevice Relocation

Test ObjectiveValidate storage device path recovery after physical port relocation.


1. Sequentially performed for each Storage Device port.2. Disconnect and reconnect port to different switch port in same fabric.

Result Validation

1. Check for any errors in the switch error logs and the switch port status at the new switch port.

# errdumpall# portstatsshow X# portshow Xroot> portshow 8portHealth: HEALTHY <-. . .portState: 1 Online <-. . .portWwn of device(s) connected: 52:4a:93:7d:f3:5f:61:11 <-

1.2.3 Storage Device Nameserver Integrity—Device Recovery with Device Relocation


Distance: normalportSpeed: N8Gbps

2. Check the multipath status on hosts to verify that the toggled path recovers.


3. Confirm that there are no errors in the Medusa log.

ResultPASS. The storage port successfully recovers after being physically moved. Paths recover, and I/Ocompletes without errors.


Test ObjectiveValidate path recovery behavior on storage and host ports for an extended duration.


1. Set up multipath on the host, and start I/O.2. Sequentially toggle each initiator and target port in the fabric (multiple iterations).3. Run the test for an extended period.

Result Validation

1. Check the switch port status after toggling, and check for any errors in the switch logs.


2. Check the multipath properties for iSCSI hosts to verify that the toggled path recovers.

On Windows: mpclaim –s –d On Linux: multipath –llOn VMware: Check the paths at Configuration > Storage > Devices > Manage Paths

3. Check host and storage logs, and verify that I/O continues without errors.

ResultPASS. 48-hour run: paths recover, and I/O completes without errors.




Test ObjectiveValidate path recovery and I/O integrity when ISL links are disabled.


1. Set up multipath on the host, and start I/O.2. Sequentially toggle each ISL path (one at a time) on all switches.

Result Validation

1. Check the FC fabric status after ISL toggles. Verify that all nodes are online.

# fabricshowroot> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:1b 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 3: fffc03 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 "B6510_066_073" 19: fffc13 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 >"B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"The Fabric has 5 switches <-Fabric Name: SSR

2. Check the switch logs for errors, and verify that I/O fails over to an alternate ISL path in the fabric.


3. Check the host and storage logs, and verify that I/O continues without errors.

ResultPASS. Paths recover, and I/O completes without errors.




Test ObjectiveValidate path recovery and I/O integrity when all ISL links on a switch are disabled.


1. Ensure ISL redundancy by provisioning multiple ISLs connected to different switches to providemultiple paths through the fabric.

2. Set up multipath on the host, and start I/O.3. On each switch, disable all ISL links simultaneously.

Result Validation

1. Check the FC fabric status after ISL toggles. Verify that all nodes are online.

# fabricshowroot> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:1b 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 3: fffc03 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 "B6510_066_073" 19: fffc13 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 >"B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"The Fabric has 5 switches <-Fabric Name: SSR








Test ObjectiveValidate path recovery and I/O integrity during director blade maintenance.


1. Uplink edge-switch ISLs to different blades on the directors.2. Sequentially toggle each director blade.3. Include blade disable/enable and blade power on/off.

Syntax:

slotpoweroff/slotpoweron, bladedisable/bladeenable, slotshowResult Validation

1. Check the FC fabric status after the blade toggles. Verify that all nodes are present in the fabric.

# fabricshowroot> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name------------------------------------------------------------------------- 1: fffc01 50:00:53:39:47:bd:6e:0a 0.0.0.0 0.0.0.0 "fcr_fd_1" 2: fffc02 10:00:00:05:1e:09:15:3e 10.38.51.5 0.0.0.0 "sw5300sus1005" 3: fffc03 10:00:c4:f5:7c:07:c2:0c 10.38.51.3 0.0.0.0 "sw300sus1003" 4: fffc04 10:00:c4:f5:7c:7b:86:1f 10.38.51.4 0.0.0.0 "sw6510sus1004" 5: fffc05 10:00:00:05:33:13:80:ef 10.38.51.24 0.0.0.0 "swBESsus1024" 13: fffc0d 10:00:00:05:1e:4c:ff:00 10.38.51.13 0.0.0.0 "sw8510sus1013" 16: fffc10 10:00:00:05:1e:44:02:00 10.38.51.16 0.0.0.0 >"sw8518sus1016" 22: fffc16 10:00:00:05:1e:07:7d:99 10.38.51.22 0.0.0.0 "sw5100sus1022" 23: fffc17 10:00:00:27:f8:2b:9f:aa 10.38.51.23 0.0.0.0 "sw6520sus1023" 25: fffc19 10:00:00:05:1e:54:8b:d4 10.38.51.25 0.0.0.0 "swBESsus1025" 26: fffc1a 10:00:00:05:33:8f:b6:77 10.38.51.26 0.0.0.0 "sw6510sus1026" 29: fffc1d 10:00:00:05:33:83:a4:00 10.38.51.29 0.0.0.0 "swDCX4sus1029" 32: fffc20 10:00:00:05:1e:b8:c1:00 10.38.51.32 0.0.0.0 "swDCXsus1032"160: fffca0 10:00:00:05:1e:a8:35:9d 10.38.51.21 0.0.0.0 "swVA40FCsus1021"171: fffcab 10:00:00:05:1e:d7:1a:04 10.38.51.171 172.172.172.26 "sw5480sus1171" 2620:100:4:fa00:205:1eff:fed7:1a04183: fffcb7 10:00:00:05:33:00:b2:e2 10.38.51.183 0.0.0.0 "swM5424sus1183"195: fffcc3 10:00:00:05:33:8d:df:d1 10.38.51.195 0.0.0.0 "sw5480sus1195"199: fffcc7 10:00:00:27:f8:44:33:61 10.38.51.199 0.0.0.0 "sw6547sus1199" 2620:100:4:fa00:227:f8ff:fe44:3362220: fffcdc 10:00:00:05:1e:86:1b:1f 10.38.51.220 0.0.0.0



"sw5450sus1220"The Fabric has 19 switches






Test ObjectiveValidate path recovery and I/O integrity during switch offline events.


1. Toggle each switch in sequential order.2. Include switch enable/disable, power on/off, and reboot testing.

Result Validation

1. Check the FC fabric status after the switch toggles. Verify that all nodes are present in the fabric.

# fabricshowroot> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:1b 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 3: fffc03 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 "B6510_066_073" 19: fffc13 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 >"B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"



The Fabric has 5 switches <-Fabric Name: SSR

2. Check the switch logs for errors, and verify that the toggled switch recovers.

# errdumpall# switchshowroot> switchshowswitchName: B6510_066_088switchType: 109.1switchState: Online <-switchMode: NativeswitchRole: SubordinateswitchDomain: 88switchId: fffc58switchWwn: 10:00:00:27:f8:06:23:28zoning: ON (SSR)switchBeacon: OFFFC Router: ONFC Router BB Fabric ID: 100Address Mode: 0Fabric Name: SSR_2


ResultPASS. Paths fail over and recover, and I/O completes without errors.


Test ObjectiveVerify that I/O continues with minimal disruption throughout switch firmware upgrade process.


1. Set up host multipath with links on different switches in the FC fabric, and start I/O.2. Sequentially perform firmware upgrades on all switches in the fabric.

Result Validation

1. Verify that the firmware upgrade completes successfully on each switch node and that the nodesmerge back in the FC fabric.

# versionroot> versionKernel: 2.6.14.2Fabric OS: v7.3.1 <-Made on: Thu Dec 11 14:30:38 2014Flash: Wed Dec 17 09:53:59 2014BootProm: 1.0.11# fabricshowroot> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:1b 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 3: fffc03 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 "B6510_066_073"



19: fffc13 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 >"B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"The Fabric has 5 switches <-Fabric Name: SSR

2. Check I/O generator tool logs to verify that I/O runs without errors throughout the firmware upgrade.3. Check the switch logs for errors, and verify that I/O resumes on the node after the firmware upgrade

is complete.

# errdumpall# portperfshow

ResultPASS. I/O fails over and recovers during the firmware download. Paths recover, and I/O completeswithout errors.

1.3 Storage Device—Fibre Channel Routing (FCR) InternetworkingTests

1.3.1 Storage Device Internetworking Validation with the FC Host

Test ObjectiveValidate that storage targets are imported successfully in a routed Fibre Channel environment.


1. Set up FCR in an edge-backbone-edge configuration.2. Set up LSAN zoning, verify host access to target LUNs, and start I/O.

Result Validation

1. Verify the name server and FCR fabric state.

# fcrfabricshowroot> fcrfabricshowFC Router WWN: 10:00:00:05:33:13:96:5a, Dom ID: 82,Info: 10.38.66.82, "B6510_066_082" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------ 40 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 41 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92"FC Router WWN: 10:00:00:05:33:5b:1d:1d, Dom ID: 83,Info: 10.38.66.83, "B6510_066_083" EX_Port FID Neighbor Switch Info (enet IP, WWN, name)

1.3 Storage Device—Fibre Channel Routing (FCR) Internetworking Tests


------------------------------------------------------------------------ 40 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 41 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088"# fcrproxydevshowroot> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric---------------------------------------------------------------------------- 40 10:00:00:05:1e:60:b4:6b 02ff02 100 132400 Imported 40 10:00:00:05:1e:60:b4:6c 02fe02 100 032400 Imported. . . 100 50:05:07:60:5e:80:76:52 01fe02 40 612800 Imported 100 50:05:07:60:5e:80:76:71 01ff02 40 642300 ImportedTotal devices displayed: 20

2. Verify I/O runs successfully without errors.

ResultPASS. All devices are available via the routed fabric. Running I/O confirms successful routing.

1.3.2 Storage Device Internetworking Validation with FCoE Test UsingVDX FlexPort

Test ObjectiveVerify storage device availability from FCoE initiators routed through the VCS fabric.


1. Set up FCoE host initiator ports on the VCS fabric using Brocade VDX 6740 switches.2. Enable Fibre Channel support on the VDX switch ports (for more details, see the "Configure

Equipment" section).3. Connect the VCS fabric to the FC fabric via FCR ports.4. Configure LSAN zones in the VCS and FC fabrics.

Result Validation

1. Verify the name server and FCR fabric state.

# fcrfabricshowroot> fcrfabricshowFC Router WWN: 10:00:00:05:33:5b:7d:06, Dom ID: 97,Info: 10.38.66.97, "B6510_066_097" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------ 18 110 10.38.66.120 10:00:50:eb:1a:62:8c:33 "VDX6740_066_120"FC Router WWN: 10:00:00:05:33:5b:10:e8, Dom ID: 100,Info: 10.38.66.100, "B6510_066_100" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------ 8 110 10.38.66.119 10:00:50:eb:1a:62:83:7b "VDX6740_066_119"# fcrproxydevshow

1.3.2 Storage Device Internetworking Validation with FCoE Test Using VDX FlexPort


root> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric---------------------------------------------------------------------------- 110 50:05:07:60:5e:80:76:52 02f001 120 612800 Imported 110 50:05:07:60:5e:80:76:71 02f101 120 642300 Imported 120 10:00:8c:7c:ff:00:48:00 03f101 110 781000 Imported 120 10:00:8c:7c:ff:00:48:01 03f001 110 771000 ImportedTotal devices displayed: 4

2. Verify that I/O runs successfully without errors.

ResultPASS. Storage targets are available through the FCoE/FC routed configuration; I/O completes withouterrors.


Test ObjectiveValidate that storage target paths recover successfully from disruptions in a routed Fibre Channelenvironment. Edge-backbone-edge configuration.


1. Set up FCR in an edge-backbone-edge configuration.2. Set up LSAN zoning.3. With I/O running, perform sequential reboots, switch disables, and ISL port toggles on the switches in

the backbone fabric.

Result Validation

1. Verify the FCR fabric state throughout the disruptions.

# fcrfabricshowroot> fcrfabricshowFC Router WWN: 10:00:00:05:33:13:96:5a, Dom ID: 82,Info: 10.38.66.82, "B6510_066_082" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------ 40 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 41 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92"FC Router WWN: 10:00:00:05:33:5b:1d:1d, Dom ID: 83,Info: 10.38.66.83, "B6510_066_083" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------ 40 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 41 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088"# fcrproxydevshowroot> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric----------------------------------------------------------------------------



40 10:00:00:05:1e:60:b4:6b 02ff02 100 132400 Imported 40 10:00:00:05:1e:60:b4:6c 02fe02 100 032400 Imported. . . 100 50:05:07:60:5e:80:76:52 01fe02 40 612800 Imported 100 50:05:07:60:5e:80:76:71 01ff02 40 642300 ImportedTotal devices displayed: 20

2. Check the switch logs for errors.


3. Check host and storage logs, and verify that I/O runs without errors.

ResultPASS. Paths fail over and recover from FCR disruptions, and I/O completes without errors.


Test ObjectiveValidate that storage target paths recover successfully from disruptions in a routed Fibre Channelenvironment. Edge-backbone configuration.


1. Set up FCR in an edge-backbone configuration.2. Set up LSAN zoning.3. With I/O running, perform sequential reboots, switch disables, and ISL port toggles on the switches

in the backbone fabric.

Result Validation

1. Verify the FCR fabric state throughout the disruptions.

# fcrfabricshowroot> fcrfabricshowFC Router WWN: 10:00:00:05:33:13:96:5a, Dom ID: 82,Info: 10.38.66.82, "B6510_066_082" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------ 40 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 41 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92"FC Router WWN: 10:00:00:05:33:5b:1d:1d, Dom ID: 83,Info: 10.38.66.83, "B6510_066_083" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------ 40 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 41 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088"# fcrproxydevshowroot> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric----------------------------------------------------------------------------



40 10:00:00:05:1e:60:b4:6b 02ff02 100 132400 Imported 40 10:00:00:05:1e:60:b4:6c 02fe02 100 032400 Imported. . . 100 50:05:07:60:5e:80:76:52 01fe02 40 612800 Imported 100 50:05:07:60:5e:80:76:71 01ff02 40 642300 ImportedTotal devices displayed: 20

2. Check the switch logs for any errors.


3. Check host and storage logs, and verify that I/O runs without errors.

ResultPASS. Paths fail over successfully, and I/O completes without errors.


1.4.1 Storage Device Firmware Update

Test ObjectiveExecute a nondisruptive firmware update on the array while running I/O, and confirm that there are noI/O errors.


1. Run continuous I/O to the array.2. Execute the code update procedure as described in vendor documentation.

Result Validation

1. Confirm the updated version on all array components.2. Check the I/O generator tool logs to verify that I/O completes without errors.3. Check the host and storage logs for errors throughout the I/O operations.4. Check the switch logs and port stats for errors or I/O drops.

# errdumpall# porterrshowroot> porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err 0: 1.7g 2.7g 0 0 0 0 0 0 0 946 1 0 1 0 0 0 0 0 1: 431.1m 431.4m 0 0 0 0 0 0 0 30 1 0 1 0 0 0 0 0 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0



ResultPASS. Update successful (from 2.2.3.0-167018-opt to 2.2.5.0-197583-opt) with proper failover and noI/O errors.


Test ObjectiveRun I/O workload generation with varying dimensions (block size, read/write/mix), and verify thatperformance characteristics are as expected.


1. Run an I/O loop at block transfer sizes of 512, 1k, 2k, 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, and1m.

2. Include a nested loop of 100-percent read, 100-percent write, and 50-percent read/write.

Repeat the test for the following configurations:

• 1 host port to 1 target port (single path)• 2 host ports to multiple target ports (multipath)• Multiple hosts to multiple target ports (multihost multipath)

Result Validation

1. Check the I/O generator tool logs to verify that I/O completes without errors.2. Check the host and storage logs for errors throughout the I/O operations.3. Check the switch error logs and port stats for errors or I/O drops.


ResultPASS. All workload runs are monitored at the host, storage, and fabric and complete without I/O errorsor faults. Performance behavior is as expected.



1.4.3 VMware I/O Analyzer Workload Test

Test ObjectiveRun a workload test suite including varying application workloads generated from multiple VMs.


1. Configure a two-host VMware cluster with multipath on two initiator ports per host, and four targetports.

2. Configure workload generation from eight worker VMs using VMware I/O Analyzer.3. Run a variety of application I/O workload patterns from the VMware I/O Analyzer suite. For this round

of testing, the following workloads are used:

• Workstation• Webserver• Video on Demand• Max Write Throughput• Max Write IOPS• Max IOPS• Max Throughput• SQL Server 16k• 4k Read• Exchange 2007• OLTP 4k

Result Validation

1. Check the I/O generator tool logs to verify that I/O completes without errors.2. Check the host and storage logs for errors throughout the I/O operations.3. Check the switch logs and port stats for errors or I/O drops.


ResultPASS. All workload runs are monitored at the host, storage, and fabric and complete without I/O errors.

