designed for integration zstuff is ... - ibm-zcouncil.com · 8/29/2017 · ibm zhyperlink delivers...

1 Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. "Broadcom" refers to Broadcom Limited and/or its subsidiaries.|Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. "Broadcom" refers to Broadcom Limited and/or its subsidiaries.Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. "Broadcom" refers to Broadcom Limited and/or its subsidiaries.

Designed for IntegrationzStuff is Faster! And so are we... Z14, Brocade Gen6, and Beyond

Tim Jeka – Field Application Engineer

[email protected]

November 2018

mailto:[email protected]

2 Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.|

This presentation contains forward-looking statements (including within the meaning of Section 21E of the United States Securities Exchange Act of 1934, as amended, and Section27A of the United States Securities Act of 1933, as amended) concerning Broadcom Limited (“Broadcom” or the “Company”). These statements include, but are not limited to,statements about historical results that may suggest trends for our business and other statements identified by words such as “will”, “expect”, “intends”, “believe”, “anticipate”,“estimate”, “should”, “intend”, “plan”, “potential”, “predict” “project”, “aim”, and similar words, phrases or expressions. These forward-looking statements are based on currentexpectations and beliefs of the management of Broadcom, as well as assumptions made by, and information currently available to, such management, current market trends andmarket conditions and involve risks and uncertainties, many of which are outside the Company’s and management’s control, and which may cause actual results to differ materiallyfrom those contained in forward-looking statements. Accordingly, you should not place undue reliance on such statements.

Particular uncertainties that could materially affect future results include risks associated with our proposal to acquire Qualcomm, including: (i) uncertainty regarding the ultimateoutcome or terms of any possible transaction between Broadcom and Qualcomm, including as to whether Qualcomm will cooperate with us regarding the proposed transaction; (ii) theeffects of the announcement of the proposed transaction on the ability of Broadcom and Qualcomm to retain customers, to retain and hire key personnel and to maintain favorablerelationships with suppliers or customers; (iii) the timing of the proposed transaction; (iv) the ability to obtain regulatory approvals and satisfy other closing conditions to the completionof the proposed transaction (including shareholders approvals); and (v) other risks related to the completion of the proposed transaction and actions related thereto; any loss of oursignificant customers and fluctuations in the timing and volume of significant customer demand; our dependence on contract manufacturers and outsourced supply chain; ourdependency on a limited number of suppliers; any acquisitions we may make, such as delays, challenges and expenses associated with receiving governmental and regulatoryapprovals and satisfying other closing conditions, and with integrating acquired companies with our existing businesses and our ability to achieve the benefits, growth prospects andsynergies expected from such acquisitions, including our acquisition of Brocade Communications Systems, Inc.; our ability to accurately estimate customers’ demand and adjust ourmanufacturing and supply chain accordingly; our significant indebtedness, including the need to generate sufficient cashflows to service and repay such debt; increased dependenceon a small number of markets and the rate of growth in these markets; dependence on and risks associated with distributors of our products; dependence on senior management;quarterly and annual fluctuations in operating results; global economic conditions and concerns; our proposed redomiciliation of our ultimate parent company to the United States; ourcompetitive performance and ability to continue achieving design wins with our customers, as well as the timing of those design wins; prolonged disruptions of our or our contractmanufacturers’ manufacturing facilities or other significant operations; our ability to improve our manufacturing efficiency and quality; our dependence on outsourced service providersfor certain key business services and their ability to execute to our requirements; our ability to maintain or improve gross margin; our effective tax rate, legislation that may impact oureffective tax rate and our ability to maintain tax concessions in certain jurisdictions; our ability to protect our intellectual property and the unpredictability of any associated litigationexpenses; any expenses or reputational damage associated with resolving customer product and warranty and indemnification claims; cyclicality in the semiconductor industry or in ourtarget markets; our ability to sell to new types of customers and to keep pace with technological advances; market acceptance of the end products into which our products aredesigned; and other events and trends on a national, regional and global scale, including those of a political, economic, business, competitive and regulatory nature.

Our filings with the Securities and Exchange Commission (“SEC”), which you may obtain for free at the SEC’s website at http://www.sec.gov, discuss some of the important risk factorsthat may affect our business, results of operations and financial condition. We undertake no intent or obligation to publicly update or revise any of these forward looking statements,whether as a result of new information, future events or otherwise, except as required by law.

Safe Harbor Statement

http://www.sec.gov/

3 Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.|

**Raffle for Fabulous Prize!**

WONDERBOOM Wireless Speaker Drawing

4 Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. "Broadcom" refers to Broadcom Limited and/or its subsidiaries.|

Agenda

Z14Brocade FICON Roadmap

• IBM and Broadcom

• Trends

• IBM z14 Evolution

• Designed for Integration

• zHyperlink

• Why 32GB?

• Transparent Cloud Tiering

• Multi-Hop FICON

• SAN Health

• Raffle for Fabulous Prizes


• Relationship– System z Brand

– System z I/O Technology Teams

– FICON Qualification Lab

• Standards– FC-SB, FC-PI, FC-SW, FC-BB

• Technology and Strategies– Joint Development for System z

IBM

System z Relationship


– System z I/O Technology

– Chris Colonna– Dale Riedy– Dave Peterson– Harry Yudenfriend– Mark Bendyk

• Standards– FC-SB, FC-PI, FC-

SW, FC-BB– Roger Hathorn– Patty Driever

• Relationship

– System z Brand– Ray Newsom

– Program Management

– Joe Turner

• FICON Qualification

– Lab– Andrew Crimmins– Mario Borelli– PJ Catalano

– Support– Dan Rinck– David Raczkowski– Len DiBella

IBM zSystem

Relationship


Brocade mainframe product timeline

FICON Introductions• 08/2002 2 Gbps FICON

• 05/2002 FICON / FCP Intermix

• 11/2001 FICON Inband Mgmt

• 04/2001 64 Port Director



• 09/2006 4 Gbps FICON

ESCON Introductions• 10/1994 9032 ESCON Directors• 08/1999 FICON Bridge

Bus/Tag and ESCON Extension• 1986 CTC Extension/B&T

• 1991 High Speed Printer Extension

• 1993 Tape Storage Extension

• 1993 T3/ATM WAN Support

• 1995 Disk Mirroring SupportESCON & FICON Extension• 1998 IBM XRC Support

• 1999 Remote Virtual Tape

• 2001 FCIP Remote Mirroring

• 2003 FICON Emulation for Disk

• 2005 FICON Emulation for Tape

1987 1990 2000 2001 2002 2003 2004 2005 2007 2008 20091997 2012

ED-5000M6140

M6064

i10K

9032

48000B24000 DCXFC9000

DCX-4S

DCX 8510

2015

Channelink

USD82xx

Edge

USDX 7500 &

FR4-18i

7800 &

FX8-24

7840

More FICON Introductions• 02/2008 DCX Backbone

• 02/2008 768 Port Platform

• 02/2008 Integrated WAN

• 03/2008 8 Gbps FICON

• 05/2008 Acceleration for FICON Tape

• 11/2009 New FCIP Platforms

• 12/2011 DCX 8510

• 01/2012 16 Gbps FICON

• 05/2016 X6 Directors

• 10/2016 32 Gbps FICON

2016

SX6

X6


In the News: Broadcom to Acquire Brocade

November 2, 2016


In the News: Broadcom to Acquire CA Technologies

July 11, 2018


The Future of Storage Protocols, June 2016

Storage performance bottlenecks are moving out of arrays and into the storage network, so Fibre Channel will remain as the data center storage protocol of choice for the next decade.

IT leaders planning to upgrade their storage networking need to use 16Gb/sec FC today, with a migration plan for moving to 32Gb/sec FC

Storage networking investments are becoming a critical top priority due to the adoption of high-throughput, solid-state and flash storage…

Industry View on Fibre Channel


Modernize all aspects of your Data Center

11

IBM 9150Scales to 10M IOPs

Flash

NVMe

Disk

STORAGE

What is missing?

The Data Path…

COMPUTE

… is critical to ensure:

Performance Security

Scalability Visibility

IOPs, Latency, Bandwidth.. Oh my!

If you do not plan for the network, then your

solution will be at risk of performance

bottlenecks, availability concerns, and other

potential issues that could jeopardize your

applications accessing their data..


Modernize all aspects of your Data Center

COMPUTE

b-type Storage Networking

DATA: Connect IT. Protect IT. Own IT.

Flash

NVMe

Disk

STORAGE

FIBRE CHANNEL is a network built for Storage!

IBM

9150

10M

IOPs

13 Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. "Broadcom" refers to Broadcom Limited and/or its subsidiaries.| 13

IT Infrastructure Modernization is StrengthenedWhen Attached with Gen 6 Networking

Storage networking ready for the Cognitive Era

Real-time Insight

Ultra-low Latency

Always-on Services

Rapid Application

Delivery

Real-timeEncryption

AutomationMassive

ScalabilityDeep

VisibilityPredictable

PerformanceGreater

AvailabilityIntegratedSecurity

Granular Application

Insights & Actions

Self LearningSelf Healing

2xmore

transactions with same storage compared to 16 Gbps data

rate

12TB

/per fabric

for 10s of thousands of workloads on scores of arrays1

50x

fasterdisaster

recovery and business

continuity5

50%fewer

I/O and storage network issues4

48%reduction in maintenance

costs2

52secrisk of downtime/yearto maintain five-9s of

availability expectations(vs 95 min average3)


Fibre Channel the Next Generation

GEN7 which doubles the speeds yet again, is already on the horizon!

• Serial and Parallel Link Speeds… 32Gb, 16Gb, 8Gb

Generation 6th Gen 7th Gen 8th Gen

Electrical / Optical Module 32GFC and

128GFCp /SFP+

and QSFP28

64GFC and

256GFCp /SFP+ and

QSFP56

128GFC and

512GFCp /SFP+ and

QSFP112

Electrical Speeds (Gbps) 1 lane of 28.05

4 lanes at 28.05

1 lanes of 56.1

4 lanes at 56.1

1 lane of 112.2

4 lanes at 112.2

QSFP28 QSFP56 QSFP112

SFP+

Best PracticesCleaning Optics at 8G and above!

SM Cable w/oil or dust.


• Two Gen 7 Director chassis models – Utilizes the X6 chassis (same CP blades, Power Supplies, Fans, Backplane etc)

– New 48p Gen 7 FC port blade having line rate backplane performance

– Support for 8/10/16/32/64G FC speeds

– 8-Slot director with of 384 x64G device ports

– 4-Slot director with of 192 x64G device ports

– New Gen 7 Core blades

• Consolidate all director offerings into Gen 7 director family

– Gen 6 blades (48p, 64p and SX6) to be supported in Gen 7 Director base chassis

– Allow for an upgrade from X6 to Gen 7 director with Core Routing blade upgrade

• Gen 7 Fixed Port Switch– Mid-Range platform

– Offering details are under development

Gen 7 Product PlansDirectors, Blades and Switches

Brocade Gen 7 Directors: same

chassis as X6

7

48-Port

blade

8-slot Core

blade4-slot Core

blade

Fixed Port Switch


Spectre: The Next Generation of SAN Management

Dramatically Simplify SAN Management. Increase Operational Efficiency.

Simplified ManagementModern Web UI, No Java

Client

User workflow focused

Enhanced MonitoringModern health summary dashboard

Context based topology view

Rapid TroubleshootingInstant view of fabric-wide conditions

Powerful log analysis, historical and real-time data

Flexible DeploymentDeploy on a single server or in a cluster of low-end servers


Emerging NVMe over Fibre Channel

17

• NVMe over Fibre Channel is relatively new, even thought its component parts are not!

Gen 5 and Gen 6 support FC-NVMe• Fibre Channel has been the leading enterprise storage technology

since the mid-1990’s

• Speeds of 16Gbps (GEN 5) are widespread.

• Gen 6 (2016), delivering twice the speed of GEN 5 selling like

hotcakes…

• GEN 7, which will double the speeds yet again, is already on the

horizon

• Fibre Channel does not have a RDMA protocol so FC-NVMe uses FCP

for data transfers.

• You can deploy NVMe over Fibre Channel on an existing Broadcom FC

infrastructure, providing it is relatively up to date!

• FC-NVMe-oF is aimed at 32GB Fabrics

• NVMe-oF will improve the performance of the FC SAN environment,

applications will receive performance enhancements!


Investment Protection: No Rip Out and Replace

Enterprise NVMe Over Fibre Channel Requires a Network!

• NVMe and SCSI coexist in the same server and SAN

• The Fibre Channel fabric must be Gen 5 (16 Gbps) or better yet Gen 6 (32 Gbps)

• Servers using NVMe over Fibre Channel need Gen 6 (HBAs); Gen 6 HBAs also work with Gen 5 fabrics.

• You need a storage device that supports FC-NVMe frame types. This can be an FC-NVMe-enabled array

• Existing Gen 5 and 6 SANs can run NVMe without code changes or disruptions

SCSI

Emulex HBAs by

SCSISCSI

Gen 5 HBAs

NVMe NVMe

NVMe Traffic

SCSI Traffic

SCSI over FC, TranslatedNVMe over FC, Natively

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwio_vPNxafJAhWUSYgKHRtKC5kQjRwIBw&url=http://icones.pro/en/schedule-today-png-image.html&bvm=bv.108194040,d.cGU&psig=AFQjCNF13h4UosjSsGvswY8swEegYpbnaw&ust=1448402396466718

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwio_vPNxafJAhWUSYgKHRtKC5kQjRwIBw&url=http://icones.pro/en/schedule-today-png-image.html&bvm=bv.108194040,d.cGU&psig=AFQjCNF13h4UosjSsGvswY8swEegYpbnaw&ust=1448402396466718

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwj65JPjxafJAhUHNIgKHUh5A_kQjRwIBw&url=http://www.chetna-dfw.org/?page_id%3D740&bvm=bv.108194040,d.cGU&psig=AFQjCNFK6L3owDPkU2ahras74YdALPzy1g&ust=1448402452962657

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwj65JPjxafJAhUHNIgKHUh5A_kQjRwIBw&url=http://www.chetna-dfw.org/?page_id%3D740&bvm=bv.108194040,d.cGU&psig=AFQjCNFK6L3owDPkU2ahras74YdALPzy1g&ust=1448402452962657


FOS v8.2 Summary…. IBM …8-28-2018 Announced

19

New Gen 6 Hardware • New ASIC – Condor 4+

- Support for I/O and VM Insight

- Supports I/O Insight w/NVMe over Fabrics

- Supports connectivity to FCoE Severs

64-port Gen6 Blade

128-port Gen6 Enterprise Switch

IBM Network Advisor 14.4 Required

FOS v8.2Platform support….. Customer Focus

New NVMe technology

Latency performance optimization

Enhanced IO performance metrics for NVMe

New automation technology

REST APIs available directly from the switch

Open source PyFOS, a Python scripting language

Integration with Ansible orchestration

Gen6 64-port Blade

Gen6 IBM SAN128B-6 128-port

Enterprise Switch

20 Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. "Broadcom" refers to Broadcom Limited and/or its subsidiaries.|SAN768B-2SAN384B-2

SAN24B-5

SAN48B-5

SAN96B-5SAN42B-RSAN06B-R

Extension Solutions

Fibre Channel

Adapters &

StorageSAN512B-6

SAN128B-6

SAN64B-6

SAN24B-6

SAN256B-6

FlashSystem 900 9150 1.7M IOPS

Scales to 10M IOP

DS8880

High End

Server

zOS

Power HA

Heterogeneous

Virtualization

FlashCore Technology

Application acceleration

Enterprise Replication

V7000 Next

GEN

Virtualization

for Mid-Range

FS9100

Enterprise Class

NVMe

Accelerated

V5030F

Virtualization for

Entry / Mid-Range

A9000

Cloud Service

Providers

A9000R

All-Flash

Data

Centers

IBM Storage and Storage Networking Portfolio

150K IOPS

3M IOPS

9150 1.7M IOPS

Scales to 10M IOP Scales to 2.5M IOPS

3M IOPS

900K IOPS


New IO Options for z14 & IBM DS8880 Storage

IBM zHyperLink ▪ zHyperLinks used for only for specific latency sensitive

applications (DB2)

▪ Must be used in conjunction with FICON paths

▪ zHyperlinks supports only direct connect to IBM DS8880

FICON Express 16S+▪ Delivers up to 3X performance for small I/O operations

compared to z13 with FICON Express 16S

▪ Uses existing SAN infrastructure. Ideal with Gen 6.

Configurations▪ zHyperLink and High Performance FICON operate

concurrently connected to DS8xxx storage

▪ zHyperLink provides short distance (150m) direct connect

option

▪ Long-distance configurations and stretch sysplex are

performed over FICON

DS8880F card enables

two zHyperLink

connections per IO bay

FICON Express

16S+ Channel

Card

zHyperLink

zHPFzHPF


Z13/z13s…I/O Infrastructure has to Keep Pace!Enhancing the value of the I/O InfrastructureZ13/z13s…I/O Infrastructure has to Keep Pace!Enhancing the value of the I/O Infrastructure

FICON Express 16S Features

FICON 16G Chip Set

Faster Transaction Performance

Forward Error Correction Code

Multi-hop zHCP

FICON Dynamic Routing

Read Diagnostic Parameters

New CUP Facility

Port Decommission

IBM zHealth Checker Enhancements

NPIV Enhancements

FICON Fabric Priority

• This Big Beast needs to be fed a hearty diet of data:

• FICON Express16S (98,000 IOPS with zHPF)

• Supports 160 FICON Express16S cards

• Supports 320 16Gbps CHPID channel ports @ 2800 MBps Full Duplex

• IBM’s z13/z13s added new Intelligent I/O with the FICON Express 16S Chip Set..

I/O

Enhancements

designed with

Brocade

Engineering

Looks Familiar…. Same

I/O Enhancements

Same 16Gb chip set as

z13/z13s

TS7760 16Gb FICON Card GA’d

March

▪ Key Hardware Enhancements to the DS888X

• 16GFC FCP/FICON Host Adapter reduces latency

Minimum Latency <1ms

▪ Key Function Enhancements to the DS888X

• Forward Error Correction

• FC Read Diagnostic Parameters

• FICON Dynamic Routing

• Fabric IO Priority

• zHPF Performance for distance

DS888X

3.0 IOPs

FICON 16G Chip Set


• 3-4x IOPS increase for application acceleration

• FICON Dynamic Routing reduce traffic congestion, simplifies configuration and

optimizes performance

• 66% I/O improvement writing data remotely with zHPF Enhanced Write Protocol

• Integration with IBM z/OS Work Load Manager SAN Priority

Performance

for Cognitive

• Integrated Read Diagnostic Parameters to non-disruptive find trouble spots

• Insight into the network health and performance with Fabric Vision technology

• Maximizes scalability with ICLs without using up ports for ISLs and adding hops Simplified

Management

Trust

• Security with In-flight Encryption between switches and sites.

• Health Checker (Z/OS) discovers single points of failure & analyze flow

characteristics

• Increase reliability with automated Port Decommission and Recommission

• Forward Error Correction enables recovery from errors in links, enhancing

transmission reliability and performance

Trust Z: Self Healing, Self Tuning, Self Diagnosing SAN InfrastructureBrocade and IBM have released a series of features and functions specifically designed to address the needs and demands of the data center…..


Z14…zStuff is Faster …and so are we!

FICON Express 16S Features

FICON 16G Chip Set

Faster links for I/O Latency.


zHPF Extended Distance II


New CUP Facility

Port Decommission

zHealth Checker

NPIV Enhancements


• This Big Beast needs to be fed a hearty diet of data:

• FICON Express16S+ (320K IOPS with zHPF)

• Supports 160 FICON Express16S+ cards

• 160 cards == 320 maximum ports 2 ports per card)

• Each “card” can deliver 320K IOPs, but each card has 2-ports, so the theoretical maximum is 160-cards x 320K IOPs

• IBM’s z14 added new Intelligent I/O with the FICON Express 16S+ Chip Set..

Looks Familiar…. Same I/O Enhancements

Same 16Gb chip set as z13/z13s

DS8880 will have 16s+ TBD

TS7760 16Gb FICON GA’d March 2018

▪ Key Hardware Enhancements to the DS888X

• 16GFC FCP/FICON Host Adapter reduces

latency

Minimum Latency <1ms

▪ Key Function Enhancements to the DS888X

• Forward Error Correction

• FC Read Diagnostic Parameters

• FICON Dynamic Routing

• Fabric IO Priority

• zHPF Performance for distance

DS888X

3.0 IOPs

FICON 16S+ Chip Set

FICON Express 16S+ Features

FICON 16G S+ Chip Set



Multi-hop zHCP



New CUP Facility

Port Decommission

IBM zHealth Checker Enhancements

NPIV Enhancements


I/O Enhancements designed

with Brocade Engineering


zStuff is Faster!

And so are we

© 2017 BROCADE COMMUNICATIONS SYSTEMS, INC.

20000 5200092000 98000

0

50000

100000

150000

200000

250000

300000

350000

300,000

I/O driver benchmark I/Os per second4k block sizeChannel 100% utilized

FICON Expres

s 8

FICON Expres

s 8zHPF

FICON Express

8SzHPF

FICON Express

16S+zHPF

z14FICON

Express 16S

zHPF

306%increase

z13EC12z196z10

STI: Self-Timed Interconnect

6 GBps

2 GBps

16 GBps

8 GBps

2.7 GBps

STIz990/z890

eSTIz9

InfiniBandzEC12/zBC12z10/z196/z114

PCIe Gen2zEC12/zBC12

z196/z114

PCIe Gen3z14

z13, z13s

https://fibrechannel.org/roadmap/

https://fibrechannel.org/roadmap/


IBM Z: The Trusted Platform for your Data Path

26

Serves up your DATA

with improvements in

speed and efficiency

Analyzes in real time

your DATA for new

offering insights

Simplifies management

of your DATA

Serves up your DATA with

improvements by 4X more

transactions

Protects your DATA with

inflight encryption

Simplifies management of

your DATA Path by automating

20 years of SAN best practices

Analyzes your DATA FLOW

for insights into application

performance issues

Protects your DATA with

pervasive encryptionz14

IBM b-type

Do We See A Pattern Here?


What is zHyperlink?


IBM zHyperLink delivers NVMe-oF like latencies for the Mainframe!

• zHyperLink Express is a direct connect short distance IBM Z I/O feature designed to work in conjunction with a FICON or High Performance FICON SAN infrastructure

• IBM zHyperLink™ dramatically reduces latency by interconnecting the z14 CPC directly to the I/O Bay of the DS8880…. 5x to 10x

• zHyperLink improves application response time, cutting I/O sensitive workload response time in half without significant application changes.

IBM z14 CPC

IBM DS8880

Db2

z/OS

IBM z14

Hardware

SANzHyperLink

Express


How does IBM zHyperLink™ change the game?

• IBM zHyperLink™ is the result of an IBM research project created to provide extreme low latency links between the IBM Z and the DS8880

• Two Fibre Options available for zHyperlink Express

– 150m: OM4 50/125 micrometer multimode fiber optic cable with a fiber bandwidth @wavelength: 4.7 GHz-km @ 850 nm.

– 100m: OM3 50/125 micrometer multimode fiber optic cable with a fiber bandwidth @wavelength: 2.0 GHz-km @ 850 nm.

• New zHyperLink compliment, do not replace,

FICON channels

− Standard FICON channel (CHPID type FC)

is required for exploiting the zHyperLink

Express feature

•

• New industry I/O link for Mainframe

SANFICON/zHPF

<20 μsec

>50,000 IOOPs/SEC

8 GB/sec

zHyperLink

IBM Z

CPC

DS8880


Primary/Secondary

Storage Subsystem

Optic

s

Node

1

Node

2

Optic

s

zHyperLink configuration

z14

Optics

Optics

Optics

Optics

zHyperLink

Adapter

zHyperLink

Adapter

FICON

FICON

Optics

Node

1

Node

2

Primary/Secondary

Storage Subsystem

Optics

SANz14

Optics

Optics

Optics

Optics

zHyperLink

Adapter

zHyperLink

Adapter

FICON

FICON

SANO

ptics

Optics

Optic

sO

ptic

s

PPRC

PPRC

If I/O Frames across the zHyperlink does not

complete.. Cache Hit, IFCC, the controller terminates

the I/O, IOS retries the I/O across the FICON Channel.


What are we announcing?

TS7700 R4.1 Phase 2- New! 16 Gb FICON support

- New! Enhanced data compression

External announce date: February 6th, 2018

GA: March 9, 2018


R4.1.2: 16Gb FICON

•Latest Generation 16Gb FICON AdapterTwo ports active per adapter (8 ports total per TS7700)

512 paths per port

Built in ASIC providing real-time compression and end to end IBM Z CRC protection

•Additional EnhancementsForward Error CorrectionImprove 16Gb error recoveryRead Diagnostics Parms

Improved link error isolation

•Improved PerformanceUp to 4.2GB/s for 32KB blocks @ 5:1 compression.Up to 380 MB/s single stream rates4.7GB/s to 4.8GB/s for large block (64KB to 256KB)


Now the Glue…. Gen 6 FICON ProductsData center modernization starts with the new Brocade/IBM FICON Director

Industry-first Gen 6 Fibre Channel director family

• 8-slot/4-slot director and 64-port switch

High-speed device and ICL connectivity

• 48x32Gbps port blade– Providing 384-ports per chassis

• Up to 32x128Gbps UltraScale ICL ports

Fibre Channel and IP storage replication solution

• Brocade SX6 Extension Blade

Brocade Directors

FC32-48 Blade

SX6 Extension Blade

SAN64B-6 Switch

All SANsManagers

SoftwareAnd

Dashboards

SAN512B-6 SAN256B-6


• Protect all application and database data

according to enterprise security policy

• Encryption of data encryption in-flight and at-

rest without application changes and no

impact to SLAs.

• The best Disaster Recovery solution in the

industry with more than “six-nines” availability

and unique 2, 3 & 4 site replication with

industry-leading RPO and RTO

• Up to eight TS7700 systems provide a flexible

grid for high-availability and disaster recovery

• Ensure the availability and security of

encrypted data with robust, centralized full-

lifecycle encryption key management

IBM TS7700,IBM DS8000 & IBM b-SAN: Designed for IBM ZTS7700

TS7700

TS7700

TS7700

TS7700TS7700

TS7700

TS7700

DS8880IBM Z

zHyperLink FICON 16S+ Ethernet

Gen 6 SAN

SAN42B-RSAN42B-R

Adding TS7700 and IBM DS8000 to IBM Z cloud improves security, availability and scalability in your mainframe infrastructure…


Designed for Integration

Designing, developing,

and testing together is

key to unlocking true value

IBM z14, Gen6 SAN, IBM DS88xx, TS7760 Systems

IBM Health Checker for

Z/OS


Forward Error Correction



Brocade CUP Diagnostics

b-type SAN

In-flight Encryption

Multi-Hop zHPF

Synergy is much more than just interoperability


Home of the IBM Qualification Letters

• IBM ResourceLink

‒ https://www.ibm.com/servers/resourcelink/

• Login

• Select “Library”

• Select “Switches and Directors”

• The qualifications letters are listed on the right hand side of the page

https://www.ibm.com/servers/resourcelink/


IBM z System Qualification Support for Brocade


SAN Software Availability

FOS 8.2.0a / BNA 14.4.2

• Availability– June 18, 2018

– https://www.ibm.com/servers/resourcelink/

• Software– FOS: 8.2.0a

– Network Advisor: 14.4.2

– Qualified for All zSystems– FCP and FICON

© 2017 BROCADE COMMUNICATIONS SYSTEMS, INC.



FOS 8.2.0a / IBMNA 14.4.2

• Schedule

– June 18, 2018

• We’re done!

• IBM Qualification Start

Refer to Director Qualification

Library

• IBM G/A – Aug 29 , 2017

• IBM POK qual FOS 8.2.0a in June.

– https://www.ibm.com/servers/resourcelink/

– https://www-304.ibm.com/servers/resourcelink/lib03020.nsf/pages/switchesAndDirectorsQualifiedForIbmSystemZRFiconRAndFcpChannels?OpenDocument

• Software Supported with z14

– FOS 8.2.0a

– 8.1.0c

– 8.0.2b2

– 8.0.1b

– 7.4.2a1

– 7.4.1d

– Network Advisor: 14.4.2

– Qualified for All System z (FCP and FICON)

– GEN4 Devices not supported

– Including the SAN06B-R


40 Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. "Broadcom" refers to Broadcom Limited and/or its subsidiaries.|© IBM Corporation 2017

2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

Gen 4

DCX

Gen 5

8510

Gen 6

X6

2021 2022

z10 z196 z13EC12 z14

z14 is supported on Gen 5 and Gen 6 SAN products only**

z14 is the last IBM Z platform tested, qualified & supported on Brocade 8510 directors

IBM FC-NVMe announced (Feb 2018).

32Gb FC HBA released on POWER9 (March 2018).

Gen 6 SAN provides best performance, features, and investment protection for IBM Z, POWER, Flash clients.

8 Gbps

16 Gbps

32 and 128 Gbps

Modernizing your Infrastructure – TODAY .

Why connect Z, POWER9 or NVMe capable Flash to a 5-10yr old network?

?

WFM

WFM*

40*Estimated 8510 WFM.

FC-NVMePOWER9

32G HBA

** z14 not supported on Gen 4 DCX (SAN768B and SAN384B). Requires RPQ.

You are here


Gen 4 does not have features to ensure optimum performance and availability

IBM Z Feature Gen 4 Gen 5 Gen 6

Port Commissioning FOS Dependent Yes Yes

Health Checker z/OS FOS Dependent Yes Yes

Read Diag Parameters FOS Dependent Yes Yes

FICON Dynamic Routing No Yes Yes

FEC No Yes Yes

Multi-hop configs No Yes Yes

I/O Statistics No No Yes, for Linux on Z

FCIP & IPEX Ext No SAN42B-R only Yes. SAN42B-R & SX6 blade

Port & ICL Speeds 4/8Gbps 8/16G, 64G 16/32G, 128G

Lifespan 2008-2018 2012-2022 2016-2026 estimate

Comparing the Integrated Tools for Z and b-type Platforms


• Given market adoption of X6 director, it is expected that the EOL announcement of the 8510 director will happen before mid CY’19

• The 8510 is expected to be in the end-of-life process at the time of zNext mainframe launch scheduled in the 2H CY’19

• BSN will not initiate qualifying the 8510 director for interoperability with the zNextmainframe

• The X6 is planned to be qualified with the zNext mainframe at time of launch

• The Gen 7 director will be available for qualification with the zNext mainframe in Q4 CY’19

• The Gen5s will most likely be End of Marketing by the time the zNext is available

• IBM will be responsible for making the support statement.

• They will work with us on the supported period of Gen5 connected to zNext and both

IBM and Brocade will announce this to the customers.

• The customers will have some time to use their existing Gen5s with the zNextbefore they must migrate to Gen6 or higher.

Gen 5 Director EOL plans and z-series FICON Qualification


SAN Modernization, Network Design Considerations

43

• Does the Network meet the Bandwidth and Latency Requirements?

• Does the network have Multipath capabilities to provide active-active designs, to enable 100% utilization of all links?

• Can the Network handle the dramatic increase in IOP throughput?

• Does the Network enable availability for high-density and critical workloads?

• Storage extension capabilities?

• In-depth visibility to enhance the Flash infrastructure?

• FC-NVMe-oF Ready?

• Like it or not! IT Planners have to evaluate the SAN!Fibre Channel Flash

and NVMe Storage

Arrays

32 Gbps SAN Core

32 Gbps HBAs

Edge Connections

Fibre Channel

Fabric


Trust the Gold Standard for Business Resiliency

b-type extension enables predictable, reliable, and secure network performance on premise, across metro or across regions

IP WAN

Primary Site Secondary Site

IBM Z

IBM SAN &

EXTENSION IBM TS7760IBM DS8880F

Encrypted

Compressed

WAN Optimized

Replication

Improvement

TS7700 Grid

100x

Up to

SAN42B-R

SX6 Director

Blade


IBM Distance Extension Product Line

| 45

Product Flexibility, Scalability, and Performance Requirements

Global 1000/ Large Enterprise

Director-Class

IBM Gen 5 Extension Blade for Gen 5 Directors

12×8 Gbps FC ports

10×1 GbE FCIP WAN ports


IBM SAN06B-R

4×8 Gbps FC ports


Small Enterprise

Medium Enterprise

IBM SAN06B-R

16×8 Gbps FC ports


Standalone

Large Enterprise

IBM SAN42B-R

24×16 Gbps FC ports

16×1/10 GbE FCIP and IPEX WAN ports


IBM Gen 6 Extension Blade for Gen 6 Directors

16×32 Gbps FC ports

16×1/10 GbE FCIP WAN ports


Purpose-built extension portfolio for business resiliency

So

lution

Price R

ang

e (

Ha

rdw

are

and

So

ftw

are

)

FCIP and IPEX


IBM SAN42B-R Extension Switch

Purpose-built for both Fibre Channel and IP storage replication

• Unprecedented 80 Gbps application throughput per platform

• Unique WAN-optimized TCP with IBM Extension Trunking and the industry’s highest jumbo frame support

• Secure data flows over distance with 256-bit IPsec encryption without a performance penalty

• Improve load balancing and network resilience with IBM Extension Trunking, Adaptive Rate Limiting, and Fabric Vision technology

• Achieve always-on business operations with the industry’s only non-disruptive firmware upgrades for FCIP

Industry-leading performance and scalability

High availability and reliability

Simplified management

• Leverage advanced monitoring and reporting on system, flows, and WAN conditions to detect WAN anomalies and avoid unplanned downtime

• Validate and troubleshoot the physical infrastructure with built-in traffic generator and Flow Vision to accelerate deployment

24x 16G FC Ports

16x 1/10Gbe WAN ports

2x 40 Gbe WAN Ports


Gen 6 Extension Blade for FC, FICON and IPEX

For IBM SAN512B-6 & SAN256B-6

• Accelerate data replication and secure data flows over distance at full line-rate speed with strong 256-bit AES encryption—up to 80 Gbpsper blade

• Maximize utilization of WAN links through protocol optimization technology and expand WAN bandwidth with hardware-based compression

• Maximize availability with redundant, hot-pluggable chassis components and non-disruptive software upgrades

• Improve load balancing and network resilience with Extension Trunking, Adaptive Rate Limiting, and Fabric Vision technology

Industry-leading performance

Enterprise-class availability

Unmatched flexibility and simplified management

• Consolidate Fibre Channel and IP replication traffic within a single blade with flexible multiprotocol port connectivity

• Consolidate operations management and extend Fabric Vision over distance for greater control and insight


Coming – New Brocade Extension Switch

Purpose-built for both Fibre Channel and IP storage replication

Optimum flexibility

High performance and availability

Simplified management

• 12 x Gen 6 FC ports and 6 x 1/10GE ports in 1U

• FC ports support 4G/8G/16G/32G FC

• 6 x 1/10GE + 2 1GE Copper ports for WAN/LAN connectivity

• Supports both FCIP and IP Extension

• Supports Compression

• IPsec encryption

• Advanced Features: Trunking, ARL

• Managed via FOS CLI, WebTools and Spectre (New Mgmt GUI Platform)

• Advanced monitoring and reporting on system, flows, and WAN conditions to detect WAN anomalies and

avoid unplanned downtime

• Validate and troubleshoot the physical infrastructure with built-in WAN traffic generator

AWING


Brocade Innovative Extension Solutions

High-speed Fibre Channel and IP storage replication

Primary Data Center

DR Site

Brocade Extension Tunneling

Technology…IPEX

• Deliver data, no matter the distance

• High-availability, high-performance, encrypted connection between the two sites

• Efficiency with compression, bandwidth mgmt.

• Higher throughput to keep up with data rates

• Encryption always on without performance penalty

• Proactive monitoring and quick alerts when problems occur

• Better tools to diagnose problems for faster resolution and minimize disruption

• Resiliency and Availability

• Security

• Visibility and Management of Network Connectivity and Application Performance

• Performance at Scale

7810 Extension Unit(Code Name: A Wing)


Brocade 7810 (A Wing) Coupled with Storage Refresh

FCIP & IP Extension Solutions

5

0

FCIP or IPEX

Extension Switch

Primary

Data CenterSecondary

Data Center

FCIP or IPEX

Extension Switch

IP WAN1/10/40G

Options

FC Array Based

Replication

FC

FC Array Based

Replication

FC

IP

Host/Server (S/W)

Based Replication

(i.e. Spectrum Protect)

IP

Host/Server (S/W)

Based Replication

(i.e. Spectrum Protect)

IP

NAS/Object Storage

Based Replication

(i.e. IBM NAS & TCT)

IP

NAS/Object Storage

Based Replication

(i.e. IBM NAS & TCT)


The IBM SAN42B-R…HA/DR GRID IPEX Solution

No tape

TS7760

Solves:

Clusters at local or metro distances

Clusters at metro or DR distances

High Availability at up to 8 locations

GRID SAN42B-R Value Proposition:

High availability with Brocade IP Extension features such as Extension Trunking

IPSec for inflight data encryption with minimal additional latency at no additional cost.

Better performance with Brocade innovations such as WAN Optimized TCP and ARL

Replication

Dec2017 16GB FICON attached to

the 16s Card. IPEX Connectivity

from the SAN42B-R to the clusters

attached to the GRID.

● Two to Eight TS7700s

● Worldwide access; as little as

one physical instance of the

data

● Automated failover

● Read / written at disk speeds,

sent at network speeds

● Where do we play?

What is a Grid?

With tape

TS7760 and TS7760T

WAN

IPEX to the GRID

SAN42B-R‘s


DS8880 Copy Services Capabilities

Out of

Region

Site C

Metro / Global Mirror

Three and four site

cascaded and multi-target

synchronous and

asynchronous mirroring

FlashCopy

Point in time copy

Within the

same

Storage

System

Out of Region

Site B

Global Mirror

Asynchronous

mirroring

Primary

Site A

Primary

Site A

Metro distance

Site B

Metro Mirror

Synchronous mirroring

Primary

Site A

Metro

Site B

The DS8880 Requires an Enterprise-Class extension solution

Site A Site B

Consider the amount of disk needed for multiple copies (Mirror Copy LUNs, FlashCopy LUNs, Consistency Group LUNs, Change Volume LUNs)

Consider BANDWIDTH requirements between sites to ensure efficient delivery of packets on mirroring solutions

• IBMs SAN42B-R’s…………………….WAN


Metro/Global Mirror Overview…GDPSYour Data just does not magically get there!

• GDPS provides a complete solution for high availability and disaster recovery in IBM Z environments

• Replication management, system management, automated workflows and deep integration with z/OS and parallel

sysplex

• Synchronous (Metro Mirror) + Asynchronous (Global Mirror)

• Continuous + near-continuous replication

• Cascaded or multi-target

• Metro Distance + Global Distance

• Almost 900 GDPS installations worldwide with high penetration in financial services and some of the largest IBM Z

environments

• 135 3-site GDPS installations and 16 4-site GDPS installations

• Application response time impacted only by distance between local and intermediate

• Fast resynchronization of sites after failures and recoveries

• Single consistency group may include open systems, IBM Z and IBM i volumes

Global Mirror

Global DistanceIntermediate Site Remote Site

Metro Mirror

Metro DistanceLocal Site

Local Site Intermediate

Site

Remote Site


zSystem ExtensionFICON,FCP/IP, and IPEX Solutions

• Increased Performance

• Strong Security

• Continuous Availability

• Simplified Management

IBM TS77xx Plus PPRC

High Availability

Architecture

WAN

IBM

SAN42B-R

IBM

SAN42B-R

FICON

IP/Ethernet

Local Site Remote Site

FICON

IP/Ethernet

DS8880

FCIP IP

IP IP

FC

TS77xx IP Replication (using IPEX)

PPRC (using FCIP)

TS77xx

Extension

(using FICON/IP)

z14 z14

TS7760TS7760


IBM/BrocadeDriving New Opportunities

Transparent Cloud Tiering

CIBM Cloud Object Storage (COS)

DS8K-SAN42B-R IPEX-TS770


IBM Transparent Cloud Tiering SolutionFind the Right Fit…Adapt to Partner Needs…Drive a Joint Plan

• IBM Goal:

– Remove CPU cycles to migrate/move data from the disk arrays to less expensive storage (i.e. Tape)

• Transparent Cloud Tiering:

– Leverages Object Storage S/W technology embedded in DS8K to move data directly between disk and tape via an IP connection

• Today’s Offering:

– Available June, 2017

– New Release Sept, 2017

– Replication using 1G Ethernet

– Local/Campus support

Local Site

SAN512B-6 SAN512B-6

All

FICON

TS7700

1G Enet

10G EnetLegend:

FICON


Remote Site

IP WAN

SAN42B-R SAN42B-RUsing IPEXTS7700

Native IP

IBM Enterprise – Transparent Cloud Tiering (TCT) Solution

Launch

• Solution Validation:– DS8K to TS7700 (TCT)

– Testing has been underway for several months

– Native IP Solution vs. IPEX

– Using 7840 for IPEX

– DS8K to “Other” Object Based Storage

– Test Configuration Is Running

– Using 7810

– No benchmarking yet

Local Site

DS8870

SAN512B-6

10G EnetLegend:

FICON

Other Object

Storage

7810 7810

10G Enet

10G Enet


FICON Multihop Supported Configurations

Native ISL Configurations


Topology ExpansionNew MultiHop Support

• MultiHop supports:

– Cascading up to Four switches

– Three Hops

– Expanding the previous restrictions of two cascaded switches, one hop.

• Solution

– Remove restrictions regarding hop counts between channel and control unit

– Verify FICON architecture over broader scope of configurations

– Establish best practice guidelines for FICON configurations

X


MultiHop FICON

• FOS 8.1 qualification

• z14 servers using Multihop have advantages for simplifying configurations in a single site or across multiple sites with ISL and cascaded FICON directors:

• Support multiple configurations for easy deployment of GDPS

• Improve utilization of FICON Directors and switches

• Simpler and easier to manage configurations for SAN availability

• IBM Technical White Paper co-authored by Brocade

• https://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102704

https://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102704


Two-Hop ArrayThree switches connected linearly

Must adhere to the FICON timeout limitations… this means….

• Longest distance a FICON packet can traverse is 300km

• The maximum distance of AB + BC <= 300km

150km 150km

SAN512B-6

A B C IBM Z

FICON

Host Channel

IBM DS8880F


Triangle ConfigurationThree switches connected in a triangle

Must observe the 300km max distance for FICON links.

• Examine the worst case scenarios in the event of path loss,

the traffic would have to traverse, to the 2 longer paths to reach its target.

If we assume BC is the shortest, then AB + AC < 300kms

A C

B

100km

75km100km


Three-Hop, Four Array Configuration

• Three switches connected linearly

• 300 km distance limit still applies

• AB + BC + CD <300km.

100km 100km 100km


The Square

• Four switches connected in a square configuration

• 300 km distance limit still applies

– Examine worst case scenario in event of path failure.

– If we assume BC is the shortest distance, then AB + AD +CD < 300km

<100km

<100km

50km

<100km


FCIP configurations

• Configurations use FCIP links to increase the distances for FICON links

• Primarily used for asynchronous replication applications

• Maximum distance supported over an FCIP connection for FICON=3000km


Two-Hop array with FCIP Bridge

• Two switches connected with native ISLs (300km)

• Third switch connected over FCIP (3000km)

SAN42B-R SAN42B-R


Three Hop Four Switch with middle FCIP

The FCIP bridge is in the middle of the configuration.The maximum distance of the bridge is 3000kms.ISLs distance AB and CD = <300kms.

SAN42B-R SAN42B-R

A B C D

<150km <150km3000kms


Three Hop array and leaf FCIP Bridge

FCIP is positioned at last hop.Maximum distance 3000kmsNative ISLs AB + BC <300kms

<150km

<150km

SAN42B-R SAN42B-R

3000kmsA B C D


FICON Multihop Design Considerations


Multihop Design Considerations

• Bandwidth planning

– Consider having additional bandwidth available between all routes

– Perform a bandwidth sizing study

• Potential adverse performance impact to applications in the event or rerouting traffic due to failure/loss of connectivity

– Increased latency traversing a longer path


Bandwidth Sizing Study Considerations

• What type of traffic is going across the cascaded FICON links (ISLs)? DASD, tape, CTC, all of the above?

• Do I have replication traffic going across the ISLs? If so, what type (synchronous, asynchronous, both)?

• Do I wish to isolate a particular traffic type to its own set of ISLs (by OS, storage type, replication type)?

• Am I using trunking/port channels in conjunction with Multi-Hop ?

• Am I using virtual fabrics and how does that effect my ISL allocation?

• Do I have an SLA that must be met on a specific replication traffic?

• Does my environment include a GDPS or similar architecture? Will I be performing a HyperSwap?


WHY SAN HEALTH?

Is your SAN Infrastructure Ready for the Future?


73Think 2018 / DOC ID / Month XX, 2018

/ © 2018 IBM Corporation

SAN Health – A free and powerful assessment tool.

6 Reasonsto Download

SAN Health

•Avoid application downtime

•Reduce troubleshooting and resolution time

•Improve capacity planning and productivity

•Gain a competitive edge

Make your SAN a business enabler •Oversubscribed ports

•Zoning issues

•Configuration anomalies

•Unusually high port errors

Quickly identify congestion risks

•Easy-to-read overview for new team members

•Professional insight for the CIO

•Greater SAN visibility for the entire storage team

Share insights with others

Audit fabrics that include any storage and

network vendor hardware—even Cisco devices

running SAN-OS 2.0 andabove.

Audit multi-vendor environments

•Eliminate whiteboard SAN management

•See down to the device and port level

•Get clear reports in Excel and Visio formats

Generate topological reports

•1 minutetoinstall

•3 minutes to audit

•0 impact on network performance

Automate it to run in the background during

your backup, or at any other scheduled time.

Easy to run

Start your free SAN Health audit. Download Today

SAN Health answers the question – is your SAN ready for Flash, NVMe, z14, Power9?

brocade.com/sanhealth


SAN Health Update

Questions:

Email: [email protected]

Downloads and more information: www.broadcom.com/sanhealthUploads [email protected]

New Online Help: SAN Health = http://community.broadcom.com/docs/DOC-2662

Are you or your customers among the 48,000 users benefitting from this?

1,800+ reports encompassing 3 million+ switch ports are generated every week!


http://www.broadcom.com/sanhealth

http://community.brocade.com/docs/DOC-2662


Summary

• Brocade and IBM – Partners with a vision

• Brocade Gen6 Introduction – The next wave of FICON

• IBM z14 Designed for Integration – The next level of integration

• The Future of Fibre Channel – More than a bandwidth boost

• Remeber the Millennials – They drive the future

• Remember the Millennials – They drive the future!


Visit the IBM Brocade Resource Center here or on the IBM Storage & SDI for Partners.

Help configure, bid, and win the storage network today!

• SAN proposal guidance: Is your proposed configuration accurate?

• SAN Health assessment tool: Is the customers existing SAN ready of Flash Storage or NVMe?

• Tools to simplify configuration: Do you have the SAN Sales Reference Manual (SSRM)? You should!

• Free Web Based Training: What would you like to learn about SAN, Storage Extension, NVMe?

https://community.brocade.com/t5/OEM-IBM-Brocade-Resource-Page/ct-p/OEM-IBM-Brocade-Resource-Page

https://apps.na.collabserv.com/wikis/home?lang=en-us#!/wiki/Wc28a326132a8_4056_937d_0034e7335406/page/Solutions%20by%20Horizontal


IBM Sales Reference Manual

• Product pages contain key product information, IBM P/N translations, and hotlinks directly to relevant online information

• Automatically translate IBM P/N to Brocade P/N and vice versa

• Contains list of frequently asked questions and messaging

• External – can be shared with IBM & IBM BP

• E-mail [email protected] will go to our WW Brocade IBM sales team

• E-mail [email protected] will go to our WW Brocade IBM technical team

• Find the latest version on Flight Deck




Brocade Education Curriculum Paths – PDF is interactive


Questions?


Thank You

designed for integration zstuff is ... - ibm-zcouncil.com · 8/29/2017 · ibm zhyperlink delivers...

Documents