virtual port channeling in nxosicon.clnchina.com.cn/pdf/vpc.pdf · virtual port channeling in nxos...

59
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 1 Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07

Upload: vuongtruc

Post on 07-Mar-2018

215 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 1

Virtual Port Channeling in NXOS Vivian Hu

Cisco China TAC

11. 07

Page 2: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Agenda

• Feature Overview

• vPC Design Guidance and Best Practices

• Scalability

• Reference Material

Page 3: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Agenda

• Feature Overview

–vPC Concept & Benefits

–How does vPC help with STP?

–vPC Terminology

–Data-Plane Loop Avoidance with vPC

• vPC Design Guidance and Best Practices

• Scalability

• Reference Material

Page 4: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

vPC is a Port-channeling concept extending link aggregation to two separate physical switches

Allows the creation of resilient L2 topologies based on Link Aggregation.

Eliminate STP blocked ports and uses all available uplink bandwidth

Virtual Port Channel

Physical Topology Logical Topology

Enable seamless VM Mobility, Server HA Clusters

Scale Available Layer 2 Bandwidth

Maintains independent control plane

Provide fast convergence upon link/device failure

Simplify Network Design

L2

Bi-sectional Bandwidth with vPC

Non-vPC vPC

SiSi SiSi SiSi SiSi SiSi SiSi SiSi

vPC Feature Overview vPC Concept & Benefits

Page 5: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

MCEC

! Enable vpc on the switch

dc11-5020-1(config)# feature vpc

! Check the feature status

dc11-5020-1(config)# show feature | include vpc

vpc 1 enabled

vPC Peers

Available on Nexus 3000, 5000/5500, 6000 and 7000

Available in NX-OS 4.1(3)N1+ on the Nexus 5000 & 4.1(3)+ on the Nexus 7000

MCEC

vPC Peers

vPC – Feature OverviewMulti-Chassis EtherChannel (MCEC)

Page 6: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Agenda

• Feature Overview

–vPC Concept & Benefits

–How does vPC help with STP?

–vPC Terminology

–Data-Plane Loop Avoidance with vPC

• vPC Design Guidance and Best Practices

• Scalability

• Reference Material

Page 7: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Feature Overview How does vPC help with STP? (1 of 2)

• Before vPC – STP blocks redundant uplinks – VLAN based load balancing – Loop Resolution relies on STP

• With vPC – No blocked uplinks – EtherChannel load balancing (hash) – Loop Free Topology

Primary

Root

Secondary

Root

Page 8: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Feature Overview How does vPC help with STP? (2 of 2)

• Reuse existing infrastructure

• Build Loop-Free Networks

smooth migration

Page 9: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Agenda

• Feature Overview

–vPC Concept & Benefits

–How does vPC help with STP?

–vPC Terminology

–Data-Plane Loop Avoidance with vPC

• vPC Design Guidance and Best Practices

• Scalability

• Reference Material

Page 10: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Feature Overview vPC Terminology (1 of 2)

• vPC Domain — A pair of vPC switches

• vPC Peer—a vPC switch, one of a pair

• vPC member port—one of a set of ports (port channels) that form a vPC

• vPC—the combined port channel between the vPC peers and the downstream device

• vPC peer-link—link used to synchronize state between vPC peer devices, must be 10GE

vPC

member

port

vPC vPC

member

port

vPC peer-link

vPC peer

vPC Domain

vPC vPC

member

port

Page 11: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Feature Overview vPC Terminology (2 of 2)

• vPC peer-keepalive link—the keepalive link between vPC peer devices

• vPC VLAN—Any of the VLANs carried

over the peer-link and used to communicate via vPC with a peer device

• non-vPC VLAN—Any of the STP VLANs not carried over the peer-link

• CFS—Cisco Fabric Services protocol, used for state synchronization and configuration validation between vPC peer devices

vPC Peer-keepalive link

CFS protocol

Page 12: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Agenda

• Feature Overview

–vPC Concept & Benefits

–How does vPC help with STP?

–vPC Terminology

–Data-Plane Loop Avoidance with vPC

• vPC Design Guidance and Best Practices

• Scalability

• Reference Material

Page 13: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Feature Overview Data-Plane Loop Avoidance with vPC

Data-Plane vs. Control-Plane Loop control

vPC peers can forward all vpc traffic locally

Orphan traffic can traverse the peer link

Non-orphan traffic on the Peer-link is marked and not allowed to egress on a vPC

vPC Domain STP Domain

STP Failure

Page 14: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Agenda • Feature Overview and Terminology • vPC Design Guidance and Best Practices

–vPC Hardware Support –Building a vPC Domain –Double Sided VPC –Layer 3 and vPC –vPC Enhancements –vPC / FEX Supported Topologies

• Scalability • Reference Material

Page 15: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

VPC Supported Hardware Nexus 7000

I/O Module vPC Peer-link

(10 GE Only) VPC Interfaces

N7K-M132XP-12 N7K-M132XP-12L

✓ ✓

N7K-M148GT-11 N7K-M148GT-11L N7K-M148GS-11 N7K-M148GS-11L

✗ ✓

N7K-M108X2-12L

✓ ✓

N7K-F132XP-15 ✓ ✓

N7K-F248XP-25 ✓ ✓

For Your Reference

Page 16: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Part Number / Chassis vPC Peer-link

(10 GE Only) VPC Member Port

N5K-C5010P-BF

✓ ✓

N5K-C5020P-BF ✓ ✓

N5K-C5548P-FA ✓ ✓

N5K-C5548UP-FA ✓

N5K-C5596UP-FA

✓ ✓

vPC Supported Hardware NEXUS 5000/5500

For Your Reference

Page 17: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

vPC supported hardware NEXUS 2000 platform

Part Number FEX

vPC Peer-link

VPC Member Port

NEXUS 5000 parent switch

NEXUS 7000 parent switch

N2K-C2148T-1GE ✗ ✓ ✗

N2K-C2224TP-1GE N2K-C2248TP-1GE

✗ ✓ ✓

N2K-C2232PP-10GE

✗ ✓ ✓

N2K-C2232TM-10GE ✗ ✓ ✓

N2K-B22-HP ✗ ✓ ✓

N2K-C2248TP-E-1GE ✗ ✓ ✓

For Your Reference

Page 18: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Agenda • Feature Overview and Terminology • vPC Design Guidance and Best Practices

–vPC Hardware Support –Building a vPC Domain –Double Sided VPC –Layer 3 and vPC –vPC Enhancements –vPC / FEX Supported Topologies

• Scalability • Reference Material

Page 19: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Building a vPC Domain Configuration Steps

• Define domains*

• Establish Peer Keepalive connectivity

• Create a Peer link

• Reuse port-channels and Create vPCs

• Make Sure Configurations are Consistent 5 6 7 8

1 2 3 4

Following steps are needed to build a vPC

(Order does Matter!)

vPC member

Routed Interface

Host Port

Page 20: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Building a vPC Domain vPC Domains

• vPC Domain defines the grouping of switches participating in the vPC

• Provides for definition of global vPC system parameters

• The vPC peer devices use the vPC domain ID to automatically assign a unique vPC system MAC address

• You MUST utilize unique Domain id’s for all vPC pairs defined in a contiguous layer 2 domain ! Configure the vPC Domain ID – It should be unique within the layer 2 domain

NX-1(config)# vpc domain 20

! Check the vPC system MAC address

NX-1# show vpc role

<snip>

vPC system-mac : 00:23:04:ee:be:14

vPC Domain 20

vPC System MAC identifies the Logical

Switch in the network topology

vPC Domain 10

Page 21: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

5K_2#sh lacp neighbor

<snip>

LACP port Admin Oper Port Port

Port Flags Priority Dev ID Age key Key Number State

Gi1/1 SA 32768 0023.04ee.be14 9s 0x0 0x801E 0x4104 0x3D

Gi1/2 SA 32768 0023.04ee.be14 21s 0x0 0x801E 0x104 0x3D

7K_2 # shvpc role

<snip>

vPC system-mac : 00:23:04:ee:be:14

vPC system-priority : 1024

vPC local system-mac : 00:0d:ec:a4:5f:7c

vPC local role-priority : 32667

7K_1# shvpc role

<snip>

vPC system-mac : 00:23:04:ee:be:14

vPC system-priority : 1024

vPC local system-mac : 00:0d:ec:a4:53:3c

vPC local role-priority : 1024

dc11-4948-1

LACP neighbor sees the same System ID from both vPC peers

The vPC ‘system-mac’ is used by both vPC peers

vPC – Virtual Port Channel Independent Control Plane + Synchronized L2 State

5K_2

7K_1 7K_2

1/4 1/5

Regular (non vPC)

Port Channel 1/1 1/2

MCEC (vPC)

EtherChannel

Page 22: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

dc11-4948-2#sh lacp neighbor

<snip>

LACP port Admin Oper Port Port

Port Flags Priority Dev ID Age key Key Number State

Gi1/4 SA 32768 000d.eca4.533c 8s 0x0 0x1D 0x108 0x3D

Gi1/5 SA 32768 000d.eca4.533c 8s 0x0 0x1D 0x108 0x3D

7k_1 # sh vpc role

<snip>

vPC system-mac : 00:23:04:ee:be:14

vPC system-priority : 1024

vPC local system-mac : 00:0d:ec:a4:53:3c

vPC local role-priority : 1024

vPC peers function as independent devices as well as peers

Local ‘system-mac’is used for all non vPC PDUs (LACP, STP, …)

MCEC (vPC)

EtherChannel

vPC – Virtual Port Channel Independent Control Plane + Synchronized L2 State

5K_2

7K_1 7K_2

1/4 1/5

Regular (non vPC)

Port Channel 1/1 1/2

dc11-4948--2

Page 23: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Virtual Port Channel (vPC) vPC Roles

• vPC primary switch election is based on role priority

• Lower priority wins if not, lower system mac wins

• Role is non-preemptive, So operational Role is what matters

• Operational Role may different from the priorities configured under the domain

• vPC Role defines which of the two vPC peers processes BPDUs

• Role matters for the behavior with peer-link failures!

dc11-5020-3(config-vpc-domain)# role priority ?

<1-65535> Specify priority value

dc11-5020-3# sh vpc

<snip>

vPC role : secondary, operational primary

Secondary (but may be

Operational Primary)

Primary (but may be

Operational Secondary)

vPC Domain 20

vPC Domain 10 Dual Layer VPC

Page 24: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Building a vPC Domain vPC Peer-Link

• Definition: –Standard 802.1Q Trunk

–Carries CFS (Cisco Fabric Services) messages

–Carries flooded traffic from the vPC peer

–Carries STP BPDUs, HSRP Hellos, IGMP updates, etc.

• Requirements: –Peer-Link member ports must be 10GE interfaces : - 32 port 10GE fibre card (M1 or F1 series) or 8 port 10GE-X2 M1 modules or 48 port 10GE F2

–- any 10G port on NEXUS 5000/5500 series

–vPC Peer-link must be a point-to-point connection (No other device between the vPC peers)

Recommendations (strong ones!) –Minimum 2x 10GE ports (on NEXUS 7000 : use 2 separate cards for best resiliency)

–10GE ports in dedicated mode (for oversubscribed modules)

vPC peer-link

Page 25: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Building a vPC Domain vPC Peer-Keepalive link

• Definition:

–Heartbeat between vPC peers

–Active/Active detection (in case vPC Peer-Link is down)

–Non- fatal to the operation of VPC

Recommendations (in order of preference):

vPC peer-

keepalive link

vPC Peer-Keepalive messages should NOT be routed over the vPC Peer-Link

NEXUS 7000 NEXUS 5000/5500

1- Dedicated link(s) (1GE LC) 1- mgmt0 interface (along with management traffic)

2- mgmt0 interface (along with management traffic)

2- Dedicated link(s) (1/10GE front panel ports)

3- As last resort, can be routed over L3 infrastructure

Page 26: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

SW3 SW4

vPC1 vPC2

vPC_PLink

vPC Peer-keepalive

Building a vPC Domain vPC Peer-Keepalive link up & vPC Peer-Link down

vPC peer-link failure (link loss):

• Check active status of the remote vPC peer via vPC peer-keepalive link (heartbeat)

• If both peers are active, then Secondary vPC peer will disable all vPCs to avoid Dual-Active scenario

• Data will automatically forward down remaining active port channel ports

Keepalive Heartbeat

Secondary vPC

P S

S

P Primary vPC

Suspend secondary vPC Member Ports

Page 27: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Virtual Port Channel - vPC vPC Control Plane - Consistency Check

Both switches in the vPC Domain maintain distinct control planes

CFS provides for protocol state synchronization between both peers (MAC Address table, IGMP state, …)

System configuration must be kept in sync

Two types of interface consistency checks

Type 1 – Will suspend interfaces to prevent invalid forwarding of packets. With Graceful Consistency check only ports on secondary switch will suspend

Type 2 – Error messages to indicate potential for undesired forwarding behavior

vPC Domain 20

vPC Domain 10

Page 28: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Type 1 Consistency Checks are intended to prevent network failures

Incorrectly forwarding of traffic

Physical network incompatibilities

vPC will be suspended

dc11-5020-2# show vpc brief

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

<snip>

vPC status

----------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

------ ----------- ------ ----------- -------------------------- -----------

201 Po201 up failed vPC type-1 configuration -

incompatible - STP

interface port guard -

Root or loop guard

inconsistent

dc11-5020-1# sh run int po 201

interface port-channel201

switchport mode trunk

switchport trunk native vlan 100

switchport trunk allowed vlan 100-105

vpc 201

spanning-tree port type network

dc11-5020-2# sh run int po 201

interface port-channel201

switchport mode trunk

switchport trunk native vlan 100

switchport trunk allowed vlan 100-105

vpc 201

spanning-tree port type network

spanning-tree guard root

vPC – Virtual Port Channel vPC Control Plane – Type 1 Consistency Check

Page 29: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Type 2 Consistency Checks are intended to prevent undesired forwarding

vPC will be modified in certain cases (e.g. VLAN mismatch)

dc11-5020-1# show vpc brief vpc 201

vPC status

----------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

------ ----------- ------ ----------- -------------------------- -----------

201 Po201 up success success 100-104

2009 May 17 21:56:28 dc11-5020-1 %ETHPORT-5-IF_ERROR_VLANS_SUSPENDED: VLANs 105 on Interface port-

channel201 are being suspended. (Reason: Vlan is not configured on remote vPC interface)

dc11-5020-1# sh run int po 201

version 4.1(3)N1(1)

interface port-channel201

switchport mode trunk

switchport trunk native vlan 100

switchport trunk allowed vlan 100-105

vpc 201

spanning-tree port type network

dc11-5020-2# sh run int po 201

version 4.1(3)N1(1)

interface port-channel201

switchport mode trunk

switchport trunk native vlan 100

switchport trunk allowed vlan 100-104

vpc 201

spanning-tree port type network

vPC – Virtual Port ChannelvPC Control Plane – Type 2 Consistency Check

Page 30: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Building a vPC Domain VDC Interaction

• VDCs are virtual instances of a device running on the Nexus 7000

• vPC works seamlessly in any VDC based environment

• Each VDC can have its own vPC domain (one vPC domain per VDC is allowed)

• Separate vPC Peer-link and Peer-keepalive link infrastructure for each VDC deployed

• Using 2 VDCs on the same N7K to form a vPC domain is not supported

L2 L3

L3

L2

L2 Channel L3 link

L2 link

L3 Channel Core

Aggregation

Access

SW-2b

VDC2

SW-2a

VDC2 SW-2a

VDC1 SW-2b

VDC1

SW-1a

VDC2 SW-1b

VDC2

SW-1a

VDC1

SW-1b

VDC1

Core2 Core1

vPC vPC

active active active standby active active active standby

vPC vPC vPC vPC

Page 31: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

dc11-5020-2#

feature vpc

feature lacp

vpc domain 10

peer-keepalive destination 1.1.1.2 source 1.1.1.1

system-priority 3000

role-priority 90

peer-gateway

vPC – Virtual Port Channel vPC Configuration

dc11-5020-1# sh run int po 1

interface port-channel 1

switchport mode trunk

vpc peer-link

spanning-tree port type network

dc11-5020-1# sh run int mgmt0

interface mgmt0

ip address 1.1.1.1

Page 32: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

dc11-5020-1# sh run int po 201

interface port-channel201

switchport mode trunk

switchport trunk allowed vlan 100-105

vpc 201

dc11-5020-2# sh run int po 201

interface port-channel201

switchport mode trunk

switchport trunk allowed vlan 100-105

vpc 201

vPC – Virtual Port Channel vPC configuration (continued)

Page 33: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Agenda

• Feature Overview and Terminology • vPC Design Guidance and Best Practices

–vPC Hardware Support –Building a vPC Domain –Double Sided VPC –Layer 3 and vPC –vPC Enhancements –vPC / FEX Supported Topologies

• Scalability • Reference Material

Page 34: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Double Sided VPC Up to 32-Way Port-Channel – Double-sided VPC

• Generally known as a ‘bowtie’ or multi-layer VPC

• Multilayer vPC can join eight active member ports of the port-channels in a unique 16-way port-channel*

• vPC peer load-balancing is LOCAL to the peer device

Nexus 7000

Nexus 5000

32-way port channel

* Possible with Any Device Supporting vPC/MCEC and Eight-Way Active Port-Channels

Double-sided vPC architecture

Page 35: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Redundancy with Enhanced vPC Data, Control and Management Plane Redundancy

• Port-channel connectivity to dual-homed FEXs –From the server perspective a single access switch with port-channel support – each line card supported by redundant supervisors

–Full redundancy for supervisor, linecard, fabric via vPC and cable or NIC failure via Port-channeling

–Logically a similar HA model to that currently provided by dual supervisor based modular switch. Suited for all types of servers.

Dual supervisor

modular chassis clustered Fabric Extender dual homed to

redundant Nexus 5000

Page 36: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Enhanced vPC ( aka Dual Tier vPC) Supported on Nexus 5500

Any Flavor of Nexus 5500 5548P/5548UP/5596UP

Any Flavor of Nexus 2000 2148T/2248TP/2224TP

2232PP/2232TM

Dual-homed Fabric Extenders

Mix of Single NIC, Active/Standby and Etherchanneled servers can

connect to same FEX

Supported on N5500 Only

N5K – 5.1 (3) Release

Page 37: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Agenda

• Feature Overview and Terminology • vPC Design Guidance and Best Practices

–vPC Hardware Support –Building a vPC Domain –Double Sided VPC –Layer 3 and vPC –vPC Enhancements –vPC / FEX Supported Topologies

• Scalability • Reference Material

Page 38: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Layer 3 and vPC Interactions Router Interconnection: different angles

vPC view

7k1 7k2

R

Layer 3 topology

Layer 3 will use ECMP

for northbound traffic

7k1 7k2

R

Layer 2 topology

Port-channel looks like

a single L2 pipe.

Hashing will decide

which link to choose

7k vPC

R

R could be any router,

L3 switch or VSS

building a port-channel

Page 39: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Layer 3 and vPC Interactions Router Interconnection: Forwarding sequence Gotcha

1) Packet arrives at Router 1 with a destination address of Switch 1

2) Router 1 does lookup in routing table and sees 2 equal paths going north (to 7k1 & 7k2)

3) Assume it chooses 7k1 (ECMP decision) 4) Router 1 now has rewrite information to which

router it needs to go (router MAC 7k1 or 7k2) 5) L2 lookup happens and outgoing

interface is port-channel 1 6) Hashing determines which port-channel member is

chosen (say to 7k2) 7) Packet is sent to 7k2 8) 7k2 sees that it needs to send it over the peer-link

to 7k1 based on MAC address Router 1

7k2

Switch 1

Po1

Po2

7k1

Page 40: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Layer 3 and vPC Interactions Router Interconnection: Forwarding sequence (continued)

9) 7k1 performs lookup and sees that it needs to send to Switch 1

10) 7k1 performs check if the frame came over peer link & is going out on a vPC.

11) Frame will ONLY be forwarded if: • Outgoing interface is NOT a vPC or • Outgoing vPC doesn’t have active

interface on other vPC peer (in our example 7k2)

Note:

• Use of Peer-Gateway allows data-path forwarding, routing/forwarding traffic for the peer-router MAC locally, but does NOT Enable Dynamic Routing on vPC VLANs

• Even with Peer-Gateway Routing protocols (e.g. OSPF) are still broken due to TTL expiry when traversing in transit the peer vPC Router device.

Router 1

7k1 7k2

Switch 1

Po1

Po2

Page 41: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Agenda

• Feature Overview and Terminology • vPC Design Guidance and Best Practices

–vPC Hardware Support –Building a vPC Domain –Double Sided VPC –Layer 3 and vPC –vPC Enhancements –vPC / FEX Supported Topologies

• Scalability • Reference Material

Page 42: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

FCoE: Asymmetry consideration Retaining FC best practices with Enhanced vPC

Enhanced vPC

FCoE + Ethernet traffic FCoE + Ethernet traffic

Ethernet only traffic

• Ethernet traffic gets equally distributed on FEX uplinks to both switches

• FCoE traffic is pinned only to uplink to one switch (for fabric isolation)

• Uplink carrying FCoE end up carrying more traffic (50% Eth + FCoE) than

other uplink (50% Eth)

Recommendation:

• Evaluate suitability of Enhanced vPC for

FCoE deployments with traffic/flow analysis

• Leverage Straight-thru vPC, if required

Page 43: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Three cases of single switch behavior addressed by vPC auto-recovery

On Failure of Peer

– Allows State changes on vPC resources

Recovery of secondary after dual failure

1. vPC peer-link goes down : vPC secondary shuts all its vPC member ports

2. vPC primary goes down. vPC secondary receive no more keep-alives

3. After 3 consecutive keep-alive timeouts, vPC secondary changes role and brings up its vPC.

Restart of a single vPC peer

1. When a vPC peer is missing, by default vPC doesn’t allow any vPC member port to either flap or for a new one to be brought online or for existing vPC member to go up after a reload

2. Auto-recovery monitors the peer device and if the vPC peer is not available it allows new ports to be brought up even if the peer missing

Switch3

Switch

1

Switch4

vPC

Primary

Peer Keep-alive “Link”

“Missing” vPC Peer

dc11-5020-3(config)# vpc domain 10

dc11-5020-3(config-vpc-domain)# auto-recovery

vPC Auto-RecoveryReload Restore Superseded by Auto-recovery

NX-OS

N7K - 5.2

N5K - 5.0(2)N1

Page 44: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

vPC Supported Server fails over

correctly

Active/Standby Server does not fail

over correctly

vPC A vPC orphan port is an non-vPC

interface on a switch where other ports in the same VLAN are configured as vPC interfaces

vPC orphan ports have historically been problematic for mixed server topologies

Prior to release 5.0(3)N2 on Nexus 5000/5500 and 5.2 on Nexus 7000 an orphan port was ‘not’ shut down on loss of vPC peer-links

With the latest NX-OS release the orphan ports on the vPC secondary peer can (configurable) also be shut down triggering NIC teaming recovery for all teaming configurations (identical to VSS behavior)

Configuration is applied to the physical port

N5K-2(config)# int eth 100/1/1

N5K-2(config-if)# vpc orphan-ports suspend

Virtual Port ChannelOrphan-ports suspend

NX-OS

N7K - 5.2

N5K - 5.0(3) N2

eth 100/1/1

Page 45: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

vPC PL

vPC PKL

L3 L2

vPC Peer-Gateway • Allows a vPC switch to act as the active

gateway for packets addressed to the peer router MAC

• Keeps forwarding of traffic local to the vPC node and avoids use of the peer-link.

• No impact on traffic and existing functionality

• Allows Interoperability with features of some NAS or load-balancer devices.

Best Practice to Enable this Feature

N7k(config-vpc-domain)# peer-gateway

RMAC A RMAC B

vPC1 vPC2

Page 46: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

vPC ARP Synchronization Needs to be enabled on both vPC devices

After the peer-link comes up perform an ARP bulk sync over CFSoE to the peer switch device

Improve Convergence for Layer 3 flows

ARP Synchronization Process

Primary vPC

Secondary vPC S

P

P S

ARP TABLE

IP1 MAC1 VLAN 100

IP2 MAC2 VLAN 200

ARP TABLE

IP1 MAC1 VLAN 100

IP2 MAC2 VLAN 200

IP1 MAC1 IP2 MAC2

SVIs

Page 47: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Problem/Impact:

• After a vPC device reloads and comes back up routing protocol needs time to reconverge. vPCs may blackhole routed traffic from access to core until layer 3 connectivity is reestablished

vPC Delay restore solution:

• Delays vPCs bringup after a vPC device reload (SVI bring-up timing is unchanged)

• Allows for Layer 3 routing protocols to converge for a more graceful restoration.

• Enabled by default with a vPC restoration default timer of 30 seconds. Timer can be tuned according to a specific layer 3 convergence baseline.

vPC

Primary

vPC

Secondary

vPC PL

vPC PKL

L3 L2

OSPF

vPC Delay Restore

Page 48: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

vPC peer-switch

• vPC peer-switch feature allows a pair of vPC peer devices to appear as a single STP Root in the L2 topology (same bridge-id)

• Simplifies STP configuration by configuring both vPC with the same STP priority • Eliminates recommendation to pin STP Root to the vPC primary switch. • Improves convergence during vPC primary switch failure/recovery avoiding Rapid-STP Sync • Supports a hybrid topology of vPC and non-vPC connections by using the spanning-tree

pseudo-information • Recommended in pure VPC Topologies

vPC Peer-link S1 S2

S3 S4

vPC Primary vPC Secondary

vPC1

Bridge Priority

VLAN 1 4K

VLAN 2 8K

Bridge Priority

VLAN 1 8K

VLAN 2 4K

STP Root

VLAN 1 STP Root

VLAN 2

No STP Topology Changes

STP Root

VLAN 1

VLAN 2

Nexus 7000(config-vpc-domain)# peer-switch

vPC2

Page 49: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

vPC Peer-link S1 S2

S3 S4

vPC Primary

STP root vPC Secondary

vPC1 vPC2

B

P

D

U

B

P

D

U

STP view of vPC

S5

vPC Peer-link S1 S2

vPC Primary

STP root vPC Secondary

S6

vPC Peer-link S1 S2

S3 S4

vPC Primary vPC Secondary

vPC1 vPC2

B

P

D

U

B

P

D

U

S5

S1 S2

vPC Primary vPC Secondary

S6

S5 S3 S4 S6

Peer-switch

Without vPC peer-switch

With vPC peer-switch

Peer-switch

Page 50: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Agenda

• Feature Overview and Terminology • vPC Design Guidance and Best Practices

–vPC Hardware Support –Building a vPC Domain –Double Sided VPC –Layer 3 and vPC –vPC Enhancements –vPC / FEX Supported Topologies

• Scalability • Reference Material

Page 51: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

1

standby active

server:

active/standby

NIC teaming

2

standby active

server:

active/standby

NIC teaming

3

active active

server:

active/active

no NIC teaming

4

vPC Supported Topologies Nexus 7000 and 5000

Local

FEX

port-

Channel server:

port-channel

NIC teaming

(active-active)

5

active active

server:

port-channel

NIC teaming

(active-active)

standby active

server:

active/standby

port-channel

NIC teaming

6 8 7

Port-Channel on HIF

(Host Interfaces)

supported

-vPC to Host supported

For Your Reference

Page 52: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

standby active

vPC

vPC

Domain

active active

vPC

vPC

Domain

vPC

vPC vPC

Domain 8 9 10

vPC Supported Topologies Nexus 5000 Only

Enhanced vPC

N5500 only

Dual-homed FEX

w/ A-S Server Dual-homed FEX

w/ Single NIC

Server

New

For Your Reference

Page 53: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

active

7

active

11

active active

vPC

Domain

vPC

8 12

VDC1 VDC2

Only 1 physical

NEXUS 7000

chassis active active

vPC

8 13

vPC

vPC

vPC / FEX Unsupported Topologies For Your Reference

Page 54: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Agenda

• Feature Overview

• vPC Design Guidance and Best Practices

• Scalability

• Reference Material

Page 56: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Agenda

• Feature Overview

• vPC Design Guidance and Best Practices

• Convergence and Scalability

• Reference Material

Page 57: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Reference Material

• vPC white Paper: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-516396.html

• vPC Best Practices: http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf

• vPC design guides: http://www.cisco.com/en/US/partner/products/ps9670/products_implementation_design_guides_list.html

• vPC and VSS Interoperability white Paper: http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_paper_c11_589890.html

• Data Center Design—IP Network Infrastructure: http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/DC_3_0/DC-3_0_IPInfra.html

• Layer 2 Extension Between Data Centers: http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_paper_c11_493718.html

• Implementing Nexus 7000 in the Data Center Aggregation Layer with Services: https://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/nx_7000_dc.html

For Your Reference

Page 58: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

BRKDCT- 2048 Recommended Reading

Page 59: Virtual Port Channeling in NXOSicon.clnchina.com.cn/pdf/VPC.pdf · Virtual Port Channeling in NXOS Vivian Hu Cisco China TAC 11. 07 . ... Available on Nexus 3000, ... •CFS—Cisco

Thank you. Thank you.