cisco - vpc concepts

Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 1

vPC Best Practices with Nexus

SAVBUTMETeam

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2

Complete Sync

Partial Sync5.1(2) 5.25.0(3) 5.1(3)

5.0(3)N2 5.2N15.0(3)N1 5.1(3)N1

5.0(3)U2 5.1(3)U15.0(3)U1

Nexus 7000

Nexus 7000

Nexus 5000

Nexus 3000

E-Rocks

Andaman

Complete sync done at major releases Architectural changes

Major enhancements

Major new features

Partial sync done at minor releases Critical flaws/bugs

Minor new features

Minor enhancements


• vPC basic components

• Hardware Specific Considerations

• vPC enhancements

• L3 and vPC

• Adding FEX

• Summary designs


• vPC is a Port-channeling concept extending link aggregation to two separate physical switches

• Allows the creation of resilient L2 topologies based on Link Aggregation.

Eliminates the need for STP in the access-distribution

• Provides increased bandwidth

All links are actively forwarding

• vPC maintains independent control planes

• vPC switches are joined together to form a ―domain‖

Virtual Port Channel

L2

SiSi SiSi

Increased BW with vPC

Non-vPC vPC

Physical Topology Logical Topology

vPC domain


• vPC peer – a vPC switch, one of a pair

• vPC member port – one of a set of ports (port

channels) that form a vPC

• vPC – the combined port channel between the

vPC peers and the downstream device

• vPC peer link – Link used to synchronize state

between vPC peer devices, must be 10GbE.

Also carry multicast/broadcast/flooding traffic

and data traffic in case of vpc member port

failure

• vPC peer keepalive link – the peer keepalive

link between vPC peer switches. It is used to

carry heartbeat packets

• CFS – Cisco Fabric Services protocol, used

for state synchronization and configuration

validation between vPC peer devices

• Orphan port—Non-vPC member port

vPC peer

keepalive linkvPC peer link

Orphan

Port

Orphan

Port

Orphan

Port

Orphan

Port

SecondaryPrimary

vPC

vPC

member

port

vPC peer


• Graceful consistency check:

On the N7k: NXOS 5.2

On the N5k: NXOS 5.0(2)N2(1)

• Per VLAN consistency check:


On the N5k: 5.0(2)N2(1)

• Autorecovery:



• Config-sync:

On the N7k: Freetown


• vPC on FEX



• Orphan Ports shutdown:

On N7k: NXOS 5.2

On N5k: E-Rocks+

• IGMP bulk sync:

On N7k: to be verified

On N5k: starting from NXOS 5.0(3)N1(1a)

• Multicast Optimization on Peer-link:

On N7k: hidden comand as of NXOS 5.1(3) (but not supported)

On N5k: starting from NXOS 5.0(3)N1(1a)

• ARP synchronization:

On N7k: NX-OS 4.2(6) and 5.0(2) (Bogota), fixed in 5.1(1) (Cairo)

On N5k: under investigation for Goldcoast

• vPC peer-switch:

On N7k: 4.2(6), 5.x

On N5k: under investigation for Goldcoast

• FEX preprovisioning:

On N7k: Freetown

On N5k: NXOS 5.0(2)N1(1)

• Dual Layer vPC:

On N7k: TBD

On N5k: Fairhaven


• vPC allows a single device to use a port channel across two neighbor switches (vPC peers)

• Eliminate STP blocked ports

• Layer 2 port channel only

• Provide fast convergence upon link/device failure

Port

channel

vPC Peers

Portchannel

vPC Peers


• Peer Link carries both vPC data and control traffic between peer switches

• Carries any flooded and/or orphan port traffic

• Carries STP BPDUs, IGMP updates, etc.

• Carries Cisco Fabric Services messages (vPCcontrol traffic)

• Carries ―multicast‖ traffic (more details follow)

• Minimum 2 x 10GbE ports

• ALL VLANS used on vPC PORTS MUST BE PRESENT ON THE PEER-LINK

5020 (config)# interface port-channel 10

5020 (config-if)# switchport mode trunk

5020 (config-if)# switchport trunk allowed <BETTER TO ALLOW ALL VLANS>

5020 (config-if)# vpc peer-link

5020 (config-if)# spanning-tree port type network

vPC Peer

Link

5k01 5k02


• Peer Keep-alive provides and out of band heartbeat between vPC peers

• Purpose is to detect and resolve roles if a Split Brain (Dual Active) occurs

• Messages sent on 1 second interval with 5 second timeout

• 3 second hold timeout on peer-link loss before triggering recovery

• Should not be carried over the Peer-Link

• Use the mgmt0 interface in the management VRF

• Can optionally be a dedicated link, 1Gb is adequate (first 16 ports on 5020 are 1/10GE ports)

• 3rd option, use a routed inband connection over L3 infrastructure (using SVI‘s in the default VRF)

dc11-5020-1(config)# vpc domain 20

dc11-5020-1(config-vpc-domain)# peer-keepalive destination 172.26.161.201 source

172.26.161.200 vrf management

Note:

--------:: Management VRF will be used as the default VRF ::--------

Peer Keepalive

can be carried

over the OOB

management

network

int mgmt 0


• Peer keep-alive is a routable protocol (both N5K and N7K)

• Primary design requirement is to have a physically different path than all other vPC traffic

• In all cases do not carry the peer-keepalivecommunication over the vPC peer-link

On Nexus 7000 when possible use dedicated VRF and front panel ports for peer-keepalive link (1G is more than adequate).

2nd best is to use the management interfaces

3rd option is to use an upstream L3 network for peer-keepalive

• If using mgmt 0 interfaces do ‘not’ connect the supervisor management interfaces back to back

In a dual supervisor configuration only one management port will be active at a given point in time!

Connect both mgmt 0 ports to the OOB network

Standby Management

Interface

Active Management

Interface




• vPC forwarding rules


• L3 and vPC

• Adding FEX

• Summary designs


• Cisco Nexus 5000 Series

• Peer keepalive:

1st option management port.

2nd option dedicated front panel port in dedicated VLAN.

3rd option upstream L3 network

• Cisco Nexus 7000 Series

• vPC works on all existing I/O modules

• Peer keepalive:

• 1st option dedicated front panel port in dedicated VRF.

• 2nd option is management interface.

• 3rd option upstream L3 network

• M1/F1 cards can be used for vPC

• Peer-link requires 10 GigE cards

• Peer-link should not span M1 and F1, peer-link should be made on either all F1cards or all M1 cards


Part number ModelvPC Peer-link

(10 GE Only)

VPC Member Port

N7K-M132XP-12

N7K-M132XP-12L ✓ ✓

N7K-M148GT-11

N7K-M148GT-11L ✗ ✓

N7K-M148GS-11

N7K-M148GS-11L ✗ ✓

N7K-M108X2-12L ✓ ✓

N7K-F132XP-15 ✓ ✓

NEXUS 7000 I/O modules


M M M F F F F

M-Series Mode

M vPC Peer-link on

M-Series Modules

Mixed Chassis Mode

M vPC Peer-link on

M-Series Modules

F-Series Mode

vPC Peer-link on

F-Series ModulesF

Mixed Chassis Mode

vPC Peer-link on

F-Series Modules (*)F

Recommendation : for mixed chassis mode (F1/M1) with vPC peer-link on F1 ports, use at least 2 M1 LC. This will provide resiliency for L3 features (FHRP, SVI).

M

(*) : command ―peer-gateway exclude-vlan <vlan list>‖ needed for backup routing path over vPCpeer-link


• NX-OS 5.1.3 introduces new behavior for handling vPC peer-gateway in mixed chassis mode (M1/F1) :

•Topology with M1 peer-link : IP/ARP packets destined to the remote Active IP/MAC get routed locally•Topology with F1 peer-link : IP/ARP packets destined to the remote Active IP/MAC use the tunneling mechanism

M M M M F F F F

M-Series

ModeMixed Chassis

Mode

F-Series

Mode

Mixed Chassis

Mode

Knob Not Required

Classic behavior of peer-gateway

Knob not Required

Peer Gateway not

required

Knob Required for transit

path/VLAN

IP/ARP Tunneling over Peer

link


vPC Peer-link

S1 S2

vPC Primary vPC Secondary

F1 F1

vPC Peer-link

S1 S2


F1M1

vPC Peer-link

S1 S2


M1M1






• L3 and vPC

• Adding FEX

• Summary designs


• With dual-active scenarios

MAC address synchronization is interrupted

IGMP synchronization is interrupted

• There is a 50% likelihood that unicast traffic is flooded and that multicast traffic is dropped

5k01 5k02

3IG

MP

rep

ort

fo

r G

1

2 - Host subscribes to G1

1

4 igmp sync lost


• There will be 2 primary switches sending independent BPDUsVPC Port-channels on upstream/downstream switches will be error-disabled by ‗EtherChannel Misconfiguration Guard‘ after ~90 secondshttp://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a008009448d.shtml

• If Nexus 7000/5000 is on the other end of VPC no action from STP as 7000/5000 do not support EtherChannel Guard

http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a008009448d.shtml

http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a008009448d.shtml


• When the peer-link is disconnected

• vPC secondary detects primary switch is alive through peer keepalive link

• The secondary vpc peer switch suspends all its vpc member ports in order to avoid traffic drop

• KEEP PEER KEEPALIVEAND PEER-LINKS SEPARATE

vPC

PrimaryvPC Secondary

Po10

5k01 5k02


dca-n7k2-vdc2

dc11-5020-1# show running int port-channel 201

version 4.1(3)N1(1)

interface port-channel201

switchport mode trunk

switchport trunk native vlan 100

switchport trunk allowed vlan 100-105

vpc 201


version 4.1(3)N1(1)





vpc 201

dca-n7k2-vdc2# sh run interface port-channel 201

version 4.1(5)




vPC supports standard 802.3ad port channels from upstream and or downstream devices

Recommended to enable LACP

―channel-group 201 mode active‖dc11-5020-2dc11-5020-1


dca-n7k2-vdc2


version 4.1(3)N1(1)





vpc 201


version 4.1(3)N1(1)





vpc 201

dca-n7k2-vdc2# sh run interface port-channel 201

version 4.1(5)





vpc 201

• vPC forwards only on locally connected members of the port channel if any exist (same principle as VSS)

• Multiple topology choices

• Square

• Full Meshdc11-5020-2dc11-5020-1


• vPC maintains layer 2 topology synchronization via CFS

• Copies of flooded frames are sent across the vPC-Link in case any single homed devices are attached

Frames received on the vPC-Link are not forwarded out vPC ports

2

3

1. Host MAC_A send packet to MAC_C

2. FEX runs hash algorithm to select one fabric uplink

3. N5K-1 learns MAC_A and flood packets to all ports

(in that VLAN). A copy of the packet is sent across

the peer link

4. N5K-2 floods the packet to any port in the VLAN

except the vPC member ports to prevent duplicated

packets

5. N7K-1 and N7K-2 repeat the same forwarding logic

6. N5K-1 updates the the MAC address learned on the

vPC port on N5K-2 via CFS

MAC_C

MAC_A

1

5

5

N5K-1 N5K-2

CFS

6

4

CFS

6


• Traffic is forwarded if destination address is known (both switches MAC address tables populated)

• Always forward via a locally attached member of a vPC if it exists

1. Host MAC_C send packet to MAC_A

2. N7K-2 forwards frame based on learned

MAC address

3. N5K-2 forwards frame based on learned

MAC address

MAC_C

MAC_A

N5K-1 N5K-2

1

2

3

N5K-1# sh mac-address-table vlan 101

VLAN MAC Address Type Age Port

---------+-----------------+-------+---------+-----

101 001b.0cdd.387f dynamic 0 Po30

101 0023.ac64.dda5 dynamic 30 Po201

Total MAC Addresses: 4



---------+-----------------+-------+---------+-----





• On loss of all of the locally attached members of the vPC MAC address table is updated to forward frames for the vPCacross the vPC Peer Link

• Note: Po20 is the vpc peer-link

MAC_C

MAC_A

N5K-1 N5K-2



---------+-----------------+-------+---------+-----






---------+-----------------+-------+---------+-----




1

3

2


√ √ X √

x


• Both switches in the vPC Domain maintain distinct control planes

• CFS provides for protocol state synchronization between both peers (MAC Address table, IGMP state, …)

• System configuration must also be kept in sync

• Currently there are 2 options to keep configuration consistent:

a manual process with an automated consistency check to ensure correct network behavior

config-sync

Two types of interface consistency checks

Type 1 – Will put interfaces into suspend state to prevent invalid forwarding of packets

Type 2 – Error messages to indicate potential for undesired forwarding behavior


• Type 1 Consistency Checks are intended to prevent network failures

• Incorrectly forwarding of traffic

• Physical network incompatibilities

• vPC will be suspended

dc11-5020-2# show vpc brief

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

<snip>

vPC status

----------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

------ ----------- ------ ----------- -------------------------- -----------

201 Po201 down failed vPC type-1 configuration -

incompatible - STP

interface port guard -

Root or loop guard

inconsistent

dc11-5020-1# sh run int po 201





vpc 201






vpc 201

spanning-tree guard root


• Type 2 Consistency Checks are intended to prevent undesired forwarding

• vPC will be modified in certain cases (e.g. VLAN mismatch)

dc11-5020-1# show vpc brief vpc 201

vPC status

----------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

------ ----------- ------ ----------- -------------------------- -----------

201 Po201 up success success 100-104

2009 May 17 21:56:28 dc11-5020-1 %ETHPORT-5-IF_ERROR_VLANS_SUSPENDED: VLANs 105 on Interface port-

channel201 are being suspended. (Reason: Vlan is not configured on remote vPC interface)


version 4.1(3)N1(1)





vpc 201


version 4.1(3)N1(1)





vpc 201


Global Spanning Tree Parameters need to be

consistent

Global QoS Parameters need to be consistent

Global Parameters

• c-nexus5010-1# show vpc consistency-parameters global

• Legend:

• Type 1 : vPC will be suspended in case of mismatch

• Name Type Local Value Peer Value

• ------------- ---- ---------------------- -----------------------

• QoS 2 ([], [3], [], [], [], ([], [3], [], [], [],

• Network QoS (MTU) 2 (1538, 2240, 0, 0, 0, (1538, 2240, 0, 0, 0,

• Network Qos (Pause) 2 (F, T, F, F, F, F) (F, T, F, F, F, F)

• STP Mode 1 Rapid-PVST Rapid-PVST

• STP Disabled 1 None None

• STP MST Region Name 1 "" ""

• STP MST Region Revision 1 0 0

• STP MST Region Instance to 1

• VLAN Mapping

• STP Loopguard 1 Disabled Disabled

• STP Bridge Assurance 1 Enabled Enabled

• STP Port Type, Edge 1 Normal, Disabled, Normal, Disabled,

• Allowed VLANs - 1,50 1

• Local suspended VLANs - 50 -


• Don‘t forget to keep global configuration in sync

• Any configuration that could cause an error in forwarding (e.g. loop) will disable all affected interfaces

• As an example if you make a change to an MST region you must make it on ‗both‘ peers

• Solution: define MST region mappings from the very beginning of the deployment, for ALL VLANs, the ones that exist as well as the ones that have not yet been created

• Defining a region mapping is orthogonal to creating a VLAN

vPCvPC vPC

mst region

vlans 1-5, 12mst region

vlans 1-5, 10






• L3 and vPC

• Adding FEX

• Summary designs


Inconsistency Type Impact Recommendation New Enhancements

VLAN to MST Region mapping mismatch

STP global settings (BA, Loop Guard, Root Guard)

1

Global

Pre-provision and MAP all VLANs on the MST region

Perform STP operations per port

Operate change during maintenance window

Leverage graceful conflict resolution Config Sync

(5.0(2)N1(1) on N5K, Freetown for N7K)

&

Graceful Conflict Resolution

(CSCtf84865,N7K -4.2(8)& 5.2, N5K –

5.0(2)N2(1))

Spanning-tree per interface settings,

switchport type (trunk/versus access)…

Port-channel mode

Per-vPC

Operate change during

maintenance window and/or

leverage graceful conflict

resolution

Quality of Service Configuration

2

Global

Minimum disruption

Per-vPCVLANs configured on vPC


• tc-nexus5010-1# show vpc consistency-parameters global

• Name Type Local Value Peer Value

• ------------- ---- ---------------------- -----------------------

• QoS 2 ([], [3], [], [], [], ([], [3], [], [], [],

• []) [])

• Network QoS (MTU) 2 (1538, 2240, 0, 0, 0, (1538, 2240, 0, 0, 0,

• 0) 0)

• Network Qos (Pause) 2 (F, T, F, F, F, F) (F, T, F, F, F, F)

• Input Queuing (Bandwidth) 2 (50, 50, 0, 0, 0, 0) (50, 50, 0, 0, 0, 0)

• Input Queuing (Absolute 2 (F, F, F, F, F, F) (F, F, F, F, F, F)

• Priority)

• Output Queuing (Bandwidth) 2 (50, 50, 0, 0, 0, 0) (50, 50, 0, 0, 0, 0)

• Output Queuing (Absolute 2 (F, F, F, F, F, F) (F, F, F, F, F, F)


• With Graceful Resolution only ports on the vPC secondary are ―suspended‖ if a Type-1 global inconsistency occurs

• This limits the impact of configuration changes.

• switch(config)# vpc domain 10

• switch(config-vpc-domain)# [no] graceful consistency-check

• Requires 5.0(2)N2(1) on the Nexus 5k

• Requires 5.2 on the Nexus 7kvPCvPC vPC

mst region


vlans 1-5, 10

vPC primary vPC secondary


Check whether STP is enabled or disabled on per-VLAN basis.

VLANs that have mismatched status will be suspended on both switches

Rest of VLANs won‘t be affected

Prior to this change all VLANs are affected

Disable STP on VLAN 5

5.2

5.0(2)N2(1)


• Config-sync allows administrators to make configuration changes on one switch and have the system automatically synchronize to its peers.

• This eliminates any user prone errors & reduces the administrative overhead of having to configure both vPC members simultaneously.

• Config-sync and Graceful conflict resolution are complementary features

• Config-sync traffic is carried over the peer keepalive link

vPCvPC vPC

mst region


vlans 1-5

+ vlan 12


• Global Configurations:

VLANs

ACLs

STP configurations

QOS

• Interface Level Configurations:

Ethernet Interfaces

Port Channel Interfaces

vPC Interfaces

• Which configurations are not synchronized?

Enabling ―Feature‖

vPC domain configuration

FCoE configuration


N5000-1#sh run switch-profile

Switch-profile Apple

sync-peers destination 10.29.170.8

N5000-1(config-if)# config sync

N5000-1(config-sync)# switch-profile Apple

N5000-1(config-sync-sp)# int ethernet

100/1/3

N5000-1(config-sync-sp-if)# switch mode

trunk

N5000-1(config-sync-sp-if)# verify

Verify Successful


Switch-profile Apple

sync-peers destination 10.29.170.7


interface ethernet 100/1/3



interface ethernet 100/1/3


N5000-1#

feature vpc

vpc domain 10

peer-keepalive destination 10.29.170.8

N5000-2#

feature vpc

vpc domain 10

peer-keepalive destination 10.29.170.7

N5000-1(config-if)# config sync

N5000-1(config-sync)# switch-profile Apple

N5000-1(config-sync-sp)# commit

Commit Successful

NOTE: Verify does not push the config to peer, user must issue “commit” for sync to

take place

If sync fails, then the config is in the BUFFER


N5K-1(config-sync-sp-if)# sh switch-profile A buffer

-----------------------------------------------------

Seq-no Command

-----------------------------------------------------

1 interface Ethernet100/1/9

1.1 switchport mode trunk

1.2 switchport trunk allowed vlan 5-10


2.1 switchport mode access

N5K-1(config-sync-sp)# ?

buffer-delete Delete buffered command(s)

buffer-move Move buffered command(s)

N5K-1(config-sync-sp)# buffer-delete 1

N5K-1(config-sync-sp)# sh switch-profile A buffer

-----------------------------------------------------

Seq-no Command

-----------------------------------------------------


2.1 switchport mode access

• Configuration is stored in a buffer until commit is applied.

• User can add/delete/move configuration.

• Once the config has been pushed via commit, it will no longer show up in buffer (it will show up in ―show running-config switch-profile X‖)

• If the commit fails due to mutexcheck or other reasons, the failed configuration still shows in the buffer, you have to explicitly remove it to continue


• Interface Ethernet1/11

• fex associate 100

• switchport mode fex-fabric

• channel-group 100

config-t area

switch-profile area• Interface Ethernet1/11

• shut/no shut

This portion is synchronized

This portion is not

synchronized


A port-channel may consist of port ethernet 1/1 on n5k01

And erthernet 1/2 on n5k02 FEX A/A has the same FEX configured to both N5ks, so

Preprovisioning has to be configured identically


• If one vPC peer needs to be disconnected completely from the vPCdomain you can still operate the remaining one

• For this you need to leverage the commands ―reload restore‖ and ―autorecovery‖

• Reload restore deals with the split brain scenario allowing a vPC peer to bring up new vPC ports even after a reload

• Autorecovery deals with the sequential loss of peer-link first, and peer-keepalive second, allowing the vPCsecondary to bring up the vPC ports (which were down previously)


• VPC needs to be able to talk to the peer (over peer-link) before bringing up VPC port-channels

Negotiate LACP/STP operating roles for the chassis

Wait for per-port peer parameters and handshake to bring up vPC ports

• Performs peer parameters consistency check on each VPC bringup

• Only after VPC port-channels are brought up.

• What if after a full DC outage (both Nexus down), only one switch is coming up ?

• Will not bring up VPCs if after a datacenter outage, only one VPCpeer comes back up


1

Switch3 Switch3

Switch2Switch1

Switch1

2

Switch3

Switch1

3

When adding a new vPC member port, the port goes up

Existing vPCs are brought up


1

Switch3

Switch2Switch1 Switch3

Switch1

2

Switch2

Switch3

Switch1

3

Switch2


S2-SecondaryS1 -Primary

Peerlink down and keepalive working

Secondary shuts vPCs

vPC peer-link

vPC 1

po1

Keepalive


Primary fails

Po1 is completely shut

vPC peer-link

vPC 1

po1

Keepalive


vPCPrimary

Peerlink down and keepalive down

vPC peer-link

vPC 1

po1

Keepalive

After 3 consecutive keepalive timeouts

Secondary changes role and brings up vPCs

vPC peer-link

vPC 1

po1

Keepalive

vPC Operational Primary

vPCSecondary


• STP for vPCs is controlled by the vPC operationally primary switch and only such device sends out BPDUs on STP designated ports.

• This happens irrespectively of where the designated STP Root is located

• The vPC operationally secondary device proxies STP BPDU messages from access switches toward the primary vPC

Primary

vPC

Secondary

vPC

BPDUs


MAC_A MAC_BSW3

SW2

SW4

SW1

vPC1 vPC2

vPC_PL

L2

L3

ECMPECMP

vPC PK-Link

Primary

vPC

Secondary

vPC

Secondary

Root

Primary

Root

vPC peer-link is a

regular STP port

vPC Primary

Switch Source

and controls

STP for vPCs

The secondary vPC

device does NOT

source STP BPDUs on

symmetrical vPCs


• Assume the following topology with vPC enabled on the vPC

• If the Primary fails over, the secondary needs to start sending BPDUs

• If the Primary was also the STP root, the secondary also has to overtake the role as a root

• If this process lasts too long, the uplink port on 5k02 may go into BA_Inconsistent state

• Better not use Bridge Assurance with vPC

• Bridge Assurance on peer-link is fine (and is the default)

5k01 5k02

Secondary

becomes

primary and

root

7k01 7k02

Primary /

Root

BPDUs prior

to the failureBA Inconsistent


Primary Secondaryleft# sh span vlan 101

VLAN0101Spanning tree enabled protocol rstpRoot ID Priority 8293

Address 0023.04ee.be01This bridge is the root

...

Bridge ID Priority 8293 (priority 8192)Address 0023.04ee.be01

...

Interface Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- ---------------Po1 Desg FWD 1 128.4096 (vPC) P2pPo100 Root FWD 2 128.4195 (vPC peer-link)

left# sh vpc role | i macvPC system-mac : 00:23:04:ee:be:01 vPC local system-mac : 00:1b:54:c2:42:43

right# sh span vlan 101

VLAN0101Spanning tree enabled protocol rstpRoot ID Priority 8293

Address 0023.04ee.be01This bridge is the root

...

Bridge ID Priority 8293 (priority 8192)Address 0023.04ee.be01

...Interface Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- ---------------Po1 Desg FWD 1 128.4096 (vPC) P2pPo100 Desg FWD 2 128.4195 (vPC peer-link)

In Peer-Switch mode bridge-ID comes from system-mac as opposed to local mac in normal mode

ROOT ROOT


• BA is default enabled on Peer-Link (and recommended to remain enable), not recommended for VPCs unless Peer-Switch feature is used

Without Peer-switch BA should be kept only on Peer-Link (no BA/Loop guard on VPCs)

• Dispute is default enabled (for both RSTP and MST on VPC)

• UDLD [normal mode] is recommended to take out bad links from channels

• BA + UDLD + Dispute (on all interswitch links when using Peer-switch) when all switches support this (nexus7000/5000)


• By default on the Nexus 5x00 series, LACP sets a port to ―I state‖ if it does not receive an LACP PDU from the peer. This behavior is different on the Nexus 7000 series where the default is to suspend a port if it doesn‘t receive LACP PDUs.

• For server facing port-channels it is better to allow LACP ports to revert to I-state if the server doesn‘t send LACP PDUs. By doing this the I-state port can operate like a regular Spanning-Tree port. Also this allows immediate server connectivity when it boots up before the full LACPnegotiation has taken place.

• For network facing ports, allowing ports to revert to I-state creates additional Spanning-tree state without any real benefit.

• This behavior can be configured on a per Port-Channel basis with the configuration ―[no] lacp suspend-individual‖ (which is the equivalent of the Catalyst IOS command ―port-channel standalone-disable‖.


• IGMP snooping shares the snooped reports with the peer vPCswitch to help with multicast forwarding

• Forwarding of IGMP protocol packets is tweaked so that IGMPreports received on one vPC switch is also forwarded to the vPC peer. Thus multicast forwarding state remains in sync on both the vPC switches.

• Do NOT DISABLE IGMP Snooping!

• If you need to support Firewalls / Clusters:

Use static IGMP entries OR

Create an IGMP querier!


• vPC maintains dual active control planes and STP still runs on both switches

• IGMP join/leave messages received on one peer is forwarded to another peer via peer link

• IP muticast packets are sent to host through local port

• Non-IP multicast and broadcast packets are flooded


vPC SecondaryvPC Primary

IGMP

join/leave

IGMP

join/leave


• So is the multicast traffic going to the peer link?

Yes, but duplicates are avoided by using the vPC loop prevention technique, which should rather be called ―duplicate prevention‖

• And how about orphan ports?

Orphan ports receive traffic because the multicast traffic is always sent over the peer-link

N7k01 N7k02

N5k01 N5k02

1 3

1 2 3 4


• Assuming that there are no orphan ports it is possible to remove multicast traffic from crossing the peer-link with the command

• no ip igmp snooping mrouter vpc-peer-link (Nexus 5k)

• ip igmp snooping vpc peer-link-exclude (hidden command on the Nexus 7k, not supported)


N5k-1 N5k-2

IGMP Group sync

• VPC peer-link is considered as mrouterport. Therefore all multicast traffic is flooded over peer-link

• A CLI was introduced in 5.0(3)N1(1) to avoid that. With the CLI multicast traffic is sent to vPC peer-link only when it is necessary, such as, there is singly connected host

• Improving multicast convergence time with peer-link down/up and switch reload

• The CLI is not supported for FEX dual-home topology in 5.0(3)N1(1). The limitation will be removed in upcoming release-5.0(3)N2(1)


vPC on the N7k

vPC on the N5k

N7k01 N7k02

N5k01 N5k02

1 2 3 4

• If the peer-link is lost the vPC secondary is going to shut down the vPC member ports

• For single attached hosts, plssee

• CSCtc49559

• and Orphan ports ―suspend‖ feature


Intended for devices that do not support port-channel. Other devices should be dually connected by vPCs (Orphan-port CLI is available only on physical ports, not on port-channels)

Configure single attached devices (like FW or LB) port as orphan-port

When vPC peer-link goes down, vPC secondary peer device shuts all its vPC member ports as well as orphan ports

CE-1


vPC peer-link

vPC 1

po1

Keepalive

Orphan port

Active or Standby

Active or Standby

S1(config)# int eth 1/1

S1(config-if)# vpc orphan-ports suspend

S2(config)# int eth 1/1

S2(config-if)# vpc orphan-ports suspend






• L3 and vPC

• Adding FEX

• Summary designs


• vPC maintains dual active control planes and STP still runs on both switches

• HSRP active process communicates the active MAC to its neighbour

• Only the HSRP active process responds to ARP requests

• HSRP active MAC is populated into the L3 hardware forwarding tables, creating a local forwarding capability on the HSRP standby device

• Consistent behavior for HSRP, VRRP and GLBP

• No need to configure aggressive FHRP hello timers as both switches are active

HSRP Active HSRP Standby

HW Programmed to forward frames

sent to the FHRP MAC address on

BOTH Switches


It recommended to ‗not‘ use HSRP link tracking in a vPC configuration

Reason: vPC will not forward a packet back on a vPC once it has crossed the peer-link, except in the case of a remote member port failure

Use an L3 point-to-point link between the vPC peers to establish a L3 backup path to the Core in case of uplinks failure

A single point-to-point VLAN/SVI will suffice to establish a L3 neighbor

VLAN 100 VLAN 200

VLAN 100, 200,300

SVI

VLAN 300

SVI

VLAN 300


• Non-RFC compliant end hosts

Device required to send packets to the MAC address returned in ARP response (HSRP virtual MAC)

Some non-compliant devices use the MAC address of the sender device (Switch physical MAC)

NAS devices (i.e. NETAPP Fast-Path or EMC IP-Reflect) have been found to do this

• vPC Peer Gateway - NX-OS 4.2(1)

Allows a vPC peer to respond both the the HSRP virtual and the real MAC address of both itself and it‘s peer

L3

L2

VLAN 100VLAN 200

“peer-gateway” command

tells the vPC to respond

to the physical MAC

address of its peer


Not enabled by default

After the peer-link comes up perform an ARP bulk sync over CFSoE to the peer switch

Improve Convergence for Layer 3 flows

ARP Synchronization Process

Primary vPC

Secondary vPCS

P

P S

ARP TABLE

IP1 MAC1 VLAN 100

IP2 MAC2 VLAN 200

ARP TABLE

IP1 MAC1 VLAN 100

IP2 MAC2 VLAN 200

IP1 MAC1 IP2 MAC2

SVIs

S1(config-vpc-domain)#

ip arp synchronize

S2(config-vpc-domain)#

ip arp synchronize

Note:

CSCti06907

has

been fixed


Feature Function Availability

VPC interaction with FHRP Both active and standby peer function as

gateway

HSRP VRRP

Peer-gateway L3 forwarding when the DMAC is peer‘s

MAC

vPC delay restore Delay bringing up vPC ports

vPC exclude VLAN CLI to specify SVI interfaces won‘t be

suspended when peer-link fails

ARP synchronization Synchronize ARP between two peer

switches

Roadmap

PIM pre-built-SPT Both N5k joins source tree as PIM last hop

router

PIM dual DR Both N5k can be DR when it is first hop

router






• L3 and vPC

• Adding FEX

• Summary designs


FE

X21

48T

sta

rtin

g f

rom

4.1

(3)N

1(1

)

FE

X2248,

2232 fro

m

4.2

(1)N

1(1

)

FE

X2248,

2232,

2224

from

4.2

(1)N

1(1

)


Fairhaven

Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 75

N7K NXOS 5.1(1)

activeactive

N

activeactive

N

Y

activeactive

N

activeactive

N7K NXOS 5.2

activeactive

activeactive

Future

Y

radar

Y

YY N Y Y


Nexus 5000 Topologies (Nexus 2248TP & 2232PP)

Redundancy model – Dual Switch with redundant fabric

Provides isolation for Storage topologies (SAN ‘A’ and ‘B’)

Port Channel and Pinning supported for Fabric Link

vPC Supported with up to 2 x 8

links

Local Etherchannel with up to 8

links

FCoE Adapters supported on 10G

N2K interfaces

Straight Through

Redundancy model – Single switch with dual ‘supervisor’ for fabric, data control & management planes

No SAN ‘A’ and ‘B’ isolation (VSAN isolation sufficient in the future?)

Dual Homed


Nexus 7000 Topologies (Nexus 2248TP & 2232PP)

Local Etherchannel with up to 8

links

NIC Teaming:TLB/ALB

NXOS 5.2Nexus 2248TP & 2232PP

Fabric links supported on N7K-M132XP-12 & N7K-M132XP-12L

Port Channel only supported for Fabric Links

Local port channel support on 2248 & 2232

No support for DCB and FCoE (parent switch fabric ports not DCB capable yet)


Redundancy model – Dual Switch (each switch supports redundant supervisors)

MCEC Etherchannel with up to 16

links

Nexus 5000 Fairhaven

Redundancy model – Single switch with dual ‘supervisor’, fabric, line card, data control & management planes

MCEC Etherchannel with up to 16

links

Nexus 7000 - vPC – NXOS 5.2


24 FEX

Nexus 2000 Straight-through deployment

n5k01

FEX100

FEX101

FEX102

max 24 with Nexus 5500 = 768 ports

max 4/8 ―fabric links‖

Active/Standby

n5k01

FEX100

FEX101

FEX102

n5k02

FEX120

FEX121

FEX122

max 24 x 2


Cisco Nexus 2000 Series

Straight-Through vPC

Cisco Nexus 2000

Active-Active

vPC

Primary

FEX120FEX100vPC 1 vPC 2

FEX120FEX100

vPC

Secondary

HIF HIF

HIF HIF

Fabric Links Fabric Links

Peer Keepalive

Peer Link

vPC Member Port

vPC

Primary

vPC

Secondary

up to 8 ports

up to 8 ports

up to 8 ports up to 8 ports

up to 24 PC

per FEX up to 24 PC

per FEX



FEX 2248


Cisco Nexus 2000 Series

Straight-Through vPC

Cisco Nexus 2000

Active-Active

vPC

Primary

FEX120FEX100vPC 1 vPC 2

FEX120FEX100

vPC

Secondary

HIF HIF

HIF HIF

Fabric Links Fabric Links

Peer Keepalive

Peer Link

vPC Member Port

vPC

Primary

vPC

Secondary

up to 8 ports

up to 8 ports


up to 16 PC

per FEX up to 16 PC

per FEX



FEX 2232

Compatible with FCoE IF server uses 2 uplinksDoesn‘t support FCoE, today


• In a Dual Tier vPC configuration FCoE traffic will NOT be load shared across both sets of fabric links

• SAN ‗A‘ and ‗B‘ isolation is maintained

• This may result in un-even sharing of traffic across the multiple fabric links

FCoE + LAN on one set of fabric links

LAN only on the other set of fabric links

• Need to plan for the aggregate traffic capacity

LAN & SAN traffic

SAN BSAN A

LAN traffic






• L3 and vPC

• Adding FEX

• Summary designs


vPC on the N7k

N7k01 N7k02

N5k01 N5k02

2/1 2/2 2/1 2/2

2/9 2/10 2/9 2/10

Po51,2

root

logical equivalent

Root


vPC on the N7k

N7k01 N7k02

N5k01 N5k02

2/1 2/2 2/1 2/2

Po10

2/9 2/10 2/9 2/10

Po51

Peer Link

primary secondary

root

regular STP priority

logical equivalent

Root


2248TPs

5500 or 50x0

7010s with

F1linecards

vPC peer link

Running vPC

only for server

attach ports

x8x8 x8

x8

vPC peer link16 port

HW

Etherchan

nel

16 port

HW

Etherchan

nel

... ...

32 PORTS


SW01SW02

N5k01 N5k02

2/1 2/2 2/1 2/2

Po10

2/9 2/10 2/9 2/10

Po51

Peer Link

primary secondary

regular STP priority

Root

HSRP primary

Secondary Root

HSRP secondary

Clear access VLANs to

create a Loop Free Topology

logical equivalent

SW01SW02

2/9 2/10 2/9 2/10

F F



Root

HSRP primary

Secondary Root

HSRP secondary


• Traffic flows are symmetric from access to aggregation

• vPC is still useful to optimize traffic flows from access to aggregation

• All traffic flows through the active HSRP switch, in this case SW01

• Client-to-Server traffic uses both SW01 and SW02

• Peer-link is almost unutilized

SW01SW02

N5k01 N5k02

2/1 2/2 2/1 2/2

Po10

2/9 2/10 2/9 2/10

Po51

Peer Link



Root

HSRP primary

Secondary Root

HSRP secondary


• Define domains

• Establish Peer Keepalive connectivity

• Create a Peer link

• Create vPCs

• Make Sure Configurations are Consistent / leverage config-sync / configure graceful conflict resolution

N7k01 N7k02

5 6 7 8

N2k01 N2k02

1 2 3 4

• Following steps are needed to build a vPC


• Ensure domain-id or system-mac differs between Agg pair and Access pair

• Connect the N7ks with redundant peer-links across linecards

• Connect the N5ks with redundant peer-links

• Create a single Port-channel leveraging LACPbetween Aggregation and Access

• Do not forget that putting a VLAN on a vPCrequires that that VLAN be on the Peer-link too

• If you foresee significant multicast traffic, or there is a high percentage of single attached devices, you may want to size the peer-link to match the uplink bandwidth utilization

N7k01 N7k02

1 2 3 4

N5k02N5k01

5 6 7 8

N2k01 N2k02

1 2 3

LACP


• If you use the ―peer switch‖ functionality, then define Identical Priorities on the Aggregation Layer switches, to make them the root

• Do not use Bridge Assurance

• Keep the default STP priorities on the access layer switches

• IF using MST, Make Sure that VLAN range Configurations are Consistent

• With MST be aware of the NXOS VLAN range and of the Global Type-1 Inconsistencies, hence configure VLAN-to-region mappings from day 1

• Use pathcost method long

• Configure STP port type edge or port type edge trunk

N7k01 N7k02

5 6 7 8

N2k01 N2k02

1 2 3 4


• Configure HSRP priorities as usual, both peers forward L3 traffic

• Configure vPC delay restore to avoid L3 traffic loss upon reboot

• Create a L3 backup ―link‖ between the N7k

• Configure peer-gateway for firewalls, load balancers, filers

• Configure regular L3 ECMP from the core to the aggregation layer

N7k01 N7k02

1 2 3 4

N5k02N5k01

5 6 7 8

N2k01 N2k02

1 2 3


N7k01 N7k02

N5k01 N5k02

2/1 2/2 2/1 2/2

2/9 2/10 2/9 2/10

• Make sure to leverage Reload Restore and auto-recovery

• Make sure to have mgmt0 connectivity for config-sync to work (you may want to use the same mgmt0 for vPC peer keep-alive)

• FEX A/A provides redundancy and each HIF

• Config-sync also helps with regular port channels

• FEX pre-provisioning is highly recommended

Thank You

cisco - vpc concepts

Documents

fwd fwd cost

graceful conflict

iparp packets

profile apple

loop free

mixed chassis

remote active

1sh run switch