nexus technology labs

7/26/2019 Nexus Technology Labs

1/313

Nexus Technology Labs - Fabric Extenders(FEX)

FEX Active Standby

Task

Configure N5K1 to pair with the Fabric Extender N2K1 as follows:

Enable the Fabric Extender feature.

Configure N5K1's link connecting to N2K1 as a FEX port.

N2K1 should be module number 101.Configure N5K2 to pair with the Fabric Extender N2K2 as follows:


Configure N5K2's link connecting to N2K2 as a FEX port.

N2K2 should be module number 102.

Configure the links between N5K1 & N5K2 as 802.1q trunk links.

Configure N5K1's links to Server 1 and the Emulex CNA Server in VLAN 10.

Configure N5K2's links to Server 2 and the Emulex CNA Server in VLAN 10.

Configure Server 1 with the IP address 10.0.0.1/24 on this link.Configure Server 2 with the IP address 10.0.0.2/24 on this link.

Configure the Emulex CNA Server to do Active Standby NIC teaming as follows:

Use the IP address 10.0.0.10/24 for the NIC Team.

Use the link to N2K1 as the primary active path and the link to N2K2 as the

secondary standby path.

Verify that both Server 1 and Server 2 have connectivity to the Emulex CNA Server,

and that traffic to the server is flowing only through N2K1.

Disable the FEX port from N5K1 to N2K1, and verify that connectivity to the CNA

Server is maintained by using the backup path through N2K2.

Configuration

N5K1:

feature fex

!

vlan 10


2/313

!

interface Ethernet1/1

switchport access vlan 10

spanning-tree port type edge

speed 1000

!

interface Ethernet1/3 - 5

switchport mode trunk

spanning-tree port type network

!


switchport mode fex-fabric

fex associate 101

!

interface Ethernet101/1/1


N5K2:

feature fex

!

vlan 10

!




speed 1000

!




!



fex associate 102

!



Verification

In Active/Standby FEX topologies, hosts are physically attached to multiple FEXes,

but only actively forward on one path. Note that this topology is not related to vPC,

as vPC is used to achieve active/active forwarding, not active/standby. This


3/313

topology also requires the end hosts support of teaming through software. In this

particular case, the teaming is achieved through the Emulex OneCommand utility,

which manages the NIC Team/Port Channel config of the end adapter.

First, the end host is configured to team its links together, with the type defined as

failover in this case. Some other utilities call this active/standby or

primary/secondary, but they essentially mean the same thing. Note that the first

connection is listed as Primary, which is the link to N2K1 (hence N5K1), whereasthe second connection goes to N2K2 (hence N5K2).

IP addressing is configured on the logical Team adapter, similar to how IOS or NX-

OS puts logical configuration on a port channel interface.

To test the traffic flows, you can use the iPerf application to generate bulk TCP or

UDP traffic. In the output below, we see the CNA Server receiving two TCP streams

of approximately 1Gbps each, one from Server 1 and one from Server 2.


4/313

From the network side, the interface counters indicate that both of these flows are

going through the link to N2K1/N5K1, while the backup link through N2K2/N5K2 is

unused.

N5K1# show interface e101/1/1 | include rate

30 seconds input rate 38455104 bits/sec, 75106 packets/sec

30 seconds output rate 1819517120 bits/sec, 149849 packets/sec input rate 38.46 Mbps, 75.11 Kpps;

output rate 1.82 Gbps

, 149.85 Kpps



30 seconds output rate 200 bits/sec, 0 packets/sec input rate 1.07 Kbps, 2 pps; output rate 200 bps

, 0 pps

A failure of the FEX port from N5K1 to N2K1 signals a link-down event to the end

host.

N5K1# config t

Enter configuration commands, one per line. End with CNTL/Z. N5K1(config)# int e1/10

N5K1(config-if)# shut

2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/10 is down(Config change)

2013 Mar 2 19:19:46 N5K1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 1 of Fex 101 that is connected with Etherne

2013 Mar 2 19:19:46 N5K1 %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-101 Off-line (Serial Number SSI16330GT8)

2013 Mar 2 19:19:46 N5K1 %PFMA-2-FEX_STATUS: Fex 101 is offline

2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet101/1/1 is down (module remov

2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet101/1/2 is down (module remov

2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/3 is down (Interface





5/313




2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/10 is down (Interfac

























N5K1(config-if)# 2013 Mar 2 19:19:47 N5K1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 1 of Fex 101 that is conne

2013 Mar 2 19:19:47 N5K1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/10 is down (Administratively do

The end hosts NIC Teaming software detects the primary link failure and begins to

forward via the backup path.


6/313

From the network view, we see that traffic is now re-routed through the backup path

via N2K2/N5K2.





, 140.58 Kpps


7/313


FEX Active Active Host vPC

Task

Configure N5K1's link to Server 1 in VLAN 10.


Configure Server 1 with the IP address 10.0.0.1/24 on this link.

Configure Server 2 with the IP address 10.0.0.2/24 on this link.Configure FEX support as follows:

Configure N5K1 to pair with N2K1 using FEX number 101.

Configure N5K2 to pair with N2K2 using FEX number 102.

Configure a vPC between N5K1 and N5K2 as follows:

Configure vPC domain 1 on the vPC peers N5K1 and N5K2.

Use the mgmt0 port as the vPC Peer Keepalive link.

Use LACP for negotiation of all port channels.

Configure all links between the vPC peers as Port-Channel 1, and use thisas the vPC Peer Link.

Configure N5K1 and N5K2's links to the Emulex CNA Server as Port-

Channel 10 and vPC 10.

Port-Channel 10 should be an access port in VLAN 10.

Configure the Emulex CNA Server with LACP NIC Teaming, and use the IP address

10.0.0.10/24 for the NIC Team.


and that traffic to the server is being load balanced across both links through

N2K1/N5K1 and N2K2/N5K2.

Configuration

N5K1:

feature lacp

feature vpc

feature fex


8/313

!

vlan 10

!

vpc domain 1

peer-keepalive destination 192.168.0.52

!

interface port-channel1



vpc peer-link

!



vpc 10

!




speed 1000

!




channel-group 1 mode active

!



fex associate 101

!




N5K2:

feature lacp

feature vpc

feature fex

!

vlan 10

!

vpc domain 1


!





9/313

vpc peer-link

!



vpc 10

!




speed 1000

!





!



fex associate 102

!




Verification

Fabric Extender (FEX) and vPC topologies currently come in three forms. The first is

a vPC from the FEX southbound to the end server, sometimes called a Host vPC;

the second is a vPC from the FEX northbound to the parent switches, sometimes

called a Fabric vPC; and the third is both a southbound and northbound vPC from

the FEX, which is considered an Enhanced vPC or EvPC. Note that EvPC is only

supported on newer hardware platforms with corresponding newer software

releases. This particular configuration is considered the first variation, a Host vPC.

This example uses the same physical topology as before, except now the server

attached to the Fabric Extenders can do active/active forwarding. This isaccomplished by configuring a vPC between the parent switches of the FEXes.

Logically, this topology would be the same as if the CNA server were physically

wired to N5K1 with one link of its NIC, and then to N5K2 with the other link. This is

again because the FEX simply acts as a remote line card of the parent switch and

behaves just like a module of a modular switch. Because of the vPC configuration,

N5K1 and N5K2 appear to be the same upstream switch from the CNA servers

perspective; therefore, it can do active/active forwarding and load balancing just as


10/313

if it was dual attached to a single switch.

From the server side, the NIC Teaming software is configured to form an LACP-

based team. Note that although the terms LACPand 802.3adare normally

interchangeable, some variations of NIC Teaming software use one term to define a

channel as mode on and the other as mode active. In the case of the Emulex

OneCommand, if you choose 802.3ad teaming, you would need to configure the

channel-group 10 mode on on the NX-OS side, while LACP means that thechannel mode can be active. As shown below, the load balancing method can also

be chosen based on load, IP address, or MAC address.

Like before, the IP address goes on the logical team adapter, not the physical links.

From the network side, the first major verification is to ensure that the vPC peering

is up between the 5Ks. Only after the keepalive is confirmed and the vPC peer link

is formed can the vPC to the end host actually form. Note that the same vPC

consistency rules apply to FEX-based vPCs as to regular vPCs.


11/313

N5K1# show vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1 Peer status : peer adjacency formed ok

vPC keep-alive status : peer is alive

Configuration consistency status: success

Per-vlan consistency status : success

Type-2 consistency status : success

vPC role : primary

Number of vPCs configured : 1

Peer Gateway : Disabled

Dual-active excluded VLANs : -

Graceful Consistency Check : Enabled

vPC Peer-link status

---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ --------------------------------------------------1 Po1 up

1,10

vPC status

----------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

------ ----------- ------ ----------- -------------------------- -----------10 Po10 up

success success 10

The FEX configuration itself it technically unrelated to the vPC, as the FEX Fabric

Ports are configured the same as before.

N5K1# show interface fex-fabric

Fabric Fabric Fex FEX

Fex Port Port State Uplink Model Serial

--------------------------------------------------------------- 101 Eth1/10 Active

1 N2K-C2232PP-10GE SSI16330GT8




--------------------------------------------------------------- 102 Eth1/11 Active

2 N2K-C2232PP-10GE SSI15030C1R

For final verification, generate traffic flows between the servers and note the

interface statistics of the FEX host ports to the Emulex CNA server. In the below

output, iPerf is used to generate bulk TCP flows from Server 1 and Server 2 to the


12/313

CNA server.

These two flows are near line-rate for the 1GigE attached Server 1 and Server 2.

The difference between this example and the last one, however, is that these 2 x1Gbps flows are distributed between the vPC member ports to the CNA server. This

can be verified as seen through the interface counters below:




output rate 934.56 Mbps

, 76.97 Kpps





, 76.98 Kpps

In the case of a link failure, traffic will automatically be rerouted to the other

available member links after LACP detects the fault. As shown below, when N5K2's

link to the downstream N2K2 FEX goes down, both 1Gbps traffic flows are rerouted

to the other FEX.

N5K2# config t

Enter configuration commands, one per line. End with CNTL/Z. N5K2(config)# int e1/11

N5K2(config-if)# shut


2013 Mar 2 22:10:07 N5K2 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 2 of Fex 102 that is connected with Etherne

2013 Mar 2 22:10:07 N5K2 %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-102 Off-line (Serial Number SSI15030C1R)

2013 Mar 2 22:10:07 N5K2 %PFMA-2-FEX_STATUS: Fex 102 is offline


13/313





, 151.41 Kpps


14/313


Fabric Extenders (FEX)

Task

Configure N5K1 to pair with the Fabric Extender N2K1 as follows:


Configure N5K1's link connecting to N2K1 as a FEX port

N2K1 should be module number 101.Configure N5K1's links to Server 1 and the Emulex CNA Server in VLAN 10.

These links should both be STP Edge Ports.


Configure the Emulex CNA Server with the IP address 10.0.0.10/24 on this link.

When complete, Server 1 and the Emulex CNA Server should have IP reachability to

each other.

Configuration


15/313

N5K1:

feature fex

!

vlan 10

!




speed 1000

!



fex associate 101

!



Verification

Fabric Extenders (FEXes) are access switches that behave as remote line cards of

a parent switch. After the FEX is paired with the parent switch, such as a Nexus 5K

or 7K, all configuration occurs on the upstream parent. From the parent switchs

perspective, the FEX is simply another module or line card, and is configured as

such.

In the output below, we can see that a Nexus 2232PP FEX is paired with the parent

switch N5K1 as FEX number 101. This means that the FEX is simply treated as

module 101 from the 5Ks perspective.

N5K1# show fex

FEX FEX FEX FEX

Number Description State Model Serial

------------------------------------------------------------------------

101 FEX0101 Online N2K-C2232PP-10GE SSI16330GT8

The detailed output below shows the specifics of this FEX, such as the software

version downloaded from the parent and what the state is. When the state is online,

the most important portion of the output shown below is how the downstream FEX

ports are pinned to the upstream Fabric Ports. In this topology, there is only one


16/313

physical uplink from the FEX to N5K1, so all FEX ports are pinned to the Fabric Port

E1/10. In a case in which more physical links are used, the pinning of FEX ports can

be controlled with the pinning max-links command under the global FEX

configuration, or the Fabric Port can be configured as a port-channel, essentially

dynamically pinning all FEX ports to all ports in the channel at the same time. In the

latter case, traffic is then load balanced based on the Port-Channel load balancing

method.

N5K1# show fex detail

FEX: 101 Description: FEX0101 state: Online

FEX version: 5.1(3)N1(1a) [Switch version: 5.1(3)N1(1a)]

FEX Interim version: 5.1(3)N1(1a)

Switch Interim version: 5.1(3)N1(1a) Extender Serial: SSI16330GT8

Extender Model: N2K-C2232PP-10GE

, Part No: 73-12533-05

Card Id: 82, Mac Addr: 54:78:1a:30:3d:c2, Num Macs: 64

Module Sw Gen: 12594 [Switch Sw Gen: 21]

post level: complete pinning-mode: static Max-links: 1

Fabric port for control traffic: Eth1/10

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state: Eth1/10 - Interface Up. State: Active

Fex Port State Fabric Port Eth101/1/1 Up Eth1/10

Eth101/1/2 Up Eth1/10

Eth101/1/3 Down None





















17/313











Logs:

03/02/2013 18:17:32.337586: Module register received

03/02/2013 18:17:32.339470: Registration response sent

03/02/2013 18:17:32.465539: Module Online Sequence 03/02/2013 18:17:35.611664: Module Online

When pairing between the FEX and the parent switch is complete, furtherconfiguration of the FEX ports is the same as any other physical link. Note that there

are some behavioral differences between FEX host ports and other physical links;

for example, the FEX ports always run as STP Edge Ports with BPDU Filter and

Guard enabled. This can be seen below; although the spanning-tree port type edge

is not configured on the FEX port, it still operationally runs in that mode.

N5K1# show spanning-tree vlan 10

VLAN0010

Spanning tree enabled protocol rstp

Root ID Priority 32778

Address 000d.eca2.edbc

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)



Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Eth1/1 Desg FWD 4 128.129 Edge P2p

Eth101/1/1 Desg FWD 2 128.1153 Edge P2p

N5K1# show spanning-tree interface e101/1/1 detail

Port 1153 (Ethernet101/1/1) of VLAN0010 is designated forwarding

Port path cost 2, Port priority 128, Port Identifier 128.1153


18/313

Designated root has priority 32778, address 000d.eca2.edbc

Designated bridge has priority 32778, address 000d.eca2.edbc

Designated port id is 128.1153, designated path cost 0

Timers: message age 0, forward delay 0, hold 0

Number of transitions to forwarding state: 1 The port type is edge

Link type is point-to-point by default Bpdu guard is enabled

Bpdu filter is enabled by default

BPDU: sent 11, received 0


19/313


FEX Active Active Fabric vPC

Task




Configure Server 2 with the IP address 10.0.0.2/24 on this link.Configure FEX support as follows:

Configure N5K1 and N5K2 to pair with N2K1 using FEX number 101.


Configure a vPC between N5K1 and N5K2 as follows:

Configure vPC domain 1 on the vPC peers N5K1 and N5K2.

Use the mgmt0 port as the vPC Peer Keepalive link.

Configure all links between the vPC peers as Port-Channel 1, and use this

as the vPC Peer Link.Configure the FEX Fabric ports from N5K1 and N5K2 to N2K1 as Port-

Channel 101, and as vPC 101.

Configure the FEX Fabric ports from N5K1 and N5K2 to N2K2 as Port-


Configure the Emulex CNA Server to do Active Standby NIC teaming as follows:

Use the link to N2K2 as the primary active path and the link to N2K1 as the

secondary standby path.

Use the IP address 10.0.0.10/24 for the NIC Team, and assign its links to

VLAN 10.


and that traffic to the CNA Server is flowing only through N2K2.

Disable the link from the CNA Server to N2K2, and verify that connectivity is

maintained by using the backup path through N2K1.


20/313

Configuration

N5K1:

feature lacp

feature vpc

feature fex

!

vlan 10

!

vpc domain 1


!




vpc peer-link

!



fex associate 101

vpc 101

!



fex associate 102

vpc 102

!




speed 1000

!





!



fex associate 101

channel-group 101 mode on

!



21/313


fex associate 102


!



!



N5K2:

feature lacp

feature vpc

feature fex

!

vlan 10

!

vpc domain 1


!




vpc peer-link

!



fex associate 101

vpc 101

!



fex associate 102

vpc 102

!




speed 1000

!





!



22/313


fex associate 101


!



fex associate 102


!



!



Verification

Fabric Extender (FEX) and vPC topologies currently come in three forms. The first is

a vPC from the FEX southbound to the end server, sometimes called a Host vPC,

the second is a vPC from the FEX northbound to the parent switches, sometimes

called a Fabric vPC, and the third is both a southbound and northbound vPC from

the FEX, which is considered an Enhanced vPC, or EvPC. Note that EvPC is only

supported on newer hardware platforms with corresponding newer software

releases. This particular configuration is considered the second variation, the Fabric

vPC.

In the Fabric vPC, the end host may be single or dual attached to one or more

FEXes, but the FEX does not perform port channeling southbound to the end host.

Instead, the FEX forms a vPC northbound to multiple parent switches. Although this

does not contribute to any load distribution between the end server and the FEX, it

does more evenly distribute the load from the FEXes northbound to their parents.

The potential danger with this design, however, is that multiple parent switches,

each with separate management and control planes, are referencing the same FEX

host ports. This means that if the configuration becomes out of sync between the

parent switches, there could be a problem in the data plane of the FEX host ports. A

possible resolution to this problem is to use the Configuration Synchronization

feature, which is demonstrated in a separate scenario.

The configuration of this scenario is similar to the other FEX pairings, with the

exception that the port channel southbound from the parent switch to the FEX does

not run LACP. This is because the FEX uplink ports do not support LACP: they only

support static channels. When complete, both parent switches must agree on


23/313

identical configurations down to the FEX Fabric Ports and to the FEX Host Ports.

The consistency of the parent switches configurations is protected against using the

vPC consistency check. Note that the show vpc output below indicates that each of

the FEX Host Ports now participates in the vPC, even though there is not

channeling configured on the host ports. Like other vPC configurations, the first

verification should be that the vPC Peer Keepalive is up and the vPC Peer Link

adjacency has been formed.

N5K1# show vpc

Legend:







vPC role : primary Number of vPCs configured : 66





---------------------------------------------------------------------


-- ---- ------ --------------------------------------------------1 Po1 up

1,10

vPC status

----------------------------------------------------------------------------


------ ----------- ------ ----------- -------------------------- ----------- 101 Po101 up

success success - 102 Po102 up

success success -

102400 Eth101/1/1 up success success 10


102402 Eth101/1/3 down* Not Consistency Check Not -

Applicable Performed






24/313
















































25/313













































26/313




























N5K2# show vpc

Legend:







vPC role : secondary Number of vPCs configured : 66





---------------------------------------------------------------------


-- ---- ------ --------------------------------------------------


27/313

1 Po1 up

1,10

vPC status

----------------------------------------------------------------------------


------ ----------- ------ ----------- -------------------------- ----------- 101 Po101 up

success success - 102 Po102 up

success success -




































28/313














































29/313















































30/313


Note that both N5K1 and N5K2 are pairing with the same downstream FEXes.




---------------------------------------------------------------

101 Eth1/10 Active 1 N2K-C2232PP-10GE SSI16330GT8

102 Eth1/11 Active 1 N2K-C2232PP-10GE SSI15030C1R




---------------------------------------------------------------

101 Eth1/10 Active 2 N2K-C2232PP-10GE SSI16330GT8

102 Eth1/11 Active 2 N2K-C2232PP-10GE SSI15030C1R

From the end servers perspective, their links have been configured in

active/standby failover teaming, with the primary path being to N2K2.

For final verification of traffic distribution, Server 1 and Server 2 generate TCP flows

toward the CNA server. The end result is multiple flows for a total nearing 2Gbps,

which is the combined line rates of Server 1 and 2.


31/313

From the network side, both N5K1 and N5K2 see that all flows towards the CNA

server exit via N2K2.




, 1 pps





, 160.63 Kpps




, 1 pps





, 160.63 Kpps

Note that these outputs are nearly identical on both parent switches, as they are

both referencing the same physical FEX host ports. The key difference with this

configuration design, however, is that traffic is load balanced from the parent

switches down to the N2K2 FEX. This can be verified by viewing the counters of the

FEX Fabric Ports of the parent switches, as seen below.

N5K1# show interface e1/10 - 11 | include rate|Ethernet1

Ethernet1/10 is up



32/313

30 seconds output rate 3184 bits/sec, 2 packets/sec input rate 18.55 Kbps, 2 pps;

output rate 2.94 Kbps

, 2 pps

Ethernet1/11 is up




, 80.56 Kpps


Ethernet1/10 is up



output rate 2.84 Kbps

, 2 pps

Ethernet1/11 is up




, 80.64 Kpps

Note that neither parent switch is sending traffic to FEX Fabric Port E1/10 because

this is the link to N2K1, which the CNA server is using as the standby connection.

Although the total of flows from Server 1 and 2 are 2Gbps, they are nearly equally

split between the FEX Fabric Ports going from N5K1 and N5K2 southbound to the

N2K2 FEX.

In the case of a failure of the servers primary uplink, traffic is automatically re-routed to the backup link via FEX N2K1, as shown below.


Ethernet1/10 is up



33/313

30 seconds output rate 987383384 bits/sec

, 80800 packets/sec

input rate 17.62 Kbps, 2 pps; output rate 518.97 Mbps, 42.43 Kpps

Ethernet1/11 is up

30 seconds input rate 8624 bits/sec, 1 packets/sec 30 seconds output rate 3128 bits/sec

, 2 packets/sec

input rate 16.26 Kbps, 2 pps; output rate 463.30 Mbps, 37.85 Kpps


Ethernet1/10 is up


, 81015 packets/sec

input rate 24.29 Mbps, 40.97 Kpps; output rate 478.05 Mbps, 39.09 Kpps

Ethernet1/11 is up


, 2 packets/sec

input rate 25.69 Mbps, 43.28 Kpps; output rate 505.58 Mbps, 41.32 Kpps


34/313


FEX and N5K Config Sync

Task

Enable the vPC, FEX, and LACP features on N5K1 and N5K2.

Enable Cisco Fabric Services over IP (CFSoIP) distribution between N5K1 and N5K2.

Configure vPC domain 1 between N5K1 and N5K2, and use the mgmt0 link for the

vPC Peer Keepalive.Create a Config Sync session on both N5K1 and N5K2, and use the switch profile

name N5K.

Use the mgmt0 IP addresses as the config sync peers destination.

Verify that N5K1 and N5K2 can reach each other over CFSoIP for the config sync

session.

Without making any additional changes on N5K2, use the switch profile on N5K1 to

replicate the following configuration to both switches:

Pre-provision FEX modules 101 and 102, both of type N2K-C2232P.Create VLAN 10.

All links to Server 1 and Server 2 should be access ports in VLAN 10.

Configure all links between the vPC peers as Port-Channel 1, and use this

as the vPC Peer Link.







Configure the links to the Emulex CNA Server in VLAN 10.

Commit the config and verify that both N5K1 and N5K2 identically accept it into their

running configuration.


35/313

Configuration

N5K2:

feature vpc

feature fex

feature lacp

cfs ipv4 distribute

!

vpc domain 1

peer-keepalive destination 192.168.0.51 vrf management

!

end

config sync

switch-profile N5K

sync-peers destination 192.168.0.51

verify

N5K1:

feature vpc

feature fex

feature lacp

cfs ipv4 distribute

!

vpc domain 1

peer-keepalive destination 192.168.0.52 vrf management

!

end

config sync

switch-profile N5K

sync-peers destination 192.168.0.52

verify

show switch-profile status

slot 101

provision model N2K-C2232P

!

slot 102

provision model N2K-C2232P

!

vlan 10

!



36/313



vpc peer-link

!



fex associate 101

vpc 101

!



fex associate 102

vpc 102

!




speed 1000

!





!



fex associate 101


!



fex associate 102


!



!



!

commit


37/313

Verification

Configuration Synchronization, also known as Config Sync for short or Switch

Profiles, is a way to apply a template of configuration onto multiple Nexus switches

at the same time. This feature is especially useful between vPC peers, or when FEX

deployments are used in active/active, where a downstream FEX peers with morethan one upstream parent switch. This feature helps to ensure that configurations

stay consistent between the vPC peers or FEX parents (or both as in the case

shown below), and avoid problems such as vPC failure caused by a consistency

check error.

Note that of current releases, this feature is not supported on the Nexus 7K.

Additionally, not all commands are supported in the switch profile mode, and instead

must be configured in regular global config. In this particular case, the unsupported

commands are the feature enablement of vPC, FEX, and LACP, as well as the vPC

domain creation, as the configuration is different between the vPC peers(specifically the peer keepalive destination address).

To use config sync, the switches must first be configured to use Cisco Fabric

Services over IP (CFSoIP), as this is the control plane protocol that is actually used

to sync the config between the switches. This is enabled simply as follows:

N5K1# conf t

Enter configuration commands, one per line. End with CNTL/Z.N5K1(config)# cfs ipv4 distribute

N5K1(config)# show cfs peers

Physical Fabric

-------------------------------------------------------------------------

Switch WWN IP Address

------------------------------------------------------------------------- 20:00:00:0d:ec:a2:ed:80

192.168.0.51

[Local] N5K1

20:00:00:0d:ec:a4:74:00 192.168.0.52

Total number of entries = 2

Next, start a config sync session, create an identically named switch profile on each

switch, specify the IP address of the peer to sync with, and then verify their

connectivity.

N5K1# config sync


38/313

Enter configuration commands, one per line. End with CNTL/Z. N5K1(config-sync)#switch-profile N5K

Switch-Profile started, Profile ID is 1 N5K1(config-sync-sp)# sync-peers destination 192.168.0.52

N5K1(config-sync-sp)# verify

Verification Successful

N5K1(config-sync-sp)#

vN5K2# config sync

Enter configuration commands, one per line. End with CNTL/Z.N5K2(config-sync)#switch-profile N5K

Switch-Profile started, Profile ID is 1 N5K2(config-sync-sp)# sync-peers destination 192.168.0.51

N5K2(config-sync-sp)# verify

Verification Successful


If the switch profile is in sync between the peers, they should both agree on the

profile revision number and show the sync status as in sync.

N5K1(config-sync-sp)# show switch-profile status

switch-profile : N5K

----------------------------------------------------------

Start-time: 382018 usecs after Sun Mar 3 15:44:48 2013

End-time: 441035 usecs after Sun Mar 3 15:44:50 2013

Profile-Revision: 1

Session-type: Initial-Exchange

Session-subtype: Init-Exchange-All

Peer-triggered: Yes

Profile-status: Sync Success

Local information:

---------------- Status: Commit Success

Error(s):

Peer information:

---------------- IP-address: 192.168.0.52

Sync-status: In sync

Status: Commit Success

Error(s):



----------------------------------------------------------




39/313

Profile-Revision: 1

Session-type: Initial-Exchange

Session-subtype: Init-Exchange-All

Peer-triggered: No

Profile-status: Sync Success

Local information:


Error(s):

Peer information:

---------------- IP-address: 192.168.0.51

Sync-status: In sync

Status: Commit Success

Error(s):

Now the switches are ready to accept the configuration changes to synchronize.

Commands are entered just like in global config, but they are not immediately

applied. Instead they are sent to the switch profile buffer, as shown below. Before

the buffer is committed, the buffer can be deleted or modified as desired. The line

numbers of the buffer show how the config will sequentially be applied. Therefore,

any configurations that are sensitive to order of operations must have the correct

line numbering in the buffer before a commit is executed.

N5K1(config-sync-sp)# slot 101

N5K1(config-sync-sp-slot)# provision model N2K-C2232P

N5K1(config-sync-sp-slot)# !

N5K1(config-sync-sp-slot)# slot 102

N5K1(config-sync-sp-slot)# provision model N2K-C2232P

N5K1(config-sync-sp-slot)# !

N5K1(config-sync-sp-slot)# vlan 10

N5K1(config-sync-sp-vlan)# !

N5K1(config-sync-sp-vlan)# interface port-channel1

N5K1(config-sync-sp-if)# switchport mode trunk

N5K1(config-sync-sp-if)# spanning-tree port type network

N5K1(config-sync-sp-if)# vpc peer-link

N5K1(config-sync-sp-if)# !

N5K1(config-sync-sp-if)# interface port-channel101

N5K1(config-sync-sp-if)# switchport mode fex-fabric

N5K1(config-sync-sp-if)# fex associate 101

N5K1(config-sync-sp-if)# vpc 101


N5K1(config-sync-sp-if)# interface port-channel102


40/313



N5K1(config-sync-sp-if)# vpc 102


N5K1(config-sync-sp-if)# interface Ethernet1/1 - 2

N5K1(config-sync-sp-if-range)# switchport access vlan 10

N5K1(config-sync-sp-if-range)# spanning-tree port type edge

N5K1(config-sync-sp-if-range)# speed 1000

N5K1(config-sync-sp-if-range)# !

N5K1(config-sync-sp-if-range)# interface Ethernet1/3 - 5

N5K1(config-sync-sp-if-range)# switchport mode trunk

N5K1(config-sync-sp-if-range)# spanning-tree port type network

N5K1(config-sync-sp-if-range)# channel-group 1 mode active

N5K1(config-sync-sp-if-range)# !

N5K1(config-sync-sp-if-range)# interface Ethernet1/10



N5K1(config-sync-sp-if)# channel-group 101 mode on


N5K1(config-sync-sp-if)# interface Ethernet1/11



N5K1(config-sync-sp-if)# channel-group 102 mode on


N5K1(config-sync-sp-if)# interface Ethernet101/1/1

N5K1(config-sync-sp-if)# switchport access vlan 10


N5K1(config-sync-sp-if)# interface Ethernet102/1/1

N5K1(config-sync-sp-if)# switchport access vlan 10

N5K1(config-sync-sp-if)#

N5K1(config-sync-sp-if)# show switch-profile buffer


----------------------------------------------------------

Seq-no Command

----------------------------------------------------------

1 slot 101

1.1 provision model N2K-C2232P

2 slot 102

2.1 provision model N2K-C2232P

3 vlan 10

4 interface port-channel1

4.1 switchport mode trunk

4.2 spanning-tree port type network

4.3 vpc peer-link


41/313


5.1 switchport mode fex-fabric

5.2 fex associate 101

5.3 vpc 101




6.3 vpc 102

7 interface Ethernet1/1-2

7.1 switchport access vlan 10

7.2 spanning-tree port type edge

7.3 speed 1000

8 interface Ethernet1/3-5

8.1 switchport mode trunk

8.2 spanning-tree port type network

8.3 channel-group 1 mode active

9 interface Ethernet1/10



9.3 channel-group 101 mode on

10 interface Ethernet1/11



10.3 channel-group 102 mode on

11 interface Ethernet101/1/1


12 interface Ethernet102/1/1


When the commands in the buffer are acceptable, the profile is committed. During

the commit procedure, the config is synchronized across to the other peer using

CFSoIP, and applied sequentially. If there is an error in applying the config, all

commands in the buffer are rolled back and the commit fails. In other words, either

the commit succeeds 100 percent, or no config is applied to either peer. In the

output below, we see that the commit was successful, and syslog messages begin

to appear as config changes, link up/down events, etc. occur just as if you hadapplied the commands manually on each switch individually.


42/313

N5K1(config-sync-sp-if)# commit

Verification successful...

Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.

Please avoid other configuration changes during this time.

2013 Mar 3 15:49:31 N5K1 %ETH_PORT_CHANNEL-5-CREATED: port-channel1 created

2013 Mar 3 15:49:31 N5K1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No



2013 Mar 3 15:49:33 N5K1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel101 is down (N

2013 Mar 3 15:49:33 N5K1 last message repeated 2 times


Commit Successful

N5K1(config-sync)#






2013 Mar 3 16:40:24 N5K2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel101 is down (N

2013 Mar 3 16:40:24 N5K2 last message repeated 2 times


When the commit is successful, N5K1 automatically exits out of the switch profile

configuration mode. If additional config changes are required, a new switch profile

session must be started, using the same session name as before. Note in the output

below that both switches agree on the switch profile revision number, and the

switches are in sync.

N5K1(config-sync)# show switch-profile status


----------------------------------------------------------



Profile-Revision: 2

Session-type: Commit

Session-subtype: -

Peer-triggered: No Profile-status: Sync Success


43/313

Sync-status: In sync Status: Commit Success

Error(s):



----------------------------------------------------------



Profile-Revision: 2

Session-type: Commit

Session-subtype: -

Peer-triggered: Yes Profile-status: Sync Success

Local information:


Error(s):

Peer information:

----------------

IP-address: 192.168.0.51

Sync-status: In sync Status: Commit Success

Error(s):

From N5K2s perspective, the configuration commands appear in the running config

just as if they had been entered manually in global configuration.

N5K2# show run interface

!Command: show running-config interface

!Time: Sun Mar 3 17:12:09 2013

version 5.1(3)N1(1a)




vpc peer-link


44/313



fex associate 101

vpc 101



fex associate 102

vpc 102

Further verification shows that the configured features such as the vPC, FEX Fabric

Ports, VLANs, etc. are functional.

N5K2# show vpc

Legend:


vPC domain id : 1

Peer status : peer adjacency formed ok





vPC role : secondary

Number of vPCs configured : 66





---------------------------------------------------------------------


-- ---- ------ --------------------------------------------------

1 Po1 up 1,10

vPC status

----------------------------------------------------------------------------


------ ----------- ------ ----------- -------------------------- -----------

101 Po101 up success success -

102 Po102 up success success -



45/313















N5K2# show spanning-tree vlan 10

VLAN0010




Cost 1

Port 4096 (port-channel1)



Address 000d.eca4.743c



---------------- ---- --- --------- -------- --------------------------------

Po1 Root FWD 1 128.4096 (vPC peer-link) Network P2p



Eth101/1/1 Desg FWD 1 128.1153 (vPC) Edge P2p

Eth102/1/1 Desg FWD 1 128.1281 (vPC) Edge P2p


46/313

Nexus Technology Labs - Virtual PortChannels (vPC)

vPC and HSRP

Task

This task applies only to INE Bootcamp partipants. Load

balancing is not included in the self-paced training curriculum at

this time.

Configure vPC between N7K1 and N7K2 as follows:

N7K1 and N7K2 are the vPC Peers.

Configure all available F1 ports between the vPC peers as Port-Channel 1,

and use this as the vPC Peer Link.

Use the mgmt0 ports for the Peer Keepalive Link.

Configure all available links from N7K1 and N7K2 to N5K1 in Port-Channel

51, and as vPC 51.

All portchannels should be trunks, STP Network Ports, and use LACP fornegotiation.

Configure VLAN assignments and the servers as follows:

Configure the link from N5K1 to Server 1 as an access port in VLAN 10.

Configure the link from N5K1 to Server 2 as an access port in VLAN 20.

Server 1 should use the IP address 10.0.0.1/24, and a default gateway of

10.0.0.254.

Server 2 should use the IP address 20.0.0.2/24, and a default gateway of

20.0.0.254.

Configure Inter-VLAN Routing and HSRP on N7K1 and N7K2 as follows:

Create interfaces VLAN 10 and VLAN 20 on N7K1 and N7K2, using the IP

address 10.0.0.X/24, where X is the last octet of the IP address on their

mgmt0 interfaces.

Configure HSRP group 10 for VLAN 10 on N7K1 and N7K2 using the virtual

address 10.0.0.254/24.


47/313

Configure HSRP group 20 for VLAN 20 on N7K1 and N7K2 using the virtual

address 20.0.0.254/24.

Set the Port Channel load balancing method on the Nexus switches to include the

source and destination layer 4 port numbers.

When complete, Server 1 and Server 2 should have IP reachability to each other,

and traffic between them should be load distributed across all links in the vPC.

Configuration

N5K1:

feature lacp

!

vlan 10,20

!

port-channel load-balance ethernet source-dest-port

!


switchport mode access


speed 1000

!


switchport mode access


speed 1000

interface Ethernet1/6-9




no shutdown

!




N7K1-1:

feature vpc

feature lacp

!

vlan 10,20

!

port-channel load-balance src-dst ip-l4port-vlan

!


48/313

vpc domain 1


!





no shutdown

!


switchport



vpc peer-link

!





no shutdown

!


switchport



vpc 51

!

interface Vlan10

no shutdown

ip address 10.0.0.71/24

hsrp 10

ip 10.0.0.254

!

interface Vlan20

no shutdown

ip address 20.0.0.71/24

hsrp 20

ip 20.0.0.254

N7K2-1:

feature vpc

feature lacp

!

vlan 10,20

!

port-channel load-balance src-dst ip-l4port-vlan


49/313

!

vpc domain 1


!





no shutdown

!


switchport



vpc peer-link

!





no shutdown

!


switchport



vpc 51

!

interface Vlan10

no shutdown

ip address 10.0.0.75/24

hsrp 10

ip 10.0.0.254

!

interface Vlan20

no shutdown

ip address 20.0.0.75/24

hsrp 20

ip 20.0.0.254

Verification

This scenario demonstrates how the forwarding pattern of vPC and HSRP combined

differs from that of just HSRP on its own. The results of this scenario would be


50/313

similar if either VRRP or GLBP were used, because all of the First Hop Redundancy

Protocols (FHRPs) have special behavior that interacts with vPC.

First, the layer 2 only switch N5K1 has access ports in VLANs 10 and 20, and a

trunking port channel that carries both VLANs. From N5K1s perspective, this port

channel logically connects to just one upstream switch, but in reality it is the two

physical vPC Peers, N7K1 and N7K2.

N5K1# show spanning-tree vlan 10,20

VLAN0010



Address 68bd.abd7.6041

Cost 2







---------------- ---- --- --------- -------- -------------------------------- Po51

Root FWD 1 128.4146 Network P2p Eth1/1

Desg FWD 4 128.129 P2p

VLAN0020



Address 68bd.abd7.6041

Cost 2







---------------- ---- --- --------- -------- -------------------------------- Po51

Root FWD 1 128.4146 Network P2p Eth1/2

Desg FWD 4 128.130 P2p

According to normal layer 2 switching vs. layer 3 routing logic, any hosts in VLAN 10


51/313

that want to talk to hosts in VLAN 20 must have their traffic switched up to the

default gateway and have the layer 2 header re-written with a new source and

destination MAC address, then switched to the final destination. In this case, there

are two default gateways for each VLAN 10 and 20, both N7K1 and N7K2 that share

the HSRP virtual IP address. In the output below we can see that N7K1 is the active

HSRP router for both groups.

N7K1-1# show hsrp

Vlan10 - Group 10

(HSRP-V1) (IPv4) Local state is Active

, priority 100 (Cfged 100)

Forwarding threshold(for vPC), lower: 1 upper: 100

Hellotime 3 sec, holdtime 10 sec

Next hello sent in 1.595000 sec(s)

Virtual IP address is 10.0.0.254 (Cfged)

Active router is local

Standby router is 10.0.0.75 , priority 100 expires in 5.957000 sec(s)

Authentication text "cisco"

Virtual mac address is 0000.0c07.ac0a (Default MAC)

4 state changes, last state change 00:48:39

IP redundancy name is hsrp-Vlan10-10 (default)

Vlan20 - Group 20







Active router is local

Standby router is 20.0.0.75 , priority 100 expires in 6.264000 sec(s)


Virtual mac address is 0000.0c07.ac14 (Default MAC)



N7K2-1# show hsrp

Vlan10 - Group 10

(HSRP-V1) (IPv4) Local state is Standby






Active router is 10.0.0.71, priority 100 expires in 9.872000 sec(s)

Standby router is local


Virtual mac address is 0000.0c07.ac0a (Default MAC)


52/313



Vlan20 - Group 20







Active router is 20.0.0.71, priority 100 expires in 5.152000 sec(s)

Standby router is local


Virtual mac address is 0000.0c07.ac14 (Default MAC)



The potential problem with this design is that if traffic is switched to N7K2, thestandby HSRP router, because of the Port Channel load balancing method of N5K1,

it would have to be sent to the active HSRP router, N7K1, to be routed. This means

that traffic would have to transit the vPC Peer Link, which is undesirable because

the aggregate of flows from vPC Member Ports would quickly overwhelm the vPC

Peer Link. To prevent this from being necessary, vPC changes the behavior of the

FHRPs so that the standby router can forward the same as the active router. The

result of this can be seen below.

Server 2 generates bulk TCP flows to Server 1 using the iPerf app. The aggregate

of flows nears 1Gbps.

When the access switch, N5K1, receives these flows, they have the destination

MAC address of the virtual HSRP address. This MAC address is reachable via the


53/313

port channel to the upstream 7K, and is then load balanced based on the layer 4

port information of the flows as configured.

N5K1# show port-channel summary

Flags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)

s - Suspended r - Module-removed

S - Switched R - Routed

U - Up (port-channel)

M - Not in use. Min-links not met

--------------------------------------------------------------------------------

Group Port- Type Protocol Member Ports

Channel

--------------------------------------------------------------------------------

51 Po51(SU) Eth LACP Eth1/6(P) Eth1/7(P) Eth1/8(P)

Eth1/9(P)

N5K1# show interface e1/6 - 9 | include rate|Ethernet

Ethernet1/6 is up

Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed8d (bia 000d.eca2.ed8d)


30 seconds output rate 117990496 bits/sec, 13644 packets/sec

input rate 2.96 Mbps, 5.28 Kpps; output rate 111.59 Mbps

, 13.04 Kpps

Ethernet1/7 is up

Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed8e (bia 000d.eca2.ed8e)




, 13.19 Kpps

Ethernet1/8 is up

Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed8f (bia 000d.eca2.ed8f)




, 28.20 Kpps

Ethernet1/9 is up

Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed90 (bia 000d.eca2.ed90)




, 28.28 Kpps

In the output above, we see that some traffic goes from N5K1 to N7K1, and some


54/313

from N5K1 to N7K2. Without the vPC modification to HSRP, this traffic shouldhave

to be switched from N7K2 to N7K1 before it can be routed, because N7K2 isnt the

active HSRP router. However, the interface counters of the vPC Peer Link, as seen

below, indicate that the flows are not switched in that direction, and instead N7K2 is

routing them itself even though it is HSRP standby.

N7K2-1# show interface port-channel 1 | include rate


30 seconds output rate 1544 bits/sec, 1 packets/sec input rate 1.56 Kbps, 1 pps; output rate 1.54 Kbps

, 1 pps

This behavior can be further verified by disabling the uplinks from N5K1 to N7K1, as

shown below.

N5K1# show cdp neighbors

Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge

S - Switch, H - Host, I - IGMP, r - Repeater,

V - VoIP-Phone, D - Remotely-Managed-Device,

s - Supports-STP-Dispute

Device-ID Local Intrfce Hldtme Capability Platform Port ID

Nexus-MGMT-SW mgmt0 175 S I WS-C3550-48 Fas0/31

N5K2(FLC12480280) Eth1/3 140 S I s N5K-C5020P-BF Eth1/3



N7K1-1(JAF1510CMLQ) Eth1/6

170 R S s N7K-C7010 Eth2/3 N7K1-1(JAF1510CMLQ) Eth1/7

169 R S s N7K-C7010 Eth2/4

N7K2-1(TBM14311481) Eth1/8 167 R S s N7K-C7010 Eth2/5

N7K2-1(TBM14311481) Eth1/9 167 R S s N7K-C7010 Eth2/6

N5K1# config t

Enter configuration commands, one per line. End with CNTL/Z.N5K1(config)# int e1/6 7

N5K1(config-if-range)# shutdown

2013 Mar 5 21:54:43 N5K1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel51: Ethernet1/7 is down


2013 Mar 5 21:54:43 N5K1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel51: Ethernet1/6 is down


2013 Mar 5 21:54:43 N5K1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/7 is down (Administratively dow

2013 Mar 5 21:54:43 N5K1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/6 is down (Administratively dow

With the links from N5K1 to N7K1 disabled, the only way for them to be switched


55/313

northbound is via N7K2, as shown below.

vN5K1# show interface e1/6 - 9 | include Ethernet1|rate

Ethernet1/6 is down (Administratively down)



input rate 0 bps, 0 pps; output rate 0 bps, 0 pps

Ethernet1/7 is down (Administratively down)



input rate 0 bps, 0 pps; output rate 0 bps, 0 pps

Ethernet1/8 is up




, 42.86 Kpps

Ethernet1/9 is up




, 42.84 Kpps

Because N7K1 still has the vPC Peer Link forwarding VLANs 10 and 20, it is still the

HSRP Active router.

N7K1-1# show hsrp | include state|Group

Vlan10 - Group 10



4 state changes, last state change 02:02:24 Vlan20 - Group 20




N7K2-1# show hsrp | include state|Group

Vlan10 - Group 10



7 state changes, last state change 01:36:48 Vlan20 - Group 20




If N7K1 is the router that is doing the layer 2 header re-write, the vPC Peer Link


56/313

should show 1Gbps input and output, which it does not according to the output

below.

N7K2-1# show interface port-channel 1 | include rate


30 seconds output rate 1504 bits/sec, 1 packets/sec input rate 1.58 Kbps, 1 pps; output rate 1.50 Kbps

, 1 pps

Note that this behavior, in which both the active and standby HSRP routers are able

to forward traffic, is the default. There is no additional configuration needed to

accomplish this. As long as HSRP/VRRP/GLBP is configured in conjunction with

vPC, this behavior will be seen.


57/313

Nexus Technology Labs - Virtual PortChannels (vPC)

Back-to-Back vPC

Task

C

nexus technology labs

Documents