trm n55k l2only-config tshoot jdinkin2 2hr 20120208

8/10/2019 TRM N55K L2only-Config Tshoot Jdinkin2 2hr 20120208

http://slidepdf.com/reader/full/trm-n55k-l2only-config-tshoot-jdinkin2-2hr-20120208 1/350

© 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 1

Cisco Advanced ServicesCisco Nexus 5500 Series

Configuration andTroubleshootingKnowledge Transfer

Instructor: Joel Dinkin ( [email protected])

Cisco Advanced Services Network Consulting Engineer

mailto:[email protected]

mailto:[email protected]



2© 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential

Agenda

Nexus 5500 Series Hardware and Architecture

Device Management

In-Service Software Upgrade (ISSU)

Layer 2 Switching

Virtual Port Channel (vPC)

Multicast

Quality of Service (QoS)

Troubleshooting




Nexus 5500 SeriesHardware and Architecture




Nexus 5500 Hardware

Nexus 5548UP32 Fixed Ports 1/10G Ethernet or 1/2/4/8 FC

Line-rate, Non-blocking 10G FCoE/IEEE DCB

1 Expansion Module Slot

IEEE 1588, FabricPath & Layer 3 Capable

Redundant Fans & Power Supplies

Nexus 5596UP

48 Fixed Ports 1/10G Ethernet or 1/2/4/8 FC

Line-rate, Non-blocking 10G FCoE/IEEE DCB

3 Expansion Module Slot

IEEE 1588, FabricPath & Layer 3 Capable

Redundant Fans & Power Supplies




Power Entry Power EntryFan Module Fan Module

Out of Band Mgmt

10/100/1000

Console

Fabric Interconnect

Not Active on Nexus

N + N Redundant FANs N + N Power Supplies

USB Flash

32 x Fixed Unified Ports 1/10 GE or 1/2/4/8 FCExpansion Module

Nexus 5500 HardwareNexus 5548 (5548P & 5548UP)




Nexus 5500 Hardware Nexus 5596UP

Fan Module

Console

N + N Redundant FANsN + N Power Supplies

Out of Band Mgmt

10/100/1000

Fabric Interconnect

Not Active on Nexus USB Flash

48 x Fixed Unified Ports 1/10 GE or 1/2/4/8 FC

3 Expansion Modules

Power Supply Fan Module Fan Module Fan Module


http://slidepdf.com/reader/full/trm-n55k-l2only-config-tshoot-jdinkin2-2hr-20120208 7/3507© 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential

Nexus 5500 HardwareNexus 5500 Expansion Modules

16 x 1/10GE8 x 1/10GE +

8 x 1/2/4/8G FC

16 unified portsindividually

configurable as 1/10GE

or 1/2/4/8G FC

L3 module for160G of L3 I/O

bandwidth

Nexus 5500 expansion slots Expansion Modules are hot swappable (Future support

for L3 OIR)

Contain forwarding ASIC (UPC-2)



1G Support on all ports

Any Ethernet port or Flexible port in N55xx switches can beconfigured in 1G mode.

Requires the use of a standard 1G SFP

GLC-T, GLC-SX-MM, GLC-LH-SM, SFP-GE-T, SFP-GE-S, SFP-GE-L (DOM capable SFP are supported)

Supports for all features at 1G speed other than Unified I/O No FCoE (no 1G Converged Network Adapters are shipping)

No Priority Flow Control (standard Pause is available)

CLI to configure 1G

switch(config)# interface Ethernet1/1

switch(config-if)# speed 1000

5.0(3)N1(1)Required for

1Gbps Support!



Nexus 5548PNexus 5548UP

Nexus 5596UP

L3 HardwareList Price

$5,000

Nexus 5548PNexus 5548UP

Nexus 5596UP

Nexus 5500 Layer 3 Options



Nexus 5500 HardwareNexus 5500 Reversible Air Flow and DC Power Supplies

Nexus 5548UP and 5596UP will support reversibleairflow (new PS and fans)

Nexus 5548UP and 5596UP will support DC powersupplies (not concurrent with reversible airflow)

Note: 5548UP and 5596UP ONLY, not 5548P

Nexus 5500Hardware

Availability

Front-to-Back Airflow, ACPower

Nexus

5548P/5548UP/5596UPToday

Back-to-Front Airflow, ACPower

Nexus 5548UP/5596UPNexus 5548UP

Nexus 5596UP (Future)

Front-to-Back Airflow, DCPower

Nexus 5548UP/5596UPNexus 5548UP

Nexus 5596UP (Future)

Back-to-Front Airflow, DCPower

N/A N/A



Reverse Air Flow - CLI

CLI enhancements to display air flow direction.

switch# show environment fan detail

---------------------------------------------------

Module Fan Airflow Speed(%) Speed(RPM)

Direction

---------------------------------------------------

1 1 Front-to-Back 40 6733










ç

Nexus 5500 InternalsData and Control Plane Elements

Gen 2 UPC

Unified Crossbar FabricGen 2

Gen 2 UPC Gen 2 UPC

Gen 2 UPC Gen 2 UPC

PEX 85254 port PCIE

Switch

SouthBridge

10 Gig

12 Gig

Mgmt 0

Console

L1

L2

PCIe x4

PCIe x8

PCIEDual Gig

0 1

CPU IntelJasperForest

. . .PCIE

Dual Gig

0 1

PCIEDual Gig

0 1

Serial

Flash

Memory

NVRAM

DRAM

DDR3

Expansion Module



Nexus 5500 Hardware OverviewData Plane Elements - Unified Port Controller (Gen 2)

Each UPC supports eight ports andcontains,

Multimode Media access controllers(MAC)

Support 1/10 G Ethernet and 1/2/4/8 G

Fibre Channel All MAC/PHY functions supported on the

UPC (5548UP and 5596UP)

Packet buffering and queuing

640 KB of buffering per port

Forwarding controller

Ethernet (Layer 2 and FabricPath) andFibre Channel Forwarding and Policy(L2/L3/L4 + all FC zoning)

Unified PortController 2

M M

A C

+ B u f f e r +

F o r w a r d i n g

M M

A C

+ B u f f e r +

F o r w a r d i n g

M M

A C

+ B u f f e r +

F o r w a r d i n g

M M

A C

+ B u f f e r +

F o r w a r d i n g

M M

A C

+ B u f f e r +

F o r w a r d i n g

M M

A C

+ B u f f e r +

F o r w a r d i n g

M M

A C

+ B u f f e r +

F o r w a r d i n g

M M

A C

+ B u f f e r +

F o r w a r d i n g



Nexus 5500 Hardware OverviewControl Plane Elements – Nexus 5500

CPU - 1.7 GHz Intel Jasper Forest (Dual Core) DRAM - 8 GB of DDR3 in two DIMM slots

Program Store - 2 GB of eUSB flash for basesystem storage and partitioned to store image,configuration, log.

Boot/BIOS Flash - 8 MB to store upgradable

and golden version of (Bios + bootloader)image

On-Board Fault Log (OBFL) - 64 MB of flash tostore hardware related fault and reset reason

NVRAM - 6 MB of SRAM to store Syslog andlicensing information

Management Interfaces

RS-232 console port: console0

10/100/1000BASE-T: mgmt0 partitionedfrom inbound-hi VLANs

PEX 85254 port PCIE

Switch

SouthBridge

PCIe x4

PCIe x8

CPU IntelJasperForest

Serial

Flash

Memory

NVRAM

DRAM

DDR3

PCIE

Dual Gig

0 1

PCIE

Dual Gig

0 1

PCIE

Dual Gig

0 1

inbound-hi Data Pathto CPU

Mgmt 0

Console



Nexus 5500 Hardware OverviewControl Plane Elements - CoPP

In-band traffic is identified by theUPC and punted to the CPU via twodedicated UPC interfaces, 5/0 and5/1, which are in turn connected toeth3 and eth4 interfaces in the CPUcomplex

Eth3 handles Rx and Tx of low priority control pkts

IGMP, CDP, TCP/UDP/IP/ARP (formanagement purpose only)

Eth4 handles Rx and Tx of high

priority control pktsSTP, LACP, DCBX, FC and FCoEcontrol frames (FC packets come toSwitch CPU as FCoE packets)

B P D U

I C M P

S D P

PEX 85254 port PCIE

Switch

CPU IntelJasper

Forest

NIC0 1




CPU queuing structure provides strict

protection and prioritization of inbound traffic

Each of the two in-band ports has 8 queuesand traffic is scheduled for those queuesbased on control plane priority (traffic CoSvalue)

Prioritization of traffic between queues on

each in-band interface CLASS 7 is configured for strict priority

scheduling (e.g. BPDU)

CLASS 6 is configured for DRR schedulingwith 50% weight

Default classes (0 to 5) are configured for DRR

scheduling with 10% weight Additionally each of the two in-band

interfaces has a priority service order fromthe CPU

Eth 4 interface has high priority to servicepackets (no interrupt moderation)

Eth3 interface has low priority (interruptmoderation)

B P D U

I C M P

S D P

PEX 85254 port PCIE

Switch

CPU IntelJasper

Forest

NIC0 1




On Nexus 5500 an additional level of controlinvoked via policers on UPC-2

Software programs a number of egresspolicers on the UPC-2 to avoid overwhelmingthe CPU (partial list)

STP: 20 Mbps

LACP: 1 Mbps

DCX: 2 Mbps

Satellite Discovery protocol: 2 Mbps

IGMP: 1 Mbps

DHCP: 1 Mbps

. . .

CLI exposed to tune CoPP (Future) B P D U

I C M P

S D P

PEX 85254 port PCIE

Switch

CPU IntelJasper

Forest

NIC0 1

EgressPoliciers



Nexus 5500 Hardware OverviewControl Plane Elements

Monitoring of in-band traffic via theNX-OS built-in ethanalyzer

Eth3 is equivalent to ‘inbound-lo’

Eth4 is equivalent to ‘inbound-hi’

dc11-5548-3# ethanalyzer local sniff-interface ?inbound-hi Inbound(high priority) interfaceinbound-low Inbound(low priority) interface

mgmt Management interface

dc11-5548-4# sh hardware internal cpu-mac inbound-hi counterseth3 Link encap:Ethernet HWaddr 00:0D:EC:B2:0C:83

UP BROADCAST RUNNING PROMISC ALLMULTI MULTICAST MTU:2200 Metric:1RX packets:3 errors:0 dropped:0 overruns:0 frame:0TX packets:630 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000RX bytes:252 (252.0 b) TX bytes:213773 (208.7 KiB)Base address:0x6020 Memory:fa4a0000-fa4c0000

eth4 Link encap:Ethernet HWaddr 00:0D:EC:B2:0C:84UP BROADCAST RUNNING PROMISC ALLMULTI MULTICAST MTU:2200 Metric:1RX packets:85379 errors:0 dropped:0 overruns:0 frame:0TX packets:92039 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:1000RX bytes:33960760 (32.3 MiB) TX bytes:25825826 (24.6 MiB)

Base address:0x6000 Memory:fa440000-fa460000

CLI view of in-band control plane data

PEX 85254 port PCIE

Switch

NX-OSEtheranalyzer

Process


NIC

0 1

NIC

0 1

Mgmt 0



UPC #7UPC #0

Nexus 5500 Hardware OverviewNexus 5500 – UPC (Gen 2) and Port Mapping

UPC-2 interfaces are indirectlymapped to front panel ports

Mapping of ports to UPC-2 ASIC

The left column identifies the Ethernetinterface identifier, xgb1/8 = e1/8

Column three and four reflect the UPCport that is associated with thephysical Ethernet port

nexus-5548# show hardware internal carmel all-ports

Carmel Port Info:

name |log|car|mac|flag|adm|opr|m:s:l|ipt|fab|xcar|xpt|if_index|diag|ucVer-------+---+---+---+----+---+---+-----+---+---+----+---+--------+----+-----xgb1/2 |1 |0 |0 -|b7 |dis|dn |0:0:f|0 |92 |0 |0 |1a001000|pass| 4.0bxgb1/1 |0 |0 |1 -|b7 |dis|dn |1:1:f|1 |88 |0 |0 |1a000000|pass| 4.0bxgb1/4 |3 |0 |2 -|b7 |dis|dn |2:2:f|2 |93 |0 |0 |1a003000|pass| 4.0bxgb1/3 |2 |0 |3 -|b7 |dis|dn |3:3:f|3 |89 |0 |0 |1a002000|pass| 4.0bxgb1/6 |5 |0 |4 -|b7 |dis|dn |4:4:f|4 |90 |0 |0 |1a005000|pass| 4.0bxgb1/5 |4 |0 |5 -|b7 |dis|dn |5:5:f|5 |94 |0 |0 |1a004000|pass| 4.0bxgb1/8 |7 |0 |6 -|b7 |dis|dn |6:6:f|6 |95 |0 |0 |1a007000|pass| 4.0b

<snip>sup0 |32 |4 |4 -|b7 |en |dn |4:4:0|4 |62 |0 |0 |15020000|pass| 0.00

sup1 |33 |4 |5 -|b7 |en |dn |5:5:1|5 |59 |0 |0 |15010000|pass| 0.00

. . .

1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8

0 1 2 3 4 5 6 70

. . .



All versions of 5500 support 1/10G on all ports

5548UP, 5596UP and N55-M16UP (Expansion Module) supportUnified Port capability on all ports

1G Ethernet Copper/Fibre

10G DCB/FCoE Copper/Fibre

1/2/4/8G Fibre Channel

Nexus 5500 Hardware Overview5548UP/5596UP – UPC (Gen-2) and Unified Ports


SFP+Cage

EthernetPHY

SFP+Cage

5548UP, 5596UP& N55-M16UP

5548PUnified PortController 2

PHY removed, all MACand PHY functions

performed on UPC-21/10G Ethernet ‘and’

1/2/4/8G FC capable on allports

Ethernet PHY1/10G on all ports



Nexus 5500 Hardware Overview5548UP/5596UP – UPC (Gen-2) and Unified Ports

Eth Ports

Eth Ports Eth Eth

FC Ports

FC FC

Slot 1

Slot 2 GEM Slot 3 GEM Slot 4 GEM

With the 5.0(3)N1 and later releases each module can define anynumber of ports as Fibre Channel (1/2/4/8 G) or Ethernet (either 1G or10G)

Initial SW releases supports only a continuous set of portsconfigured as Ethernet or FC within each ‘slot’

Eth ports have to be the first set and they have to be one contiguousrange

FC ports have to be second set and they have to be contiguous as well

Future SW release will support per port dynamic configuration

n5k(config)# slot <slot-num>n5k(config-slot)# port <port-range> type <fc | ethernet>



Nexus 5500Station (MAC) Table allocation

Nexus 5500 has a 32K Station table entries

4k reserved for multicast (Multicast MAC addresses)

3k assumed for hashing conflicts (very conservative)

25k effective Layer 2 unicast MAC address entries

N e x u s 5 5 0 0

U

P C

S t a t i o n

T a b l e

3 2 k

e n t r i e s

4k entries forIGMP

3k entries for potential hash collision space

25k effective MAC entries for unicast



Nexus 5500 Packet ForwardingPacket Forwarding—Cut Thru Switching

Unified CrossbarFabric

Packet Header Re-Write, MAC Learning

and then serialized out egress port

Egress Queue is

only used ifPause Frame

Received whilepacket in-flight

Packet Header

is serialized intoUPC

Forwarding

Forwarding

Nexus 5500 utilizes a CutThru architecture whenpossible

Bits are serialized in from theingress port until enough ofthe packet header has beenreceived to perform a

forwarding and policy lookup Once a lookup decision has

been made and the fabric hasgranted access to the egressport bits are forwardedthrough the fabric

Egress port performs anyheader rewrite (e.g. CoSmarking) and MAC beginsserialization of bits out theegress port

Packet is serializedacross Fabric onceforwarding decision

is made




Nexus 5500 Packet ForwardingPacket Forwarding—Cut-Through Switching

Ingress10G

Nexus 5500 utilizes both cut-through and store and forward switching

Cut-through switching can only be performed when packets are beingsent out as fast as they are received over the fabric

1G to 1G always does store and forward because the fabric is runningat 10Gig

The fabric is designed to forward 10G packets in cut-through whichrequires that 1G to 1G switching is store and forward mode

Egress10G

0 1 2 3


Ingress10G

Egress1G


Ingress1G

Egress10G


Ingress1G

Egress1G


Cut-Through

Mode

Cut-Through

Mode

Store and Forward

Mode

Store and Forward

Mode

Direction ofPacket Flow




Nexus 5500 Packet ForwardingForwarding Mode Behavior (Cut-Through or Store and Forward)

Source Interface Destination Interface Switching Mode

10 GigabitEthernet 10 GigabitEthernet Cut-Through

10 GigabitEthernet 1 GigabitEthernet Cut-Through

1 GigabitEthernet 1 GigabitEthernet Store-and-Forward

1 GigabitEthernet 10 GigabitEthernet Store-and-Forward

FCoE Fibre Channel Cut-Through

Fibre Channel FCoE Store-and-Forward

Fibre Channel Fibre Channel Store-and-Forward

FCoE FCoE Cut-Through

For YourReference




Nexus 5500 Packet ForwardingPacket Forwarding - Cut Through Switching

In Cut-Through switching frames are notdropped due to bad CRC

Nexus 5500 implements a CRC ‘stomp’mechanism to identify frames that have beendetected with a bad CRC upstream

A packet with a bad CRC is “stomped”, by

replacing the “bad” CRC with the original CRCexclusive-OR’d with the STOMP value( a 1’s inverse operation on the CRC)

In Cut Through switching frames with invalidMTU (frames with a larger MTU than allowed)

are not dropped Frames with a “> MTU” length are truncated

and have a stomped CRC included inthe frame

Bad Fibre

Corrupt Framewith original

CRC

Ingress

UPC

EgressUPC


Corrupt Framewith “Stomped

CRC”

CorruptFrame with“Stomped

CRC”




Nexus 5500 Packet ForwardingPacket Forwarding—Cut Through Switching

Corrupt or Jumbo frames arriving inbound will

count against the Rx Jumbo or CRC counters

Corrupt or Jumbo frames exiting will be identifiedvia the Tx output error and Jumbo counters

dc11-5548-4# sh int eth 2/4 <snip>TX

112 unicast packets 349327 multicast packets 56083 broadcast packets405553 output packets 53600658 bytes31 jumbo packets31 output errors 0 collision 0 deferred 0 late collision0 lost carrier 0 no carrier 0 babble0 Tx pause

0 1

dc11-5548-4# sh int eth 1/39 <snip>RX

576 unicast packets 4813153 multicast packets 55273 broadcast packets4869002 input packets 313150983 bytes31 jumbo packets 0 storm suppression packets0 runts 0 giants 0 CRC 0 no buffer0 input error 0 short frame 0 overrun 0 underrun 0 ignored0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop0 input with dribble 0 input discard0 Rx pause

Ingress

UPC

EgressUPC


Eth1/39

Eth2/4




Nexus 5500 Packet ForwardingPacket Forwarding—Cut Thru Switching

CRC and ‘stomped’ frames are tracked internally

between ASIC’s within the switch as well as on theinterface to determine internal HW errors areoccurring

dc11-5548-4# show hardware internal carmel asic 2 counters interrupt <snip>Carmel 2 interrupt statistics:Interrupt name |Count |ThresRch|ThresCnt|Ivls

-----------------------------------------------+--------+--------+--------+---- <snip>car_bm_port0_INT_err_ig_mtu_vio |1f |0 |1f

<snip>

dc11-5548-4# show hardware internal carmel asic 13 counters interrupt <snip>Carmel 13 interrupt statistics:Interrupt name |Count |ThresRch|ThresCnt|Ivls-----------------------------------------------+--------+--------+--------+----

<snip>car_fw2_INT_eg_pkt_err_cb_bm_eof_err |1f |0 |1 |0car_fw2_INT_eg_pkt_err_eth_crc_stomp |1f |0 |1 |0car_fw2_INT_eg_pkt_err_ip_pyld_len_err |1f |0 |1 |0car_mm2_INT_rlp_tx_pkt_crc_err |1f |0 |1 |0

<snip>

Ingress

UPC

EgressUPC





Nexus 5500 Packet ForwardingPacket Forwarding—Ingress Queuing

In typical Data Center accessdesigns, multiple ingressaccess ports transmit to a fewuplink ports

Nexus 5500 utilizes anIngress Queuing architecture

Packets are stored in ingressbuffers until egress port isfree to transmit

Ingress queuing provides anadditive effective

The total queue size availableis equal to [numb er of ingress

por ts x q ueue depth per por t ]

Statistically ingress queuingprovides the sameadvantages as shared buffermemory architectures

Egress Queue0 is full, linkcongested

Traffic is Queued on all ingress interfacebuffers providing a cumulative scaling of

buffers for congested ports

v




Nexus 5500 Packet ForwardingPacket Forwarding—Virtual Output Queues


EgressQueue 0

is full

EgressQueue 0is free

PacketsQueued for

Eth 1/20

Eth 1/20

VoQ Eth1/20

VoQ Eth1/8

Eth 1/8

Packet is able tobe sent to the

fabric for Eth 1/8

Nexus 5500 use an 8 Queue QoS

model for unicast traffic

Traffic is Queued on the Ingressbuffer until the egress port is free totransmit the packet

To prevent Head of Line Blocking(HOLB) Nexus 5500 use a Virtual

Output Queue (VoQ) Model Each ingress port has a unique set

of 8 virtual output queues for everyegress port (1024 Ingress VOQs =128 destinations * 8 classes onevery ingress port)

If Queue 0 is congested for any porttraffic then Queue 0 in all the otherports is still able to be transmitted

Common shared buffer on ingress,VoQ are pointer lists and notphysical buffers




Nexus 5500 SeriesVRF-Lite Support

Prior to 5.0(3)N1(1) , N5k support two VRFs

VRF management & VRF default

With 5.0(3)N1(1) user can create additional VRFs

VRF-lite,

VRF aware Unicast -BGP/OSPF/RIP

VRF Aware Multicast

Hardware supports 1K VRF

Current Solution testing limit – 64 VRF’s

Similar to N7K ‘if’ user data ports are used as

keepalive link, it is now recommended to creatededicate VRF for keepalive link

interface Vlan123vrf member vpc_keepaliveip address 123.1.1.2/30no shutdown

vpc domain 1 peer-keepalive destination 123.1.1.1 source 123.1.1.2 vrf vpc_keepalive

vPC Keepalive – Dedicated VRF ifusing data ports rather than mgmt

port for keepalive



© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 33

Nexus 5000 & 5500Reference

Product Features & Specs Nexus5010 Nexus5020 Nexus5548P Nexus5548UP Nexus5596UP

Switch Fabric Throughput 520Gbps 1.04Tbps 960Gbps 960Gbps 1.92Tbps

Switch Footprint 1RU 2RU 1RU 1RU 2RU

1 Gigabit Ethernet Port Density 8 16 48 48 96

10 Gigabit Ethernet Port Density 26 52 48 48 96

8G Native Fibre Channel Port Density 6 12 16

48

96

Port-to-Port Latency ~ 3.2us ~ 3.2us ~2.0us ~1.8us ~ 1.8us

No. of VLANs 512 512 4096 4096 4096

Layer 3 Capability ✔ ✔ ✔

1 Gigabit Ethernet FEX PortScalability (L2 mode) 576 576 1152

1152

1152

10 Gigabit Ethernet FEX Port

Scalability (L2 mode)

384 384 768

768

768

40 Gigabit Ethernet Capable ✔ ✔ ✔

Reversed Airflow ✔ ✔

For YourReference




Device Management




Cisco Nexus 5500Fundamentals

Config and Troubleshooting




FundamentalsImportant Cisco NX-OS and Cisco IOS Differences

In Cisco NX-OS:

• When you first log into the NX-OS, you go directly into EXEC mode.

• Role Based Access Control (RBAC) determines a user’s permissions by

default. NX-OS 5.0(2a) introduced privilege levels and two-stage authenticationusing an enable secret that can be enabled with the global feature privilegeconfiguration command.

• By default, the admin user has network-admin rights that allow full read/write

access. Additional users can be created with very granular rights to permit ordeny specific CLI commands.

• The Cisco NX-OS has a Setup Utility that allows a user to specify the systemdefaults, perform basic configuration, and apply a pre-defined Control PlanePolicing (CoPP) security policy.

• The Cisco NX-OS uses a feature based license model. An Enterprise Services,

Advanced Services, Transport Services, Scalable Feature and Enhanced Layer 2license is required depending on the features required. Additional licenses maybe required in the future.

• A 120 day license grace period is supported for testing, but features areautomatically removed from the running configuration after the expiration date isreached. Some features such as Cisco Trustsec that require an Advanced

Services license cannot be configured with a grace period.




Fundamentals (cont’d) Important Cisco NX-OS and Cisco IOS Differences

In Cisco NX-OS:

• The Cisco NX-OS has the ability to enable and disable features such as OSPF,BGP, etc… using the feature configuration command. Configuration and

verification commands are not available until you enable the specific feature.

• Interfaces are labeled in the configuration as Ethernet. There aren’t any speed

designations.

• The Cisco NX-OS has two preconfigured VRF instances by default

(management, default). The management VRF is applied to the supervisormodule out-of-band Ethernet port (mgmt0), and the default VRF instance isapplied to all other I/O module Ethernet ports. The mgmt0 port is the only portpermitted in the management VRF instance and cannot be assigned to anotherVRF instance.

• SSHv2 server/client functionality is enabled by default. TELNET server

functionality is disabled by default. (The TELNET client is enabled by default andcannot be disabled.)

• VTY and Auxiliary port configurations do not show up in the defaultconfiguration unless a parameter is modified (The Console port is included inthe default configuration). The VTY port supports 32 simultaneous sessions andthe timeout is disabled by default for all three port types




Fundamentals (cont’d) Important Cisco NX-OS and Cisco IOS Differences

In Cisco NX-OS:

• The Console and VTY ports always prompt the user for a username/passwordpair for authentication before granting access to the CLI. The Cisco IOS appliesthe login command to the Console and VTY ports by default to enable passwordauthentication (If the no login command is applied, a user can gain accesswithout a password.).

• A user can execute show commands in configuration mode without using the

do command as in Cisco IOS Software.• When executing a show command, a user has several more options whenusing the pipe (|) option such as grep for parsing the output, perl for activating ascript, and xml to format the output for network management applications.




FundamentalsThings You Should Know

• The default administer user is predefined as admin. An admin user password

has to be specified when the system is powered up for the first time, or if therunning configuration is erased with the write erase command and system isrepowered.

• The license grace-period can be disabled without any impact if the properlicense is installed for a feature within the 120 day grace period.

• If you remove a feature with the global no feature configuration command, all

relevant commands related to that feature are removed from the runningconfiguration. Some features such as LaCP and vPC will not allow you to disablethe feature if they are configured.

• The NX-OS uses a kickstart image and a system image. Both images areidentified in the configuration file as the kickstart and system boot variables. Theboot variables determine what version of NX-OS is loaded when the system is

powered on. (The kickstart and system boot variables have to be configured forthe same NX-OS version.)

• The show running-config command accepts several options, such as OSPF,BGP, etc… that will display the runtime configuration for a specific feature.

• The show tech command accepts several options that will display informationfor a specific feature.




FundamentalsThings You Should Know

• The NX-OS has a configuration checkpoint/rollback feature that should be

used when making changes to a production network. A checkpoint configurationcan be saved in EXEC mode with the global checkpoint command and therollback procedure can be executed with the rollback command.




FundamentalsCommand Comparison: NX-OS vs IOS

Cisco IOS CLI Cisco NX-OS CLI

Default User Prompt

c6500> n5000#

Entering Configuration Mode

c6500# configure terminal n5000# configure terminal

Saving the Running Config to the Startup Config (nvram)

c6500# write memory or

c6500# copy running-config startup-config

n5000# copy running-config startup-config

Erasing the startup config (nvram)

c6500# write erase n5000# write erase




FundamentalsCommand Comparison: NX-OS vs IOS (cont’d)


Installing a License

Cisco IOS Software does not require alicense file installation.

n5000# install licensebootflash:license_file.lic

Interface Naming Convention

interface Ethernet 1/1 interface

FastEthernet 1/1

interface GigabitEthernet 1/1

interface TenGigabitEthernet 1/1

interface Ethernet 1/1

Cisco IOS Software doesn’t enable VRFsby default.

vrf context management

Default VRF Configuration (management)






Configuring the Software Image Boot Variables

boot system flash sup-bootdisk:s72033-

ipservicesk9_wan-mz.122-33.SXH1.bin

boot kickstart bootflash:/n5000-uk9-kickstart.5.0.3.N2.2.bin

boot system bootflash:/n5000-uk9.5.0.3.N2.2.bin

Enabling Features

Cisco IOS Software does not have the

functionality to enable or disable features.feature ospf

Enabling TELNET (SSH is recommended)

Cisco IOS Software enables TELNET by

default.feature telnet






Configuring the Console Timeout

line console 0

exec-timeout 15 0 (minutes seconds)

login

line console

exec-timeout 15 (minutes only)

line vty 0 9

exec-timeout 15 0 (minutes seconds)

login

line vty

session-limit 10

exec-timeout 15 (minutes only)

Configuring the VTY Timeout and Session Limit




FundamentalsTroubleshooting and Verification Commands

Cisco NX-OSInterface

Cisco IOS SoftwareInterface

Command Description

show running-config show running-config Displays the running configuration

show startup-config show startup-config Displays the startup configuration

- - -

show interface show interface Displays the status for all of the interfaces

show interface ethernet <x/x> show interface <int type> Displays the status for a specific interface

show interface mgmt 0 - Displays the status for the mgmt interface

- -

show boot show boot Displays the current boot variables

- -

show clock show clockDisplays the system clock and time zone

configurationshow clock detail show clock detail Displays the summer-time configuration

- - -

show environment show environment Displays all environment parameters




FundamentalsTroubleshooting and Verification Commands (cont’d)

Cisco NX-OS Cisco IOS

Software

Command Description

show environmentclock

show environment

status clockDisplays clock status for A/B and active clock

show environment fan show environment

cooling fan-trayDisplays fan status

show environmentpower

show power Displays power budget

show environmenttemperature

show environment

temperature Displays environment data

- - -

show feature -Displays the features and routing processes

enabled

- - -

show log logfile show log Displays the local log

show log nvram -Displays persistent log messages (severity 0-2)

stored in NVRAMshow module show module Displays installed modules and their status

show module uptime -Displays how long each module has be powered

up






Software

Command Description

show process cpu show process cpu Displays the processes running on the CPU

show process cpuhistory

show process cpu

history

Displays the process history of the CPU in chart

form

show process cpusorted

show process cpu

sortedDisplays sorted processes running on the CPU

- - -

show system cores - Displays the core dump files if present

show systemexception-info

show exception Displays last exception log

show systemresources

show process cpu Displays CPU and memory usage data

show system uptime -Displays system and kernel start time (Displays

active supervisor uptime)

- - -

show tech-support show tech-support Displays system technical information for CiscoTAC

show tech-support <name>

show tech-support

<name>

Displays feature specific technical information for

Cisco TAC

Hint: Show proc cpu | ex 0.0






Software

Command Description

show version show versionDisplays running software version, basic

hardware, CMP status and system uptime

- - -

show line show line Displays console and auxiliary port information

show line com1 - Displays auxiliary port information

show line console show line console 0 Displays console port information

show line console

connected - States if the console port is physically connectedshow terminal show terminal Displays terminal settings

show users show users Displays current virtual terminal settings






Software

Command Description

show vrf show ip vrf Displays a list of all configured VRFs

show vrf <name> show ip vrf <name> Displays a specified VRF

show vrf <name>

detail show vrf detail <name> Displays details for a specified

show vrf <name>

interface - Displays interface assignment for a specified VRF

show vrf default - Displays a summary of the default VRF

show vrf detail show vrf detail Displays details for all VRF'sshow vrf interface show ip vrf interface Displays VRF interface assignment

show vrfmanagement

- Displays a summary of the management VRF






Software

Command Description

show license - Displays all license file information

show license brief - Displays the license file names installed

show license file <name> -Displays license contents based on a specified

name

show license host-id -Displays the chassis Host-ID used for creating a

license

show license usage - Displays all licenses used by the system

show license usage <license-type>

- Displays all licenses used by the system per type




Cisco Nexus 5500Interface





InterfacesImportant Cisco NX-OS and Cisco IOS Differences

In Cisco NX-OS:

• SVI command-line interface (CLI) configuration and verification commands arenot available until you enable the SVI feature with the feature interface-vlancommand.

• Only 802.1q trunks are supported, so the encapsulation command isn'tnecessary when configuring a layer-2 switched trunk interface. (Cisco ISL is notsupported)

• An IP subnet mask can be applied using /xx or xxx.xxx.xxx.xxx notation whenconfiguring an IP address on a layer-3 interface. The IP subnet mask is displayedas /xx in the configuration and show interface command output regardlesswhich configuration method is used.

• The CLI syntax for specifying multiple interfaces is different in Cisco NX-OSSoftware. The range keyword has been omitted from the syntax (IE: interfaceethernet 1/1-2)

• When monitoring interface statistics with the show interface CLI command, aconfigurable load-interval can be configured per interface with the load-intervalcounters command to specify sampling rates for bit-rate and packet-ratestatistics. The Cisco IOS Software supports the load-interval interface command,but doesn't support multiple sampling rates.

• A locator-LED (beacon) that allows remote-hands-support personnel to easilyidentify a specific port. The beacon light can be enabled per interface in interface

configuration mode with the beacon CLI command.




Interfaces (cont’d) Important Cisco NX-OS and Cisco IOS Differences

In Cisco NX-OS:

• An administrator can configure port profiles as templates that can be appliedto a large number of interfaces to simplify the CLI configuration process. Portprofiles are "live" configuration templates, so modifications to a port profile areautomatically applied to the associated interfaces. Cisco IOS uses port macrosto simplify the CLI configuration process, but unlike Port Profiles they areapplied one time.

• The out-of-band management ethernet port is configured with the interface

mgmt 0 CLI command.• Proxy ARP is disabled on all interfaces by default.




InterfacesThings You Should Know

• The default port type is configurable for L3 routed or L2 switched in

the setup startup script. (L3 is the default port type prior to running thescript)

• A layer-2 switched trunk port sends and receives traffic for all VLANsby default (This is the same as Cisco IOS Software). Use the switchporttrunk allowed vlan interface CLI command to specify the VLANs allowedon the trunk.

• The clear counters interface ethernet <x/x> CLI command resets thecounters for a specific interface.

• An interface configuration can be reset to its default values with thedefault interface <x/x> global configuration command.




InterfacesCommand Comparison: NX-OS vs IOS


Configuring a Routed Interface

interface gigabitethernet 1/1

ip address 192.168.1.1 255.255.255.0

no shutdown

interface ethernet 1/1

ip address 192.168.1.1/24

no shutdown

Configuring a Switched Interface (VLAN 10)

vlan 10


switchport

switchport mode access

switchport access vlan 10

no shutdown

vlan 10


switchport

switchport mode access


no shutdown




InterfacesCommand Comparison: NX-OS vs IOS (cont’d)


Configuring a Switched Virtual Interface (SVI)


ability to enable or disable SVI interfaces

using the feature command.

interface vlan 10

ip address 192.168.1.1 255.255.255.0

no shutdown

feature interface-vlan

interface vlan 10

ip address 192.168.1.1./24

no shutdown






Configuring a Switched Trunk Interface

interface GigabitEthernet 1/1

switchport

switchport trunk encapsulation dot1q

switchport trunk native vlan 2

switchport trunk allowed vlan 10,20

switchport mode trunk

no shutdown



switchport trunk allowed vlan 10,20

switchport trunk native vlan 2

no shutdown






Configuring a Routed Trunk Sub-Interface


no switchport

no shutdown

interface gigabitethernet1/1.10

encapsulation dot1Q 10

ip address 192.168.1.1 255.255.255.0

no shutdown


no switchport

no shutdown

interface ethernet 1/1.10

encapsulation dot1q 10

ip address 192.168.1.1/24

no shutdown






Configuring Multiple Interfaces

(Examples)

interface range gigabitethernet 1/1-2

or

interface range gigabitethernet 1/1,

gigabitethernet 2/1

interface ethernet 1/1-1

or

interface ethernet 1/1, ethernet 2/1

Configuring the Interface Locator-LED

(Beacon)


ability to enable a located-led per interface.


beacon

I f






Configuring Port Profiles


ability to configure port profiles.

port-profile type ethernet Email-Template

switchport


spanning-tree port type edge

no shutdown

description Email Server Port

state enabled

interface ethernet 2/1-48

inherit port-profile Email-Template

I t f




InterfacesTroubleshooting and Verification Commands

Cisco NX-OS

Interface

Cisco IOS Software

Interface Command Description

show interface show interfaceDisplays the status and statistics for all

interfaces or a specific interface

show interface ethernet

<x/x/x> -

Displays the status and statistics for a

FEX host interface

show interface brief -Displays a brief list of the interfaces (type,

mode, status, speed, MTU)

show interface

capabilities show interface capabilities Displays interface capabilities

show interface

counters show interface counters

Displays interface counters (input/output

unicast, multicast & broadcast)

show interface

description show interface description

Displays all interfaces with configured

descriptions

show interface ethernet show interface ethernetDisplays status and statistics for a specific

interface

show interface fex-

fabric - Displays FEX fabric interface status

I t f




InterfacesTroubleshooting and Verification Commands (cont’d)

Cisco NX-OS InterfaceCisco IOS Software

InterfaceCommand Description

show interface flowcontrol show interface

flowcontrol

Displays Flow Control (802.1p) status

and state for all interfaces

show interface loopback show interface loopbackDisplays status and statistics for a

specific loopback interface

show interface mac-

address

-Displays all interfaces and their

associated MAC Addresses

show interface mgmt -

Displays status and statistics for the

management interface located on the

supervisor

show interface port-

channel

show interface port-

channel

Displays status and statistics for a

specific port-channel

show interface priority-flow-control

- Displays PFC information

show interface pruning show interface pruningDisplays trunk interfaces VTP pruning

information

show interface snmp-

ifindex - Displays SNMP interface index

I t f





Cisco NX-OS Interface

Cisco IOS Software

Interface Command Description

show interface status show interface statusDisplays all interfaces and their current

status

show interface switchport show interface switchportDisplays a list of all interfaces that are

configured as switchports

show interface transceiver show interface transceiver

Displays a list of all interfaces and

optic information (calibrations, details)

show interface trunk show interface trunkDisplays a list of all interfaces

configured as trunks

show interface tunnel <#> show interface tunnel <#> Displays status and statistics for a

specific tunnel interface

show interface vlan <#> show interface vlan <#> Displays status and statistics for a

specific VLAN interface

I t f





Cisco NX-OS InterfaceCisco IOSSoftware

Interface

Command Description

show port-profile - Displays all port profile information

show port-profile brief - Displays brief port profile information

show port-profile expand-

interface-

Displays active profile configuration applied to an

interfaceshow port-profile name - Displays specific port profile

show port-profile sync-status - Displays interfaces out of sync with port profiles

show port-profile usage - Displays interfaces inherited to a port profile




CLI Overview

The Cisco NX-OX CLI shares a lot of concepts as Cisco IOS software, so initial

configuration is very simple. The commands can be abbreviated, the ? providesonline help, and the <TAB> key auto-fills command options.

Entering Configuration Mode:

User Exec Mode:

n5500# configure terminal

n5500(config)#

n5500#

Saving Running Configuration to Startup:

n5500# copy running-config startup-config

Erasing the Startup Configuration:

n5500# write erase

No “write memory” command

Default prompt - Type “exit” to log out

Attaching to a Module:

Type “exit” or “$” to log out of the module n5500# attach module 1 Attaching to module 1 ...module-1#

User is prompted to continue

Show Running & Startup Configuration:n5500# show running-confign5500# show startup-config

Several additional options exist to view the

configuration related to a specific feature




Enabling NX-OS Features

n5500(config)# feature ?

bgp Enable/Disable Border Gateway Protocol (BGP)cts Enable/Disable CTSdhcp Enable/Disable DHCP Snoopingdot1x Enable/Disable dot1x

eigrp Enable/Disable Enhanced Interior Gateway Routing Protocol (EIGRP)eou Enable/Disable eou(l2nac)

glbp Enable/Disable Gateway Load Balancing Protocol (GLBP)hsrp Enable/Disable Hot Standby Router Protocol (HSRP)interface-vlan Enable/Disable interface vlan

isis Enable/Disable IS-IS Unicast Routing Protocol (IS-IS)lacp Enable/Disable LACP

msdp Enable/Disable Multicast Source Discovery Protocol (MSDP)

netflow Enable/Disable NetFlow

ospf Enable/Disable Open Shortest Path First Protocol (OSPF)ospfv3 Enable/Disable Open Shortest Path First Version 3 Protocol(OSPFv3)pbr Enable/Disable Policy Based Routing(PBR)pim Enable/Disable Protocol Independent Multicast (PIM)

pim6 Enable/Disable Protocol Independent Multicast (PIM) for IPv6

port-security Enable/Disable port-securityprivate-vlan Enable/Disable private-vlanrip Enable/Disable Routing Information Protocol (RIP)scheduler Enable/Disable schedulerssh Enable/Disable ssh

tacacs+ Enable/Disable tacacs+

telnet Enable/Disable telnet

tunnel Enable/Disable Tunnel Managerudld Enable/Disable UDLD

vpc Enable/Disable VPC (Virtual Port Channel)

vrrp Enable/Disable Virtual Router Redundancy Protocol (VRRP)vtp Enable/Disable VTP

wccp Enable/Disable Web Cache Communication Protocol (WCCP)

The Cisco NX-OS provides the capability to enable and disable features using

the feature command. Configuration CLI and show commands are notavailable(displayed) for a feature if it isn’t enabled.

V if i S ft V i




Verifying Software Version

File locations

NX-OS versions

Bootflash (Size)Expansion flash

System DRAM (KB)

NX-OS software

Use the show version command to obtain general hardware/software information.

System uptime

Cisco Nexus Operating System (NX-OS) SoftwareTAC support: http://www.cisco.com/tacCopyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.

The copyrights to certain works contained herein are owned byother third parties and are used and distributed under license.Some parts of this software are covered under the GNU PublicLicense. A copy of the license is available athttp://www.gnu.org/licenses/gpl.html.

SoftwareBIOS: version 1.8.0loader: version N/Akickstart: version 5.0(2)N1(1)system: version 5.0(2)N1(1)

power-seq: version v3.0, gem: version v1.0uC: version v1.0.0.14

BIOS compile time: 10/06/2010kickstart image file is: bootflash:/n5500-uk9 kickstart.5.0.3.N2.2.binkickstart compile time: 10/15/2010 0:00:00 [10/15/2010 04:00:43]system image file is: bootflash:/n5500-uk9.5.0.3.N2.2.binsystem compile time: 10/15/2010 0:00:00 [10/15/2010 05:34:05]

Hardwarecisco Nexus5548 Chassis ("O2 32X10GE/Modular Supervisor")Intel(R) Xeon(R) CPU with 8299548 kB of memory.Processor Board ID JAF1445APSP

Device name: USPA833NEXUS5548-01

bootflash: 2007040 kB

Kernel uptime is 143 day(s), 1 hour(s), 1 minute(s), 8 second(s)

Last resetReason: UnknownSystem version: 5.0(2)N1(1)Service:

pluginCore Plugin, Ethernet Plugin

`<truncated>

Basic Configuration: Configuring the




Basic Configuration: Configuring theManagement VRF Context

1. Configuring switch name

2. Configuring the management interface

3. Configuring the management VRF context

switch# configureswitch(config)# switchname N5K

N5K(config)# interface mgmt0 N5K(config-if)# ip address 172.18.217.80 255.255.255.0 N5K(config-if)# no shut

N5K(config-if)# exit

N5K(config)# vrf context management N5K(config-vrf)# ip route 0.0.0.0/0 172.18.217.1/24



© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID © 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

ACL on mgmt0 / VTY

N5k supports mgmt0 for OOB Mgmt

N5k supports SVI for inband management

- Enable ‘feature interface-vlan’

inter mgmt0ip access-group xx in/out

line vty

Ip access-class xx in/out




Management Interface Verification

The following commands verify the “management” VRF routing table as well as

the interface statistics.

n5500# show ip route vrf managementIP Route Table for VRF "management"'*' denotes best ucast next-hop '**' denotes best mcast next-hop'[x/y]' denotes [preference/metric]

0.0.0.0/0, 1 ucast next-hops, 0 mcast next-hops*via 159.142.1.10, mgmt0, [1/0], 00:01:27, static

VRF “management” Routing Table:

“management” VRF default route

Routing table for “management“ VRF

Management Interface Statistics:

n5500# show interface mgmt 0mgmt0 is upHardware is GigabitEthernet, address is 001b.54c0.feb8 (bia 001b.54c0.feb8)Internet Address is 159.142.1.1/24MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255Encapsulation ARPAfull-duplex, 1000 Mb/s Auto-Negotiation is turned on30 minute input rate 1102814 bytes/sec, 16317 packets/sec30 minute output rate 42224 bytes/sec, 251 packets/secRx16422 input packets 6 unicast packets 11734 multicast packets4682 broadcast packets 1110256 bytes

Tx254 output packets 164 unicast packets 74 multicast packets

16 broadcast packets 42547 bytes




Configuring User Accounts

Creating user accounts N5K# configure N5K(config)# username admin password cae123rtp role network-admin N5K(config)# username operator password oper1234 role network-operatoruser:operator is reserved N5K(config)# username paul password oper1234 role network-operator

N5K(config)# sh run | incl usernameusername admin password 5 $1$6KdEue0H$vexPxI/qjJNZrRmg8nsIo. role network-adminusername paul password 5 $1$PvSqwWxh$gxL46OnByOVe8ZC5zOj0b. role network-operator

N5K(config)# sh run | incl snmp-serversnmp-server user paul network-operator auth md5

0x72fffc91ff1de08468c5b1c3c0acd111 priv 0x72fffc91ff1de08468c5b1c3c0acd111 localizedkeysnmp-server user admin network-admin auth md50x25bb8f4349b3217abb2672edc84981ac priv 0x25bb8f4349b3217abb2672edc84981ac localizedkey




Configuring Administrative Access:RSA and SSH

Configure RSA keys (may have to disable SSH server first)

Enable the SSH server process (enabled by default)

Verify that the SSH server is running

N5K(config)# ssh key rsa 1024 forcedeleting old rsa key.....generating rsa key(1024 bits)......

generated rsa key N5K(config)# ssh server enable N5K(config)# sh ssh serverssh is enabledversion 2 enabled




Role Based Access Control (RBAC)

Default User Roles Role Description

network-admin read / write access for “default” VDC

network-operator read access for the “default” VDC

vdc-admin read / write access for a VDC

vdc-operator read access for a VDC

Users and associated roles are created to secure access to the Cisco NX-OS.RBAC allows you to create a granular security policy that limits a user’s access to

the device, so they can only perform the actions they are authorized for. RBACcan work in conjunction with AAA.

Note: a user is assigned to the “network-operator” role if a role isn’t specified when the user is created.

Default User User Description

admin “admin” user with “network-admin” role




RBAC Configuration Example

The following example illustrates how to create a role with multiple rules and

assign it to a user.

Only a user with the “network-admin” or “vdc-admin” role can create users androles.

n5500(config)# role name ospf-adminn5500(config-role)# rule 1 permit command show interface *n5500(config-role)# rule 2 permit command show running-confign5500(config-role)# rule 3 permit read-write feature router-ospfn5500(config-role)# rule 4 permit command config t ; interface *n5500(config-role)# rule 5 permit command copy running-config startup-config

Create a Role:

Create a User and Assign a Role:

n5500(config)# username ospf-admin password xxxxxxxx role ospf-admin

Allow a user to

configure OSPF, verify

the configuration and

save the running-

configuration

If a users role is modified, the changes do not take effect until that user logsout and back into the system.




Logging Configuration and Verification

n5500# show logging info

Logging console: enabled (Severity: critical)Logging monitor: enabled (Severity: notifications)Logging linecard: enabled (Severity: notifications)Logging timestamp: SecondsLogging loopback : disabledLogging server: enabled

{159.142.1.10}server severity: debuggingserver facility: local7server VRF: management

Logging logflash: enabled (Severity: notifications)Logging logfile: enabled

Name - messages: Severity - notifications Size – 4194304<Text Omitted>

n5500(config)# logging server 159.142.1.10 ?<CR><0-7> 0-emerg;1-alert;2-crit;3-err;4-warn;5-notif;6-inform;7-debug

n5500(config)# logging server 159.142.1.10 7 use-vrf management

Syslog Server 159.142.1.10 is enabled

Configured in the “management” VRF

Specify the VRF the server

should use to send logs

Logging (Syslog) Configuration:

Logging (Syslog) Verification:

Multiple logging servers can be enabled with different severity levels. Use the

use-vrf option to specify the VRF where the Syslog server resides.

n5500# clear logging logfile Clears the“logfile”

Other common options (“logfile” & “nvram”)

Specify the logging severity

level per server




SNMP Configuration and Verification

n5500(config)# snmp-server community secret ron5500(config)# snmp-server host 159.142.1.10 version 1 secret n5500(config)# snmp-server host 159.142.1.10 use_vrf management n5500(config)# snmp-server enable traps n5500(config)# snmp-server contact Lab Manager

n5500# show snmp host

--------------------------------------------------------------------------------Host Port Version Level Type SecName

--------------------------------------------------------------------------------159.142.1.10 162 v1 noauth trap secret

Use VRF: management-------------------------------------------------------------------

n5500# show snmp trap

Trap type Enabled--------- -------aaa server-state-change Yescallhome Noentity fru Yeslicense Yessnmp authentication Yeslink Yesbridge topologychange Nobridge newroot Nostpx inconsistency Nostpx loop-inconsistency No

stpx root-inconsistency No

SNMP Traps enabled by default

Configured Host

V1 is the default

Enable default Traps

Basic SNMPv1 configuration. SNMP versions 2c and 3 are also supported.

The VRF the host is associated with

Configured SNMP host

Community String (RO or RW)




SNMP Community ACL Configuration

n5500(config)# interface mgmt0

n5500(config-if)# ip address 10.20.1.21/24

n5500(config)# ip access-list snmp-ron5500(config-acl)# permit udp 10.20.0.20/32 10.20.1.21/32 eq snmp

n5500(config)# snmp-server community cisco123 ron5500(config)# snmp-server community cisco123 use-acl snmp-ro

Configuration:

Verification:n5500# show snmp community

Community Group / Access context acl_filter--------- -------------- ------- ----------cisco123 network-operator snmp-ro

“snmp-ro” ACL associated with the“cisco123” community string

Define the SNMP community

string and associate the ACL

Define an ACL “UDP port 161”

An extended ACL can be applied to an SNMP community string to limit access to

SNMP data. An ACL can be applied for read-only and read-write community strings.The following example restricts SNMP access to one host when accessing the IPaddress associated to the “mgmt 0” interface.




TACACS+ Configuration and Verification

n5500(config)# feature tacacs+n5500(config)# tacacs-server host 159.142.1.10warning: no key is configured for the hostn5500(config)# tacacs-server key cisco123

n5500(config)# aaa group server tacacs+ AAA-Servern5500(config-tacacs+)# use-vrf managementn5500(config-tacacs+)# server 159.142.1.10

n5500(config)# aaa authentication login default group AAA-Servern5500(config)# aaa authorization commands default group AAA-Server localn5500(config)# aaa authorization config-commands default group AAA-Server localn5500(config)# aaa accounting default group AAA-Server

TACACS+ Configuration:

TACACS+ Server Verification:

A basic AAA/TACACS+ configuration is illustrated below that is very similar to theprevious RADIUS configuration. The “tacacs+” feature needs be enabled first.TACACS+ supports command and config-command AAA authorization.

Enable the TACACS+ feature first!

Specify which VRF to use for TACACS+

Optional: Enable AAA

command & config-

command authorization

with local fallback

n5500# show tacacsGlobal TACACS+ shared secret:********timeout value:5deadtime value:0total number of servers:1

following TACACS+ servers are configured:159.142.1.10:

available on port:49

n5500# show tacacs groupstotal number of groups:1

following TACACS+ server groups are configured:group AAA-Server:

server 159.142.1.10 on port 49deadtime is 0vrf is management

Optional: Enable AAA Accounting




Cisco Nexus 5500AAA, RADIUS, and TACACS+Config and Troubleshooting

AAA RADIUS and TACACS+




AAA, RADIUS and TACACS+Important Cisco NX-OS and Cisco IOS Differences

In Cisco NX-OS:

• TACACS+ command-line interface (CLI) configuration and verification commands are notavailable until you enable the TACACS+ feature with the feature tacacs+ command (TheRADIUS feature is enabled by default and cannot be disabled).

• The aaa new-model command is not required to enable AAA authentication,authorization, or accounting.

• The RADIUS vendor-specific attributes (VSA) feature is enabled by default. Cisco IOSSoftware requires the global radius-server vsa send configuration command to enable IETF

attribute 26.• Local command authorization can be performed using privilege-levels or role-basedaccess control (RBAC) without a AAA server. Local privilege-levels or RBAC roles can beassociated to users configured on the AAA server using VSAs (TACACS+ supportscommand authorization that can be configured on the AAA server).

• If a configured AAA server is not available for authentication, the local database(username/password) is automatically used for device access.

• The RADIUS and TACACS+ host keys are Triple Data Encryption Standard (3DES)encrypted in the configuration. Cisco IOS Software requires the service password command.

AAA, RADIUS, and TACACS+




AAA, RADIUS, and TACACS+Important Cisco NX-OS and Cisco IOS Differences

In Cisco NX-OS:

• All configuration commands are recorded in a local log (NVRAM) with user and timestamp information by default (no AAA configuration required). The log can be viewed withthe show accounting log command.

• The aaa accounting default command enables accounting for start and stop records aswell as command accounting (Exec mode and configuration mode). Cisco IOS Softwarerequires additional aaa accounting commands to enable both types of accounting.





AAA, RADIUS, and TACACSThings You Should Know

• Configuring a protocol for AAA is a multi-step configuration process: Definethe server(s), create the server group, and associate the server group to therequired AAA commands.

• If you remove a feature such as TACACS+ with the global no feature <name> command, all relevant configuration information is removed from the running-configuration for the specified feature.

• AAA server groups are associated with the default Virtual Route Forwarding(VRF) instance by default. Associate the proper VRF instance with the AAAserver group if you are using the management port on the supervisor module orif the AAA server is in a non-default VRF instance.

• A RADIUS and TACACS+ source interface can be configured globally or perAAA server group to specify the source IP address for packets destined toremote AAA services.

• RADIUS and TACACS+ server keys can be specified for a group of servers orper individual server.





AAA, RADIUS, and TACACSThings You Should Know

• By default, RADIUS uses UDP ports 1812 (authentication) and 1813(accounting), and TACACS+ uses TCP port 49. All server ports can be configuredto use different values.

• Directed server requests are enabled by default for RADIUS and TACACS+.

• The local option can be used with AAA authorization to fallback to localprivilege-levels or RBAC in the event a AAA server is not available for commandauthorization.

• RADIUS and TACACS+ support global server test monitoring (Per servermonitoring takes precedence over global monitoring).

• Use the show running-config command with the AAA, radius or tacacs+option to display the running configuration for a specific feature.





AAA, RADIUS, and TACACSCommand Comparison: NX-OS vs IOS


Configuring a RADIUS Server with a Key

radius-server host 192.168.1.1 key cisco123 radius-server host 192.168.1.1 key 7"fewhg123" (7=encrypted or 0=cleartext)

Specifying Non defualt RADIUS UDP Ports radius-server host 192.16.1.1 auth-port 1645acct-port 1646

radius-server 192.168.1.1 auth-port 1645acct-port 1646

Specifying the RADIUS Timeout Value (Global)

radius-server host 192.168.1.1 timeout 10 radius-server timeout 10

ip radius source-interface loopback0 ip radius source-interface loopback0

Specifying the RADIUS Source Interface (Global)





Command Comparison: NX-OS vs IOS (cont’d)


Enabling TACACS+

Cisco IOS Software does not have the ability to

enable or disable TACACS+.feature tacacs+

Configuring a TACACS+ Server with a Key

tacacs-server host 192.168.1.1 key cisco123 tacacs-server host 192.168.1.1 key 7"fewhg123" (7=encrypted or 0=cleartext)

Specifying a Nondefualt TACACS+ TCP Port

tacacs-server host 192.168.1.1 port 85 tacacs-server host 192.168.1.1 port 85

Specifying the TACACS+ Timeout Value (Global) tacacs-server timeout 10 tacacs-server timeout 10

Specifying the TACACS+ Source Interface (Global)

ip tacacs source-interface loopback0 ip tacacs source-interface loopback0







Configuring an AAA Server Group (RADIUS)

aaa group server radius AAA-Servers

server 192.168.1.1


server 192.168.1.1

Configuring an AAA Server Group for a VRF Instance (RADIUS)


server 192.168.1.1

ip vrf forwarding management


server 192.168.1.1

use-vrf management







Configuring the AAA Server Group Dead Time (RADIUS)


deadtime 5


deadtime 5

Configuring an AAA Server Group (TACACS+)

aaa group server tacacs+ AAA-Servers

server 192.168.1.1

aaa group server tacacs+ AAA-Servers

server 192.168.1.1

Enabling AAA Authentication with an AAA Server Group

aaa new-model aaa authentication login default group AAA-

Servers

aaa authentication login default groupAAA-Servers







Enabling AAA Authorization with an AAA Server Group

aaa new-model

aaa authorization config-commandsaaa authorization commands 1 default group

AAA-Servers

aaa authorization

config-commands default group AAA-Servers aaa authorization commands default group AAA-

Servers

Enabling AAA Accounting with an AAA Server Group

aaa new-model aaa accounting exec default start-stop group

AAA-Servers

aaa accounting default group AAA-Servers





, ,Troubleshooting and Verification Commands

Cisco NX-OS Interface

Cisco IOS

Software Interface Command Description

show aaa accounting - Displays the status of AAA accounting

show aaa authentication -Displays the default and console login

methods

show aaa authenticationlogin ascii-authentication

-Displays the status of ascii authentication;

enabled or disabled

show aaa authenticationlogin chap

-Displays the status of the ChallengeHandshake authentication protocol (CHAP);

enabled or disabled

show aaa authenticationlogin error-enable

-Displays the login error message status;

enabled or disabled.

show aaa authenticationlogin mschap

-Displays the status of Microsoft CHAP (MS-

CHAP); enabled or disabled.

show aaa authenticationlogin mschapv2 - Displays the status of MS-CHAPv2; enabledor disabled)

show aaa authorization - Displays the AAA authorization configuration

show aaa groups - Displays the AAA groups that are configured

show aaa users show aaa userDisplays the AAA users that authenticated

remotely





, ,Troubleshooting and Verification Commands (cont’d)

Cisco NX-OS Interface Cisco IOS SoftwareInterface Command Description

show accounting log -Displays the local AAA configuration

accounting log

- - -

show radius-server -Displays the RADIUS server configuration

for all servers

show radius-server<x.x.x.x>

- Displays a specific RADIUS serverconfiguration

show radius-serverdirected-request

-Displays the status of the directed-request

feature (enabled or disabled)

show radius-server groups show radius server-group Displays RADIUS server groups

show radius-server sorted - Displays RADIUS servers sorted by name

show radius-server

statistics <x.x.x.x> show radius statistics

Displays RADIUS statistics for a specific

server





Troubleshooting and Verification Commands (cont’d)



show tacacs-server show tacacsDisplays the TACACS+ server

configuration for all servers

show tacacs-server<x.x.x.x>

-Displays a specific TACACS+ server

configuration

show tacacs-serverdirected-request

-Displays the status of the directed-request

feature (enabled or disabled)

show tacacs-server groups - Displays TACACS+ server groups

show tacacs-server sorted -Displays TACACS+ servers sorted by

name

show tacacs-serverstatistics <x.x.x.x>

-Displays TACACS+ statistics for a specific

server

- - -

show user-account - Displays a list of locally configured users

show users show users Displays the users who are logged in

C f




Network Time Protocol Configuration

NTP Configuration:

n5500(config)# ntp server 10.20.8.129 prefer use-vrf managementn5500(config)# ntp server 10.20.8.130 use-vrf management

n5500(config)# ntp source 10.205.225.43

Use the “prefer” option to specify theprimary NTP Server

Specify the source IP address (Optional)

n5500(config)# ntp ?peer NTP Peer addressserver NTP server addresssource Source of NTP packets

NTP Configuration Options:

Configures the NX-OS to sync its clock from an NTP server

The Network Time Protocol (NTP) can be used to synchronize the clockfrom a reliable time source. The NX-OS can be configured tosynchronize its time with a “peer” or a “server”. The NX-OS cannot actas an NTP “server” for non-peering clients.

n5500(config)# clock ?summer-time Configure summer (daylight savings) timetimezone Configure time zone

Timezone Configuration:

The default time zone is UTC

N t k Ti P t l V ifi ti




Network Time Protocol Verification

n5500# show ntp peer-statusTotal peers : 2* - selected for sync, + - peer mode(active),

- - peer mode(passive), = - polled in client moderemote local st poll reach delay vrf

---------------------------------------------------------------------------------------------*10.20.8.129 10.205.225.43 2 64 17 0.00142 management=10.20.8.130 10.205.225.43 2 64 17 0.00133 management

Preferred Peer selected for sync

n5500# show ntp peers

--------------------------------------------------Peer IP Address Serv/Peer--------------------------------------------------10.20.8.129 Server (configured)10.20.8.130 Server (configured) Configured NTP “servers”

n5500# show ntp statistics peer ipaddr 10.20.8.129remote host: 10.20.8.129local interface: 10.205.225.43

time last received: 30stime until next send: 21sreachability change: 190spackets sent: 26packets received: 25bad authentication: 0bogus origin: 0duplicate: 0bad dispersion: 0bad reference time: 0candidate order: 6

NTP packets exchanged with NTP server




In-Service Software Upgrade

Nexus 5500 ISSUDifferences from Nexus 7000




Differences from Nexus 7000

Although the high-level steps associated with ISSU is common between both the Nexus5500 and Nexus 7000 platforms, the 2 platforms differ in key fundamental ways. The

Nexus 5500 supports a single “supervisor” ISSU architecture and performs a statefulrestart of the entire operating system upon execution, whilst leaving data planeforwarding intact…

During this time, control plane functions of the switch undergoing ISSU are temporarilysuspended, and configuration changes disallowed. The control plane will be broughtonline again within 80 seconds to allow protocol communications again.

Nexus 5500 ISSUPreconditions




Preconditions

The ISSU process is executed through the installer, and certain conditions must be satisfiedbefore it can proceed.

Restriction on Configuration changes Restriction on Topology Changes

CLI and SNMP config change requests aredenied during ISSU operations

Network/Topology changes like STP, FCFabric changes that affect zoning, FSPF,

domain manager, Module insertion are notexpected during ISSU operation

Nexus 5500 ISSUVPC Topologies




VPC Topologies

VPC topologies are fully supported with ISSU. Three types of VPC topologies are supportedfor the Nexus 5500 and Nexus 2000 FEX.

Throughout the ISSU process, VPC roles will remain intact. It is the peer switch’s responsibility to holdonto its state until ISSU process is complete

Blade or Access Switch FEX Active-Active FEX Straight-Through

Nexus 5500 ISSUSTP Topologies




STP Topologies

There are some restrictions that need to be placed on Ethernet STP topologies if a non-disruptive ISSU process is required:

The Nexus 5500/2000 switch undergoing ISSU must be a leaf on the spanning tree.The switch should not be a root switch or have any designated non-edge ports in theSTP topology

Bridge Assurance must be disabled for non-disruptive ISSU

1

2

STP Primary Root STP Secondary Root

STP Edge Ports STP Edge Ports

Non-Disruptive ISSUOK Here

Non-Disruptive ISSUNot OK Here

Cisco Confidential – Do Not Distribute

Nexus 5500 ISSUManagement Services



9999© 2009, Cisco Systems, Inc. All rights reserved.

Management Services

Prior to the switch being reset for ISSU, inbound-hi and management ports are brought

down, and are brought back up after ISSU completes. Services that depend on inbound-hi

and management ports are impacted during this time…

Telnet/SSH The Telnet/SSH daemons rely on the startup configs of the switch. As the device is

restarted, all Telnet/SSH sessions will be disconnected and need to be re-established

after ISSU completes

AAA/RADIUS Applications that leverage the AAA Service (such as “Login”) will be disabled during ISSUprocess. Since all Network Management services are disabled during this time, this

behavior is consistent.

HTTP The HTTP sessions to the Switch will be disconnected during ISSU reboot. After ISSU

reboot, the HTTPd will be restarted and switch will accept HTTP sessions after ISSU

reboot.

NTP The ntp sessions to and from the switch are disrupted during ISSU reboot. After ISSU

reboot, ntp session will be re-established based on the saved startup configuration.

Telnet/SSH will be dropped, perform ISSU from theConsole!

ISSU Req irements




Ensure you have enough space to store the images on bootflash:

Ensure no power interruptions occur during any install procedure.

Ensure the system and kickstart images are compatible with eachother.

Run only one installation on a switch at a time ***

Do not issue another command while running the installation

If the fabric extenders are not compatible with the software imageyou install on the Nexus 5500 switch, some traffic disruption mayoccur depending on the configuration. The “install all” command

output identifies these commands.

ISSU Requirements

Pre ISSU Check #1




Pre-ISSU Check #1DCN-N5K1# show spanning issu-impact

For ISSU to Proceed, Check the Following Criteria :

1. No Topology change must be active in any STP instance2. Bridge assurance(BA) should not be active on any port(except vPC peer-link)3. There should not be any Non Edge DesignatedForwarding port (except vPC peer-link)4. ISSU criteria must be met on the VPC Peer Switch as well

Following are the statistics on this switch

No Active Topology change Found!Criteria 1 PASSED !!

No Ports with BA Enabled Found!Criteria 2 PASSED!!

List of all the Non-Edge Ports

Port VLAN Role Sts Tree Type Instance---------------- ---- ---- --- --------- ---------

Ethernet1/1 49 Desg FWD PVRST 49port-channel20 50 Desg FWD PVRST 50port-channel20 51 Desg FWD PVRST 51port-channel20 52 Desg FWD PVRST 52port-channel20 77 Desg FWD PVRST 77port-channel20 201 Desg FWD PVRST 201

Criteria 3 FAILED !!

ISSU Cannot Proceed! Change the above Config

Spanning Tree designated portspresent, upgrade will be

disruptive

Pre ISSU Check #2




Pre-ISSU Check #2

show install all impact kickstart <image> system <image>

Displays information describing the impact of the upgrade oneach fabric extender including details such as upgrade imageversions.

This command will also display if the upgrade isdisruptive/non-disruptive and the reason why.

Compatibility check is done: Module bootable Impact Install-type Reason------ -------- -------------- ------------ ------

1 yes non-disruptive reset100 yes non-disruptive rolling

FEXInstallation will be non-disruptive

“rolling” upgrademeans each FEX

updated one at a time




Layer 2 Switching

VLAN Scalability




VLAN Scalability

The Cisco Nexus 5500 SeriesHardware supports 4096VLANs

Software allows users toconfigure the following VLANs:

1 – 3967 and 4048 to 4093 =4012 VLANs

This is true with or without vPC

The NXOS reserved VLANrange doesn’t match theCatalyst reserved VLAN range

But the internal NXOS VLANscan be mapped to an MST

instance Future optimization allows to

shift the reserved VLAN range

NXOS Reserved Range




NXOS Reserved Range

The Cisco Nexus 5500 Series Hardware supports 4096 VLANs

NXOS Reserves the following VLANs:

3968-4031 To support Multicast4032 Online diagnostics vlan1 - used for internal diags4033 Online diagnostics vlan24034 Online diagnostics vlan3

4035 Online diagnostics vlan44036-4047 Reserved - for future use, not used right now

4094 Reserved - for ERSPAN

Out of the NXOS Range, Nexus 5500 Series use:

4041 – RSVD_VLAN_DOT1Q_TAG_NATIVE 4041

4042 - for communication with FEX 4043 – for communication with adapter



VTP (*)




VTP ( )

NXOS 5.0(2)N1(1) introducedVTP client/server

Feature vtp

VTP v1 and v2

VLANs in the range 1 – 1006can be configured in VTP

VLANs beyond this range arenot propagated by VTP

VTPv3 is needed for the full 4krange, but it is not in thisrelease

Inconsistent VTPconfigurations are a Type 2

misconfiguration (so it is notdisruptive to vPC)

PVLANs requires VTP to betransparent or off

vPC + VTP is to be verified

VLAN Trunking Protocol (VTP)




VLAN Trunking Protocol (VTP)

n5500(config)# feature vtp

n5500(config)# vtp domain cisco.comn5500(config)# vtp version 2

Enable the VTP feature first!

Configure the VTP domain name

Configuration:

n5500# show vtp statusVTP Version : 2Configuration Revision : 0Maximum VLANs supported locally : 1005VTP Operating Mode : TransparentVTP Domain Name : cisco.comVTP Pruning Mode : DisabledVTP V2 Mode : EnabledVTP Traps Generation : Disabled

Verification:

All VTP packets received on the Nexus 5500 are dropped by default. Enable VTP intransparent mode to extend a VTP domain through a Nexus. Once, enabled, VTPpackets received on a trunk port are relayed to all other trunk ports.

Enables version 2 – version 1 is the default

Note: Select the VTP domain name and version that match the values used in the existing VTP domain.

Spanning TreeNX-OS - Spanning Tree Design




NX OS Spanning Tree Design

N

NN

RN R

N

N

E E E AccessEdge PortsNo BPDUs

Network PortsAll Send BPDUs

Network portN

R Root GuardDesignated port

Root port

Alternate port

E Edge port

NX-OS STP modes

Rapid-PVST+ (Default mode)

MST (Supported)

PVST (Not supported, butinteroperable)

NX-OS always uses Extended System ID

NX-OS uses a fixed STP link cost forEtherchannel links (based on number oflinks configured, not number active as inIOS)

Understand the three port modes

“Edge” port type replacesspanning-tree portfast

“Network” port type for bridge-to-bridge links

“Normal” for generic links inspanning tree

Spanning-Tree Port Types




Spanning-Tree Port Types

n5500(config-if-range)# spanning-tree port type ?edge Consider the interface as edge port (enable portfast)network Consider the interface as inter-switch linknormal Consider the interface as normal spanning tree port

Port Configuration:

n5500# show spanning-tree vlan 10VLAN0010Spanning tree enabled protocol rstp

<Text Omitted>

Interface Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- -----------------------Eth2/3 Desg FWD 4 128.259 P2pEth2/4 Desg FWD 4 128.260 Edge P2pEth2/5 Desg FWD 4 128.261 Network P2p

Port Verification:

STP supports three different port types. The default port type is normal. Anedge port type can be configured, so an interface immediately forwards traffic

(IOS “Portfast”) and the network port type can be configured to enable BridgeAssurance on an interface.

“edge” ports can beconfigured on trunks with the

additional “trunk” option

Port Types: Edge * Network Normal (Default)

* Note: Trunk ports for L3 hosts can be configured with the edge trunk option

Normal (Default)

Edge

Network

Optimizing the Layer 2 DesignBridge Assurance




Bridge Assurance

Root

Blocked

BPDUs

Network

Network Network

Network

BPDUs

EdgeEdge

Network

Network

BPDUs

Malfunctioning

switch

Stopped receivingBPDUS!

BA Inconsistent

BA Inconsistent


Specifies bi-direct ional

transmission of BPDUs on all ports of type “network”.

Protects againstunidirectional links and peerswitch software issues

Provides IGP like hello-deadtimer behaviour for SpanningTree

In all versions of NX-OS,available in IOS on theCatalyst 6500 beginning

12.2(33) SXI

Recommended in STPtopologies

Not recommend in vPCtopologies

interface port-channel200switchport mode trunkswitchport trunk allowed vlan 200-202spanning-tree port type network

Without Bridge Assurance




Without Bridge Assurance

Root

Blocked

BPDUs

BPDUs

BPDUs

Malfunctioningswitch

Loop!

With Bridge Assurance




With Bridge Assurance

Root

Blocked

BPDUs

Network

Network Network

Network

BPDUs

EdgeEdge

Network

Network

BPDUs

Malfunctioning

switch



BA Inconsistent

BA Inconsistent

%STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port Ethernet2/48VLAN0700.tstevens-dc3-2# sh spanning vl 700 | in -i bknEth2/48 Altn BKN*4 128.304 Network P2p *BA_Inctstevens-dc3-2#

STP Bridge Assurance




STP Bridge Assurance

Bridge Assurance prevents a spanning-tree domain from failing in an “open”state. When a port configured for Bridge Assurance stops receiving BPDU’s, theport transitions into a “blocking” state as opposed to remaining in a“forwarding” state. This “closed” state reduces the likelihood for mis-configured devices from creating STP loops.

n5500(config)# spanning-tree bridge assurance Enabled by default

n5500(config)# interface ethernet 1/25, ethernet 1/26n5500(config-if-range)# spanning-tree port type network Change the port type to “network”

Configuration:

Verification:

n5500# show spanning-tree summarySwitch is in mst mode (IEEE Standard)Root bridge for: MST0002Port Type Default is disabledEdge Port [PortFast] BPDU Guard Default is disabledEdge Port [PortFast] BPDU Filter Default is disabledBridge Assurance is enabledLoopguard Default is disabledPathcost method used is longPVST Simulation is enabled

<Text Omitted>

Enabled on all “network” port types

Note: Both ends of the link must have Bridge Assurance enabled

STP (Rapid-PVST+) Configuration



© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 116

STP (Rapid PVST+) Configuration

n5500(config)# vlan 20,30

n5500(config)# spanning-tree mode rapid-pvst

n5500(config)# spanning-tree vlan 20 root primary

n5500(config)# spanning-tree vlan 30 root secondary

n5500# show spanning-tree rootRoot Hello Max Fwd

Vlan Root ID Cost Time Age Dly Root Port---------------- -------------------- ------- ----- --- --- ----------------VLAN0020 24596 0018.bad8.58a5 0 2 20 15 This bridge is rootVLAN0030 24606 0018.bad8.5825 4 2 20 15 Ethernet1/13

Verifying STP Root Summary:

Rapid-PVST is the default

Decrements Priority to 24,596 to increase

the probability for it to become root

Rapid-PVST is defined in IEEE 802.1w. Rapid-PVST enables one STP instance perVLAN. Rapid-PVST is enabled by default, so there are very few commandsrequired to set up a Rapid-PVST domain.

Specifies the root or

root port

Decrements Priority to 28,672 to increase the

probability for it to become the backup for the root

Make sure you create the VLAN(s)

n5500(config)# spanning-tree vlan 20,30 priority 4096 The preferred method to influence the root

selection is to manually set the bridge priority

-OR-

STP (Rapid-PVST+) Verification




STP (Rapid PVST+) Verification

n5500# show spanning-tree VLAN0020Spanning tree enabled protocol rstpRoot ID Priority 24596

Address 0018.bad8.58a5This bridge is the rootHello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 24596 (priority 24576 sys-id-ext 20)Address 0018.bad8.58a5Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- --------------------------------Eth1/13 Desg FWD 4 128.141 P2pEth1/14 Desg FWD 4 128.142 P2p

VLAN0030Spanning tree enabled protocol rstpRoot ID Priority 24606

Address 0018.bad8.5825Cost 4Port 141 (Ethernet1/13)Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 28672 (priority 28672 sys-id-ext 30) Address 0018.bad8.58a5Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- --------------------------------Eth1/13 Root FWD 4 128.141 P2pEth1/14 Altn BLK 4 128.142 P2p

Spanning-Tree port States (IE: FWD, BLK)

STP Protocol = Rapid-PVST

Root Priority

Root STP ID (MAC Address)

Root Bridge or Root Port

This Bridges Priority and ID

Multiple Spanning Tree Configuration




Multiple Spanning Tree Configuration

n5500(config)# spanning-tree mst configuration

n5500(config-mst)# instance 1 vlan 10

n5500(config-mst)# instance 2 vlan 20

n5500(config-mst)# exit

n5500(config)# vlan 10,20

n5500(config)# spanning-tree mode mst

Enable MST:

Configure MST Instances:

Make sure you create the VLAN(s)

n5500(config)# spanning-tree mst 1 root secondary

n5500(config)# spanning-tree mst 2 root primary

Configure the MST Bridge Priority (Optional):

Change from the default RAPID-PVST mode to MST

MST is defined in IEEE 802.1s. MST maps multiple VLANs into “instances” that

maintain their own STP topology. MST improves STP scalability by reducing thenumber of STP instances and providing fault isolation between STP domains.

Map VLANs to MST Instances

Multiple Spanning Tree Verification




Multiple Spanning Tree Verification

n5500# show spanning-tree mst

##### MST0 vlans mapped: 1-9,11-4094Bridge address 0018.bad8.5825 priority 32768 (32768 sysid 0)Root this switch for the CISTRegional Root this switchOperational hello time 2 , forward delay 15, max age 20, txholdcount 6Configured hello time 2 , forward delay 15, max age 20, max hops 20

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------Eth1/25 Desg FWD 20000 128.153 P2p

##### MST1 vlans mapped: 10Bridge address 0018.bad8.5825 priority 28673 (28672 sysid 1)Root address 0018.bad8.58a5 priority 24577 (24576 sysid 1)

port Eth1/25 cost 20000 rem hops 19

Interface Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- --------------------------------Eth1/25 Root FWD 20000 128.153 P2p

MST1 with VLAN 10 mapped

Root Bridge information

Ports in MST1 instance

n5500# show spanning-tree mst ?<CR><0-4094> MST instance range, example: 0-3,5,7-9> Redirect it to a fileconfiguration MST current region configurationdetail Detailed informationinterface Spanning Tree interface status and configuration| Pipe command output to filter

Additional MST Options:

MST verification is very similar to Rapid-PVST. Several common show commandsexist for both protocols.

Data Center ArchitectureSpanning Tree - Layer 2 Loops




3/2 3/2

3/1 3/1

Switch 1 Switch 2

DST MAC 0000.0000.4444

DST MAC 0000.0000.4444

Spanning Tree Layer 2 Loops

Layer 2 topologies have sometimes proven an operational ordesign challenge

Spanning tree protocol itself is not usually the problem, it’s theexternal events that triggers the loop or flooding

L2 has had no native mechanism to dampen down a problem andno solution to provide link redundancy other than STP

Additional STP Features




Additional STP Features

The Cisco NX-OS supports several other Spanning-Tree Protocol features that

can be very useful to speed up convergence and reduce the likelihood for layer-2 loops. All of the following STP extensions are documented on Cisco.com.

STP Extensions:

BPDU Guard Shuts down an interface if a BPDU is received.

BPDU FilteringPrevents a device from sending or receiving BPDU’s onspecific ports.

Loop Guard Prevents a unidirectional-link from creating a bridging loop.

PVST Simulation Allows MST to interoperate with Rapid-PVST+.

Root Guard Prevents a specified port from becoming a root port.

BPDU Guard




BPDU Guard

Prevents a switch from being plugged in on an Edge port

Port will move to STP BKN (show spanning-tree vlan x)

Recommended on access layer Edge or Edge Trunk ports

Two options for deployment in NX-OS:

DCN-N5K1(config-if)# spanning-tree bpduguard enable

DCN-N5K1(config)# spanning-tree port type edge bpduguard default

Option 1: Enable on an interface:

Option 2: Enable by default on all Edge ports:

Global BPDU Filtering




dc11-5548-3(config)# spanning-tree port type edge bpdufilter default

dc11-5548-3(config)# interface ethernet 1/7dc11-5548-3(config-if)# spanning-tree port type edge trunk

dc11-5548-3# show spanning-tree interface ethernet 1/7 detail <snip>

The port type is edgeLink type is point-to-point by defaultBpdu filter is enabled by defaultBPDU: sent 11, received 0

Edge ports should have BPDU Guard enabled

If a BPDU is received port will transition toerr-disable state

Global BPDU Filter compliments BPDU Guard

On link up port will send 10-12 BPDUs and thenstop (in order to reduce CPU load)

If BPDU is received the port will err-disable

Improves CPU scaling in cases with trunk edgeports (e.g. VMWare servers)

This is NOT interface level BPDU Filtering

E

N NN N

E E

1. X-Connectedpatch cable

3. BPDU Guard

err-disablesedge port andprevents loop

2. BPDU Sent onLink-Up

4. BPDU are notsent once link is

up and active

Loop Guard




p

Prevents a port from moving to forwarding upon loss ofBPDUs

Puts the port into loop_inconsistent state until BPDUsare received again

Minimal benefit and not recommended for switchesrunning vPC

Deploy on access layer switches that are NOTconnected to the Agg layer using vPC

n5K-1(config)#spanning-tree loopguard default

Global Configuration Interface Configuration

n5k-1(config-if)#spanning-tree guard loop

Root Guard




Prevents Unwanted Changes to

STP Topology

Enable Root Guard on linksconnecting to access layer toprotect from edge switchesbecoming root and causing sub-optimal traffic flow

Forces Layer 2 LAN interface tobe a designated port. If portreceives a superior BPDU, RootGuard puts the interface into theroot-inconsistent (blocked) state

Channel the trunk betweenDistribution Switches so failuredoesn’t break topology interface Ethernet1/32

description dc10-5548-4switchport mode trunkswitchport trunk allowed vlan 15,98,180-183spanning-tree port type network

spanning-tree guard root

N

NN

RN R

N

N

Network portN

R Root GuardDesignated port

Root port

Alternate port

E Edge port

Should neverreceive a superior

BPDU

Root Bridge Secondary RootBridge

Should neverreceive a superior

BPDU

Spanning Tree Recommendations Port Configuration Overview

N Network port



g

Aggregation

Access

Data Center Core

B

RR

N N

- - -

-

-

- - -

RRRRRR

--

B

E

BB

E

B

E

Layer 3

Layer 2 (STP + Rootguard)

Layer 2 (STP + BPDUguard)

L

E

Secondary

Root

HSRP

STANDBY

Primary

Root

HSRP

ACTIVE

E

-

Primary

vPC

Secondary

vPC

vPC

Domain

Nexus

1000v

B

L

R

E

BPDU Guard

Loopguard

Rootguard

Edge or portfast port type

- Normal port type

Edge TrunkT

T

B

L

N5K config defaults




loopguard Spanning tree loopguard options

mode Spanning Tree operating modemst Multiple spanning tree configurationpathcost Spanning tree pathcost optionsport Spanning tree port optionsvlan VLAN Switch Spanning Trees

TM3# show spanning-tree summarySwitch is in rapid-pvst modeRoot bridge for: nonePort Type Default is disableEdge Port [PortFast] BPDU Guard Default is disabledEdge Port [PortFast] BPDU Filter Default is disabledBridge Assurance is disabledLoopguard Default is disabledPathcost method used is short

Name Blocking Listening Learning Forwarding STP Active--------------------------------------------- -------- --------- -------- ---------- ----------VLAN0001 0 0 0 2 2VLAN0213 0 0 0 3 3--------------------------------------------- -------- --------- -------- ---------- ----------2 vlans 0 0 0 5 5

Data Center Access ArchitectureSpanning Tree Design Considerations




p g g

Nexus-5500# show spanning-tree interface ethernet 100/1/48 detail

Port 560 (Ethernet100/1/48) of VLAN0100 is designated forwardingPort path cost 4, Port priority 128, Port Identifier 128.560Designated root has priority 24776, address 0023.ac64.73c3Designated bridge has priority 32968, address 000d.eca4.533cDesignated port id is 128.560, designated path cost 2Timers: message age 0, forward delay 0, hold 0Number of transitions to forwarding state: 1The port type is edgeLink type is point-to-point by defaultBpdu guard is enabledBPDU: sent 215784, received 0

BPDU Guard Is Enabled by Default andCannot be Disabled on FEX Server Ports

interface port-channel200switchport mode trunkswitchport trunk allowed vlan 200-202spanning-tree port type network

interface Ethernet1/33switchport mode trunkswitchport trunk allowed vlan 200-202udld enable

channel-group 200 mode active

interface Ethernet1/37switchport mode trunkswitchport trunk allowed vlan 200-202udld enablechannel-group 200 mode active

Nexus5500(config)# spanning-tree port type edge bpdufilter default

Bridge Assurance Requiresthe Port Type to be

Configured as ‘network’

Global BPDU Filter

Spanning Tree Path Cost Method




p g

Default in NX-OS is short (16-bit values) for link costs Using the Short method, a 10Gbps interface has a cost

of 2. A port-channel 20Gbps and above will have costof 1.

Recommended to change the Path Cost Method toLong in order to accommodate larger link sizes.

All switches must be configured to use the same PathCost Method

DCN-N5K1(config)# spanning-tree pathcost method long

Configuring N5K Ethernet Trunk Ports




cae-n5k(config)# int ethernet 1/3, ethernet1/11, ethernet 1/8, ethernet 1/12

cae-n5k(config-if)# switchport mode trunk

cae-n5k(config-if)# switchport trunk allowedvlan except 4093

cae-n5k(config-if)# no shut

‘encapsulation dot1q’ not required, it is the default. ISL is not supported

Verifying N5K Trunk Ports




cae-n5k# show runinterface Ethernet1/3switchport mode trunkswitchport trunk allowed vlan 1-3967,4048-4092

[snip]interface Ethernet1/8switchport mode trunkswitchport trunk allowed vlan 1-3967,4048-4092

[snip]

cae-n5k# show interface ethernet 1/3Ethernet1/3 is down (linkNotConnected)

Hardware is 10000 Ethernet, address is 000d.ec6b.cd4a (bia000d.ec6b.cd4a)

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is trunk[snip]

Port-channel Count




UPC/Carmel supports 48“hardware” port-channels

In Summary Every port can bea port-channel with either 5548or 5596

You can bundle up to 16 portsin a single port-channel

Portchannels configuredon FEX do not take anyresource from the Nexus5500 switch

More details in thefollowing slides

All ports can be part of a port-channel simultaneously

LACP Turn on LACP globally first




Turn on LACP globally first

switch(config)# feature lacp

Channel mode needs to be either “active” or “passive” andone side has to be “active”

No cisco PAgP supported

Switch 1 mode Switch 2 mode Port added to EtherChannel

active passive Yes

passive active Yes

active active Yes

passive passive Noactive or passive on No

on active or passive No

on on Yes but no LACP negotiation

Creating EtherChannel




Best practice is to use LACP in active mode on both sides ofthe link

Channel mode Description

active Initiates LACP negotiation

passive Responds to LACP negotiation

on No LACP. Adds port to EtherChannel

Three channel group modes: active , passive and on.

Switch(conf)#interface e1/1

switch(config-if)# channel-group 1 mode ?

active Set channeling mode to ACTIVE

on Set channeling mode to ON

passive Set channeling mode to PASSIVE



Etherchannel - Force Keyword




If the physical port parameters do not match that of theport-channel, the interface cannot be joined to theEtherchannel

You could try and fix the inconsistency, or you can

force the interface into the channel-group The config is pushed down from the port-channel to the

physical interface

switch(config)# int ethernet 1/2switch(config-if)# channel-group 1 force mode active

Port-Channel (LACP) Verification




n5500# show port-channel summaryFlags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)s - Suspended r - Module-removedS - Switched R - RoutedU - Up (port-channel)

--------------------------------------------------------------------------------Group Port- Type Protocol Member Ports

Channel--------------------------------------------------------------------------------1 Po1(RU) Eth LACP Eth1/13(P) Eth1/14(P)

n5500# show port-channel trafficChanId Port Rx-Ucst Tx-Ucst Rx-Mcst Tx-Mcst Rx-Bcst Tx-Bcst------ - -------- ------- ------- ------- ------- ------- -------

1 Eth1/13 100.00% 100.00% 94.16% 71.15% 100.00% 100.00%1 Eth1/14 0.0% 0.0% 5.83% 28.84% 0.0% 0.0%

n5500# show port-channel usageTotally 1 port-channel numbers used====================================Used : 1Unused: 2 - 4096

Usage:

Traffic Distribution:

Port-Channel Summary:

Receive and transmit percentages

1 LaCP Port-Channel with 2 members

Port-Channel (LACP) Statistics




n5500# show lacp neighborFlags: S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs

A - Device is in Active mode P - Device is in Passive modeport-channel1 neighbors

Partner's informationPartner Partner Partner

Port System ID Port Number Age FlagsEth1/13 32768,0-18-ba-d8-58-250x10d 365 SA

LACP Partner Partner PartnerPort Priority Oper Key Port State32768 0x0 0x3d

Partner's informationPartner Partner Partner

Port System ID Port Number Age FlagsEth1/14 32768,0-18-ba-d8-58-250x10e 284 SA

LACP Partner Partner PartnerPort Priority Oper Key Port State32768 0x0 0x3d

n5500# show lacp countersLACPDUs Marker Marker Response LACPDUs

Port Sent Recv Sent Recv Sent Recv Pkts Err-------------------------------------------------------------------------------------------------port-channel1Ethernet1/13 34 21 0 0 0 0 0 Ethernet1/14 20 19 0 0 0 0 0 PDU errors

Successful PDU’s

Neighboring device isconfigured for “Active” modeand sending “Slow” PDU’s

Hash algorithm CLI CLI to select the fields of frame into the hash calculation




CLI to select the fields of frame into the hash calculationNexus5500(config)# port-channel load-balance ethernet ?

destination-ip Destination IP address

destination-mac Destination MAC address

destination-port Destination TCP/UDP port

source-destination-ip Source & Destination IP address

source-destination-mac Source & Destination MAC address

source-destination-port Source & Destination TCP/UDP port

source-ip Source IP address

source-mac Source MAC address

source-port Source TCP/UDP port

Check the hash algorithmNexus5500# sh port-channel load-balance

Port Channel Load-Balancing Configuration:

System: destination-mac

Port Channel Load-Balancing Addresses Used Per-Protocol:

Non-IP: destination-mac

IPv4: destination-mac

IPv6: destination-mac

Port-channel Load Balancing

CLI to help the user know about the port Nexus 5K picks




CLI to help the user know about the port Nexus 5K picksfor load balancing on a Ethernet port-channel.

show port-channel load-balance [forwarding-path interface port-channel channel-number ] {dst-ip | dst-mac | dst-ipv6 | src-dst-ip | l4-src-port | l4-dst-port | src-ip | src-mac | src-ipv6 }

5548-2# sh port-channel load-balance

Port Channel Load-Balancing Configuration:System: source-dest-ip

Port Channel Load-Balancing Addresses Used Per-Protocol:Non-IP: source-dest-macIP: source-dest-ip source-dest-mac

DCN-N5k2# show port-channel load-balance forwarding-path interface po20 src-interface e1/1 vlan 49 src-ip 10.122.49.10 dst-ip172.18.84.183Missing params will be substituted by 0's.Load-balance Algorithm on switch: source-dest-ipcrc8_hash: 148 Outgoing port id: Ethernet1/17Param(s) used to calculate load-balance:

dst-ip: 172.18.84.183src-ip: 10.122.49.10dst-mac: 0000.0000.0000src-mac: 0000.0000.0000

Example:

Configuring N5K Port Channels




cae-n5k(config)# conf tcae-n5k(config)# interface ethernet 1/3, ethernet 1/11cae-n5k(config-if)# channel-group 5 force mode active

Ethernet1/3 Ethernet1/11 added to port channel 5

cae-n5k(config-if)# interface port-channel 5cae-n5k(config-if)# switchport mode trunkcae-n5k(config-if)# switchport trunk allowed vlan except 4093cae-n5k(config-if)# no shut




Virtual Port Channel (vPC)

Virtual Port-ChannelFeature Overview




Allow a single device to use aport channel across twoupstream switches

Eliminate STP blocked ports

Uses all available uplink

bandwidth Dual-homed server operate in

active-active mode

Provide fast convergence uponlink/device failure

Virtual Port Channel

L2

SiSi SiSi

Increased BW with vPC

Non-vPC vPC

Physical Topology Logical Topology

Feature OverviewHow does vPC help with STP?




Before vPC

STP blocks redundant uplinks

VLAN based load balancing

Loop Resolution relies on STP

Protocol Failure

With vPC

No blocked uplinks

Lower oversubscriptionEtherChannel load balancing (hash)

Loop Free Topology

PrimaryRoot

SecondaryRoot

vPC Terminology onN5K-N2K

vPC peer – a vPC switch, one of a pair

vPC member port – one of a set of ports

(port channels) that form a vPC




(port channels) that form a vPC

vPC – the combined port channel

between the vPC peers and thedownstream device

vPC peer link – Link used to synchronize

state between vPC peer devices, must

be 10GbE. Also carry

multicast/broadcast/flooding traffic and

data traffic in case of vpc member portfailure

vPC peer keepalive link – the peer

keepalive link between vPC peer

switches. It is used to carry heartbeat

packets

CFS – Cisco Fabric Services protocol,used for state synchronization and

configuration validation between vPC

peer devices

vPC peer

keepalive linkvPC peer link

vPC peer

vPCvPC

member

port

How to Configure vPC




vPC configuration on the Cisco Nexus 5500 Series includes these steps:

• Enable the vPC feature.

• Create a vPC domain and enter vpc-domain mode.

• Configure the vPC peer keepalive link.

• (Optional) Configure system priority.

• (Optional) Configure vPC role priority. • Create the vPC peer link.

• Move the PortChannel to vPC.

How to Configure vPCEnable VPC feature on both N5Ks




Configure VPC domain

N5K-1(config)# feature vpc

N5K-1(config)# vpc domain 1

VPC domain ID is an unique number (from 1 to 1000).

Note: The same VPC Domain ID will be configured on the other Nexus5500.

Note: Each pair of devices in the same layer 2 domain running vPC mustalways use a unique Domain ID.

How to Configure vPCConfigure system-priority (optional)




N5K-1(config-vpc-domain)# system-priority 4000

Enter the system priority that you want for the specified vPC domain. The range ofvalues is 1 to 65535. The default value is 32667.

You should manually configure the vPC system priority when you are runningLink Aggregation Control Protocol (LACP) to help ensure that the vPC peer

devices are the primary devices on LACP.

When you manually configure the system priority, make sure that you configurethe same priority value on both vPC peer devices. If these values do not match,vPC will not be activated.

How to Configure vPCConfigure VPC role priority (optional)




N5K-1(config-vpc-domain)# role priority 8192

Each VPC member has a role (primary or secondary), it is calculated by the rolepriority value plus local system mac, the lowest value will be elected as primary.The default role priority is 32768.

Configure one N5500 as primary and the other as secondary by setting rolepriority.

Once the election is completed, the VPC role will not change unless the VPC peerlink connection is reset.

Warning: vPCs will be flapped on current primary vPC switch while attempting

role changeNote: VPC Role will indicate “none established” and have a vPC local role-priorityof zero in the ‘show vpc role’ command output until the VPC peer link comes up.

How to Configure vPCConfigure the VPC peer keepalive link




N5K-1(config-vpc-domain)# peer-keepalive destination 14.1.83.214source 14.1.83.213 vrf management

It is recommended as best practice to use a separate L3 link for VPC keepaliveexchange and to put the peer keepalive link in a separate VRF.

Typically we will use interface mgmt0 with IP address 14.1.83.213/24 which is usesvrf management for the peer-keepalive link.

For the destination address, use the mgmt0 IP address of the other N5K.

How to Configure vPCConfigure the VPC peer link




Configure interfaces e2/1 and e2/2 as members of PO10 and configure PO10 asthe peer link.

N5K-1(config-if)# int e2/1-2N5K-1(config-if-range)# switchport mode trunkN5K-1(config-if-range)#channel-group 10N5K-1(config)# int po10N5K-1(config-if)# switchport mode trunk

N5K-1(config-if)# vpc peer-link

First create a port-channel interface, in this example we use PO10 for the peer-link.The peer-link must be a 10GE link between the VPC members.Configure trunking on the L2 port-channel interfaces between the two Nexus 5500.The supported channeling mode is On (which is the default) or LACP (i.e. mode active).The port mode for interface port-channel 10 is configured as trunk.

NOTE: Spanning tree port type is changed to "network" port type on vPC peer-link.This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP BridgeAssurance (which is enabled by default) is not disabled.NOTE: The port-channel for the peer link and the peer keepalive link will not come up until theother N5500 is also configured identically.

How to Configure vPC (cont’d) Move the Downstream PortChannel to vPC




interface port-channel channel-number vpc number

switch(config)#interface e1/1switch(config-if)channel-group 20switch(config-if)# interface port-channel 20switch(config-if)# vpc 100

Add the interface to the PortChannel and then move the PortChannel to the vPCto connect to the downstream device. The vPC number ranges from 1 to 4096.The vPC number does not need to match the PortChannel number, but it mustmatch the number of the vPC peer switch for that vPC bundle.A PortChannel is needed even if there is only one member interface for thePortChannel. When there is only one member for the PortChannel, the hardwarePortChannel resource will not be created.

Configuring vPC




N5K(config)# feature lacpN5K(config)# feature vpc

N5K(config)# vpc domain 1N5K(config-vpc-domain)# peer-keepalive destination 10.20.0.191 source 10.20.0.190Note:--------:: Management VRF will be used as the default VRF ::--------

N5K(config)# interface ethernet 3/1,ethernet 4/1N5K(config-if-range)# switchportN5K(config-if-range)# switchport mode trunkN5K(config-if-range)# switchport trunk allowed vlan 9,11-14N5K(config-if-range)# channel-group 10 mode activeN5K(config-if-range)# no shut

N5K(config-if-range)# interface port-channel 10N5K(config-if)# vpc peer-link Please note that spanning tree port type is changed to "network" port type on vPC peer-link.This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP Bridge Assurance(which is enabled by default) is not disabled.

N5K(config)# interface ethernet 3/2,ethernet 4/2N5K(config-if-range)# switchportN5K(config-if-range)# switchport mode trunkN5K(config-if-range)# switchport trunk allowed vlan 11-14N5K(config-if-range)# channel-group 20 mode activeN5K(config-if-range)# no shut

N5K(config-if-range)# interface port-channel 20N5K(config-if)# vpc 20

The following example enables vPC with LaCP on one side of the vPCDomain. The same config is required on the other vPC Domain member.

Enable the LaCP and vPC features first!

Configure the vPC Peer-Link

Configure the downstream link

Configure the

vPC domain and

keep-alive link

Define the vPC Port-Channel # for the downstream link

Define the vPC Peer-Link



Virtual Port-ChannelDomain ID




vPC System MAC is used for both LACP System Identifier and STP bridgeID. Uses IETF assigned range of 00:23:04:ee:be:00 -> 00:23:04:ee:c1:ff.

vPC Domain ID is encoded in the vPC System MAC within the last octetand the trailing 2 bits of the previous octet

10 bits

vPC Domain ID

System Identifier used by LACP to identifylinks connected to the same neighbor

Duplicate System ID would result in anLACP error condition

Could also result in two switches with thesame STP Bridge ID

You MUST use a unique vPC domain ID foreach pair of adjacent vPC peers!

vPC Domain 20

vPC Domain 10

Note: This also applies to VSS domains as well. Always use a unique domain

ID when connecting a vPC domain to VSS

Virtual Port Channel (vPC)802.3ad & LACP – System MAC

LACP i hb d h S ID f b h PC




dc11-4948-1#sh lacp neighbor <snip>

LACP port Admin Oper Port PortPort Flags Priority Dev ID Age key Key Number StateGi1/33 SA 32768 0023.04ee.be14 9s 0x0 0x801E 0x4104 0x3DGi1/34 SA 32768 0023.04ee.be14 21s 0x0 0x801E 0x104 0x3D

dc11-5548-2# sh vpc role <snip>vPC system-mac : 00:23:04:ee:be:14vPC system-priority : 1024vPC local system-mac : 00:0d:ec:a4:5f:7cvPC local role-priority : 32667

dc11-5548-1# sh vpc role <snip>vPC system-mac : 00:23:04:ee:be:14vPC system-priority : 1024vPC local system-mac : 00:0d:ec:a4:53:3cvPC local role-priority : 1024

dc11-4948-1

LACP neighbour needs to see the same System ID from both vPC peers

The vPC ‘system-mac’ is used by both vPC peers

dc11-5548-1 dc11-5548-2

1/331/34

Virtual Port Channel (vPC)802.3ad & LACP – System MAC

PC f ti i d d t d i ll




dc11-4948-2#sh lacp neighbor <snip>

LACP port Admin Oper Port PortPort Flags Priority Dev ID Age key Key Number StateGi1/4 SA 32768 000d.eca4.533c 8s 0x0 0x1D 0x108 0x3DGi1/5 SA 32768 000d.eca4.533c 8s 0x0 0x1D 0x108 0x3D

dc11-5548-1# sh vpc role <snip>vPC system-mac : 00:23:04:ee:be:14vPC system-priority : 1024vPC local system-mac : 00:0d:ec:a4:53:3cvPC local role-priority : 1024

dc11-4948-1

dc11-5548-1 dc11-5548-2

vPC peers function as independent devices as well as peers

Local ‘system-mac’ is used for all non vPC PDUs (LACP, STP, …)

1/4 1/5

dc11-4948-2

MCEC (vPC)Etherchannel

Regular (non vPC)Etherchannel

Virtual Port-ChannelPeer Keepalive Link

P K li id f b d




Peer Keepalive provides an out-of-bandheartbeat between vPC peers

Purpose is to detect and resolve roles ifa Split Brain (Dual Active) occurs

Messages sent on 1 second interval with5 second timeout

3 second hold timeout on peer-link lossbefore triggering recovery

Should no t be carried over the Peer-Link

Keepalives sourced and destined to themgmt0 interface

Keep-alives can be routed over L3infrastructure

dc11-5548-1(config)# vpc domain 20dc11-5548-1(config-vpc-domain)# peer-keepalive destination 172.26.161.201 source172.26.161.200 vrf management

Note:--------:: Management VRF will be used as the default VRF ::--------

Peer Keepalivecarried over the

OOB managementnetwork

int mgmt 0

Virtual Port-ChannelvPC Peer Link




dc11-5548-1(config)# interface port-channel 20dc11-5548-1(config-if)# switchport mode trunkdc11-5548-1(config-if)# switchport trunk native vlan 100dc11-5548-1(config-if)# switchport trunk allowed vlan 100-105dc11-5548-1(config-if)# vpc peer-linkdc11-5548-1(config-if)# spanning-tree port type network

vPC PeerLink

Peer Link carries both vPC data andcontrol traffic between peer switches

Carries any flooded and/ororphan port traffic

Carries STP BPDUs, HSRPHellos, IGMP updates, etc.

Carries Cisco Fabric Servicesmessages (vPC control traffic)

Minimum 2 x 10GbE ports

It is no t recommended to share vPCand non-vPC traffic on the same PeerLink

Virtual Port Channel (vPC)vPC Roles

Role is defined under the domain




dc11-5548-3(config-vpc-domain)# role priority ? <1-65535> Specify priority value

dc11-5548-3# sh vpc <snip>vPC role : secondary, operational primary

Role is defined under the domainconfiguration

Lower priority wins if not, lower system macwins

Role is non-preemptive so Operational Role iswhat matters

Operational Role may different from thepriorities configured under the domain

vPC Role defines which of the two vPC peersprocesses BPDUs

Role matters for the behavior with peer-linkfailures!

Secondary(but may beOperational

Primary)

Primary (but may beOperational Secondary)

Virtual Port-ChannelvPC Control Fabric – Cisco Fabric Services




dc11-5548-2# show CFS statusDistribution : EnabledDistribution over IP : DisabledIPv4 multicast address : 239.255.70.83IPv6 multicast address : ff15::efff:4653Distribution over Ethernet : Enabled

CiscoFabric

Services

CFSoE

Cisco Fabric Services provides thecontrol plane synchronization betweenvPC peers

Configuration validation/comparison

MAC member port synchronization

vPC member port status

IGMP snooping synchronization vPC status

Highly Reliable - Inherited from MDS

CFS messages are encapsulated instandard Ethernet frames (with CoS 6)

Virtual Port-ChannelvPC Control Plane – Cisco Fabric Services

d 7k2 d 2




dca-n7k2-vdc2

dc11-5548-1# show running int port-channel 201version 4.1(3)N1(1)

interface port-channel201switchport mode trunkswitchport trunk native vlan 100switchport trunk allowed vlan 100-105vpc 201spanning-tree port type network



dca-n7k2-vdc2# sh run interface port-channel 201

version 4.1(5)

interface port-channel201switchport mode trunkswitchport trunk allowed vlan 100-105spanning-tree port type networklogging event port link-statuslogging event port trunk-status

vPC supports standard 802.3ad portchannels from upstream and ordownstream devices

Recommended to enable LACP

“channel-group 201 mode active”

dc11-5548-2dc11-5548-1



Virtual Port Channel - vPCvPC Control Plane – Type 1 Consistency Check

T 1 C i t Ch k



167© 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential – Internal Use Only

Type 1 Consistency Checks areintended to prevent network failures

Incorrectly forwarding of traffic

Physical network incompatibilities

vPC will be suspended

dc11-5548-2# show vpc briefLegend:

(*) - local vPC is down, forwarding via vPC peer-link <snip>vPC status----------------------------------------------------------------------------id Port Status Consistency Reason Active vlans------ ----------- ------ ----------- -------------------------- -----------201 Po201 up failed vPC type-1 configuration -

incompatible - STPinterface port guard -Root or loop guardinconsistent

dc11-5548-1# sh run int po 201


dc11-5548-2# sh run int po 201

interface port-channel201switchport mode trunkswitchport trunk native vlan 100switchport trunk allowed vlan 100-105vpc 201spanning-tree port type networkspanning-tree guard root

Virtual Port Channel - vPCvPC Control Plane – Type 2 Consistency Check




Type 2 Consistency Checks are

intended to prevent undesiredforwarding

vPC will be modified in certain cases(e.g. VLAN mismatch)

dc11-5548-1# show vpc brief vpc 201

vPC status----------------------------------------------------------------------------id Port Status Consistency Reason Active vlans------ ----------- ------ ----------- -------------------------- -----------201 Po201 up success success 100-104

2009 May 17 21:56:28 dc11-5548-1 %ETHPORT-5-IF_ERROR_VLANS_SUSPENDED: VLANs 105 on Interface port-channel201 are being suspended. (Reason: Vlan is not configured on remote vPC interface)

dc11-5548-1# sh run int po 201version 4.1(3)N1(1)


dc11-5548-2# sh run int po 201version 4.1(3)N1(1)


Virtual Port Channel - vPCvPC Control Plane – Global Consistency Checks




Don’t forget to keep global configurationin sync

Any configuration that could causean error in forwarding (e.g. loop) willdisable all affected interfaces

As an example if you make a change toan MST region you must make it on‘both’ peers

Solution: define MST region mappingsfrom the very beginning of thedeployment, for ALL VLANs, the onesthat exist as well as the ones that have

not yet been created Defining a region mapping is orthogonal

to creating a VLAN

vPCvPC vPC

mst regionvlans 1-5, 12

mst regionvlans 1-5, 10

This behavior equally applies to Nexus7000 and Nexus 5500 when configured

as vPC peers

Virtual Port Channel - vPCvPC Consistency Check – Global Configuration Parameters




GlobalSpanning

TreeParametersneed to beconsistent

Global QoSParameters

need to beconsistent

Global Parameters

are type 1

Global vs. Interface Consistency Check

Global consistency check failure for type 1 will result all vPC suspended




n5k-1# show vpc consistency-parameters interface port-channel 200

Legend:Type 1 : vPC will be suspended in case of mismatch

Name Type Local Value Peer Value------------- ---- ---------------------- -----------------------

STP Port Type 1 Default DefaultSTP Port Guard 1 None NoneSTP MST Simulate PVST 1 Default Defaultlag-id 1 [(7f9b, [(7f9b,

0-23-4-ee-be-64, 80c8, 0-23-4-ee-be-64, 80c8,0, 0), (8000, 0, 0), (8000,

0-1e-13-15-7-40, 1, 0, 0-1e-13-15-7-40, 1, 0,0)] 0)]

mode 1 active activeSpeed 1 10 Gb/s 10 Gb/s

Duplex 1 full fullPort Mode 1 trunk trunkNative Vlan 1 1 1Allowed VLANs - 1-999,1001-3967,4048-4093 1-3967,4048-4093

n5k-1#

Type 2 consistency

check parameter

Interface level consistency check failure only affects the involved interfaces




vPC Forwarding

Virtual Port Channel - vPCvPC provides optimized forwarding

dca-n7k2-vdc2 vPC forwards only on locally connected




dca-n7k2-vdc2





dca-n7k2-vdc2# sh run interface port-channel 201version 4.1(5)


vPC forwards only on locally connectedmembers of the port channel if any exist

(same principle as VSS)

Multiple topology choices

Square

Full Mesh dc11-5548-2dc11-5548-1

Virtual Port Channel - vPCvPC Forwarding- Unicast Learning

vPC maintains layer 2 topology

MAC_C

5




vPC maintains layer 2 topologysynchronization via CFS

Copies of flooded frames are sent acrossthe vPC-Link in case any single homeddevices are attached

Frames received on the vPC-Link are notforwarded out vPC ports

2

3

1. Host MAC_A send packet to MAC_C2. FEX runs hash algorithm to select one fabric uplink3. N5K-1 learns MAC_A and flood packets to all ports

(in that VLAN). A copy of the packet is sent acrossthe peer link

4. N5K-2 floods the packet to any port in the VLANexcept the vPC member ports to prevent duplicated

packets5. N7K-1 and N7K-2 repeat the same forwarding logic6. N5K-1 updates the the MAC address learned on the

vPC port on N5K-2 via CFS MAC_A

1

5

N5K-1 N5K-2

CFS

6

4

Double SidedvPC

Virtual Port Channel - vPCvPC Forwarding- Unicast Learning

Traffic is forwarded if destination address is

MAC_C

1




Traffic is forwarded if destination address isknown (both switches MAC address tables

populated)

Always forward via a locally attachedmember of a vPC if it exists

1. Host MAC_C send packet to MAC_A2. N7K-2 forwards frame based on learned

MAC address3. N5K-2 forwards frame based on learned

MAC address

MAC_A

N5K-1 N5K-2

2

3

N5K-1# sh mac-address-table vlan 101 VLAN MAC Address Type Age Port---------+-----------------+-------+---------+-----101 001b.0cdd.387f dynamic 0 Po30

101 0023.ac64.dda5 dynamic 30 Po201Total MAC Addresses: 4

N5K-2# sh mac-address-table vlan 101 VLAN MAC Address Type Age Port---------+-----------------+-------+---------+-----101 001b.0cdd.387f dynamic 0 Po30101 0023.ac64.dda5 dynamic 30 Po201Total MAC Addresses: 4




vPC Failure Scenarios

on N55K

vPC Failure ReactionvPC member port failure




vPCPrimary

vPCSecondry

MAC_A

When vPC member port fails,N5k updates the MAC table forall the address points to theaffected vPC bundle

On the right N5k, MAC_A pointsto peer link “Po1” after thefailure occurs

Before the failure, MAC_Apoints to Po2

vPC member port status changeis updated to peer via CFSmessage

Po1

Po2

vPC Failure Reaction (FEX Straight Thru)Peer-link failure

vPCmember port




vPCPrimary

vPCSecondry

When peer link fails,secondary vpc peer switchsuspends all its vpc memberports

vPC secondary detectsprimary switch is alivethrough peer keepalive link

vPCmember portis suspended

vPCmember portis suspended

vPC Failure Reaction (FEX A/A)Peer-link failure




vPCPrimary

vPCSecondry

When peer link fails, secondaryvpc peer switch suspends all itsvpc member ports

FEX will be only connected to

primary switch.

FEX ports remain up

vPC Failure Reactionkeepalive link failure




vPCPrimary

vPCSecondry

Don’t care as long as peer linkis up

vPC Double Failure ReactionPeer-link failure followed by keepalive link failure




vPCPrimary

vPCSecondry

When peer link fails, secondaryvpc peer switch suspends all itsvpc member ports

Keepalive failure has no impact

vPC Double Failure ReactionPeer-link failure followed by keepalive link failure




vPCPrimary

vPCSecondry

With the failure of both peer linkand peer keepalive link, FEX willbe connected ONLY to primaryvPC switch.

vPC Double Failure ReactionKeepalive link failure followed by Peer Link failure




vPCPrimary

vPCSecondry

With the peer keepalive linkdown, vPC secondary switchdoesn’t know if the primary isalive when the peer link fails

Both switch run as primaryswitch

STP ensures no loop




vPC Enhancements

QoS Config Checks have been lowered to Type-2NX-OS 5.0(2)N1(1)

S l f t h th i fi ti t l d f T 1 t T 2




tc-nexus5548-1# show vpc consistency-parameters global

Name Type Local Value Peer Value

------------- ---- ---------------------- -----------------------

QoS 2 ([], [3], [], [], [], ([], [3], [], [], [],

[]) [])

Network QoS (MTU) 2 (1538, 2240, 0, 0, 0, (1538, 2240, 0, 0, 0,

0) 0)

Network Qos (Pause) 2 (F, T, F, F, F, F) (F, T, F, F, F, F)

Input Queuing (Bandwidth) 2 (50, 50, 0, 0, 0, 0) (50, 50, 0, 0, 0, 0)

Input Queuing (Absolute 2 (F, F, F, F, F, F) (F, F, F, F, F, F)

Priority)

Output Queuing (Bandwidth) 2 (50, 50, 0, 0, 0, 0) (50, 50, 0, 0, 0, 0)

Output Queuing (Absolute 2 (F, F, F, F, F, F) (F, F, F, F, F, F)

Several features have the misconfiguration type lowered from Type 1 to Type 2

Configurations can be synched between vPC member ports by using the Config-syncfeature

vPC graceful type-1 checks

S2-SecondaryS1 -Primary

Keepalive

NX-OS 5.0(2)N2(1)




CE-1

vPC peer-link

vPC 1

po1

vPC member ports on S1 and S2 shouldhave identical parameters (MTU, speed,…)

Any inconsistency in such parametersis Type1. As a consequence, all vlans on

both vpc legs are brought down in suchinconsistency

With graceful type-1 check, onlySecondary vPC members are broughtdown. vPC member ports on primary

peer device remain up

S1(config-vpc-domain)# gracefulconsistency-check

S2(config-vpc-domain)# gracefulconsistency-check

Graceful Type-1 check enabled bydefault.

Type-1Inconsistency



vPC Auto-Recovery




If enabled (default is disabled)

On switch reload, vPC listensto switch online notification(indicates all LCs are up)

Starts reload-delay timer(user configurable), default 240

secondsIf peer-link port comesphysically up or peer-keepalive works, stop timer, wait forpeer adjacency to form

Normal behavior, peer

presumed alive

S2S1

po1 po2

vPC peer-link

S4

A

C

B

vPC 2vPC 1

Primary Secondary

vPC Auto-Recovery




If enabled

If after reload-delay timerexpiration, no peer-keep alive orno peer-link up received

Assume primary STP role

Assume primary LACP role(internal role between LACP and

vPC, currently based on switchmac comparison)

Reinitialize vPCs

On vPC port bringup, consistencycheck is bypassed for vPCs

S2S1

po1 po2

vPC peer-link

S4

A

C

B

vPC 2vPC 1

Primary

vPC auto-recovery


Keepalive


vPC peer-link

Keepalive

1

2

NX-OS 5.0(2)N2(1)




CE-1

vPC peer-link

vPC 1

po1

CE-1

vPC 1

po1

CE-1

S1 -Primary

vPC peer-link

vPC 1

po1

Keepalive S2-OperationalPrimary

1. vPC peer-link goes down : vPCsecondary peer device shuts all itsvPC member ports

2. S1 goes down. S2 receive no moremessages on vPC peer-keepalive link

3. After 3 consecutive keepalivetimeouts, vPC secondary peer device(S2) changes role and brings up itsvPC.

3

S1(config-vpc-domain)# auto-recoveryS2(config-vpc-domain)# auto-recovery



Virtual Port Channel – vPCvpc orphan-por t suspend – new knob

Supported only on physical Ethernet

NX-OS 5.0(3)N2(1)




interfaces

Suspends/Disables orphan ports onvPC secondary switch during peer-link failure

Orphan ports are re-enabled alongwith vPCs on peer-link recovery

“show vpc orphan-port” to displayconfigured orphan ports

Best practices

Eliminate orphan ports withdual-homing when you can

If not, identify orphan ports anduse new configuration knob 1. Orphan Ports are disabled

Primary Secondary

2. Standby link takes over

PC T bl h ti




vPC Troubleshooting

vPC troubleshooting

Basic checks




Nexus# sh vpc...

vPC domain id : 111

Peer status : peer adjacency formed ok

vPC keep-alive status : peer is alive

Configuration consistency status: success

vPC role : primary

vPC Peer-link status---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ --------------------------------------------------

1 Po100 up 1 34-35

vPC status

----------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

-- ---- ------ ----------- -------------------------- ------------

1 Po1 up success success 34-35

vPC troubleshooting

Config check (vPC default parameters not shown)




g ( p )

Nexus# sh run vpc

version 4.1(5)

feature vpc

vpc domain 111

peer-keepalive destination 7.7.7.77source 7.7.7.7 vrf v1

interface port-channel1

vpc 1


vpc peer-link

Nexus-dg# sh run vpc

version 4.1(5)

feature vpc

vpc domain 111

peer-keepalive destination 7.7.7.7source 7.7.7.77 vrf v1


vpc 1


vpc peer-link

vPC troubleshooting

vPC peer-keepalive check




Nexus# show vpc peer-keepalive


--Send status : Success

--Last send at : 2009.06.19 00:41:15 589 ms

--Sent on interface : Eth2/35

--Receive status : Success--Last receive at : 2009.06.19 00:41:14 580 ms

--Received on interface : Eth2/35

--Last update from peer : (1) seconds, (9) msec

vPC Keep-alive parameters

--Destination : 7.7.7.77

--Keepalive interval : 1000 msec

--Keepalive timeout : 5 seconds--Keepalive hold timeout : 3 seconds

--Keepalive vrf : v1

--Keepalive udp port : 3200

--Keepalive tos : 192

vPC timers check

vPC troubleshooting

vPC peer-keepalive statistics




Nexus# show vpc statistics peer-keepalive


vPC keep-alive statistics

----------------------------------------------------

peer-keepalive tx count: 9773

peer-keepalive rx count: 8985

average interval for peer rx: 991

Count of peer state changes: 159

p p

vPC troubleshooting

vPC role (primary / secondary) and system-mac




(p y y) y

Nexus# show vpc role

vPC Role status

----------------------------------------------------

vPC role : primary

Dual Active Detection Status : 0

vPC system-mac : 00:23:04:ee:be:6f

vPC system-priority : 32667vPC local system-mac : 00:1b:54:c2:42:41

vPC local role-priority : 32667

vPC troubleshooting

Global consistency parameters




Nexus# show vpc consistency-parameters global

Legend:

Type 1 : vPC will be suspended in case of mismatch


------------- ---- ---------------------- -----------------------

STP Mode 1 Rapid-PVST Rapid-PVST

STP Disabled 1 None NoneSTP MST Region Name 1 "" ""

STP MST Region Revision 1 0 0

STP MST Region Instance to 1

VLAN Mapping

STP Loopguard 1 Disabled Disabled

STP Bridge Assurance 1 Enabled Enabled

STP Port Type 1 Normal Normal

STP MST Simulate PVST 1 Enabled Enabled

Allowed VLANs - 1,34-35,51,69-70,99,20 1-2,34-35

Note currently it is user responsibility to ensure same L3 interfaces arepresent and are in the same operational state on both peer devices

vPC troubleshooting

Interface consistency parameters




Nexus# show vpc consistency-parameters interface port-channel 1

Legend:

Type 1 : vPC will be suspended in case of mismatch


------------- ---- ---------------------- -----------------------

STP Port Type 1 Default Default

STP Port Guard 1 None NoneSTP MST Simulate PVST 1 Default Default

lag-id 1 [(7f9b, [(7f9b,

0-23-4-ee-be-6f, 8001, 0-23-4-ee-be-6f, 8001,

0, 0), (8000, 0, 0), (8000,

0-12-da-65-9e-c0, 1, 0-12-da-65-9e-c0, 1,

0, 0)] 0, 0)]

mode 1 active active

Speed 1 1000 Mb/s 1000 Mb/s

Duplex 1 full full

Port Mode 1 trunk trunk

Native Vlan 1 2 2

MTU 1 1500 1500

Allowed VLANs - 34-35 34-35

VLAN Err-Disabled Status On Trunk

PKN5K-1 N5K-2

i t 20




PO20

PO10

PL

PK

N5K-1# show int po20 trunk

Port Native Status PortVlan Channel

Po20 1 trunking --

Port Vlans Allowed on TrunkPo20 1,10-11,100,176,208-209,3001

Port Vlans Err-disabled on TrunkPo20 100 VLAN shows up as err-disabled

int po20

switchport trunk allowedvlan 1,10-11,176,208-209,3001

int po20switchport trunk allowed

vlan 1,10-11,100,176,208-209,3001

VL100 is missing on

vPC Peer Link

VL100 must be in theallowed list on bothN5K-1 and N5K-2 forerr-disabled to clear!

Type-1 Global Inconsistency

PKN5K-1 N5K-2

N5K-2# spanning-tree loopguard default




PO20

PO10

PL

PK

N5K-1# show vpc briefLegend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 3Peer status : peer adjacency formed okvPC keep-alive status : peer is aliveConfiguration consistency status: failed

Configuration consistency reason: vPC type-1 configurationincompatible - STP global loop guard inconsistentType-2 consistency status : failedType-2 consistency reason : SVI type-2 configuration incompatiblevPC role : secondaryNumber of vPCs configured : 4Peer Gateway : EnabledPeer gateway excluded VLANs : -Dual-active excluded VLANs : -


id Port Status Active vlans-- ---- ------ --------------------------------------------------1 Po10 up -

vPC status----------------------------------------------------------------------id Port Status Consistency Reason Active vlans-- ---- ------ ----------- ------ ------------20 Po20 down* failed Global compat check failed -

p g pg

All vPCMember Portsare takendown!

N5K-1# show port-channel sum int p20Flags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)s - Suspended r - Module-removedS - Switched R - RoutedU - Up (port-channel)M - Not in use. Min-links not met


Channel--------------------------------------------------------------------------------20 Po20(SD) Eth LACP Eth2/17(D)

Type-1 Global Inconsistency




Type-1 Interface Inconsistency

PKN5K-1 N5K-2

N5K-1f)# show vpc brief




PO20

PO10

PL

Legend:(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 3Peer status : peer adjacency formed okvPC keep-alive status : peer is aliveConfiguration consistency status: successType-2 consistency status : failedType-2 consistency reason : SVI type-2 configuration incompatiblevPC role : primaryNumber of vPCs configured : 4Peer Gateway : EnabledPeer gateway excluded VLANs : -

Dual-active excluded VLANs : -

vPC Peer-link status---------------------------------------------------------------------id Port Status Active vlans-- ---- ------ --------------------------------------------------1 Po10 up 1,10-11,176,208-209,3001

vPC status----------------------------------------------------------------------id Port Status Consistency Reason Active vlans-- ---- ------ ----------- ------ ------------

20 Po20 up failed vPC type-1 configuration -incompatible - STPinterface port guard -Root or loop guardinconsistent

N5K-1# spanning-tree guard root

vPC member ports shut down until

both N5K-1 and N5K-2 configured. Only PO20 is affected, other vPCs

remain operational.

Graceful Type-1 Recovery

PKN5K-1 N5K-2

N5K-2# spanning-tree loopguard default




PO20

PO10

PLN5K-1# show vpc brieLegend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 3Peer status : peer adjacency formed okvPC keep-alive status : peer is aliveConfiguration consistency status: failed

Configuration consistency reason: vPC type-1 configurationincompatible - STP global loop guard inconsistentType-2 consistency status : failedType-2 consistency reason : SVI type-2 configuration incompatiblevPC role : secondaryNumber of vPCs configured : 4Peer Gateway : EnabledPeer gateway excluded VLANs : -Dual-active excluded VLANs : -


id Port Status Active vlans-- ---- ------ --------------------------------------------------1 Po10 up -

vPC status----------------------------------------------------------------------id Port Status Consistency Reason Active vlans-- ---- ------ ----------- ------ ------------20 Po20 down* failed Global compat check failed -

Peer holding Secondary vPC role shuts downvPC member ports

N5K-1# show port-channel sum int p20Flags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)s - Suspended r - Module-removedS - Switched R - RoutedU - Up (port-channel)M - Not in use. Min-links not met


Channel--------------------------------------------------------------------------------20 Po20(SD) Eth LACP Eth2/17(P)

Local Suspended VLAN




Common Causes: VLAN not permitted on vPC Peer Link VLAN doesn’t exist in VL database on vPC peer In case of global inconsistency, all VLANs suspended

What Happened?

N5K-1g)# show logging level vpc

Facility Default Severity Current Session

Default severity levelfor vPC is 2.

Recommended to




N5K-1(config)#logging level vpc 3

N5K-1# show logging | i %VPC

2011 Aug 25 13:14:34 N5K-1 %VPC-3-GLOBAL_CONSISTENCY_FAILED: In domain 3, global configuration is not consistent (vPC type-1 configuration incompatible - STP global loop guard inconsistent)

Facility Default Severity Current SessionSeverity-------- ---------------- ------------------------vpc 2 3

0(emergencies) 1(alerts) 2(critical)3(errors) 4(warnings) 5(notifications)6(information) 7(debugging)

Recommended to

change this to at least3 to see msgs such asbelow

N5K-1# show accounting log | b “Aug 25 13:14”

Thu Aug 25 13:14:34 2011:type=update:id=10.116.186.217@pts/28:user=admin:cmd=configure terminal ; spanning-tree loopguard default (SUCCESS)

Who Done It?

STP and vPC

Peer link is running STP

It is possible to see situation when

there are 2 root ports: peer-link and

vPC toward the root

This happens on vPC peer holding

the vPC secondary role




DCN-N5K1# show spanning vlan 176

VLAN0176

Spanning tree enabled protocol rstp

Root ID Priority 8368

Address 0023.04ee.be01

Cost 2

Port 4096 (port-channel1)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32944 (priority 32768 sys-id-ext 176)

Address 000d.ecb2.2afc

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Po1 Root FWD 1 128.4096 (vPC peer-link) Network P2p

Po20 Root FWD 1 128.4115 (vPC) P2p

Po27 Desg FWD 1 128.4122 (vPC) Edge P2p

Po28 Desg FWD 1 128.4123 (vPC) Edge P2p

the vPC secondary role

This is perfectly normal in a vPCenvironment!

sh tech-support vpc

Collect for TAC/engineering to look at the issue




Collects the following

`show version``show module``show vpc brief``show vpc role``show running-config vpc``show system internal vpcm event-history global``show system internal vpcm event-history errors`

`show system internal vpcm event-history msgs``show system internal vpcm event-history interactions``show system internal vpcm mem-stats detail``show system internal vpcm info all``show system internal vpcm info global``show CFS internal ethernet-peer database``show spanning-tree`

Most often information about other components would be needed as well, so best isto start with ‘sh tech detail’ – this includes in it ‘sh tech vpc’ and most otherrelevant outputs




vPC Config Sync

Nexus 5500 Config-SyncOverview

Starting from NX-OS 5.0.2 release, the Nexus 5500 introduces the config-sync feature forvPC. Config-sync allows administrators to make configuration changes on one switch andh h ll h h l

NX-OS 5.0(2)N2(1)




have the system automatically synchronize to its peers. This eliminates any user proneerrors & reduces the administrative overhead of having to configure both vPC memberssimultaneously.

PO5

interface Ethernet1/47fex associate 100switchport mode fex-fabricchannel-group 5

interface Ethernet1/47fex associate 100switchport mode fex-fabricchannel-group 5



Nexus 5500 Config-SyncWhat features are supported with config sync?

Config sync is used to ensure configuration consistency betweenpeers who require it (i.e. vPC peers). Under the switch-profile the




p q ( p ) p

following features are configurable for synchronization

The following are NOT automatically synchronized

Must be configured manually on each switch•Enabling the specific feature set (i.e. feature vpc, feature vlan, etc)•vPC Domain Configuration•vPC peer-keepalive configuration•FCOE configurations (not supported in a switch-profile)

VLANsACLS’s

STPQOS

Interface Level Configurations:(Ethernet Interfaces)

(Port Channel Interfaces)(vPC Interfaces)

Nexus 5500 Config-SyncPrerequisites – 3 steps required

Config sync feature is supported today on the Nexus 5500 platform running 5.0.2. Inaddition, CFSoIP, Switch-profiles, and Peer-configuration must be configured on each peer




CFSoIPTransport protocol for theconfiguration across peers

Both peers need to haveCFSoIP enabled

N5500-1# config tN5500-1(config)# CFS ippv4 distribute

N5500-2# config tN5500-2(config)# CFS ippv4 distribute

Both peers require identicalswitch profiles

N5500-1# config syncN5500-1(config-sync)# switch-profileApple


Both peers require to configureeach other as their peer

N5500-1# config syncN5500-1(config-sync)# switch-profileAppleN5500-1(config-sync)# sync-peersdestination 10.29.170.8


N5500-2(config-sync)# sync-peersdestination 10.29.170.7

Switch-profileUsed to create the config thatneeds to be sync across peers

Peer ConfigurationTo indicate which peer willreceive the configuration

Only one switch profile per switch is configurable today.A new mode “config sync”, similar to “config t” is introduced to create switch -profiles

Step 3:Step 1: Step 2:

Nexus 5500 Config-SyncConfig-Sync example – New Switch

This example assumes that N5K’s are new switches that will be configured for vPC. It is

assumed that only the basic vPC parameters have been enabled for vPC to operate




Enable CFSoIP N5500-1# config t N5500-1(config)# CFS ippv4 distribute

Configure identicalswitch-profile on eachswitch

N5500-1# config sync N5500-1(config-sync)# switch-profile Apple

N5500-2# config t N5500-2(config)# CFS ippv4 distribute


Configure peer

relationship underswitch-profile


N5500-1(config-sync)# sync-peers destination 10.29.170.8

N5500-2# config sync N5500-2(config-sync)# switch-profile Apple N5500-2(config-sync)# sync-peers destination 10.29.170.7


Continued…




Once config has beenverified, issue

“commit”

N5K-1# sh running-config

Verify theconfiguration was

merged sucessfully

N5K-1(config-sync-sp)# verify

Verify Successful N5K-1(config-sync-sp)# commit

Commit Successful


Repeat as needed

We recommend to copy smaller chunks of the profile to ensure each syncis smooth

N5K-1(config-sync-sp)# interface Ethernet1/10 <snip>interface Ethernet100/1/2

switchport mode trunkswitchport access vlan 5switchport trunk allowed vlan 5

<snip>

Enter all the configunder the switch-

profile and VERIFY

config “show switch-profile buffer”

Nexus 5500 Config-Sync

Once a configuration is applied using config-sync, thatconfig exists under the switch profile




No changes are allowed to the physical interface,changes must be made within the switch profile

Deleting switch profile deletes the configuration!

DCN-N5K1(config)# interface e199/1/2DCN-N5K1(config-if)# sw trunk allowed vlan add 200

Error: Command is not mutually exclusive

DCN-N5K1(config-if)# config syncDCN-N5K1(config-sync)# switch-profile FEX_Ports

Switch-Profile started, Profile ID is 1DCN-N5K1(config-sync-sp)# interface e199/1/2DCN-N5K1(config-sync-sp-if)# switchport trunk allowed vlan add 200DCN-N5K1(config-sync-sp-if)# verifyVerification SuccessfulDCN-N5K1(config-sync-sp)# commitVerification successful...Proceeding to apply configuration. This might take a while depending onamount of configuration in buffer.Please avoid other configuration changes during this time.Commit Successful

DCN-N5K1# show run int e199/1/2

!Command: show running-config interface Ethernet199/1/2!Time: Fri Aug 26 15:02:23 2011

version 5.0(3)N1(1b)

interface Ethernet199/1/2switchport mode trunkswitchport trunk allowed vlan 176,200

Command is denied onphysical interface. Configmust be applied underswitch-profile

DCN-N5k2# show run int e199/1/2

!Command: show running-config interface Ethernet199/1/2!Time: Fri Aug 26 13:52:47 2011

version 5.0(3)N1(1b)

interface Ethernet199/1/2switchport mode trunkswitchport trunk allowed vlan 176,200

Nexus 5500 Config-SyncMutual Exclusion Check

Mutual Exclusion (Mutex) –Verifies configuration between inside and outside the profile. Ifthere is a conflict, a “verify” or “commit” will fail. Applies to both adding and removingconfigurations from inside/outside profile




configurations from inside/outside profile.

N5500-1#sh run int ether 100/1/3int ether 100/1/3switchport mode trunk Outside of Profile

N5500-1(config-if)# config syncN5500-1(config-sync)# switch-profile ASwitch-Profile started, Profile ID is 1N5500-1(config-sync-sp)# int ethernet 100/1/3N5500-1(config-sync-sp-if)# switchport mode access Inside of ProfileN5500-1(config-sync-sp-if)# verify

Failed: Verify Failed

N5500-1(config-sync-sp)# show switch-profile A status… Session-type: CommitStatus: Verify FailureError(s): Following commands failed mutual-exclusion checks:interface Ethernet100/1/3

switchportmode access

Mismatch between the outside and inside the profile results in a failure in a mutex verify

To resolve this, user needs to manually remove the configuration outside/inside profile

Inside profile includes all the configuration under a “switch-profile”. Outside profile includes all theglobal/interface level configuration that is done outside of a switch-profile

Nexus 5500 Config-SyncMerge Exchange Check

Merge Check – occurs after peer-reachability is established in one of two scenarios.1) Peers interacting for the first time (i.e. after a reload, or a peer being reloaded)2) Peers interacting after an intermittent network down time If there is a conflict between




2) Peers interacting after an intermittent network down time. If there is a conflict between

the 2 devices, a “verify” and “commit” will fail

N5500-1#sh run switch-profileSwitch-profile Applesync-peers destination 10.29.170.8

Peer becomes unreachable due to a network outage, config sync will not occur across mgmt0.vPC peer link is up, but vPC PKL is down due to mgm0 not reachable

Local changes on N5K-1 and N5K-2 are possible

N5500-1(config-if)# config sync N5500-1(config-sync)# switch-profile Apple N5500-1(config-sync-sp)# int ethernet100/1/3

N5500-1(config-sync-sp-if)# switch modetrunk

N5500-1(config-sync-sp-if)# commit

Commit Successful

N5500-2#sh run switch-profileSwitch-profile Applesync-peers destination 10.29.170.7

N5500-2(config-if)# config sync N5500-1(config-sync)# switch-profile Apple N5500-2(config-sync-sp)# int ethernet100/1/3

N5500-2(config-sync-sp-if)# switch mode fex-fabric

N5500-2(config-sync-sp-if)# commitCommit Successful

N5500-1#sh run switch-profileinterface ethernet 1/10switchport mode trunk

N5500-2#sh run switch-profileinterface ethernet 1/10switchport mode fex-fabric

Nexus 5500 Config-SyncMerge Exchange Check - continued

Once peer-reachability is established again, the Merge will fail dueto conflicting/overlapping changes. Configuration of peers remains




g/ pp g g g p

unchanged.

N5500-1#sh run switch-profileinterface ethernet 1/10switchport mode trunk

Peer becomes reachable, mgmt0 is up

N5K-1(config-sync-sp)# commit N5K-1(config-sync-sp)# sh switch-profile AstatusProfile-status: Merge Failed

Status: Verify FailureError(s):Following commands failed merge checks:interface Ethernet1/10


N5500-2#sh run switch-profileinterface ethernet 1/10switchport mode fex-fabric

Mismatch bothethernet1/10 interfacesresults in a failure in amerge check

To resolve this, userneeds to manually

remove the configurationoutside/inside profile


This example assumes that N5K’s are new switches that will be configured for vPC. It isassumed that only the basic vPC parameters have been enabled for vPC to operate





Configure identicalswitch-profile on each

switch




Configure peerrelationship under

switch-profile




Continued…




Once config has beenreviewed, issue

“commit”


Verify theconfiguration wasmerged sucessfully

N5K-1(config-sync-sp)# commit

Commit Successful


Repeat as needed

We recommend to copy smaller chunks of the profile to ensure each sync is smooth

N5K-1(config-sync-sp)# interface Ethernet1/10 <snip>interface Ethernet100/1/2

switchport mode trunkswitchport access vlan 5switchport trunk allowed vlan 5

<snip>

Enter all the configunder the switch-profile and VERIFY

config “show switch-profile buffer”

Nexus 5500 Config-SyncConfig-Sync example – (i.e. Dee Why Plus -> Eaglehawk)

This example assumes that N5K’s are already working in vPC, with configurations alreadymanually synced. User now wants to continue with config-sync





Configure identicalswitch-profile on each

switch

Option1: N5K-1(config-sync-sp)# import running-config

We recommend to copy smaller chunks of the profile to ensure each sync is smooth

Option2: N5K-1(config-sync-sp)# interface Ethernet1/10

<snip>interface Ethernet100/1/2switchport mode trunkswitchport access vlan 5switchport trunk allowed vlan 5

<snip>

Import config underthe switch-profile and

VERIFY runningconfiguration “show

switch-profile buffer”




Continued…

Once reviewed, issue N5K-1(config-sync-sp)# commit

Nexus 5500 Config-SyncConfig-Sync example – (i.e. Dee Why Plus -> Eaglehawk)




Then, configure peersto initiate a merge and

bring both in sync


Verify theconfiguration was

merged successfully


N5K-2(config-sync-sp)# commit

Commit Successful


Repeat as needed

In this example, the peers are defined only after the configurations are put under a profile. Thereason is to eliminate any sync from occurring before user is able to review the configuration

Once reviewed, issue

“commit” on BOTHsides to “import” the

config locally first

Commit Successful


Any failures shall bereported as merge-

failures and need to bemanually correctedinside/outside the

switch-profile

Event Reaction

vPC peer-link down No impact if config-sync is over mgmt0

Nexus 5500 Config-SyncFailure Scenarios




vPC peer-link down No impact if config-sync is over mgmt0

CFS keepalive failure CFS issues a “peer not reachable” notification,

config-sync becomes non-operational with that

peer

Switch reload Peer switches get a “peer unreachable”

notification from CFS and stop communicating

with this switch

Commit failure on peer Rollback to previously taken checkpoint

Merge failure Syslogs gets generated and user shall use 'show

switch-profile status' to determine the errors

and correct.

When ISSU is in progress on a peer, then a 'verify/commit' is not permitted

on this peer

Nexus 5500 Config-SyncISSU interaction




If a commit is issued from other peer, that shall fail only if the peerundergoing ISSU was still reachable but can't accept configuration due to

ISSU, otherwise the 'commit' will become a local-operation by default

behavior.

When a verify/commit is in progress between the peers, then ISSU shall be

blocked on both peers. However, if there is no reachability then a local-commit on one peer won't be affect ISSU on the other peer.

It is recommended to choose only one switch as the initiator. Initiator can be vPC

primary/secondary. The roles are NOT dependent one each other.

Nexus 5500 Config-SyncHeads up !




Commit should be issued on initiator. Only one session (verify/commit/merge) can be inprogress at a time. A session attempted while another session is in progress shall fail

All configuration changes are prevented when a switch-profile session is in progress i.e.

even changes through config-terminal for all supported commands (ACL, QoS etc) are

also blocked when a session is in progress.

Ensure that the specific feature is enabled on each switch (i.e. feature vpc, feature vlan,etc

When migrating to config-sync (vPC is running with configurations already synced),

ensure you add smaller sections under the profile and commit versus doing everything

in one chunk

vPC and config sync are independent features. If peer-link is down, config-sync will stillwork

Config sync is ONLY transported across mgmt0 interface

FEX pre provisioning can also be done using switch-profiles.



Nexus 5500 Config RollbackOverview

Starting from NX-OS 5.0(2) release, the Nexus 5500 will introduces the config rollbackfeature. This feature allows the end user to take a snapshot (checkpoint) of the Cisco NX-OS configuration and then reapply that configuration to the device at any point without




have to reload the device. A rollback allows any authorized admin to apply the checkpointconfiguration without requiring expert knowledge of features configured in a checkpoint

Prior to 5.0(2), the system required a reload to run another configuration file

--------------------

Currentrunning-config

--------------------

--------------------

Checkpointrunning-config

--------------------

User wants to revert back tothe original configuration

Configurationcheckpoint

Today’sconfiguration



Nexus 5500 Config RollbackHow to verify the config captured for a rollback

The user can verify the configuration that is captured in a checkpoint beforeexecuting a rollback.




N5K-1(config)# show checkpoint ? <CR>> Redirect it to a file>> Redirect it to a file in append modeTest-Config Checkpoint nameall (no abbrev) Show default configsummary (no abbrev) Show configuration rollback checkpoints summary

system (no abbrev) Show only system configuration rollback checkpointsuser (no abbrev) Show only user configuration rollback checkpoints| Pipe command output to filter



Nexus 5500 Config RollbackHow to execute a rollback

When a rollback is trigged, the Nexus 5500 only supports the atomic method. Theatomic rollback implements a rollback only if no errors occur. If an error doesoccur, we go back to the last running-configuration the system was using .




occur, we go back to the last running configuration the system was using .

N5K-1: rollback running-config checkpoint Test-Config Note: Applying config parallelly may fail Rollback verificationCollecting Running-ConfigGenerating Rollback patch for switch profileRollback Patch is EmptyCollecting Running-Config#Generating Rollback Patch

Rollback Patch is Empty

Rollback completed successfully.

Nexus 5500 only supports atomic rollback at FCS

We don’t support config rollback for fiber channel interface/configuration.

The CLI will get disabled if “feature fcoe” is enabled

Nexus 5500 Config RollbackHeads up !




The Nexus 5500 only supports atomic rollback. If an error is encountered(i.e. a command does not go through), we will rollback to the “show

running-config” at the time when rollback was issued

N5K does not support auto checkpoints, only manually configured ones

If you create a configuration checkpoint and upgrade or downgrade to adifferent software release, the rollback procedure is not officially supported.

However, the rollback procedure may still work depending on the

configuration changes being executed

Multicast




Nexus 5500 Multicast ForwardingFabric-Based Replication

Multicast Framesare Queued in

Nexus 5500 use fabricbased egress replication





MCAST packet isreplicated in the

Fabric

Eth 1/20Eth 1/8

dedicatedmulticast queueson Ingress

Traffic is queued in theingress UPC for eachMCAST group

When the schedulerpermits the traffic if

forwarded into the fabricand replicated to allegress ports

When possible, traffic issuper-framed (multiplepackets are sent with asingle fabric schedulergrant) to improvethroughput

Multicast

Scheduler

Nexus 5500Multicast Fabric Replication (Animated)

Ingress Interface Switch Egress




Fabricg

Interface

PacketBuffer

Mcast

A

Ucast

BMcast

C

Unicast VOQ

Multicast VOQMcast

A

Mcast

A

Mcast

A

Mcast

A

128 MCAST VOQ per port

4 Crosspoints – Shared acrossunicast and MCAST

8 Dedicated Egress MCAST Queues

Nexus 5500 Multicast ForwardingNexus 5500 Data Plane Changes

Nexus 5500 supports 4000 IGMP snooping entries

Dedicated Unicast & Multicast Queuing and Scheduling

128 MCAST VOQ per port




Resources 128 MCAST VOQ per port

8 for egress queues for unicast and 8 for multicast

4 Egress cross-points (fabric buffer) per egress port

Out of 4 fabric buffer, one is used for unicast, one formulticast and two are shared between unicast and

multicast Two configurable Multicast scheduler modes

Overloaded mode (Proxy Queue)

Congested egress ports are ignored

Multicast packets are sent to non-congested port only

Reliable mode Packets are sent to switch fabric when all OIF ports are

ready, ie, have fabric buffer and egress buffer to acceptthe multicast packets

Multicast

Scheduler

4 FabricCrosspointsper port (10KX-Bar buffer)

8 DedicatedEgress MCAST

Queues per Port

8 DedicatedEgress UCAST

Queues per Port

...

Multicast Optimization and VOQ Assignment

128 Multicast VOQ for each ingress port. Separate VOQ for multicast andunicast traffic




One multicast VOQ per class of service without multicast optimization

Multicast optimization can be turned on for one class of service

With multicast optimization multicast traffic assigned to VOQ based on fanout

Multicast VOQ

Class 1 Q1Q2

Q3

Q128

Multicast VOQ

Q1Q2

Q3

Without “multicastoptimization”

Class 2

Class 3

Class 8 Q8

Class 1

Class 2

Class 3

Class 8

Q127

With “multicast optimization”

class with“multicast

optimization”

Q8

Q9

Q127

Q128

Multicast Optimization Configuration Multicast optimization is turned on by default for “class-default ”. It means

all multi-destination traffic will be assigned to multicast VOQ according totheir fanout




Multi-destination traffic includes:

IP multicast

Unknown unicast flooding

Broadcast traffic

L2 multicast traffic

User can choose to turn on multicast optimization for selected multi-destination traffic, such as, IP multicast traffic

Multicast optimization can only be turned on for one system class.

8 multicast VOQ reserved for QoS queuing. The rest of 120 queues formulticast optimization

Multicast Optimization Sample Configuration

Multicast optimization can be turned on for user defined system




class. Multicast optimization for “class-default ” will be disabled

automatically

No change for unicast traffic

N5k(config-cmap-qos)# policy-map type qos Mcast_optimize N5k(config-pmap-qos)# class type qos class-ip-multicast N5k(config-pmap-c-qos)# set qos-group 2 N5k(config-pmap-c-qos)# exit N5k(config-pmap-qos)# class type network-qos IP_mcast

N5k(config-cmap-nq)# match qos-group 2 N5k(config-cmap-nq)# policy-map type network-qos Mcast_optimize N5k(config-pmap-nq)# class type network-qos IP_mcast N5k(config-pmap-nq-c)# multicast-optimize N5k(config-pmap-nq-c)# queue-limit 170000

Nexus 5500 Multicast ForwardingNexus 5500 Data Plane Changes

Proxy queues to detect congestion at egress

One proxy queue for each hardware egress queue




Bytes are added to proxy queue when packets arriveat egress hardware queue

Proxy queues are drained at 98% of port speedusing DWRR

When proxy queue is full egress port sends

“overload” message to central scheduler Central scheduler excludes the port in multicast

scheduling calculation when overload bit is set ANDthere is no fabric buffer available. Multicast packetis sent over to non-congested port

In case of congestion there is a delay for proxyqueue to signal overload

Multicast

Scheduler

Proxy Queue sends overloadsignal to scheduler when port

congested

...

N5k(config)#hardware multicast disable-slow-port-pruning

Multicast Load-sharing Over Port-Channel

Load-sharing influenced by ingress port and VOQ number

Each interface is assigned a unique seed number for hashcalculation




Multicast optimization (turned on by default for “class-default”)required for better distribution.

The Port-Channel load-sharing option configuration doesn’t apply tomulticast traffic

Po10

Receivers

1/1

1/10 1/111/2

1/3

Source

1.1.1.1 224.1.1.2

Multicast MAC Table LookupOIF : 1/2 ,1/3, Po10(1/10, 1/11)

VOQ # 20

Hashing calculationChoose 1/10 for Po10

VOQ # 20

1/10

Seed numberfor eth1/1

Request to central schedulerwith OIF 1/2, 1/3 and 1/10

Switch fabric replicatespackets to 1/2 , 1/3 and 1/10

Nexus 5500Station (MAC) Table allocation

Nexus 5500 has a 32K Station table entries

4k reserved for multicast (Multicast MAC addresses)




3k assumed for hashing conflicts (very conservative)

25k effective Layer 2 unicast MAC address entries

N e x u s 5 5 0 0

U P C

S t a t i o n

T a b l e

3

2 k e n t r i e s 4k entries forIGMP

3k entries for potential hash collision space

25k effective MAC entries for unicast




Cisco Nexus 5500Multicast


MulticastImportant Cisco NX-OS and Cisco IOS Differences

In Cisco NX-OS:

• PIM and MSDP protocols require a LAN Enterprise Services license.

• The global ip multicast-routing command does not exist in NXOS and is not required to




enable multicast forwarding/routing. (It is required in Cisco IOS Software to enable multicastforwarding/routing)

• PIM command-line interface (CLI) configuration and verification commands are not availableuntil you enable the PIM feature with the “feature pim” command.

• MSDP CLI configuration and verification commands are not available until you enable theMSDP feature with the “feature msdp” command.

• IGMP versions 2 and 3 are supported. IGMP version 1 and Version 3 Lite are not supported.• An IGMP Snooping Querier is configured under the layer-2 VLAN with the ip igmp snoopingquerier CLI command (Physical L3 interfaces cannot be configured as IGMP SnoopingQueriers). In Cisco IOS Software, an IGMP Snooping Querier is configured under the layer-3interface.

• PIM version 2 Sparse Mode is supported. Cisco NX-OS does not support PIM version 1Sparse Mode or Dense Mode. The NX-OS cannot fallback to Dense Mode operation.

• When configuring a PIM Auto-RP Candidate or BSR RP-Candidate the NX-OS requires aconfigured group-list (i.e. x.x.x.x/x), whereas Cisco IOS Software defaults to 224.0.0.0/4. Anoptional standard ACL can be configured to specify multicast groups in Cisco IOS Software.

• When configuring PIM Auto-RP Mapping-Agent's or Candidate-RP's, Cisco NX-OS uses adefault scope of 32, whereas Cisco IOS Software requires it to be specified with the scopeoption (1-255).


In Cisco NX-OS:

• When configuring PIM Auto-RP, Cisco NX-OS multicast devices must be enabled to listenand/or forward RP advertisements with the ip pim auto-rp forward listen global CLI

fi ti d Ci IOS S ft h t b fi d f S D M d




configuration command. Cisco IOS Software has to be configured for Sparse-Dense Mode orSparse Mode with the global ip pim autorp listener CLI configuration command.

• When configuring PIM BSR, Cisco NX-OS multicast devices must be enabled to listenand/or forward RP advertisements with the ip pim bsr forward listen global CLI configurationcommand. Cisco IOS Software doesn’t require additional configuration, but does not have

the ability to enable/disable RP forwarding and listening capabilities.

• BSR-Candidate routers have a default priority of 64. Cisco IOS Software defaults to 0. The

priority value can be configured between 0 – 255 in both operating systems using the priorityoption. A higher numeric value is preferred when comparing priorities.

• BSR RP-Candidate routers have a default priority of 192. Cisco IOS Software defaults to0. The priority value can be configured between 0 – 255 in both operating systems using thepriority option. The lower numeric value is preferred when comparing priorities.

• When configuring a Static-RP, the NX-OS does not have an override option like Cisco IOSSoftware that forces the Static-RP to be elected for it’s specified multicast group list. Cisco

IOS Software prefers dynamically learned RP’s over Static RP’s if the override option is not

configured.

• When comparing PIM Static-RP’s to dynamically learned RP’s (Auto-RP and BSR) duringthe election process: The RP with the most specific multicast group-list is elected. If thegroup-lists are identical, the router with the highest RP IP address is elected.


In Cisco NX-OS:

• When configuring a PIM domain border, the ip pim border interface CLI commandprevents BSR and Auto-RP packets from being sent or received on an interface. The

Ci IOS S ft d i l t (i i b b d ) l t BSR




Cisco IOS Software command equivalent (ip pim bsr-border) only prevents BSRpackets. Cisco IOS Software requires the ip multicast boundary interface command toprevent Auto-RP packets.

• PIM neighbor authentication (IPSec ah-md5) can be enabled to authenticate directlyconnected neighbors to increase security. Cisco IOS Software does not support thisfunctionality.

• PIM neighbor logging can be enabled with the global ip pim log-neighbor-changesCLI command. (Cisco IOS Software enables PIM neighbor logging by default)

• The data in the MSDP Source-Active (SA) messages are cached by default,whereas Cisco IOS Software requires the global ip msdp cache-sa-state and ip msdpcache-rejected-sa CLI commands.

• PIM is configured with the Source Specific Multicast (SSM) group range 232.0.0.0/8

by default (ip pim ssm range 232.0.0.0/8).• PIM does not support Bidirectional Forwarding Detection (BFD) for rapid failuredetection on the Nexus 5500 series yet, but it is being targeted for the Goldcoastrelease. However, on the Nexus 7000 series, beginning with NX-OS 5.0(2a), PIMsupports BFD.

MulticastThings You Should Know

• If you remove the feature pim command, all relevant PIM configurationinformation is also removed.

• If you remove the feature msdp command, all relevant MSDP configurationi f ti i l d




information is also removed.

• IGMP Snooping is enabled globally by default. It can be disabled globally, orper layer-2 VLAN with the no igmp snooping command.

• IGMP version 2 is enabled by default when PIM Sparse Mode is configured onan interface.

• PIM configuration is supported under IP Tunnel (GRE) interfaces in Cisco NX-OS 5.2(1) and onward (PIM was previously not supported in IP Tunnels).

• PIM supports three modes of operation: Any Source Multicast (ASM), SingleSource Multicast (SSM), Bidirectional Shared Tree (Bidir). The default mode isASM. Bidir can be configured with the bidir option when configuring a RP.

• The Cisco NX-OS supports four types of PIM Rendezvous Points: Static,

Bootstrap router (BSR), Auto-RP and Anycast-RP. (Do not configure Auto-RP andBSR in the same network)

MulticastThings You Should Know

• When configuring a PIM Static-RP, the group-list defaults to 224.0.0.0/4 if oneis not specified.

• The Cisco NX-OS has two different CLI syntax options when configuring BSRd A t RP' (N Ci NX OS t d b k d tibl Ci IOS




and Auto RP's (New Cisco NX-OS syntax, and backwards compatible Cisco IOSSoftware syntax).

• The Cisco NX-OS supports multicast routing per layer-3 Virtual Routing andForwarding (VRF) instance.

• PIM SSM and Bidir are not supported on Virtual Port-Channels (vPCs).

• A topology that has a PIM router connected to a pair of Cisco Nexus 5500Platform switches through vPC is not supported.

• Configure candidate RP intervals to a minimum of 15 seconds.

• A vPC peer link is a valid link for IGMP multicast forwarding.

• If the vPC link on a switch is configured as an output interface (OIF) for amulticast group or router port, the vPC link on the peer switch must also be

configured as an output interface for a multicast group or router port.

• In SVI VLANs, the vPC peers must have the multicast forwarding stateconfigured for the vPC VLANs to forward multicast traffic directly through thevPC link instead of the peer link.

MulticastCommand Comparison: NX-OS vs IOS


E bli M lti t F di




Enabling Multicast Forwarding

ip multicast-routing

The Cisco NX-OS does not have a singleglobal command to enable multicastforwarding/routing.

Cisco IOS Software does not have theability to enable or disable PIM.

feature pim

Configuring PIM Sparse Mode on an Interface

interface TenGigabitEthernet1/1

ip address 192.168.10.1 255.255.255.0

ip pim sparse-mode

interface Ethernet1/1

ip address 192.168.10.1/24

ip pim sparse-mode

Enabling the PIM Feature

MulticastCommand Comparison: NX-OS vs IOS (cont’d)


C fi i PIM A t RP




Configuring a PIM Auto-RP

interface Loopback10 I

p address 172.16.1.1 255.255.255.255

ip pim sparse-mode

ip pim send-rp-announce Loopback10

scope 32

ip pim send-rp-discovery Loopback10

scope 32

ip pim autorp listener

interface loopback10

ip address 172.16.1.1/32

ip pim sparse-mode

ip pim auto-rp rp-candidate loopback10

group-list 224.0.0.0/4

ip pim auto-rp mapping-agent loopback10

ip pim auto-rp forward listen

or

ip pim send-rp-announce loopback10

group-list 224.0.0.0/4

ip pim send-rp-discovery loopback10

ip pim auto-rp forward listen



C fi i PIM BSR RP




Configuring a PIM BSR RP

interface Loopback10

ip address 172.16.1.1 255.255.255.255ip pim sparse-mode

ip pim bsr-candidate Loopback10

ip pim rp-candidate Loopback10


ip address 172.16.1.1/32

ip pim sparse-mode

ip pim bsr bsr-candidate loopback10ip pim bsr rp-candidate loopback10 group-

list 224.0.0.0/4

ip pim bsr forward listen

or

ip pim bsr-candidate loopback10

ip pim rp-candidate loopback10 group-list

224.0.0.0/4ip pim bsr forward listen



C fi i PIM A t RP (BSR E l )




Configuring a PIM Anycast-RP (BSR Example)

Cisco IOS Software does not have theability to enable the PIM Anycast RPfeature.

interface loopback0

ip address 192.168.10.1/32

ip pim sparse-mode

interface loopback10description Anycast-RP-Address

ip address 172.16.1.1/32

ip pim sparse-mode

ip pim bsr bsr-candidate loopback0

ip pim bsr rp-candidate loopback10 group-

list 224.0.0.0/4ip pim anycast-rp 172.16.1.1 192.168.10.1

ip pim anycast-rp 172.16.1.1 192.168.10.2

ip pim bsr forward listen



Configuring a PIM Static RP




Configuring a PIM Static-RP

ip pim rp-address 172.16.1.1 ip pim rp-address 172.16.1.1

Configuring PIM Neighbor Authentication

Cisco IOS Software does not have theability to enable neighbor authentication.


ip address 192.168.10.1/24ip pim sparse-mode

ip pim hello-authentication ah-md5 3

a667d47acc18ea6b






Configuring a PIM BSR Border on an Interface

interface TenGigabitEthernet1/1 I

p address 192.168.10.1 255.255.255.0

ip pim bsr-border

ip pim sparse-mode

ip multicast boundary 10

access-list 10 deny 224.0.1.39

access-list 10 deny 224.0.1.40

access-list 10 permit 224.0.0.0

15.255.255.255


ip address 192.168.10.1/24

ip pim sparse-mode

ip pim border



Configuring PIM in a Non Default VRF Instance




Configuring PIM in a Non-Default VRF Instance

ip vrf production

ip multicast-routing vrf production


ip vrf forwarding productionip address 172.16.1.1 255.255.255.255

ip pim sparse-mode


ip vrf forwarding production

ip address 192.168.10.1 255.255.255.0

ip pim sparse-mode

ip pim vrf production rp-address 172.16.1.1

vrf context production

ip pim rp-address 172.16.1.1 group-list

224.0.0.0/4


vrf member production

ip address 172.16.1.1/32


vrf member production

ip address 192.168.10.1/24ip pim sparse-mode



Configuring IGMP Version 3 for an Interface




Configuring IGMP Version 3 for an Interface


ip address 192.168.10.1 255.255.255.0

ip pim sparse-mode

ip igmp version 3


ip address 192.168.10.1/24

ip pim sparse-mode

ip igmp version 3

Configuring an IGMP Snooping Querier for a VLAN

interface Vlan10

ip address 192.168.10.1 255.255.255.0

ip igmp snooping querier

vlan 10

ip igmp snooping querier 192.168.10.1

Note: there is no subnet mask on the IP address of the nexus querier configcommand.



Configuring MSDP (Anycast RP)




Configuring MSDP (Anycast-RP)

interface Loopback0

description MSDP Peer Address

ip address 192.168.1.1 255.255.255.255


description PIM RP Address

ip address 1.1.1.1 255.255.255.255

ip pim rp-address 1.1.1.1

ip msdp peer 192.168.2.1 connect-source

Loopback0ip msdp cache-sa-state

interface loopback0

description MSDP Peer Address

ip address 192.168.1.1/32


description PIM RP Address

ip address 1.1.1.1/32

ip pim rp-address 1.1.1.1 group-list

224.0.0.0/4

ip msdp peer 192.168.2.1 connect-sourceloopback0

MulticastTroubleshooting and Verification Commands



h i i h i i Displays all IGMP attached group




show ip igmp groups show ip igmp groups Displays all IGMP attached groupmembership information

show ip igmp interface show ip igmp interface Displays IGMP information for all interfaces

show ip igmp interfacebrief

-Displays a one line summary status per

interface

show ip igmp interface int-

type

show ip igmp interface

int-type

Displays IGMP information for a specific

interface

show ip igmp interface vrf name

show ip igmp vrf name Displays IGMP information for a specificVRF instance

show ip igmp local-groups

int-type -

Displays IGMP local groups associated to a

specific interface

show ip igmp local-groupsvrf name

-Displays IGMP local groups associated to a

specific VRF instance

show ip igmp route -

Displays IGMP attached group membership

information

show ip igmp route x.x.x.x -Displays IGMP attached group membership

for a specific group

show ip igmp route int-type -Displays IGMP attached group membership

for a specific interface

MulticastTroubleshooting and Verification Commands (cont’d)



show ip igmp route vrf name -

Displays IGMP attached group

membership for a specific VRF instance




name membership for a specific VRF instance

show ip igmp snooping -Displays global and per interface IGMP

Snooping information

show ip igmp snoopingexplicit-tracking

show ip igmp snooping

explicit-tracking

Displays explicit tracking information for

IGMPv3

show ip igmp snoopinggroups

show mac-address-table

multicast igmp-snooping

Displays IGMP Snooping groups

information

show ip igmp snoopingmrouter


mrouterDisplays detected multicast routers

show ip igmp snoopingotv

-Displays IGMP Snooping OTV

information

show ip igmp snoopingquerier

-Displays IGMP Snooping querier

information

show ip igmp snoopingstatistics


statistics Displays packet/error counter statistics

show ip igmp snoopingvlan #

-Displays IGMP Snooping information per

specific VLAN




Cisco NX-OS InterfaceCisco IOS

Software InterfaceCommand Description

show ip pim df show ip pim interface df Displays Bidir designated forwarders




show ip pim df show ip pim interface df Displays Bidir designated forwarders

show ip pim df x.x.x.x show ip pim interface df

x.x.x.x

Displays Bidir designated forwarders for

a specific RP or group

show ip pim df vrf name -Displays Bidir designated forwarders for

a specific VRF instance

show ip pim group-range - Displays the PIM group-ranges

show ip pim group-range x.x.x.x - Displays a specific PIM group-range

show ip pim group-range vrf name

-Displays the PIM group-ranges for a


show ip pim interface - Displays all PIM enabled interfaces

show ip pim interface brief x.x.x.x

-Displays a one line summary of all PIM

enabled interfaces

show ip pim interface int-type show ip pim interfaceint-type Displays information for a specific PIMinterface

show ip pim interface vrf name

-Displays the PIM interfaces for a





show ip pim neighbor show ip pim neighbor Displays all PIM neighbors




show ip pim neighbor show ip pim neighbor Displays all PIM neighbors

show ip pim neighbor x.x.x.x show ip pim neighbor

x.x.x.x

Displays a specific PIM neighbor for a

specific IP address

show ip pim neighborinterface int-type

show ip pim neighbor

int-type

Displays a specific PIM neighbor for a

specific interface

show ip pim neighbor vrf name

-Displays PIM neighbors for a specific

VRF instance

show ip pim oif-list x.x.x.x -Displays PIM OIF-List for a specific

multicast group address

show ip pim policy statistics - Displays PIM statistics

show ip pim route - Displays PIM routes

show ip pim route x.x.x.x - Displays a specific PIM route

show ip pim route vrf name -Displays PIM routes for a specific VRF

instanceshow ip pim rp show ip pim rp mapping Displays PIM RP information

show ip pim rp x.x.x.x show ip pim rp x.x.x.x Displays information for a specific PIM

group address




Displays information for PIM RP's in a




show ip pim rp vrf name -Displays information for PIM RP s in a


show ip pim rp-hash x.x.x.x show ip pim rp-hash

x.x.x.x

Displays PIM RP-Hash value for a

specific group

show ip pim statistics - Displays PIM packet statistics

show ip pim statistics vrf name

-Displays per packet statistics for a


show ip pim vrf name show ip pim vrf name Displays detailed PIM information per


- - -

show ip mroute show ip mroute Displays the multicast routing table

show ip mroute summary show ip mroute

summary

Displays the multicast routing table with

packet counts and bit rates

show ip mroute x.x.x.x show ip mroute x.x.x.x Displays a specific multicast routeshow ip mroute vrf name

show ip mroute vrf

name

Displays the multicast routing table for a





show ip pim rp x.x.x.x show ip pim rp x.x.x.x

Displays information for a specific PIM group

address




s o p p p p p p

show ip pim rp vrf name -Displays information for PIM RP's in a specific

VRF instance

show ip pim rp-hash x.x.x.x show ip pim rp-hash

x.x.x.x

Displays PIM RP-Hash value for a specific

group

show ip pim statistics - Displays PIM packet statistics

show ip pim statistics vrf

name -

Displays per packet statistics for a specific

VRF instance

show ip pim vrf name show ip pim vrf name Displays detailed PIM information per specific

VRF instance

- - -

show ip mroute show ip mroute Displays the multicast routing table

show ip mroute summary show ip mroute summaryDisplays the multicast routing table with

packet counts and bit rates

show ip mroute x.x.x.x show ip mroute x.x.x.x Displays a specific multicast route

show ip mroute vrf name show ip mroute vrf name Displays the multicast routing table for a


show ip route rpf show ip rpfDisplays the Reverse Path Forwarding (RPF)

table used for multicast source lookup

QoS




Nexus 5500 QoSQoS Capabilities and Configuration

Nexus 5500 supports a new set of QoS capabilities designed toprovide per system class based traffic control

Lossless Ethernet—Priority Flow Control (IEEE 802 1Qbb)




Lossless Ethernet Priority Flow Control (IEEE 802.1Qbb) Traffic Protection—Bandwidth Management (IEEE

802.1Qaz)

Configuration signaling to end points—DCBX (part of IEEE802.1Qaz)

These new capabilities are added to and managed by thecommon Cisco MQC (Modular QoS CLI) which defines a three-step configuration model

Define matching criteria via a class-map

Associate action with each defined class via a pol icy-map

Apply policy to entire system or an interface via a service- pol icy

Nexus 5500/7000 leverage the MQC qos-group capabilities toidentify and define traffic in policy configuration

Supported QoS Features

Eight class of service with eight hardware queue

Two reserved for internal control traffic

DSCP CoS or ACL based classification at ingress




DSCP, CoS or ACL based classification at ingress

DSCP marking and CoS marking

Support no-drop class of service to achieve lossless end-to-end

MTU per class of service

Queuing and bandwidth management

Strict priority queue and DWRR (Deficit Weigh RoundRobin)

Buffer tuning for drop and no-drop class

DSCP Marking

Only available with Nexus 5500 platform

Configured with policy-map type qos




Configured with policy map type qos

Independent of CoS marking

Without DSCP marking the DSCP value in the incoming packets ispreserved

ip access-list High-ACL10 permit ip 30.30.1.0/24 any

class-map type qos match-all High-ACL match access-group name High-ACL

policy-map type qos Policy-Classify

class High-ACLset qos-group 2set dscp 46

Nexus 5500 QoSQoS Policy Types

There are three QoS policy types used to definesystem behavior (qos, queuing, network-qos)

There are three policy attachment points tol th li i t

Ingress UPC

Unified Crossbar




There are three policy attachment points toapply these policies to

Ingress interface

System as a whole (defines global behavior)

Egress interface

Egress UPC


Policy Type Function Attach Point

qos Define traffic classification rulessystem qos

ingress Interface

queuingStrict Priority queue

Deficit Weight Round Robin

system qos

egress Interfaceingress Interface

network-qosSystem class characteristics (drop or no-

drop, MTU), Buffer size, Markingsystem qos

Nexus 5500 QoSUPC (Gen 2) QoS Defaults

QoS is enabled by default (not possible to turn it off)

Three default class of services defined when systemboots up

Gen 2 UPC




# Predefined FCoE service policies

service-policy type qos input fcoe-default-in-policyservice-policy type queuing input fcoe-default-in-policyservice-policy type queuing output fcoe-default-out-policyservice-policy type network-qos fcoe-default-nq-policy

p

Two for control traffic (CoS 6 & 7)

Default Ethernet class (class-default – all others)

Cisco Nexus 5500 switch supports five user-definedclasses and the one default drop system class

FCoE queues are ‘not’ pre-allocated

When configuring FCoE the predefined servicepolicies must be added to existing QoSconfigurations

Gen 2 UPC

Unified Crossbar

Fabric

Gen 2 UPC

VoQs for unicast

(8 per egress port)Classify

CoS/DSCPL2/L3/L4 ACL

If buffer usage crosses threshold:• Tail drop for drop class

• Assert pause signal to MACfor no-drop system class

Central

Scheduler

Nexus 5500 QoSUPC (Gen 2) QoS Capabilities (*Not Currently Supported)




C

r o s s b ar

F

a b r i c

MACTraffic

Classification

IngressCos/DSCP

Marking

MTUchecking

Per-class

Buffer usage

Monitoring

Egress Queues

EgressCOS/DCSP

Marking

PAUSE ON/OFF signal

Truncate or droppackets if MTU is violated

Strict priority +DWRR scheduling

128 muticast queues

MAC

unicast

multicast

Ingress

Policing*

ECN

Marking*Egress

Policing*Egress

scheduling

Proxy Queues

UPC Gen 2

Nexus 5000 Traffic Classification

Packets are classified at ingress forwarding engine

No egress classification

Classification occurs before queuing

Classification rules share the 2K TCAM space with other features




Classification rules share the 2K TCAM space with other features

192 CAM entries for QoS classification rules

Port ACL

VLAN ACL

SPAN

Control Traffic redirection

Matching Criteria

CoS MAC

IP, UDP/TCP port, DSCP, IP Precedence

Protocol Type

Traffic is assigned to one of 8 qos-group

Qos-group is internal to Nexus 5000

Each qos-group represents one class of service

Queueing and network-qos policy are applied to qos-group after classification

Scheduling and Bandwidth Sharing Each qos-group is mapped to one egress queue

Scheduler controls how bandwidth is shared among 8 egressqueues

C t l t ffi i d t t i t i it




Control traffic is mapped to strict priority queue

One qos-group can be mapped to strict priority queue

Non-strict priority queues share bandwidth using Deficit WeightRound Robin (DWRR)

Is control trafficSP queue empty

Schedule non-SP queueUsing DWRR

Schedule the queue

Is userSP queue empty

Schedule the queue

Y

N N

Y

Nexus 5500 QoSUPC (Gen 2) Buffering

640KB dedicated packet buffer per one 10GE port

Buffer is shared between ingress and egress with majority of buffer

being allocated for ingress Ingress buffering model




g g Ingress buffering model

Buffer is allocated per system class

Egress buffer only for in flight packet absorption

Buffer size of ingress queues for drop class can be adjusted usingnetwork-qos

policy

Class of Service Ingress Buffer(KB) Egress Buffer(KB) Class-fcoe

78 19Sup-Hi & Sup-Lo 18.0 & 18.0 9.6 & 9.6

User defined no-drop class of servicewith MTU<2240

78 19

User defined no-drop class of servicewith MTU>2240

88 19

User defined tail drop class of servicewith MTU<2240

22 19

User defined tail drop class of servicewith MTU>2240

29 19

Class-default All remaining buffer 19

DefaultClasses

Nexus 5500 QoSPriority Flow Control and No-Drop Queues




Nexus 5000 supports a number of new QoS conceptsand capabilities

Priority Flow Control is an extension of standard 802.3x

pause frames

No-drop queues provide the ability to support loss-lessEthernet using PFC as a per queue congestion controlsignaling mechanism


Actions when congestion occurs depending onpolicy configuration

PAUSE upstream transmitter for losslesstraffic

SFP SFP SFP SFP

EgressUPC

1. Congestionor Flow

Control onEgress Port




Tail drop for regular traffic when buffer isexhausted

Priority Flow Control (PFC) or 802.3X PAUSEcan be deployed to ensure lossless for

application that can’t tolerate packet loss Buffer management module monitors buffer

usage for no-drop class of service. It signalsMAC to generate PFC (or link level PAUSE)when the buffer usage crosses threshold

FCoE traffic is assigned to class-fcoe, which is

a no-drop system class Other class of service by default have normal

drop behavior (tail drop) but can be configuredas no-drop SFP SFP SFP SFP

Unified

CrossbarFabric

UPC

ingressUPC

2. EgressUPC does notallow Fabric

Grants

3. Traffic isQueued on

Ingress

4. If queue ismarked as no-drop or flowcontrol then

Pause is sent

Tuning of the lossless queues to support avariety of use cases

Extended switch to switch no drop traffic lanes

S t f 3k ith N 5500





Configs for

3000m no-drop

class

Buffer sizePause Threshold

(XOFF)

Resume

Threshold (XON)

N5020 143680 bytes 58860 bytes 38400 bytes

N5548 152000 bytes 103360 bytes 83520 bytes

Support for 3km with Nexus 5500

Increased number of no drop serviceslanes (4) for RDMA and other multi-queueHPC and compute applications

Support for 3 km nodrop switch to

switch linksInter Building DCB

FCoE links

5548-FCoE(config)# policy-map type network-qos 3km-FCoE5548-FCoE(config-pmap-nq)# class type network-qos 3km-FCoE5548-FCoE(config-pmap-nq-c)# pause no-drop buffer-size 152000 pause-threshold 103360resume-threshold 83520

Gen 2 UPC


Gen 2 UPC

Nexus 5500 QoSMTU per Class of Service (CoS Queue)

MTU can be configured for each class of service (no interface level MTU)

No fragmentation since Nexus 5000 is a L2 switch

When forwarded using cut-through, frames are truncated if they are larger




g g , y gthan MTU

When forwarded using store-and-forward, frames are dropped if they arelarger than MTU

class-map type qos iSCSI match cos 2class-map type queuing iSCSI match qos-group 2 policy-map type qos iSCSI

class iSCSIset qos-group 2

class-map type network-qos iSCSI match qos-group 2

policy-map type network-qos iSCSIclass type network-qos iSCSI mtu 9216

system qosservice-policy type qos input iSCSIservice-policy type network-qos iSCSI

Each CoS queue on theNexus 5000 supports a

unique MTU

QoS Configuration — MQC

MQC(Modular QoS CLI) defines three-step configuration

model




Define matching criteria

class-map

Associate action with each defined class

policy-map

Apply policy to entire system or an interface

service-policy

Policy Types

Policy Type Function Attach Point

qos Define traffic classification rules System qos

Ingress Interface




queuing Strict Priority queue

Deficit Weight Round Robin

System qos

Egress Interface

Ingress Interface*

network-qos

System class type(drop or no-drop)

MTU per class of service Buffer size

Marking

System qos

*Queuing policy applied under ingress interface is advertised to server using DCBX protocol

Prefer service policy attached under interface when same type ofservice policy is attached at both system qos and interface Qos and network-qos policy-map are required to create new system

classes

Some key commands to remember

class-map and policy-map type qos

Mostly used for classification and marking (forDSCP)




y g (DSCP)

class-map and policy-map type network-qos

Mostly used for network properties such as queue-size, drop vs no drop / MTU, multicast optimize and

marking (for CoS) class-map and policy-map type queueing

Mostly used for bandwidth allocation (in egress) andassigning the priority

Or to communicate the bandwidth allocation to aCNA (in ingress)

Classification Options – type qos Remember the “qos- group” concept

untagged CoS:

Specifies CoS for untagged frames received on an interfaceswitch(config)# interface ethernet 1/1




switch(config)# interface ethernet 1/1

switch(config-if)# untagged cos 5

Or via policy-map type qos:policy-map type qos classify-5548-global

class voice-global

set qos-group 5

class video-signal-global

set qos-group 4

class critical-global

set qos-group 3

class scavenger-globalset qos-group 2

This could be ACL based

Classification Options - type qosExample of Class i f icat ion

class-map type qos match-anycfy-video

match cos 4 policy-map type qos classify

Order matters




match cos 4

match dscp 34

match access-group …

class-map type qos match-anycfy-transact

match cos 2

match dscp 18

match access-group …

class cfy-video

set qos-group 4

set dscp 34

class cfy-transact set qos-group 3

set dscp 18

Setting Network Properties – type network-qos Drop/no Drop, MTU , multicast optimize etc…

Class-map type network justmatches the qos-group (youcannot match anything else

policy-map type network-qos<name>

class type network-qos video




y gclass-map type network-qos video

match qos-group 4

class-map type network-qos nfs

match qos-group 2

You can set:

MTU

Drop/No Drop

Multicast Optimize Queue size

CoS (notice DSCP is in type qos)

queue-limit <Bytes>

class type network-qos nfs

mtu 9216

set cos 2

Setting Scheduling – type queueing Bandwidth Al locat ion

Class-map type queuing justmatches the qos-group (you

cannot match anything elsel t i id

policy-map type queuing <name>

class type queuing video

bandwidth percent 40




class-map type queuing video

match qos-group 4

class-map type queuing nfs

match qos-group 2

You can set:

Bandwidth allocation

Priority scheduling

p

class type queuing nfs

bandwidth percent 10

priority

Policy Attach Point

System qos configuration context Apply service policy to whole system, i.e., all

interfaces All three types of policy can be applied under




All three types of policy can be applied undersystem qos

Ingress Interface Pol icy- type qos for classification rules

Pol icy- type queuing for strict priority and DWRR.Input queuing policy defines egress queuing policyfor device connected to Nexus 5000, such as CNAor FEX

Egress Interface Output queuing policy for strict priority and DWRR

Set Jumbo MTU

Nexus 5000 supports different MTU for each system class

MTU is defined in network-qos policy-map

No interface level MTU support on Nexus 5000




pp

Following example configures jumbo MTU for all interfaces

N5k(config)# policy-map type network-qos policy-MTUN5k(config-pmap-uf)# class type network-qos class-defaultN5k(config-pmap-uf-c)# mtu 9216N5k(config-pmap-uf-c)# system qosN5k(config-sys-qos)# service-policy type network-qos policy-MTUN5k(config-sys-qos)#

Adjust N5k Ingress Buffer Size

Step 1 Define qos class-map

N5k(config)# ip access-list acl-1N5k(config-acl)# permit ip 100.1.1.0/24 anyN5k(config-acl)# exit

N5k(config)# ip access-list acl-2N5k(config-acl)# permit ip 200.1.1.0/24 anyN5k(config)# class map t pe qos class 1

Step 4 Define network-qos Class-Map

N5k(config)# class-map type network-qos class-1N5k(config-cmap-nq)# match qos-group 2N5k(config-cmap-nq)# class-map type network-qos class-2

N5k(config-cmap-nq)# match qos-group 3




Step 2 Define qos policy-map

N5k(config)# class-map type qos class-1N5k(config-cmap-qos)# match access-group name acl-1N5k(config-cmap-qos)# class-map type qos class-2N5k(config-cmap-qos)# match access-group name acl-2N5k(config-cmap-qos)#

N5k(config)# policy-map type qos policy-qosN5k(config-pmap-qos)# class type qos class-1N5k(config-pmap-c-qos)# set qos-group 2N5k(config-pmap-c-qos)# class type qos class-2N5k(config-pmap-c-qos)# set qos-group 3

N5k(config)# system qosN5k(config-sys-qos)# service-policy type qos input policy-qos

Step 3 Apply qos policy-map under

system qo s

Step 5 Set ingress buffer size forclass-1 in network-qos policy-map

N5k(config)# policy-map type network-qos policy-nqN5k(config-pmap-nq)# class type network-qos class-1

N5k(config-pmap-nq-c) queue-limit 81920 bytes N5k(config-pmap-nq-c)# class type network-qos class-2

N5k(config-pmap-nq-c)# system qosN5k(config-sys-qos)# service-policy type network-qos

policy-nqN5k(config-sys-qos)#

Step 6 Apply network-qos policy-map

under system q os context

Step 7 Configure bandwidth allocationusing queuing policy-map

Configure no-drop system classStep 1 Define qos class-map

N5k(config)# class-map type qos class-nodropN5k(config-cmap-qos)# match cos 4N5k(config-cmap-qos)#


N5k(config)# class-map type network-qos class-1N5k(config-cmap-nq)# match qos-group 2





N5k(config)# policy-map type qos policy-qosN5k(config-pmap-qos)# class type qos class-nodropN5k(config-pmap-c-qos)# set qos-group 2

N5k(config)# system qos

N5k(config-sys-qos)# service-policy type qos input policy-qos

Step 3 Apply qos policy-map undersystem qo s

N5k(config)# policy-map type network-qos policy-nq

N5k(config-pmap-nq)# class type network-qos class-nodropN5k(config-pmap-nq-c) pause no-drop



Step 5 Configure class-nodrop as no-drop class in network-qos policy-map



Step 7 Configure bandwidth allocation

using queuing policy-map

Configure CoS Marking

Step 1 Define qos class-map

N5k(config)# ip access-list acl-1N5k(config-acl)# permit ip 100.1.1.0/24 any

N5k(config-acl)# exitN5k(config)# class-map type qos class-1N5k(config-cmap-qos)# match access-group name acl-1


N5k(config)# class-map type network-qos class-1N5k(config-cmap-nq)# match qos-group 2





( g p q ) g pN5k(config-cmap-qos)#

N5k(config)# policy-map type qos policy-qosN5k(config-pmap-qos)# class type qos class-1N5k(config-pmap-c-qos)# set qos-group 2

N5k(config)# system qosN5k(config-sys-qos)# service-policy type qos input policy-qos

Step 3 Apply qos policy-map undersystem qo s

Step 5 Enable CoS marking for class-1

in network-qos policy-map

N5k(config)# policy-map type network-qos policy-nqN5k(config-pmap-nq)# class type network-qos class-1

N5k(config-pmap-nq-c) set cos 4





Step 7 Configure bandwidth allocationfor new system class using queuing policy-map

DSCP/IP Precedence Marking on 5548

On the N5548 ‘dscp’ or ‘ip precedence’ marking can beconfigured in ‘type qos input’ policy (attached at“system qos” or “interface”)




Switch-6(config-cmap-qos)# policy-map type qos cos1-dscp-IFSwitch-6(config-pmap-qos)# class type qos class-1

Switch-6(config-pmap-c-qos)# set dscp efSwitch-6(config-pmap-c-qos)# set qos-group 2

Switch-6(config-cmap-qos)# policy-map type qos cos1-precedenceSwitch-6(config-pmap-qos)# class type qos class-1Switch-6(config-pmap-c-qos)# set precedence 2Switch-6(config-pmap-c-qos)# set qos-group 2

Revert QoS policy to default configuration

Display service policy under system qos

context

N5k# sh run | begin "system qos"system qosservice-policy type qos input policy-qos

service-policy type network-qos policy-nqservice-policy type queuing output policy-BW




Display default policy-map name with show pol icy-map Name of the default policy-map starts with default

Default qos policy-map: default- in-pol icy Default network-qos policy-map: defaul t -nq-pol icy

Default egress queuing policy-map: default- in-pol icy

N5k(config)# system qosN5k(config-sys-qos)# service-policy type qos input default-in-policyN5k(config-sys-qos)# service-policy type network-qos default-nq-policyN5k(config-sys-qos)# service-policy type queuing output default-

out-policy

N5k(config-sys-qos)#interface e1/1N5k(config-if)# no service-policy type qos input policy-qos

Revert QoS service policy to defaultpolicy by applying default policy-mapunder system q os

no s erv ice-pol icy command doesn’texist under system qos

Interface level service policy can be

removed with no s erv ice-pol icy

command

Nexus 5500 QoSMapping the Switch Architecture to ‘show queuing’

dc11-5548-4# sh queuing int eth 1/39

Interface Ethernet1/39 TX Queuingqos-group sched-type oper-bandwidth

0 WRR 50

SFP SFP SFP SFP

Egress (Tx) QueuingConfiguration




1 WRR 50

Interface Ethernet1/39 RX Queuingqos-group 0

q-size: 243200, HW MTU: 1600 (1500 configured)drop-type: drop, xon: 0, xoff: 1520

Statistics:Pkts received over the port : 85257Ucast pkts sent to the cross-bar : 930 Mcast pkts sent to the cross-bar : 84327Ucast pkts received from the cross-bar : 249Pkts sent to the port : 133878Pkts discarded on ingress : 0Per-priority-pause status : Rx (Inactive), Tx (Inactive)

<snip – other classes repeated>

Total Multicast crossbar statistics: Mcast pkts received from the cross-bar : 283558

UnifiedCrossbar

Fabric

UPC

Packets Arriving on this portbut dropped from ingress

queue due to congestion onegress port




Troubleshooting

SPAN




Nexus 5500 SPAN Features

4 active SPAN sessions

Protects data traffic when experiencing congestionith SPAN




with SPAN

ACL based SPAN to monitor selected flows (Future)

For ingress SPAN, replicate packets before the packets

are rewritten. For egress SPAN replicate packets afterpackets are rewritten

Support ERSPAN. Accurately timestamp packets byincluding IEEE 1588 timestamp in ERSPAN header

Option to truncate SPAN packets to reducebandwidth (Future)

Support FEX ports as SPAN destination port (Future)

Ingress SPAN Packet Flow

Data is replicated at ingress port ASIC-Unified PortController(UPC)

SPAN packets is queued at the SPAN destination port VOQ

E h t h 12Gb ti t it h f b i D t k t d




Ingress interface (rx SPAN source) U

ni f i e d

F a b r i c C

on t r ol l er

Egress Interface

PacketBuffer

Unicast VOQ

Multicast VOQ

SPAN

Destination

12Gbps

12Gbps

12Gbps

Each port has 12Gbps connection to switch fabric. Data packets andSPAN packets share the 12Gbps fabric connection at SPAN source.

data

span

data

span

Egress SPAN Packet Flow SPAN copy is made at egress pipe of the TX SPAN source port.

SPAN packets are looped back to ingress pipe of UPC and

sent to switch fabric SPAN and data share the 12Gbps fabric link




SPAN and data share the 12Gbps fabric link

Ingress Interface

Egress Interface(tx SPAN source)

PacketBuffer

Unicast VOQ

Multicast VOQ SPANDestination

12Gbps

12Gbps

12Gbps

data

span

dataspanUnicast VOQ

data U

ni f i e d

F a b r i c C

on t r ol l er

Protecting Data TrafficRX SPAN

Ingress interface measures thefabric link(connection between10GE port and switch fabric)

utilization at SPAN source port SPAN policing kicks in when

Ingress Interface

(rx SPAN source)




SPAN policing kicks in whenincoming data traffic rate is close to6Gbps for RX SPAN source. Forsmall frame size, policing kicks in at5Gbps due to internal header

SPAN policing regulates the allowedbandwidth for SPAN traffic.Production data traffic always getfabric bandwidth

SPAN and data traffic are stored inseparate packet buffer pools.

SPAN traffic won’t affect datatraffic when SPAN destination portis congested

Packet Buffer

12Gbps

dataspan

Traffic meter

SPAN Policing

Un

i f i e d

F a b r i c C o

n t r ol l er

Protecting Data TrafficTX SPAN

Egress Interface

(tx SPAN source)

TX SPAN source interface measures thereceived traffic rate

SPAN policing is enabled ONLY when RX

traffic rate is higher than 6Gbps for TXSPAN source port. For small framepolicing kicks in with 5Gbps RX traffic




Ingress Interface

SPANDestination

12Gbps

12Gbps

12Gbpsspan

TX data

span

Traffic meter

SPAN Policingpolicing kicks in with 5Gbps RX traffic

Separate buffer pool for SPAN and data

Uni f i e d

F a b r i c

C on t r ol l er

TX data

RX dataRX data

Expected SPAN Performance for Each SPANSource

8

10

12




0

2

4

6

1 2 3 4 5 6 7 8 9 10

Received traffic rate

Data throughput

SPAN throughput per source

This charts assume the SPAN policing kicks in at 5.5Gbps traffic and policing

rate for SPAN traffic is set to 0.75Gbps per SPAN source interface.

SPAN PerformanceScenario 1: No oversubscription

Monitor session 1

source interface eth1/1 rx


destination interface eth1/12

eth1/1 Eth1/2

5Gbps 5Gbps

Unified Port Controller




Two rx SPAN source interfaces each

carries 5Gbps traffic

Total traffic need to be monitored is10Gbps

No congestion point. All data and SPAN

traffic are received at egress

Eth1/10 Eth1/11

eth1/5Unified Port Controller

Unified Fabric Controller

5Gbps 5Gbps

Eth1/12

10Gbps

Sniffer



SPAN PerformanceScenario 3-Fabric Link Oversubscription

Monitor session 1



destination interface eth1/12

eth1/1 Eth1/2

8Gbps 8Gbps

Unified Port Controller




SPAN source interface carries 8Gbps

Fabric link between SPAN source port and

switch fabric is congestion point SPAN policing kicks in and rate limits the

SPAN traffic

Data traffic is not affected. SPAN

throughput for each SPAN source will be

the pre-configured poling rate( Assumepolicing is configured as 0.75Gbps in this

example)

Eth1/10 Eth1/11

eth1/5Unified Port Controller

Unified Fabric Controller

8Gbps 8Gbps

Eth1/12

1.5Gbps

Sniffer

SPAN Configuration

Configure the Destination SPAN Port:

A SPAN destination port needs to be configured as a switchport monitor port forthe session to become active.




n5000(config)# interface ethernet 2/14n5000(config-if)# switchportn5000(config-if)# switchport monitor

Configure the Monitor (SPAN) Session:

n5000(config)# monitor session 1n5000(config-monitor)# description Inbound(rx) SPAN on Eth 2/13n5000(config-monitor)# source interface ethernet 2/13 rxn5000(config-monitor)# destination interface ethernet 2/14n5000(config-monitor)# no shut

Monitor (SPAN) Options:

n5000(config-monitor)# ?

description Session description (max 32 characters)destination Destination configurationexit Exit from command interpreterfilter Filter configurationno Negate a command or set its defaultsshut Shut a monitor sessionsource Source configuration

Configure destination “monitor” port

VLAN Filter for 802.1q tagged trunks

Sessions must be activated

Port = “ethernet”, “port-channel”, or “sup-eth” Traffic = “rx”, “tx”, or “both”

SPAN VerificationVerifying the Destination Port Type:

n5500# show interface ethernet 2/14Ethernet2/14 is up

Hardware is 10/100/1000 Ethernet, address is 001b.54c0.fedd (bia 001b.54c0.fedd)MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA




n5500# show monitor session 1session 1

---------------description : Inbound(rx) SPAN on Eth 2/13type : localstate : upsource intf :

rx : Eth2/13tx :both :

source VLANs :rx :tx :both :

filter VLANs : filter not specifieddestination ports : Eth2/14

Verifying the SPAN Session:

Port mode is accessfull-duplex, 1000 Mb/sBeacon is turned offAuto-Negotiation is turned onInput flow-control is off, output flow-control is offAuto-mdix is turned onSwitchport monitor is onLast clearing of "show interface" counters never

Switchport mode

Operational monitor session = “up”

Other options:

= down (Session admin shut)

= down (No hardware resource)

Source Interface = rx

Destination interface

Ethanalyzer




Ethanalyzer (Control Plane Traffic)

Ethanalyzer is an internal CLI based protocol analyzer that captures packets onthe CPU control plane (ingress or egress). Ethanalyzer is useful whentroubleshooting CPU and/or control plane related issues.

The packets can be viewed using the CLI or exported to a Wireshark protocolanalyzer on an external host for GUI analysis




Configured in “user-exec” mode

Three interface options can be specified - “inbound-hi”, “inbound-low”, “mgmt”

10 packet capture limit by default – Configurable up to 2.1 billion packets

Packet contents scroll on the console by default

Packet capture can be redirected to a destination file - Recommended

Brief or Detailed analysis available (Brief is enabled by default)

User configurable Frame-Size, with Capture and Display Filter options

analyzer on an external host for GUI analysis.

Ethanalyzer Guidelines:

Nexus 5500 Hardware OverviewControl Plane Elements

CPUIntel LV Xeon1.66 GHz

SouthB id

Monitoring of in-band traffic via NX-OSbuilt-in ethanalyzer (sniffer)

Eth3 is equivalent to ‘inbound-lo’

Eth4 is equivalent to ‘inbound-hi’




Bridge

NIC

Unified PortController

eth3 eth4

N5k-2# ethanalyzer local interface ?inbound-hi Inbound(high priority) interfaceinbound-low Inbound(low priority) interface


DCN-N5K1# show hardware internal cpu-mac inband counterseth3 Link encap:Ethernet HWaddr 00:0D:EC:B2:2A:C3

UP BROADCAST RUNNING PROMISC ALLMULTI MULTICAST MTU:9216 Metric:1RX packets:5603201 errors:0 dropped:0 overruns:0 frame:0TX packets:30249490 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:1000RX bytes:682915556 (651.2 MiB) TX bytes:5638322004 (5.2 GiB)Base address:0x6020 Memory:fa4a0000-fa4c0000

eth4 Link encap:Ethernet HWaddr 00:0D:EC:B2:2A:C4UP BROADCAST RUNNING PROMISC ALLMULTI MULTICAST MTU:2200 Metric:1RX packets:81560230 errors:0 dropped:0 overruns:0 frame:0TX packets:38145612 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:1000RX bytes:24429668210 (22.7 GiB) TX bytes:4141361337 (3.8 GiB)Base address:0x6000 Memory:fa440000-fa460000

CLI view of in-band control plane data

Ethanalyzer Configuration

Capture using Defaults and Write to a File on Bootflash:

Create a Capture:n5500# ethanalyzer local interface

inbound-hi inbound-hi/Outband interface





n5500# ethanalyzer local interface inbound-hi write bootflash:ethanalyzer-data

Capturing on inbound-hi10

n5500# ethanalyzer local interface inbound-hi ?

<CR>> Redirect it to a file>> Redirect it to a file in append modecapture-filter Filter on ethanalyzer capturedecode-internal Include internal system header decodingdetailed-dissection Display detailed protocol informationdisplay-filter Display filter on frames captureddump-pkt Hex/Ascii dump the packet with possibly one line summarylimit-captured-frames Maximum number of frames to be captured (default is 10)limit-frame-size Capture only a subset of a frame

write Filename to save capture to

p g

Additional Capture Options:

Writes to a file instead of the console

Applies a capture-filter to limit data

n5500# ethanalyzer local interface inbound-hi limit-frame-size ?

<64-65536> Size in bytes

Limit Captured Frame Size:

Slice packets for headers only

Real-Time counter

Ethanalyzer Capture-Filter Configuration

Capture filters can be used to reduce the amount of data collected whentroubleshooting. The following CLI illustrates some basic examples.

The capture filter syntax is the same as tcpdump (also same as Wireshark).




n5500# ethanalyzer local

interface

inbound-hi capture-filter icmp


interface

inbound-hi capture-filter tcp


interface

inbound-hi capture-filter udp


interface

inbound-hi capture-filter ip proto ospfn5500# ethanalyzer local

interface

inbound-hi capture-filter ip proto eigrp


interface

inbound-hi capture-filter src net 192.168.204.2

n5500# ethanalyzer local interface inbound-hi capture-filter dst net 224.0.0.2


interface

inbound-hi capture-filter tcp dst port 23


interface

inbound-hi capture-filter tcp src port 23


interface

inbound-hi capture-filter udp dst port 23


interface

inbound-hi capture-filter udp src port 23


interface

inbound-hi capture-filter src net 10.20.0.190 and tcp dst port 23


interface

inbound-hi capture-filter dst net 224.0.0.2 and udp dst port 1985

Ethanalyzer “Brief” Output (Console)

The Ethanalyzer output defaults to brief mode for collecting an initial snapshot ofpackets on the CPU control plane. If more information is needed, perform adetailed capture and specify a capture-filter for a more specific match.

Packets will scroll on the screen to the specified capture limit (Default is 10)




n5500# ethanalyzer local interface inbound-hi

Capturing on inbound-hi2008-06-02 20:44:40.327808 192.168.20.1 -> 224.0.0.5 OSPF Hello Packet2008-06-02 20:44:41.480658 192.168.20.2 -> 207.68.169.104 DNS Standard query A print.cisco.com2008-06-02 20:44:41.730633 192.168.20.2 -> 207.68.169.104 DNS Standard query A print.cisco.com2008-06-02 20:44:41.730638 192.168.20.2 -> 65.54.238.85 DNS Standard query A print.cisco.com2008-06-02 20:44:42.480586 192.168.20.2 -> 65.54.238.85 DNS Standard query A print.cisco.com2008-06-02 20:44:43.480513 192.168.20.2 -> 207.68.169.104 DNS Standard query A print.cisco.com2008-06-02 20:44:45.480499 192.168.20.2 -> 207.68.169.104 DNS Standard query A print.cisco.com2008-06-02 20:44:45.480506 192.168.20.2 -> 65.54.238.85 DNS Standard query A print.cisco.com2008-06-02 20:44:46.308177 192.168.10.1 -> 224.0.0.5 OSPF Hello Packet2008-06-02 20:44:46.974771 192.168.10.2 -> 224.0.0.5 OSPF Hello Packet

Packets will scroll on the screen to the specified capture limit. (Default is 10)

The output can also be copied to a local flash (i.e. bootflash, logflash, usb1, usb2)

Ethanalyzer “Detailed” Output (Console)

Packets will scroll on the screen to the specified capture limit. (The default is 10)

Use the detail option to capture detailed packet information.




n5500# ethanalyzer local interface inbound-hi detail

Capturing on inbound-hi

Capturing on inbound-hiFrame 1 (60 bytes on wire, 60 bytes captured)

Arrival Time: Nov 2, 2009 22:07:57.150394000[Time delta from previous captured frame: 0.000000000 seconds][Time delta from previous displayed frame: 0.000000000 seconds]

[Time since reference or first frame: 0.000000000 seconds]Frame Number: 1Frame Length: 60 bytesCapture Length: 60 bytes[Frame is marked: False][Protocols in frame: eth:llc:stp]

IEEE 802.3 EthernetDestination: 01:80:c2:00:00:00 (01:80:c2:00:00:00)

Address: 01:80:c2:00:00:00 (01:80:c2:00:00:00).... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast).... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)

Source: 00:0d:ec:6d:96:6f (00:0d:ec:6d:96:6f)

Address: 00:0d:ec:6d:96:6f (00:0d:ec:6d:96:6f).... ...0 .... .... .... .... = IG bit: Individual address (unicast).... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)

Length: 39Trailer: 00000000000000

<Text Omitted>

Reading Ethanalyzer Output Locally

You don’t need to specify an output option when writing a capture to a localdestination. Use the detail option if you want to see the packet details.

Brief:




n5500# ethanalyzer local read bootflash:ethanalyzer-data

00:0d:ec:6d:96:6f -> 01:00:0c:cc:cc:cc CDP Device ID: MSDC-N5K-01(FLC12100023) Port ID: Ethernet1/4000:1b:54:c1:0a:69 -> 01:00:0c:cc:cc:cd STP RST. Root = 32788/00:18:ba:d8:58:25 Cost = 2 Port = 0x900900:1b:54:c1:0a:69 -> 01:80:c2:00:00:00 STP RST. Root = 32769/00:18:ba:d8:58:25 Cost = 2 Port = 0x900900:1b:54:c1:0a:69 -> 01:00:0c:cc:cc:cd STP RST. Root = 32769/00:18:ba:d8:58:25 Cost = 2 Port = 0x9009192.168.1.2 -> 224.0.0.10 EIGRP Hello

Note: Timestamps Omitted

n5500# ethanalyzer local read bootflash:ethanalyzer-data detail

Frame 1 (268 bytes on wire, 268 bytes captured)Arrival Time: Nov 2, 2009 21:50:18.794493000[Time delta from previous captured frame: 0.000000000 seconds][Time delta from previous displayed frame: 0.000000000 seconds][Time since reference or first frame: 0.000000000 seconds]Frame Number: 1Frame Length: 268 bytesCapture Length: 268 bytes[Frame is marked: False][Protocols in frame: eth:llc:cdp:data]

IEEE 802.3 EthernetDestination: 01:00:0c:cc:cc:cc (01:00:0c:cc:cc:cc)

Address: 01:00:0c:cc:cc:cc (01:00:0c:cc:cc:cc)

Reading detailed output from local bootflash: Detailed:

Core Files & Logging




System Crash

Determine the reset reason and how long since lastreset:

DCN-N5K1# show system reset-reason----- reset reason for Supervisor-module 1 (from Supervisorin slot 1) ---1) At 574259 usecs after Thu Jul 21 18:59:24 2011

Reason: Reset Requested by CLI command reloadService:

( ) ( )




Version: 5.0(3)N1(1b)

2) At 605182 usecs after Tue Apr 19 20:53:24 2011Reason: Disruptive upgradeService:Version: 4.2(1)N2(1a)

3) At 465315 usecs after Tue Apr 19 20:33:43 2011Reason: Reset by installerService:Version: 4.1(3)N2(1)

4) At 370523 usecs after Tue Apr 19 20:02:18 2011Reason: Reset Requested by CLI command reloadService:Version: 4.1(3)N2(1)

DCN-N5K1# show system uptimeSystem start time: Thu Jul 21 19:04:28 2011System uptime: 34 days, 6 hours, 41 minutes, 30 secondsKernel uptime: 34 days, 6 hours, 48 minutes, 10 secondsActive supervisor uptime: 34 days, 6 hours, 41 minutes, 30 seconds

Process Crash

Investigate syslog file for errors:

switch# show log logfile | include error

R th h d St t f ER




Run the show processes command. State of ER indicates process should be running but is not.

Check the process log for a stack trace or core dump:

DCN-N5K1# show process log

Process PID Normal-exit Stack Core Log-create-time--------------- ------ ----------- ----- ----- ---------------installer 24484 N N N Wed Jun 23 16:26:47 2010installer 24493 N N N Wed Jun 23 16:27:18 2010installer 24508 N N N Wed Jun 23 16:28:14 2010

Core Files & Logging

switch# show cores

Module-num Process-name PID Core-create-time

---------- ------------ --- ----------------

1 fwm 2834 Aug 13 16:3

Show cores:

Copy to a remote server:

switch# copy core:?




switch# copy core:?

core: Enter URL "core://<module-number>/<process-id>"

switch# copy core://1/2834 ftp://128.107.65.217/ vrf management

Enter username: anonymousPassword:

***** Transfer of file Completed Successfully *****

OBFL Logging: N5K-S003-LAB# sh logg onboard exception-log

----------------------------

OBFL Data for

Module: 1

----------------------------

N5K-S003-LAB# sh logg last 20

Grab a “show tech-support”

Sometimes too general

Large file, time consuming

Or not…




If time permits, use targeted outputs or a specificshow tech

If there is no time, use tac-pac and copy offMuch quicker than transmitting to terminal

Zips entire output to file in volatile:

Copy file off of switch for analysis

N5k-1# tac-pacN5k-1# dir volatile:

180242 Jan 28 4:37:26 2011 show_tech_out.gz

Which show tech? As of 5.0(3), There Are 68

N5k-1# show tech-support ?

aaa Display aaa information

aclmgr ACL commands

adjmgr Display Adjmgr informationarp Display ARP information

ascii-cfg Show ascii-cfg information for technical support personnel

assoc mgr Gather detailed information for assoc mgr troubleshooting




assoc_mgr Gather detailed information for assoc_mgr troubleshooting

bcm-usd Gather detailed information for BCM USD troubleshooting

bootvar Gather detailed information for bootvar troubleshooting

brief Display the switch summary

btcm Gather detailed information for BTCM component

callhome Callhome troubleshooting information

cdp Gather information for CDP trouble shooting...

session-mgr Gather information for troubleshooting session manager

snmp Gather info related to snmp

sockets Display sockets status and configuration

spm Service Policy Manager

stp Gather detailed information for STP troubleshooting

sysmgr Gather detailed information for sysmgr troubleshooting

time-optimized Gather tech-support faster, requires more memory & disk space

track Show track tech-support informationvdc Gather detailed information for VDC troubleshooting

vpc Gather detailed information for VPC troubleshooting

vtp Gather detailed information for vtp troubleshooting

xml Gather information for xml trouble shooting

Logging

show logging logfile

Basis for tracing events chronologicallyTry using start-time or last

Often Overlooked, but very Important




show accounting log

Basis for tracing configuration changes

terminal log-all to also log show commands

All commands end with (SUCCESS) or (FAILURE)

N5k-1# show logging logfile start-time 2011 Mar 9 20:00:00

2011 Mar 9 20:17:18 esc-n5548-1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/1 is down (None)

2011 Mar 9 20:17:18 esc-n5548-1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/3 is down (None)

N5k-1# show logging last ?

<1-9999> Enter number of lines to display

Other System Logsshow logging nvram

Survives reloads – helpful for crash or reload issues

show process log details

P f il it




Process failure or exit reason

Onboard Failure Logging

show logging onboard obfl-logsshow logging onboard obfl-history

show logging onboard exception log

show logging onboard kernel-trace

show system reset-reason

Hardware Issues




POST and OHMS (Online Health Monitoring System)Types Of Errors Types of Reaction

Failures causing NXOSnot be able to comeup properly

Console continuous print error messages every 30 seconds.

System LED sets to Flashing Amber . Example of such failure:

DRAM, backplane SPROM checksum failure, PCIe enumeration

failure Failures not fatal and

NXOS can boot upSystem comes all the way up. Syslog, OBFL and callhomeinitiated to indicate failure. Example of such failure: OBFL flash,




p pCTS keystore.

Failure causing portfailures

System comes all the way up. Syslog, OBFL and callhomeinitiated to indicate failure. Example of such failure: ASIC ECCerror found during POST or OHMS

N5K-C5548P-L11-01# sh platform nohms errors1) Event:E_DEBUG, length:79, at 806296 usecs after Sun Apr 1809:57:02 2010

[102] nohms_process_lc_online(350): FEX-100 On-line (Serial Number JAF1307BHCD)

2) Event:E_DEBUG, length:57, at 498025 usecs after Sun Apr 1809:57:00 2010

[102] nohms_handle_lc_inserted(191): n_errs 0 n_notices 0

NOHM (Online Health Monitoring) loggingswitch# show logging |grep NOHMS

2008 Apr 18 23:00:01 switch %NOHMS-2-NOHMS_DIAG_ERROR: Module 1: Runtimediag detected major event: Port failure: Ethernet1/1






diag detected major event: Port failure: Ethernet1/5


2008 Apr 19 01:45:25 swor35p %NOHMS-2-NOHMS_ENV_ERROR: Module 1temperature sensor 1 failed.

2008 Apr 19 01:45:25 swor35p %NOHMS-2-NOHMS_ENV_ERROR: Module 1temperature sensor 2 failed.

2008 Apr 19 01:45:25 swor35p %NOHMS-2-NOHMS_ENV_ERROR: System majortemperature alarm on Module 1. Sensor 9 Temperature 42 MajorThreshold 0

Environmental Monitoring

switch# show environment

Displays following status:




Fan

Temperature

Power Supply

Power Usage Summary

Diagnostic Resultswitch# show diagnostic result module 1Current bootup diagnostic level: complete

Module 1: 40x10GE/Supervisor SerialNo : JAB1208005T

Overall Diagnostic Result for Module 1 : PASS

Diagnostic level at card bootup: complete

Test results: (. = Pass, F = Fail, I = Incomplete,U = Untested, A = Abort)

1) TestUSBFlash ------------------------> .

15) TestFabricPort :

Eth 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Port ------------------------------------------------------------

. . . . . . . . . . . . . . . . . . . .

Eth 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

Port ------------------------------------------------------------

. . . . . . . . . . . . . . . . . . . .




2) TestSPROM ---------------------------> .

3) TestPCIe ----------------------------> .

4) TestLED -----------------------------> .

5) TestOBFL ----------------------------> .

6) TestNVRAM ---------------------------> .

7) TestPowerSupply ---------------------> F

8) TestTemperatureSensor ---------------> .9) TestFan -----------------------------> .

10) TestVoltage -------------------------> .

11) TestGPIO ----------------------------> .

12) TestSupervisorPort ------------------> .

13) TestMemory --------------------------> .

14) TestFabricEngine :

Eth 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Port ------------------------------------------------------------

. . . . . . . . . . . . . . . . . . . .

Eth 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

Port ------------------------------------------------------------

. . . . . . . . . . . . . . . . . . . .

16) TestForwardingEngine :

Eth 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Port ------------------------------------------------------------

. . . . . . . . . . . . . . . . . . . .

Eth 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

Port ------------------------------------------------------------

. . . . . . . . . . . . . . . . . . . .

17) TestForwardingEnginePort :

Eth 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Port ------------------------------------------------------------

. . . . . . . . . . . . . . . . . . . .

Eth 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

Port ------------------------------------------------------------

. . . . . . . . . . . . . . . . . . . .

Show tech

switch# terminal length 0

switch# show tech-support details

`show switchname`

switch

`show system uptime`

System start time: Mon Aug 11 15:33:17 2008

Capture to terminal emulator buffer or log file:




System start time: Mon Aug 11 15:33:17 2008

System uptime: 2 days, 0 hours, 46 minutes, 4 seconds

.

.

.

OrCapture to file in volatile:

switch# tac-pac

switch# dir volatile:

66860 Aug 13 16:23:03 2008 show_tech_out.gz

switch# copy volatile:show_tech_out.gz sftp://[email protected]/ vrf management

Port Issues




Ethernet Interface Countersswitch# show interface eth1/21Ethernet1/21 is up

Hardware is 10000 Ethernet, address is 000d.ec6d.84dc(bia 000d.ec6d.84dc)

MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255Encapsulation ARPA

Port mode is access

full-duplex, 10000 Mb/s




Input flow-control is off, output flow-control is off

5 minute input rate 22203 bytes/sec, 346 packets/sec

5 minute output rate 640597 bytes/sec, 10000 packets/sec

Rx

16501327 Input Packets 9 Unicast Packets 16500923Multicast Packets

395 Broadcast Packets 0 Jumbo Packets 0 StormSuppression Packets

1056159080 Bytes

0 No buffer 0 runt 0 crc 0 ecc

0 Overrun 0 Underrun 0 Ignored 0 Bad etype drop

0 Bad proto drop 0 If down drop 0 Collision

0 Late collision 0 Lost carrier 0 No carrier

0 Babble

Tx

433943286 Output Packets 26171 Multicast Packets

0 Broadcast Packets 0 Jumbo Packets

27772499094 Bytes

0 Ouput errors

16499333 Rx pause 0 Tx pause 0 Reset

Ethernet Interface Countersswitch# sh interface ethernet 1/17 counters detailed all

64 bit counters:

0. rxHCTotalPkts = 475168

1. txHCTotalPks = 3445907

2. rxHCUnicastPkts = 1390

3. txHCUnicastPkts = 2053

4. rxHCMulticastPkts = 191780

5. txHCMulticastPkts = 473324

6. rxHCBroadcastPkts = 281998

7. txHCBroadcastPkts = 2970530




14. rxTxHCpkts512to1023Octets = 195759



All Port Counters:

0. InPackets = 47516827. ShortFrames = 0

28. Collisions = 0

29. SingleCol = 0

30. MultiCol = 0

31. LateCol = 0

32. ExcessiveCol = 0

33. LostCarrier = 0

34. NoCarrier = 0

35. Runts = 0

36. Giants = 0

N5K# show interface E1/13 counters errors

--------------------------------------------------------------------------------

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards

--------------------------------------------------------------------------------

Eth1/13 0 0 0 0 0 0

--------------------------------------------------------------------------------

Port Single-Col Multi-Col Late-Col Exces-Col Carri-Sen Runts

--------------------------------------------------------------------------------

Eth1/13 0 0 0 0 0 0

Interface Error Counters




--------------------------------------------------------------------------------

Port Giants SQETest-Err Deferred-Tx IntMacTx-Er IntMacRx-Er Symbol-Err

--------------------------------------------------------------------------------

Eth1/13 0 -- 0 0 0 0

N5K# show interface e1/13 flowcontrol

--------------------------------------------------------------------------------

Port Send FlowControl Receive FlowControl RxPause TxPause

admin oper admin oper

--------------------------------------------------------------------------------

Eth1/13 off off off off 0 0

N5K# show interface e1/13 priority-flow-control============================================================

Port Mode Oper(VL bmap) RxPPP TxPPP

============================================================

Ethernet1/13 Auto Off 0 0

QoS Countersd14-switch-1# show policy-map interface ethernet 3/1

Ethernet3/1

Service-policy system: global

class-map: class-fcoe

Statistics:Pkts received over the port : 0

Ucast pkts sent to the cross-bar : 0

Ucast pkts received from the cross-bar : 0




Ucast pkts received from the cross bar : 0

Pkts sent to the port : 0

Pkts discarded on ingress : 0

Per-priority-pause status : Rx (Inactive), Tx (Inactive)

class-map: class-defaultStatistics:

Pkts received over the port : 761951066

Ucast pkts sent to the cross-bar : 429740044

Ucast pkts received from the cross-bar : 3127717414

Pkts sent to the port : 3308485758

Pkts discarded on ingress : 9038

Per-priority-pause status : Rx (Inactive), Tx (Inactive)

Multicast crossbar statistics:

Mcast pkts sent to the cross-bar : 140042101

Mcast pkts received from the cross-bar : 357560270

QoS CountersDCN-N5K1(config-if)# show queuing interface e1/1

Ethernet1/1 queuing information:TX Queuingqos-group sched-type oper-bandwidth

0 WRR 50

1 WRR 50

RX Queuingqos-group 0q-size: 243200, HW MTU: 1600 (1500 configured)




q size: 243200, HW MTU: 1600 (1500 configured)drop-type: drop, xon: 0, xoff: 1520Statistics:

Pkts received over the port : 6330629Ucast pkts sent to the cross-bar : 5580600Mcast pkts sent to the cross-bar : 750029Ucast pkts received from the cross-bar : 7695639

Pkts sent to the port : 10598898Pkts discarded on ingress : 0Per-priority-pause status : Rx (Inactive), Tx (Inactive)

qos-group 1q-size: 76800, HW MTU: 2240 (2158 configured)drop-type: no-drop, xon: 128, xoff: 240Statistics:

Pkts received over the port : 0Ucast pkts sent to the cross-bar : 0Mcast pkts sent to the cross-bar : 0Ucast pkts received from the cross-bar : 1Pkts sent to the port : 1Pkts discarded on ingress : 0Per-priority-pause status : Rx (Inactive), Tx (Inactive)

Total Multicast crossbar statistics:Mcast pkts received from the cross-bar : 2905930

Monitoring PAUSE frame counters

switch# show int ethernet 1/5 priority-flow-control

-------------------------------------------------------------------------------

-Port Mode Oper RxPPP TxPPP

-------------------------------------------------------------------------------




-

Eth1/5 auto on 2967222 0

switch# show interface ethernet 1/6 flowcontrol

-------------------------------------------------------------------------------

-

Port Send FlowControl Receive FlowControl RxPause TxPause

admin oper admin oper

-------------------------------------------------------------------------------

-

Eth1/5 off off off off 3127212 0

N5K# show interface e1/13 transceiver detailsEthernet1/13

sfp is present

name is CISCO-AVAGOpart number is SFBR-7700SDZ

revision is B4

serial number is AGD121321JFnominal bitrate is 10300 MBits/sec

Interface Transceiver Details




nominal bitrate is 10300 MBits/sec

Link length supported for 50/125um fiber is 82 m(s)

Link length supported for 62.5/125um fiber is 26 m(s)

cisco id is --cisco extended id number is 4

SFP Detail Diagnostics Information (internal calibration)----------------------------------------------------------------------------

Alarms Warnings

High Low High Low----------------------------------------------------------------------------

Temperature 35.87 C 75.00 C -5.00 C 70.00 C 0.00 C

Voltage 3.26 V 3.59 V 3.00 V 3.46 V 3.13 V

Current 6.43 mA 10.50 mA 2.50 mA 10.50 mA 2.50 mATx Power -2.46 dBm 1.49 dBm -11.30 dBm -1.50 dBm -7.30 dBm

Rx Power -2.63 dBm 1.99 dBm -13.97 dBm -1.00 dBm -9.91 dBm

----------------------------------------------------------------------------Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning

Troubleshooting “sfpInvalid” Status

switch# show logging | grep 1/7

2005 Jul 1 16:07:41 switch %ETHPORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver forinterface Ethernet1/7 is not supported

DCN-N5K1(config-if)# show int e1/1Ethernet1/1 is down (SFP validation failed)




pp

2005 Jul 1 16:07:41 switch %ETHPORT-3-IF_UNSUPPORTED_TRANSCEIVER_VENDOR:Transceiver vendor for interface Ethernet1/7 is not supported

switch#

switch# show system internal ethpm event-history errors | grep 1/7 [102] Ifindex (Ethernet1/7)0x2006000, SFP security check: CRC failed, rcvd CRC0x0 calculated CRC 0xe9777080

Most Common Reason for sfpInvalid:‘speed 1000’ missing from a 1Gig SFP

Error Disabled Interfaceswitch# show interface e1 14

e1/7 is down (errDisabled)

switch# show system internal ethpm event-history interface e1/7

>>>>FSM: <e1/7> has 86 logged transitions<<<<<1) FSM:<e1/7> Transition at 647054 usecs after Tue Jan 1 22:44..

i [ ]

View internal state transition info:




Previous state: [ETH_PORT_FSM_ST_NOT_INIT]Triggered event: [ETH_PORT_FSM_EV_MODULE_INIT_DONE]Next state: [ETH_PORT_FSM_ST_IF_INIT_EVAL]

2) FSM:<e1/7> Transition at 647114 usecs after Tue Jan 1 22:43..Previous state: [ETH_PORT_FSM_ST_INIT_EVAL]Triggered event: [ETH_PORT_FSM_EV_IE_ERR_DISABLED_CAP_MISMATCH]Next state: [ETH_PORT_FSM_ST_IF_DOWN_STATE]

switch# show logging logfile. . .Jan 4 06:54:04 switch %PORT_CHANNEL-5-CREATED: port-channel 7 created

Jan 4 06:54:24 switch %PORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel 7 is down (No operational members)Jan 4 06:54:40 switch %PORT_CHANNEL-5-PORT_ADDED: e1/8 added to port-channel 7Jan 4 06:54:56 switch %PORT-5-IF_DOWN_ADMIN_DOWN: Interface e1/7 is down(Admnistratively down)Jan 4 06:54:59 switch %PORT_CHANNEL-3-COMPAT_CHECK_FAILURE: speed is not compatibleJan 4 06:55:56 switch%PORT_CHANNEL-5-PORT_ADDED: e1/7 added to port-channel 7

Examine the log file for port state transitions:

N5K# splf interface port-channel 200 dst-mac ffff.ffff.ffffMissing params will be substituted by 0's.

Load-balance Algorithm: source-ip

crc8_hash: 0 Outgoing port id: Ethernet1/33

N5K# splf int port-ch 200 src-mac 0050.5646.3e72 dst-mac ffff.ffff.ffffMissing params will be substituted by 0's.

Load-balance Algorithm: source-ipcrc8_hash: 126 Outgoing port id: Ethernet1/33

N5K# splf interface port-channel 200 src-mac 0050.5646.3e72 dst-mac 0050.5646.582b

Port Channel Link Selection




Missing params will be substituted by 0's.

Load-balance Algorithm: source-ip


N5K# show port-channel load-balance forwarding-path interface po200 src-ip 14.17.104.32

Missing params will be substituted by 0's.Load-balance Algorithm: source-ip


N5K# show platform fwm info pc port-channel 200 | grep hashPo200: hash params - l2_da 0 l2_sa 1 l3_da 0 l3_sa 1 Po200: hash params - l4_da 0 l4_sa 0 xor_sa_da 1 hash_elect 1

N5K# show port-channel load-balance

Port Channel Load-Balancing Configuration:System: source-ip

Port Channel Load-Balancing Addresses Used Per-Protocol:

Non-IP: source-macIP: source-ip source-mac

Note: To fit the output ontothe slide – splf is used for“show port-channel loadforwarding-path”

LACP Not Coming Up?

DCN-N5K1# show lacp interface e1/18

Interface Ethernet1/18 is upChannel group is 20 port channel is Po20

PDUs sent: 94993PDUs rcvd: 95702Markers sent: 0Markers rcvd: 0Marker response sent: 0

Are PDUs being received? If not, LACPconfigured on neighbor?




Marker response sent: 0Marker response rcvd: 0Unknown packets rcvd: 0Illegal packets rcvd: 0

Lag Id: [ [(7f9b, 0-23-4-ee-be-1, 8014, 8000, 204), (7f9b, 0-23-4-ee-be-2, 8014,

8000, 112)] ]Operational as aggregated link since Wed Jul 27 17:47:492011

Are there any Unknown or Illegal packetsreceived? If so, get a sniffer capture of thepackets on the wire and open a TAC case.

Common LACP Misconfiguration





Server configured to tagdot1Q VL100

N5K can see LACP PDUsfrom the host on VLAN 100

N5K sends the packetsuntagged, whereas the host isexpecting them tagged withVL100.

To remediate, either change the switch port to

a trunk or do not tag at the server

Feature Comparisons




Nexus 5000 to 5500 Comparison

Features Nexus 5000 Next Generation N55K

Numer of ports per ASIC (Gatos / Carmel) 4 8

Numer of LIF per ASIC (Gatos / Carmel) 512 (128 per port) 4K (flexible allocation)

Buffer per port 480 KB 640 KB

Number of unicas VoQ per ingress port 416 1024 (800 with sunnyvale)




Number of unicas VoQ per ingress port 416 1024 (800 with sunnyvale)

Number of mulicast VoQ per ingress por 8 128

Number of Egress queues 8 16 (8 for unicat and 8 for multicast)

COS marking Ingress Ingress & EgressDSCP marking NO Ingress & Egress

ECN marking NO YES

ACL based buffering and queuing YES YES

Station Table (MAC table) 16K 32K

VLAN Table 1K 4K

Number of active VLAN 512 4K

Mulicast index Table 4K 8K

Number of IGMP entries 1K 4K

The items marked in RED will NOT be available in Eagle Hawk release

Nexus 5000 to 5500 Comparison (cont)Features Nexus 5000 Nexus 5500

Multiple egress SPAN source NO YES (up to 4)

Port Channel can be egress SPAN source NO YES

VLAN can be egress SPAN source NO YESERSPAN YES YES

ERSPAN v3 NO YES




FEX port as destination SPAN NO YES

Latency 3.2 us 2 us

IEEE 1518 No Yes

Number of Port channel per box 16 48Number of port in a port channel 16 16

Port Channel load balancing L2/L3/L4 SA/DA L2/L3/L4 SA/DA , VLAN

Port Channel Load balancing for multicast flow

destination NO YES

LID multipathing NO YES

Superframing YES YES

Flexible output buffer selection between unicast

and multicast NO YES

Proxy queue mulicast overload NO YES


Nexus 5000 to 5500 Comparison (cont’d)

Features Nexus 5000 Nexus 5500

TCAM size 2K 4K

FC Forwarding YES YESFCoE Forwarding YES YES

FCF lookup table 4K 8K




DCE Forwarding NO YES

DCE lookup table N/A 8K

TRILL Forwarding NO YES

TRILL lookup table N/A 8K

L3 binding table 2K 4K

FC zoning table 2K 4K

RBAC table 2K 2K

Policers 256 512

Number of acive SPAN session 2 4Dedicated buffer allocated for SPAN NO YES

Multiple ingress SPAN source YES YES


Nexus Layer 3 Functional Comparison7000 vs 5500

L3 Functional Areas Nexus 7000 / M1 Modules Nexus 5500 + L3 Module

Routing ProtocolsOSPF, EIGRP, RIPv2, BGP, IS-IS, PIM,

IGMP, BiDir PIM

Base Enterprise: Static, OSPF*, EIGRP Stub,

RIPv2, PIM, IGMP

LAN Enterprise: BGP, OSPF, EIGRP

IPv6 Dual Stack, OSPFv3, EIGRP, HSRPv6 For Management

Base: VRF Lite, VRF Aware Features, VRFB E t i VRF (M t)




L3 SegmentationBase: VRF Lite, VRF Aware Features, VRF

Import/Export

MPLS License: MPLS VPNs

Base Enterprise: VRF (Management)

LAN Enterprise: VRF-Lite

High AvailabilityISSU, NSF, Graceful Restart, Multicast NSF,

IGP NSRISSU Edge – L2 Only

Fast Convergence BFD, Next Hop Tracking, BGP PIC, MPLS-TE No

MonitoringFlexible Netflow, Sampled Netflow, MPLS

OAM, ERSPANERSPAN**

L2 over L3 Overlay Transport Virtualization (OTV) No

Traffic SteeringPolicy-Based Routing, VRF Select, WCCPv2,

Static Multicast MACNo

Tunneling / Mobility Unicast over GRE, LISP No

* 256 dynamically Learned routes

** CY11 Roadmap. Not available in existing release

Nexus Layer 3 Scale Comparison7000 vs 5500


L3 Interfaces 4K 4K

IPv4 Unicast FIB 1M 8K*




IPv4 Multicast FIB M1/XL: 32K 4K

L3 ECMP 16 Way 16 Way

ARP 50K 8K

Routing Adjacency 128K 8K

FHRP 4K HSRP Groups 1K HSRP Groups

L3 ACLs 128KIngress: 2K

Egress: 1K

Segmentation 1K VRFs 1K VRFs

* With Enterprise LAN License

Nexus Layer 3 System Comparison7000 vs 5500


Redundant Route

ProcessorsYes No

Control Plane Protection Extensive CoPP Granularity Single Rate Limiter, Basic CoPP**

Y Di t ib t d M lti t li ti




Distributed ProcessingYes, Distributed Multicast replication

and BFDNo for L3

FEX Routed Port Yes No

FEX Scale – L3 32 8

ISSU Yes – L2 or L3 Edge – L2 Only

Stateful Process Restart Yes – OSPF, IS-IS No

L3 over VPC No No

** CY11 Roadmap. Not available in existing release

trm n55k l2only-config tshoot jdinkin2 2hr 20120208

Documents