nexus validation test phase 4 - cisco · 4 service provider mpls: the topology validates the inter...

1

NexusValidationTestPhase4.0

2

1 Introduction.....................................................................................................................................................3

2 NVTTopologyandScaleAchieved...........................................................................................................52.1 VxLANSolution...................................................................................................................................................................52.2 MPLSProfile1withF3,M1andM2andFabricPathandCEAccess........................................................62.3 ServiceProviderMPLSF3..............................................................................................................................................82.4 DynamicFabricAutomation(Vinci)Solution...................................................................................................102.5 DataCenter1...................................................................................................................................................................122.6 Enterprise1.......................................................................................................................................................................132.7 MSDCScaleProfiles.......................................................................................................................................................142.8 M1vPC.................................................................................................................................................................................16

3 ISSUMatrix....................................................................................................................................................17

4 ProfileDetails...............................................................................................................................................184.1 VxLANSolution................................................................................................................................................................184.1.1 NetworkTopologyandDesignReview..............................................................................................184.1.2 HardwareandSoftwareOverview.......................................................................................................214.1.3 VxLANSolutionConfigurationGuide..................................................................................................214.1.4 VxLANSolutionScale.................................................................................................................................21

4.2 MPLSProfile1withF3,M1andM2andFabricPathandClassicalEthernetAccess.....................224.2.1 MPLSTopologyDesignReview..............................................................................................................224.2.2 HardwareandSoftwareOverview.......................................................................................................234.2.3 Topology..........................................................................................................................................................244.2.4 SPMPLSScaleCoverage...........................................................................................................................244.2.5 SPMPLSISSUMatrix..................................................................................................................................25

4.3 SPMPLSSolution............................................................................................................................................................254.3.1 NetworkLogicalTopologyDesignOverview...................................................................................254.3.2 Descriptionofthetestnetwork.............................................................................................................264.3.3 HardwareandSoftwareOverview.......................................................................................................264.3.4 Topology..........................................................................................................................................................274.3.5 Scale...................................................................................................................................................................274.3.6 ISSUMatrix.....................................................................................................................................................284.3.7 TrafficandConvergence...........................................................................................................................28

4.4 DynamicFabricAutomation(Vinci).....................................................................................................................294.4.1 TopologyDesignReview..........................................................................................................................294.4.2 HardwareandSoftwareOverview.......................................................................................................304.4.3 Topology..........................................................................................................................................................314.4.4 TrafficandMetrics......................................................................................................................................31

4.5 DC1........................................................................................................................................................................................324.5.1 HardwareandSoftwareOverview.......................................................................................................324.5.2 NetworkLogicalTopologyDesignOverview...................................................................................324.5.3 ConfigurationDetails.................................................................................................................................33

4.6 ENT.......................................................................................................................................................................................364.6.1 HardwareandSoftwareOverview.......................................................................................................364.6.2 NetworkLogicalTopologyDesignOverview...................................................................................36

4.7 ConfigurationDetails...................................................................................................................................................374.8 ENTTopology...................................................................................................................................................................384.9 MSDCScale........................................................................................................................................................................394.9.1 TopologyDesignOverview......................................................................................................................394.9.2 ISSUMatrix.....................................................................................................................................................40

3

4.10 M1vPCScale..................................................................................................................................................................404.10.1 HardwareandSoftwareOverview....................................................................................................404.10.2 NetworkLogicalTopologyDesignOverview................................................................................404.10.3 ConfigurationDetails...............................................................................................................................41

4.11 M1vPCScaleTopology.............................................................................................................................................424.12 DC1,ENT1andM1vPCFeature/ScaleCoverage.......................................................................................424.13 ISSUMatrixforDC1,ENTandM1vPC..............................................................................................................43

5 NVTFindings/Conclusion/Recommendations.................................................................................44

6 Appendix........................................................................................................................................................476.1 VxLANSolutionConfigurationGuide....................................................................................................................476.2 ServiceProviderMPLSConfigurationGuideline.............................................................................................576.3 DC1ConfigurationGuideline....................................................................................................................................626.4 ENT1ConfigurationGuide.........................................................................................................................................656.5 M1vPCScaleConfigurationGuideline.................................................................................................................68

1 Introduction The Cisco Nexus line of data center product hardware and software must pass Cisco'scomprehensive quality assurance process, which includes a multistage approach comprisingextensive unit test, feature test, and system‐level test. Each successive stage in the process addsincreasinglyhigherlevelsofcomplexityinamultidimensionalmixoffeaturesandtopologies.This document describes the NVT Phase 4.0 network topologies, hardware and softwareconfigurations,testproceduresandfindings.NVTPhase4.0testingisperformedonthefollowingnetworks:

VxLANSolution:ThistestprofileisdevelopedbasedonBGPEVPNtechnologyusingVxLANfabric. It is intend toprovidehostmobility and subnet extensionacrossdata centers andsimplify network operation using anycast gateway and does not require a first hopredundancyprotocol.

MPLSProfile1:Thisisdividedintoaccesstechnologies.o ClassicEthernetAccess:Thisnetwork focusesonbuildingandoperatinganMPLS

corenetwork thatprovidesend‐to‐endL3VPNandMVPNserviceswith theNexus7000 Sup2E andNexus 7700 Sup2E between two different sites: Site 8 ismainlyfocusedonlegacyM1moduleswhereasSite10isbuilttotestandverifytheMPLSfeaturesetover thenewF3modules.BothsitesusevirtualPort‐Channel (vPC) todeliverhighlyavailableunicastandmulticastservices.

o Fabric Path Access: In addition to the Phase 1 topology, the new Site 6 has beenintegratedtotheMPLStopology.ThisnetworkusesthenewF3moduleswithbothvirtual Port‐Channel (vPC+) and FabricPath solutions to deliver highly availableunicast and multicast services. Furthermore, the deployment of FabricPath willincreasetheoverallbandwidth(sincevPCsolutionislimitedtoonlytwovPCpeers)allowing to deploymulti‐spine network layer,while eliminating the need for STPand allowing a “plug‐and‐play” type of network as the additionof newdevices ateitherthespineortheleaflayerdoesnotpresentanysignificanttrafficlossforthepre‐establishedflows.

4

ServiceProviderMPLS:ThetopologyvalidatestheinterDCintersitedesignwithCarrierssupportingCarriersCSCbackboneforMPLSL3VPN.Twodatacenternetworksaresimulated.N7710withF3cardisusedinonedatacenterandN7010withF3isusedinthesecondnetwork.TestingisfocusedonthenetworkwithN7710.N7710andN7010withF3cardareusedasaggregationandcoreswitches.ASR9KisusedasCSCPE.N5Kisusedforlayer2accessaswellasforvPC.ASR1KisusedassecondaryRRsforredundancypurposeinthenetwork

DynamicFabricAutomation(Vinci)Profile:DFAprofileusespoweronauto‐provisioningtoconfigurenetworkdevicesandsupportdistributedgatewayfunctionandautomateddatacenter interconnect. The topology contains N7000/N7700/N6004 spines, N7000/N6001leafs, andN7004border leafs. The topology uses the latestDCNM for cablemanagementandworkflow automation. It also uses emulated leafs (Virtual ToR or VToRs) to achievescale. It supports unicast and multicast for IPv4 and IPv6 hosts and auto configurationbasedondynamicframesnoopingbasedonthetrafficandVDPrunningonthevSwitch.

DataCenter1(DC1):ThisnetworkfocusesonbuildingandoperatingadatacenterwiththeNexus7000SUP1atcoreandNexus7000SUP2ataggregationasroutingandswitchingcomponents.ItalsocoversinteroperabilitywiththeNexus5000,Nexus3000,Nexus2000,Catalyst6500/4500switches.ThisnetworkusesvirtualPort‐Channel(vPC)andFabric‐Path(vPC+)todeliverhighlyavailableunicastandmulticastservices.

ENT:ThistestprofileprimarilyisbuilttovalidateEnterprisecustomerprofiles.Inthefirstphase,wehavevalidatedtheTIERIEnterprisescustomerprofile.Thetestbedisbuiltwitha new hardware covering the existing feature set for future deployments. Nexus7000SUP2E/M2 is used at the core layer and N7700 SUP2E/F2E/F3(40G) at the aggregationlayer.TheprofilealsocoversinteroperabilitywithNexus6000andNexus3000switchesandcoversL3AggwithL3ToRandL3AggwithL2ToR.

MSDC:ThisprofilefocusesonscalerequirementsofMassivelyScalableDataCenters.Itusesafullyloaded7018peeringwithanother7018andusesF2linecardandSup2.TestsweredonewithBGPandOSPFasIGPprotocols.

M1VPC:ThistestbedfocusesonscalingthevirtualPort‐Channel(vPC]withNexus7000.Italso covers interoperability with Catalyst 6500. This network uses vPC and PVLAN todeliverhighavailabilitytoserversconnectingtodatacenters.

Operation:NetworkmanagementincludingSNMPpolingandinventorycollectionisperformedthroughDCNMfromCiscoandnetMRIfromInfoblox,TACAS+authenticationandsyslogserver.NetFlowisconfiguredtoexportthirdpartyNetflowcollectorScrutinizeroncertaintestbeds.RealhostsareconnectedbyNexusaccessswitchesusingNICteaming(inbothactivemodeandOnmode).UCS‐BseriesareconnectedtoNexusaccessswitchesthrufabricinterconnect.NTPissyncedtotheserver.

5

2 NVT Topology and Scale Achieved

2.1 VxLAN Solution Topology

Scale

Feature Scale

Maximum VLANs 40

VLANs per TOR 14

VRFs per ToR 10

Max VNI per ToR 24

Max VNI per network 10 VRFs + 30 VLANs = 40

Max underlay Multicast Groups 10 (to support broadcast + multicast traffic

Max ToR per ‘network’ 116 ( 112 ToRs per POD and 4 on peer

6

POD)

Max VTEP peers per ‘network’ 116 ( 112 ToRs per POD and 4 on peer POD) �

Overlay MAC scale 16K per POD * 2 = 32 K per ‘network’

Overlay IPv4 routes 16K per POD * 2 = 32 K per ‘network’

LISP dynamic routes 6600

Undelay v4 routes 400 across all PODs

Undelay v6 routes none

2.2 MPLS Profile 1 with F3, M1 and M2 and Fabric Path and CE Access Topology

Scale

ClassicalEthernet FabricPathFeature/Parameter Site10

(F3vPC)Site8

(M1vPC)Site6

(F3FP/vPC+)VLAN 1000 1000 1000SVI 1000 1000 1000IPv4hostsxsubnet 32 32 32HostsperVLAN 47+ 47+ 47+vPC/(vPC+) 20/na 2/na 2/(1)VLANSxvPC/(vPC+) 50/na 500/na 250/(500)FabricPathIS‐IS na na 6

7

adjacenciesPort‐channel(excludingpeer‐link,vPC&vPC+)

4 4 4

HSRPv4version2 1000 1000 1000HSRPv6version2 1000 1000 1000VRF 100 100 100VLANsperVRF 10 10 10ARPxVRF 350+ 350+ 350+mVRF 50 50 50PIMneighbordefaultVRF

4 4 4

PIMneighborxVRF 36 36 36MDTdefaulttunnel 50 50 50MDTdatatunnel 200 200 200Numberoflocalcustomergroups

10 10 10

Numberofsourcesxcustomergroup

8 8 10

NumberofOIFxgroup 10 10 10OSPFPeersdefaultVRF

4 4 4

OSPFRoutes/(Path) 48/(54) 48/(54) 48/(54)LDPSessions 4 4 4iBGPIPv4Sessions 2 2 2iBGPVPNv4Sessions 2 2 2iBGPVPNv6Sessions 2 2 2

Hardware

ClassicalEthernet‐vPC

ModelNo. Software

N7K N7KSUP2E/F3/M2/M1XL

7.2.0

C6K VS‐SUP720‐10G 151‐2.SY2CSR1000v Na 15.4.3.S1FabricPath–vPC+

ModelNo.

N7700 N7700SUP2E/F340G 7.2(0)N7000 N7KSUP1/M1 7.2(0)

8

2.3 Service Provider MPLS F3 Topology

Scale

Feature Scale

6VPEVRFs(dualstack) 100

ARPaddressesoneachswitch 21,120

GlobalMulticastroutes 100

HSRPgroups 1,000

IGMPgroups 100

L3Physicalinterfaces 300

L3VPNv4VRFs 100

MVPNVRFs 10

NumberofIPv6Host 2,000

L3Portchannel 2

9

Feature Scale

SVI 1,000

TotalMulticastprefixesinMVPN 100

TotalnumberofIPv4prefixesindefaultroutingtablewith2IGPPath 200

TotalVPNv4PrefixesfromIBGPpeerswith2IGPPath 8,000


vPClinks 8

VPNv4Prefixesacross6VRFswithoneIGPPath 6,000


Hardware

ModelNo. ImageVersionN77K N77‐SUP2E/ N77‐

F324FQ‐257.2.0

N7K N7K‐SUP2E/ N7K‐F312FQ‐25

7.2.0

N5K N5K‐C5548UP‐SUP 7.0(5)N1(1)ASR9K ASR9001‐RP/ A9K‐MPA‐

20X1GE5.1.1

ASR1K ASR1000‐RP2/ ASR1000‐ESP20/SPA‐8X1GE‐V2

3.14

10

2.4 Dynamic Fabric Automation (Vinci) Solution Topology

Scale

Spine/SuperSpine

VRFs(vni) Transit‐Mode

SVI/BDI Transit‐Mode

VLAN/BD Transit‐Mode

BGPRoute‐ReflectorPeer 180

FabricCorePorts(Portstoleaves) 192

FabricBFDSessions 12

FabricBGPSessions 192

FabricPathLayer‐2BFD 12

Leaf FabricFacingFabricCorePorts 16FabricBFDSessions 16FabricBGPSessions 1RR VRFs(vni) 410SVI/BDI(Core) 410VLAN/BD(Core) 410

11

HostFacing VLAN(vn‐segment) 1640SVI/BDI 1640VPC+/VPC 96FEX NAPhysical/VirtualServerIPAddresses(ARP/ND)–1:6IPv4toIPv6ratio

IPv4Routes(/32+SubnetRoute) 6kIPv6Routes(/128+SubnetRoute) 4kMulticastRoutes(S,G) 2kTotalVNI/vn‐segmentperLeaf 2050

BORDERLEAF

FabricFacing FabricCorePorts 16FabricBFDSessions 16FabriciBGPSessions 1 VRFs(vni) 1000SVI/BDI(Core) 1000VLAN/BD(Core) 1000 Edge‐RouterFacing VLAN(vn‐segment) 1000Sub‐InterfaceperBorderLeaf 1000(ifVRF‐lite)

IPv4Routes(/32+SubnetRoute) 17K*

IPv6Routes(/128+SubnetRoute) 5K*MulticastRoutes(S,G) 1K* eBGP/MP‐BGPpeers 1000VPC+/VPC 0Labels/VRF OptionB

TotalVNI/vn‐segmentperBorderLeaf 1000

12

2.5 Data Center 1 Topology

Scale

Feature ScaleF1 2M1 5VDC 4FEX 6FEX‐HIF 96ACL/ACE 500POMembers 2VLAN/SVI 100VLANperFEX 10FPVLAN 50MAC 7KvPC 120VLANpervPC 1,10,16HSRPv2 100OSPFNeighbor 16BGPNeighbor 2PIMNeighbor 256PBRSequence 20UnicastRoute 10K

13

MulticastRoute 500MulticastSource 20MulticastOIF 10MulticastGroup 100FabricExtender 3FPIS‐ISadjacencies 6FPNumberofSwitchID’s 20PrimaryVLAN 20SecondaryVLAN 25Port‐channel 125Port‐channelinPVLAN 18VPCPOwithPVLAN 10Hostmode 18PromiscuousAccess 2PromiscuousTrunk 2TrunkSecondary 7PVLAN 50

2.6 Enterprise 1 Topology

Scale

14

Feature ScaleACL/ACE 500VLAN/SVI 2000MAC 1KHSRPv2 2000BGPNeighbor 18PIMNeighbor 2000PBRSequence 200UnicastRoute 22KMulticastRoute 3KMulticastGroup 390PVLAN 60vPCConfig‐Sync 120Netflow 84WCCP 58

2.7 MSDC Scale Profiles Topology

15

Scale

Parameters Profile1(L3ToR+L3Agg)

Profile2‐1(L2ToR+L3Agg)

Profile1(L3ToR+L3Agg)w/RFC5549

REQ‐ID NXOS‐MD‐OTT‐002‐001

NXOS‐MD‐OTT‐002‐003

Description F2‐Seriesbased; M2‐based;Dual‐stack;

F2‐Seriesbased;

ToR 4948EorN3K 4948Eoranyother

4948Eoranyother

Hardware *FullyloadedF2‐series

*4MSeries *FullyloadedF2‐series

SingleSup:Test1Sup2;Test2Sup1

DualSup DualSup

VDC 1VDC 1VDC 1VDCPortchannels 8‐bundlePort‐

channel8‐bundlePort‐channel

8‐bundlePort‐channel

L3ECMP Test1:16‐wayECMP;Test2:32‐wayECMP

16‐wayECMP 32‐wayECMPv4–3khostroutesanddefaultroute

UnicastRoutingProtocol

Test1‐iBGP754Adjacencies;Test2‐OSPF754Adj

OSPF,OSPFv348Adj

23iBGPv4Adjacencies[leafnodesthatdonotsupportrfc5549and16simulatedgateway/loadbalancer/firewallnodes]755iBGPv6Adjacenciesthatsupportrfc55491N7700leafnodethatsupportrfc5549

MulticastRoutingProtocols

N/A PIM‐SM,MSDPw/anycastRP

N/A

FHRP N/A 2000HSRPgroups(1000HSRPv4+1000HSRPv6)

NA

FastDetection(forIGP,PIM,FHRP)

Test1:BFD:250msx3;Test2:DefaultTimers



DualStack Yes Yes Yes,onsupportingleafnodesNumberof/32&/128hostentries

N/A 80Ksplit(40KIPv4+40KIPv6)

N/A

16

NumberofUnicastIGP

10KIPv4300IPv6

10K 10K

NumberofMulticastRoutes(S,G)

N/A 15K N/A

VLANs N/A 250 N/A

SVI(sameasVLANs)

N/A 250 N/A

ACLs N/A 250 N/A

ARP/NDP(30secondsARPrefreshrate)

N/A 80K(40KARPand40KNDP)

N/A

ARP/NDPLearningRate

N/A 5000pps N/A

ARP/NDPGleanrate

N/A 1000pps N/A

DHCPRelay N/A 100pps N/AMIB KPIX KPIX KPIX

2.8 M1 vPC Scale

FeatureMax M1vPCPrimaryVLAN 140SecondaryVLAN 280Phy.PortsusedforPVLAN 468Port‐channel 388Port‐channelinPVLAN 388VPCPOwithPVLAN 256Hostmode 32PromiscuousAccess 64PromiscuousTrunk 186TrunkSecondary 186

17

Topology

3 ISSU Matrix Image ISSU/COLDBOOT MPLS

Profile1

DC1 ENT MSDC M1vPC

7.2.0>7.2.0.upg ISSU PASS PASS PASS PASS PASS6.2.12>7.2.0 ISSU PASS PASS PASS PASS 6.2.10>7.2.0 ISSU PASS PASS PASS 6.2.8a>7.2.0 ISSU PASS 6.2.8b>7.2.0 ISSU PASS PASS 7.2.0>7.2.0.upg COLDBOOT PASS 6.2.12>7.2.0 COLDBOOT PASS PASS 6.2.10>7.2.0 COLDBOOT PASS PASS 6.2.8a>7.2.0 COLDBOOT PASS 6.2.8b>7.2.0 COLDBOOT PASS PASS PASS PASS7.2.0>6.2.12 COLDBOOT PASS PASS 7.2.0>6.2.10 COLDBOOT PASS 7.2.0>6.2.8a COLDBOOT PASS 7.2.0>6.2.8b COLDBOOT PASS PASS

18

4 Profile Details

4.1 VxLAN Solution

4.1.1 Network Topology and Design Review TheDCdesignisstructuredinpairsofPODsbetweenwhichmobilityandsegmentationshouldbesupported.ThechoiceistouseVXLANwithanEVPNcontrolplanewithineachPODandforeast‐westconnectivityacrosspeerPODs;LISPisrequiredforoptimizationofNorth‐Southtraffic.Thereare4aggregationboxesperPODthatuseSUP2EandF3linecardsand40GEinterfaces.EachaggregationboxisconnectedtoallToRs.The4aggregationboxesarealsoconnectedtoeachotherinaring.EachRackhastwoToRsinavPCarray.vPCrunsfromthephysicalhoststotheToRs.CommunicationbetweenPODsinthesameDChappensoverthecore.North‐SouthCommunicationinandoutofthePODsalsohappensoverthecore.Segmentationisonlystretchedbetweenpeer‐PODs;segmentationisnotstretchedoutsideofthePODstotheCore.OnlyoneVRF(theproductionVRF)isroutableoverthecore.Thus,inter‐PODandNorth‐SouthcommunicationislimitedtotheproductionVRFonly.East‐WestcommunicationisoptimizedbetweenpairsofPODsacrossDCs.EachPODwillhaveapeerPODattheremoteDCandtheaggregationswitchesofthesepeerPODswillbedirectlyconnectedoverfiber.Thefollowingconnectionswillbeestablished:POD1toPOD1’East‐WestcommunicationsbetweenPeerPODswillfollowthelowlatency/high‐capacitydirectconnectionbetweenAggregationswitches.CommunicationacrossPODsthatarenotpeerswillhappenoverthecoreonlyforthe“ProductionVRF”DesignChoices

Spine to Core is eBGP, peering is interface based. On Spine only Production_vrf routes are routable in core

Core is single AS OSPF will be running between all Spines in the ring topology Spines are iBGP peering over interface to TORs. Loopback are redistributed with network

statement. All four spines are IPv4 RR to all ToRs Each N9396 are connected to all four Spines. Connection is 40G links. Some connection uses are

port‐channel with 2‐3 40G member link Legacy TORs has 10 VRFs. Correspondingly Spine will have 10 VRF‐lite for Intra‐POD legacy to

fabric traffic Legacy TORs eBGP peer to Spine thru "fake iBGP" by manipulating "local AS" in as‐path. So BGP

peering is iBGP, but local AS are different so route get redistributed to MP‐iBGP L2VPN EVPN.

Spines are route reflector for IPv4 and L2VPN EVPN

All leafs in the POD are EVPN RR clients to Spins. IBGP peering is with loopback

Total 10 VRF in system, one is Production VRF which will carry hosts that will require inter‐POD

and N‐S (i.e branch to host) access

19

Other 9 VRFs will be connecting within the POD or peer‐POD only. No N‐S access for these VRFs.

There is no route‐leaking into Production VRF. These 9 VRFs require connectivity to legacy ToRs

also

VM hosts requiring mobility are separated into different VLANs (we call it mobility VLAN)

Production VRF will have 10 Mobility Vlan with total of 6600 host and 500 Non‐mobility vlans

with 4 Vlan each in each TORs (125 TORs X 4 Vlans = 500)

Each Mobility Vlan will have 3 host (10 Vlan x 3 host X 125 Racks = 3750 hosts) in POD A and

2850 hosts in POD A”

For Non Mobility Vlans, 4 Vlan X 50 host X 125 TORs = 25000 hosts in POD A

Inter‐peer‐POD is EBGP interface peering over high speed direct link. VXLAN tunnels are TOR to

TOR

For N‐S for Production VRF, VXLAN tunnel will terminate on Spine and L3VRF handoff to

production‐vrf handoff

IPv4 and L2VPN EVPN iBGP peering between inter‐spine ring links and Set local pref low for inter

spine link

POD A has 250 emulated TORs using IXIA eVPN emulation

Each TORs N9K have 40 host VPCs with N3K fan out switches

10 Legacy TORs are connected to PODA

2 map server (CSR1KV) connected with DDT to Branch router ASR1K.

LISP traffic will load balance between 4 spines by the map‐server

Spine we have configured route-map to allow only non-mobility prefixes to get advertise to core and branch for south to north traffic

2 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

BGP Design

N9300 N9300 N9300 N9300

N7700 N7700

N3000 N3000

N9300 N9300 N9300 N9300

N7700 N7700

N3000 N3000

AS 100 local as 300 remote-as 200 prepend

AS 100

“fake” eBGP ipv4

eBGP pfx list allow non-mobility pfxs and legacy pfxs

IGP

iBGP lo0 Ipv4, evpn local_pref 50

iBGP ipv4 intf RR clients each other local_pref 50 next-hop-peer-addr

iBGP ipv4 intf iBGP evpn lo0 Spine: next-hop-peer-addr

eBGP intf ipv4 lo0 evpn

AS 100 local as 200 remote-as 300 prepend

20


LISP

N9300 N9300 N9300 N9300

N7700 N7700

N3000 N3000

N9300 N9300 N9300 N9300

N7700 N7700

N3000 N3000

Branch MS

MS

ASR1K

1

2

3a

3b 5

6

4

7

1. GARP 2. BGP EVPN update (MAC, IP) 3. a. eBGP EVPN update 3. b. LISP update MS 4. Branch XTR send traffic to XTR old POD 5. LISP SMR to Branch XTR to check MS 6. Branch XTR query MS 7. Traffic send to new POD XTR


Data Packet Flow

N9300 N9300 N9300 N9300

N7700 N7700

N3000 N3000

N9300 N9300 N9300 N9300

N7700 N7700

N3000 N3000

MS

MS

ASR1K

Native IPv4

VxLAN Encaped

LISP Encaped

21

4.1.1.1 VxLAN Solution Topology

4.1.2 Hardware and Software Overview

ModelNo. NVT4.0N77K N77‐SUP2E/N77‐

F324FQ‐257.2.0

N9K N9K‐C9396PX/N9K‐M12PQ

7.0.3

CSR1000v Na 15.4(3)S1ASR1K ASR1000‐RP2/

ASR1000‐SIP10/SPA‐10X1GE‐V2

155‐1.S1

ASR9K A9K‐RSP440‐TR/A9K‐40GE‐E/A9K‐2x100GE‐SE

5.3.0

4.1.3 VxLAN Solution Configuration Guide Pleaseseetheappendixforconfigurationdetails

4.1.4 VxLAN Solution Scale

Feature Scale

Maximum VLANs 510

VLANs per TOR 14

VRFs per ToR 10

Max VNI per ToR 24

Max VNI per network 10 VRFs + 500 VLANs = 510

22

Max underlay Multicast Groups 510 (to support broadcast + multicast traffic

Max ToR per ‘network’ 250�(125 ToRs per POD * 2) = We build in pairs.

Max VTEP peers per ‘network’ 250�(125 VTEPs per POD * 2) = We build in pairs.

Overlay MAC scale 16K per POD * 2 = 32 K per ‘network’

Overlay IPv4 routes 16K per POD * 2 = 32 K per ‘network’

LISP dynamic routes 6600

Undelay v4 routes 400 across all PODs

Undelay v6 routes none

4.2 MPLS Profile 1 with F3, M1 and M2 and Fabric Path and Classical Ethernet Access TheMPLSprofileisperformedintwoaccessmethodtechnologies.

ClassicalEthernet:Corenetwork,Site8andSite10.Thisnetwork focusesonbuildingandoperatinganMPLScorenetworkthatprovidesend‐to‐endL3VPNandMVPNserviceswiththeNexus7000Sup2EandNexus7700Sup2Ebetweentwodifferentsites:Site8ismainlyfocusedonlegacyM1modules,whereasSite10isbuilttotestandverifyMPLSfeaturesetover the new F3 modules. Both sites use virtual Port‐Channel (vPC) to deliver highlyavailableunicastandmulticastservices.

FabricPath: In addition to theabove topology, thenewSite6hasbeen integrated to theMPLS topology. This network uses the new F3 modules with both virtual Port‐Channel(vPC+)andFabricPathsolutionstodeliverhighlyavailableunicastandmulticastservices.Furthermore,thedeploymentofFabricPathwillincreasetheoverallbandwidth(sincevPCsolution is limited to only two vPC peers) allowing to deploymulti‐spine network layer,whileeliminatingtheneed forSTPandallowinga“plug‐and‐play” typeofnetworkas theadditionofnewdevicesateitherthespineoftheleaflayerdoesnotpresentanysignificanttrafficlossforthepre‐establishedflows.

4.2.1 MPLS Topology Design Review Thetopologiesandtestcasesvalidatehigh‐availabledatacenternetworksinordertoprovideunifiedfabricandcomputingservices.ThisisachievedbyusingtheNexus7000andN7700,withfeaturessuchasvPCwithdifferentNexus7000modules.Core Network ThecorenetworkisbuiltaroundtwoNexus7000chassiswithSup2E.EachchassisisconfiguredtohavetwoProutersintwodifferentVDCs(foratotaloffourProuters).

Nexus7000withdistributedport‐channelsconnectedinasquaretopology CSR1kvareconfiguredasaroute‐reflectorclusterhenceconnectedtotwooftheoppositeP

routers(P1andP3)Site 10 Network – Classical Ethernet F3 ThissiteisbuiltaroundtheNexus7700withSup2EwithF3modules:

23

TwoNexus7700withvPCactingasredundantmulti‐homingPEs,whichareconnectedtotwootherNexus7700foraggregation/accessactingasCEsdevicesconfiguredondifferentVDCs.

Site 8 Network – Classical Ethernet M1 ThissiteisbuiltaroundtheNexus7000withSup2EwithM1andM1XLmodules:

TwoNexus7700withvPCactingasredundantmulti‐homingPEswhichareconnectedtotwoCatalyst6500foraggregation/accessactingasCEsdevices

WhilethemajorityoftestcasesfocusonintegratedsolutionsusingNexusswitching,modularCatalystswitchesarealsoincludedforinteroperabilitybetweenNX‐OSandIOS.Site 6 Network – Fabric Path Thetopologiesandtestcasesvalidatehigh‐availabledatacenternetworksinordertoprovideunifiedfabricandcomputingservices.ThisisachievedbyusingtheNexusN7700,withfeaturessuchasFabricPathandvPC+withF3/40GNexusmodulesInFigure1,Site6isthesitethathadbeendeployedtoverifyandtesttheFabricPathfeaturesetontheNexus7700withSup2EandF3/40Gmodules.

Nexus7700/F3‐40GmoduleswithvPC+toNexus7000/M1modules. Nexus7700/F3‐40GmoduleswithvPCtoNexus7700/F3‐40Gmodulesfor

FabricPathwithdifferenttopologies.


ClassicalEthernet‐vPC

ModelNo. Software

N7K N7KSUP2E/F3/M2/M1XL

7.2.0

C6K VS‐SUP720‐10G 151‐2.SY2CSR1000v Na 15.4.3.S1FabricPath–vPC+

ModelNo.

N7700 N7700SUP2E/F340G 7.2(0)N7000 N7KSUP1/M1 7.2(0)

24

4.2.3 Topology

4.2.4 SP MPLS Scale Coverage

Phase1 Phase2Feature/Parameter Site10

(F3vPC)Site8

(M1vPC)Site6

(F3FP/vPC+)VLAN 1000 1000 1000SVI 1000 1000 1000IPv4hostsxsubnet 32 32 32HostsperVLAN 47+ 47+ 47+vPC/(vPC+) 20/na 2/na 2/(1)VLANSxvPC/(vPC+) 50/na 500/na 250/(500)FabricPathIS‐ISadjacencies

na na 6

Port‐channel(excludingpeer‐link,vPC&vPC+)

4 4 4

HSRPv4version2 1000 1000 1000HSRPv6version2 1000 1000 1000VRF 100 100 100VLANsperVRF 10 10 10ARPxVRF 350+ 350+ 350+mVRF 50 50 50

25

PIMneighbordefaultVRF

4 4 4

PIMneighborxVRF 36 36 36MDTdefaulttunnel 50 50 50MDTdatatunnel 200 200 200Numberoflocalcustomergroups

10 10 10

Numberofsourcesxcustomergroup

8 8 10

NumberofOIFxgroup 10 10 10OSPFPeersdefaultVRF

4 4 4

OSPFRoutes/(Path) 48/(54) 48/(54) 48/(54)LDPSessions 4 4 4iBGPIPv4Sessions 2 2 2iBGPVPNv4Sessions 2 2 2iBGPVPNv6Sessions 2 2 2

4.2.5 SP MPLS ISSU Matrix

ISSU/Coldboot (SUP2E)ISSU6.2.10‐>7.2.1 PassColdboot6.2.10‐>7.2.1 PassISSU6.2.8b‐>7.2.1 PassColdboot6.2.8b‐>7.2.1 PassISSU6.2.12‐>7.2.1 PassColdboot6.2.12‐>7.2.1 Pass

4.3 SP MPLS Solution

4.3.1 Network Logical Topology Design Overview MPLSSolution:ThefocusofthistestbedistotesttheMPLSsolutionproposedtothedatacentercustomersingeneral.Scaleandtopologyarealignedtothedeploymentproposedtothetoaspecificcustomer.SolutiontestingisdoneforinterDC(layer3DCIwithMPLS)withCSCbackbone.TestingisprimarilyfocusedonMPLSfeaturesconfiguredonN77KwithF3cardpositionedasaggregationandcoreswitchesinthedatacenternetwork.ThisnetworkusesvPCtowardsaggregationtodeliverhighlyavailableunicastandmulticastservicesandCSCintheMPLSbackbone.ASR9KsareusedasCSCPEs.TheoverallgoalistoidentifyanystabilityissueswhenconfiguredwithmultiplefeaturesandmeasuretrafficconvergencemetricwithkeytriggerssuchasSSO,processcrashes/restarts,VDCreloadandOIRwithmultiplefeaturesconfiguredatpre‐definedscales.Theresultsinthisdocumentarecloselytiedwiththesetuputilizedforthisprofile.Theseresultsmayvarydependinguponthedeploymentandoptimizationstunedfortheconvergence.

26

ThetopologyvalidatestheinterDCintersitedesignwithCSCbackboneforMPLSL3VPN.Twodatacenternetworksaresimulated.N7710withF3cardisusedinonedatacenterandN7010withF3isusedinthesecondnetwork.TestingisfocusedonthenetworkwithN7710.N7710andN7010withF3cardareusedasaggregationandcoreswitches.ASR9KisusedasCSCPE.N5Kisusedforlayer2accessaswellasforvPC.ASR1KisusedassecondaryRRsforredundancypurposeinthenetworkasshowninFigure1.

4.3.2 Description of the test network

N7710andN7010withF3cardareusedasaggregationandcoreswitches‐FeaturesconfiguredareL3VPNwithVRFs,ISIS,iBGP,HSRP,SVIs,vPC,LDP,6vPE,mvpn,multicast.

ASR9KisusedasCSCPE–FeaturesconfiguredareOSPF,iBGP,LDP,VRF. N5Kisusedforlayer2accessaswellasforvPC. ASR1KisusedassecondaryRRsforredundancypurposeinthenetwork‐Features

configuredareOSPF,iBGP,LDP Ixiaisusedtogeneratehostroutes(/32)aswellforendtoendtrafficvalidation.

4.3.3 Hardware and Software Overview ModelNo. ImageVersion

N77K N77‐SUP2E/ N77‐F324FQ‐25

7.2.0

N7K N7K‐SUP2E/ N7K‐F312FQ‐25

7.2.0

N5K N5K‐C5548UP‐SUP 7.0(5)N1(1)ASR9K ASR9001‐RP/ A9K‐MPA‐

20X1GE5.1.1

ASR1K ASR1000‐RP2/ ASR1000‐ESP20/SPA‐8X1GE‐V2

3.14

27

4.3.4 Topology

4.3.5 Scale ThescalenumberslistedbelowareconfiguredintheN77K‐PE1andN77K‐PE2(asshownintheTopologyabove)Aggregationswitches.Feature Scale

6VPEVRFs(dualstack) 100

ARPaddressesoneachswitch 21,120

GlobalMulticastroutes 100

HSRPgroups 1,000

IGMPgroups 100

L3Physicalinterfaces 300

L3VPNv4VRFs 100

MVPNVRFs 10

NumberofIPv6Host 2,000

28

Feature Scale

L3Portchannel 2

SVI 1,000

TotalMulticastprefixesinMVPN 100

TotalnumberofIPv4prefixesindefaultroutingtablewith2IGPPath 200



vPClinks 8



4.3.6 ISSU Matrix Sincethisisanewdeployment,ISSUfrom7.2to7.2upgradeimageswastested

4.3.7 Traffic and Convergence Asmentionedpreviously,theoverallgoalofthistestingistocheckthestabilityoftheN77KasaggregationandRRcuminlinecoreandmeasurethetrafficconvergenceduringvariousoperationaltriggers.Thetrafficconvergencemeasurementsareintheattachedspreadsheet.AllthetrafficconvergencemeasurementsarefocusedonN77KusedaseitheraggregationswitchorN77KasRRcuminlinecore.Thetrafficconvergencemetricsforeachtriggerarenotconsistenteverytime.Soeachtriggerisrepeated3to4timesandthemaximumvalueobservedduringanyoftheiterationisrecordedasshowninthetablebelow.ThefollowingtrafficflowmetricsareconfiguredandtestedusingIxia.Fixedframesizeof100isusedforallstreams.TrafficStreams Description TrafficRateV4trafficbethostsondiffDCs

BidirectionalFlowbetween900v4hosts

100KFrames/Sec

V6trafficbethostsondiffDCs

BidirectionalFlowbetween200v6hosts

100KFrames/Sec

V4Scaletraffic Bidirectionalflowforrestofthev4prefixesfromremoteDCforscaleasspecifiedinSection2

100KFrames/Sec

V6ScaleTraffic Bidirectionalflowforrestofthev6prefixes

100KFrames/Sec

29

fromremoteDCforscaleasspecifiedinSection2

Multicast(mvpn)withN77KconnectedtoReceiversonIxia

100flowsfor100groups

10KFrames/Sec

Multicast(mvpn)withN77KconnectedtoSourceonIxia


10KFrames/Sec

GlobalMulticastwithN77KconnectedtoReceiversonIxia


10KFrames/Sec

GlobalMulticastwithN77KconnectedtoSourceonIxia


10KFrames/Sec

V4InterVRFtraffic IntervrftrafficgoingthroughfirewallwithintheDC

200KFrames/Sec

V6InterVRFTraffic IntervrftrafficgoingthroughfirewallwithintheDC

200KFrames/Sec

CPUandmemoryutilizationontheN77K‐PE1andN77K‐PE2measuredundersteadystateareshownbelow.Thesteadystatemeasurementsaredonewhenthesystemisstablewithallthefeaturesconfiguredand100%endtoendtrafficrunning.SNMPpollingofMPLSMIBsaredonecontinuouslyinthebackground.Device AverageCPUUtilization

(5minwith16coreCPU)MemoryUtilization

N77K‐PE1 1.49 6738104KN77K‐PE2 0.51 7957180KDuetotheredistributionoftheISISroutesintoOSPFandvice‐versatowardstheCSCbackbone,routingloopsareboundtohappenduringcertaintriggerslikelinkfailure,vdcreloadetc.Toavoidthis,itisrecommendedtohaveroute‐mapsattachedsuchthat,forlocallylearntroutesISIShashigherpriorityandforremoteroutes,OSPFhashigherpriority.

4.4 Dynamic Fabric Automation (Vinci)

4.4.1 Topology Design Review Thereare9Spines,180Leafsand2BorderLeafsinthistopology.SpineS1andRemoteDC‐SpineareNexus6004Switches.SpineS8isNexus7710ChassiswithF3CardandSpineS2

30

toS7areVDCswithinNexus7009ChassiswithF2eCards.LeafsL174toL177areNexus6001Switches.LeafsL178andLeaf179areNexus7009ChassiswithF3CardsandLeaf0toLeaf173areemulatedusingTitanium+VTORs.Border39andBorderLeaf40areNexus7706andNexus7710ChassisrespectivelywithF3cards.FabricConnectivityRealLeafs(L174‐L179)andBorderLeafs(BL38‐BL40)areconnectedto8spines.Spine8isconnectedto174EmulatedLeafs(L0‐L173)withthehelpofFanoutswitch(Nexus7018).UCSsusedforemulatingVTORsareconnectedtoFanoutswitchwithaTrunkLinkasdescribedinVinciHybridClusterapproach.ServerLeafConnectivityForconfiguringVM,weuseEmulatedhosts.IXIAtrafficgeneratorwillbeusedforemulatinghostwithVDPandARP/NDSupportconnectedtoRealLeafs.DCIandL3VPNConnectivityatBorderLeafBGPbasedVRFLitewillbeconfiguredasborderLeafnodeandDCInode.BGPMPLSVPNwillbeconfiguredbetweenDCIPeers.IXIAemulationwillbeusedtoemulateremotebranchroutes.

4.4.2 Hardware and Software Overview ModelNo. ImageVersion

N77K N7F‐SUP2E /N77‐F324FQ‐25/N77‐F248XP‐23E/N7F‐F248XP‐23E

7.2(0)D1(1)

N7K N7K‐SUP2E / N7K‐F312FQ‐25/N7K‐F248XP‐25E/N7K‐F248XP‐25

7.2(0)D1(1)

N6K N6K‐C6001‐64P‐SUP /N6K‐C6001‐M4Q

7.1(1)N1(1)

N6K N6K‐C6004‐96Q‐SUP /N6K‐FIXED‐LEM/N6K‐C6004‐M12Q

7.1(0)N1(1)

N6K N6K‐C6004‐96Q‐SUP /N6K‐FIXED‐LEM

7.0(2)N1(1)

CiscoPrime DataCenterNetworkManager

7.2(0.14)

31

4.4.3 Topology

4.4.4 Traffic and Metrics

IPv4 Unicast Traffic Across Different Leaf/Across Border Leaf

IPv6 Unicast Traffic Across Different Leaf/Across Border Leaf

IPv4 Multicast Traffic Across Different Leaf/Across Border Leaf

Device AverageCPU

Utilization/LoadAverage15mins

MemoryUtilization

Leaf–PrimaryvPC 0.34 Memoryusage:5965956Kused

Leaf–SecondaryvPC 0.25 Memoryusage:5967084Kused

Spine 0.36 Memoryusage:5757104Kused

BL 0.43 Memoryusage:6308256Kused

32

4.5 DC1


Platform ModelNo. SoftwareN7K N7KSUP2/F1/M1 7.2.0N5K N5K‐C5548UP‐SUP 7.0.6.N1.1N3K N3K‐C3048TP‐1GE‐SUP 5.0.3.U5.1cC6K VS‐SUP720‐10G 151‐1.SY WS‐SUP720 122‐33.SXJ4C4K WS‐X45‐SUP7‐E 03.03.02.SG.151‐1.SG2 WS‐C4948 150‐2.SG6‐6.10N2K C2224TP;C2248TP;C2232PP

4.5.2 Network Logical Topology Design Overview

Thetopologiesandtestcasesvalidatehigh‐availabledatacenternetworksinordertoprovideunifiedfabricandcomputingservices.ThisisachievedbyusingtheNexus7010andNexus5548withfeaturessuchasvPCandFabric‐Path.Nexus7010areinstalledwithF1andM1linecardstoprovidelegacyL2&L3andvPClegportchannelconnectivitywithpeerdevicesandF1portsareusedforfabric‐pathport‐channelwithN5000peers.ArrayofNexus2000areconnectedwithNexus7010asfabric‐extenders(FEX).AccesslayerswitchesinthisdatacenterextendedtoIXIA(TrafficGenerator)portstosimulateendhosts/serverstosendandreceiveunicast&multicasttrafficThedatacentersiteisbuiltaroundtheNexus7010SUP2ataggregation.Thisdatacentersiteissplitintotwohalves:DC101:(vPC)

Nexus7000withvPCtoNexus5000,C4K/C6Kforaccess Nexus7000withlegacyether‐channel(trunk)withC4K Nexus7000withNexus2000FEX Nexus7000withL3toNexus3048

DC102:(vPC+)

Nexus7000(spine)withFabric‐PathtoNexus5000(leaf) Nexus7000withvPCtoC6Kforaccess Nexus7000withL3toNexus3048

33

DC101Device Platform Position VPCStatusDC101‐5(VPCSec.) N7K Aggregation VPCDC101‐6(VPCPri.) N7K Aggregation VPCDC101‐7 C6K Access VPCDC101‐8 C6K Access VPCDC101‐17(VPCPri.) N5K Access DualsidedVPCDC101‐18(VPCSec.) N5K Access DualsidedVPCDC101‐27(VPCPri.) N5K Access DualsidedVPCDC101‐28(VPCSec.) N5K Access DualsidedVPCDC101‐37 C4K Access NonVPCDC101‐38 C4K Access NonVPCFEX101,102,103 N2K AccessFEX VPC‐HIFDC101‐47 N5K Access L3

DC102Device Platform Position VPCStatusDC102‐51(VPC+Sec.) N7K Aggregation Fabric‐path‐SpineDC102‐52(VPC+Pri.) N7K Aggregation Fabric‐path‐SpineDC102‐53(VPC+Sec.) N7K Aggregation Fabric‐path‐SpineDC102‐54(VPC+Pri.) N7K Aggregation Fabric‐path‐SpineDC102‐701(VPC+Pri.) N5K Access Fabric‐path‐LeafDC102‐702(VPC+Sec.) N5K Access Fabric‐path‐LeafDC102‐703(VPC+Pri.) N5K Access Fabric‐path‐LeafDC102‐704(VPC+Sec.) N5K Access Fabric‐path‐LeafDC102‐705(VPC+Pri.) N5K Access Fabric‐path‐LeafDC102‐706(VPC+Sec.) N5K Access Fabric‐path‐LeafDC102‐7011(VPC+Pri.) N5K Access DualsidedVPCDC102‐7012(VPC+Sec.) N5K Access DualsidedVPCDC102‐17 C6K Access VPCDC102‐18 C6K Access VPCDC102‐27 C6K Access VPCDC102‐47 N3K Access L3

4.5.3 Configuration Details

Thefollowingconfigurationsareappliedtothetestnetwork:

34

Commonsystemcontrol,managementandaccounting:CommonsystemfeatureslikeSSH,TACACS+,Syslog,SNMP,NTP,SPAN,DNSandManagementVRFareconfigured

BGP:eBGPisconfiguredbetweenthecoreswitchesandthepubliccloud.

OSPF:OSPFistheIGPrunningacrossthenetwork.Eachaggregation‐accessblockisconfiguredasauniqueareawiththecoreswitchesplayingtheroleoftheABR.

PIM‐SM:PIMSparseMode/PIMAnySourceMulticastisdeployedacrossthenetworktosupportmulticast.Eachaggregation‐accessblockisconfiguredwiththeRPforlocallysourcedgroups.

MSDPAnycastRP:MSDPisdeployedtoexchangesourceinformationbetweenAnycastRPs.

vPC:vPCtechnologyisdeployedintheaggregation‐accessblockDC101asshowninFigure1.Inaddition,dual‐sidedvPCisconfiguredbetweentheNexus7000andNexus5000switches

FP:FabricPathisdeployedintheaggregationblockDC102.Thespinelayercomprises

Nexus7000switchesandtheleafswitchesaredeployedusingNexus5000switches.

VLANtrunking:VLANtrunkingisusedintheaggregation‐accessblockstomaintainsegregationandsecurity.

STP:RapidSpanningTreeProtocolisusedtopreventLayer2loopsintheaggregation‐

accessblocks.Thespanningtreerootisplacedontheaggregationlevel.RootGuardisconfiguredontheaggregationleveltoenforcerootplacement.BPDUFilter,BPDUGuardandPort‐FastEdgeareconfiguredontheaccessportstowardshosts.

HSRP:HSRPv2isusedasthefirsthopgatewayprotocolforhosts;HSRPconfiguredatN7KaggregationVDCs(DC101‐5,DC101‐6&DC102‐51,DC102‐52,DC102‐53,DC102‐54)

FEX:ThreesetofFabricExtenders(Nexus2000)FEX101,FEX103&FEX104aredeployed

onNexus7000atDC101

FEXuplinksarelegacyportchannelswhereasFEXhostportsareconfiguredasvPC.

IGMP:IGMPisusedbyhoststojoinmulticastgroupsofinterest.IGMPsnoopingisenabledonallswitchesintheaggregation‐accessblockstopreventfloodingofmulticastdatatraffic.

LACP:LACPisusedforlinkaggregationtoformport‐channelsacrossthenetwork.

UDLD:UDLDaggressivemodeisconfiguredacrossthenetworktodetectandprevent

unidirectionallinks

PVLAN:Promiscuous/Isolated/CommunityPVLANconfiguredbetweenNexus7000toNexus5000,C6KandC4KovervPCaswellasnonVPCatDC101.PVLANtrafficconfiguredoverFabric‐pathPOatDC102

35

4.5.3.1 DC1 Topology

36

4.6 ENT

4.6.1 Hardware and Software Overview Platform ModelNo. SoftwareN7000 SUP2E,M2 7.2.0N7700 SUP2E,F2E,F3(40Gwith

breakout)7.2.0

N6000 N6K‐C6004‐96Q‐SUP,N6K‐C6001‐64P‐SUP

7.1.0.N1

N3000 N3K‐C3548P‐10G‐SUP,N3K‐C3048TP‐1GE‐SUP

6.0.2

4.6.2 Network Logical Topology Design Overview Thetopologiesandtestcasesvalidatehigh‐availabledatacenternetworksinordertoprovideunifiedfabricandcomputingservices.ThisisachievedbyusingtheNexus7010andNexus7700withfeaturessuchasvPCscale,VRF,PVLAN,ACL,netflow,wccp,multicast,dualvpc,etc.Nexus7710installedwithF2EandF3linecardsprovidelegacyL2&L3andvPClegportchannelconnectivitywithpeerdevices.AccesslayerswitchesinthissetupareextendedtoIXIA(TrafficGenerator)portstosimulateendhosts/serverstosendandreceiveunicast&multicasttrafficThedatacentersiteisbuiltaroundtheNexus7010withSUP2Eatcore&Nexus7710withSUP2Eataggregation.ENT1Aggregation:Nexus7710with120VPCPOtoN6004Nexus7710withDualVPCPOtoN6001Nexus7710withVRFsonF2physicalportsand6ECMPtoN3kNexus7710withVRFsonF3Port‐channels/sub‐interfacetoN3kNexus7710withPVLANonF3andF2toN3K

ENT1Device Platform Position StatusEnt‐1(VDC) N7000 Backbone L3Ent‐3 N7000 Core L3Ent‐4 N7000 Core L3Ent‐5 N7700 Aggregation VPCEnt‐6 N7700 Aggregation VPC

37

VPC‐sim1 N6004 VPCsimulator FanoutVPC‐sim2 N6004 VPCsimulator FanoutVPC‐sim3 N3548 Access No‐VPC/RegularL2Ent‐701 N6001 Access DualsidedVPCEnt‐702 N6001 Access DualsidedVPCEnt‐703 N3048 Access L3withVRFEnt‐704 N3048 Access L3withVRFEnt‐705 N3048 Access L3withVRFEnt‐706 N3048 Access L3withVRFEnt‐FO1 N3048 Fanout Fanout/RegularL2

4.7 Configuration Details Thefollowingconfigurationsareappliedtothetestnetwork:

Commonsystemcontrol,managementandaccounting:CommonsystemfeatureslikeSSH,TACACS+,Syslog,SNMP,NTP,SPAN,DNSandManagementVRFareconfigured


PIM‐SM:PIMSparseMode/PIMAnySourceMulticastisdeployedacrossthenetworktosupportmulticast.Eachaggregation‐accessblockisconfiguredwiththeRPforthelocallysourcedgroups.


vPC:vPCtechnologyisdeployedintheaggregation‐accessasshowninFigure1.Inaddition,dual‐sidedvPCisconfiguredbetweentheNexus7000andNexus6000switches


accessblocks.Thespanningtreerootisplacedontheaggregationlevel.RootGuardisconfiguredontheaggregationleveltoenforcerootplacement.BPDUFilter,BPDUGuardandPort‐FastEdgeareconfiguredontheaccessportstowardshosts.

HSRP:HSRPv2isusedasthefirsthopgatewayprotocolforhosts;HSRPconfiguredatN7Kaggregation

IGMP:IGMPisusedbyhoststojoinmulticastgroupsofinterest.IGMPsnoopingisenabledonallswitchesintheaggregation‐accessblockstopreventfloodingofmulticastdatatraffic.

LACP:LACPisusedforlinkaggregationtoformport‐channelsacrossthenetwork.

38

4.8 ENT Topology

39

4.9 MSDC Scale ThetopologiesandtestcasesvalidatetherequirementoffeatureandscaleinaMassiveScalableDatacenter(MSDC)deployment.ThisisachievedbyusingtheNexus7010,Nexus3k,andNexus5548withprotocolssuchasOSPF,BGP,andBFDtocreatevariousprofiles.Thefollowingfeaturesarecoveredinprofile1:‐ BGPorOSPF BFDwithdefaulttimers(50ms)or250ms SNMP NTP SYSLOG

Thefollowingfeaturesarecoveredinprofile2‐1: OSPF/OSPFv3 BFD(v4andv6)withtimersof250ms vPC MulticastwithMSDP(v4) IGMP ACL DHCPrelay Highavailability SNMP NTP SYSLOG

4.9.1 Topology Design Overview MSDCcovered2mainprofiles–profile1withBGP/OSPFandprofile2‐1

1) Profile1–leafnodeona7018with16F2,runningeitherBGPorOSPFasroutingprotocolwithBFDusingdifferentbfdtimers,default(50ms)or250ms.

2) Profile2‐1–L2/L3ona7018with2M2,runningospf/ospfv3asroutingprotocolwithBFDusing250mswithvpc,multicast.

RolesandRouters/devicesRoles RoutersandDevices

Spinenode N7K‐114‐101(UUT1),N7K‐114‐102defaultVDC(UUT2)andVDC1andN7k‐114‐108VDC1.16/32ECMPisbetweenspinenodeandgateway/loadbalancersandfirewall.

Leafnode/ToR N3Kandsimulatedby4510‐1‐fanout/4510‐2‐fanout/4510‐3fanout+Ixia

Gateway,Loadbalancers,Firewall

SimulatedbyN7K‐114‐108defaultVDCand4507‐1‐fanout+Ixia

40

4.9.2 ISSU Matrix

ISSU/ColdbootISSU6.2.8a‐>7.2ISSU6.2.10‐>7.2 PassISSU6.2.12‐>7.2 PassISSU7.2‐>upg Pass

4.10 M1 vPC Scale


Platform ModelNo. SoftwareN7K N7KSUP2EM1 7.2.0.C6K VS‐SUP720‐10G 122‐33.SXJ1

4.10.2 Network Logical Topology Design Overview

41

Thetopologyvalidateshigh‐availablenetworksthatdepictthevariousprivateVLANfeatureimplementationsinordertoprovideanideaoftheprivateVLANscalenumberssupported.ThisisachievedbyusingtheNexus7000andCatalyst6500switches.Figure 3 illustrates thenetworkbuilt around2Nexus 7000 switcheswith Sup2e andM1modules.Thetopologycontains:

Nexus7000withvPCtoCatalyst6500VSSswitchesforaccessconfigured. Nexus7000connectedtoCatalyst6500switchwithclassicalPort‐channels. Nexus7000connectedtoCatalyst6500switchesusingorphanports.

4.10.3 Configuration Details

Thefollowingconfigurationsareappliedtothetestnetwork:

Commonsystemcontrol,managementandaccounting:CommonsystemfeatureslikeSSH,Syslog,SNMP,NTPandManagementVRFareconfigured.

vPC:vPCtechnologyisdeployedinthenetworkbetweentheN7kandtheCatalystVSSswitchesasshowninthefigure3.

VLANtrunking:VLANtrunkingisusedintheaggregation‐accessblockstomaintainsegregationandsecurity.

STP:RapidSpanningTreeProtocolisusedtopreventLayer2loopsintheaggregation‐accessblocks.Thespanningtreerootisplacedontheaggregationlevel.RootGuardisconfiguredontheaggregationleveltoenforcerootplacement.BPDUFilter,BPDUGuardandPortFastEdgeareconfiguredontheaccessportstowardshosts.

LACP:LACPisusedforlinkaggregationtoformport‐channelsacrossthenetwork PVLAN:PVLANisconfiguredinthenetworkandisthemainfocusoftesting.The

followingPVLANcomponentsarecoveredinthenetwork: PVLANprimaryandsecondaryvlan(CommunityandIsolated) Promiscuoustrunkandsecondarytrunkonvpc PrivateVLANhostonclassicport‐channel. PrivateVLANpromiscuoustrunk,secondarytrunkonclassicPort‐channel

42

4.11 M1 vPC Scale Topology

4.12 DC1, ENT1 and M1 vPC Feature / Scale Coverage

FeatureMax DC1 ENT M1vPCF1 2 M1 5VDC 4FEX 6FEX‐HIF 96 ACL/ACE 500 500 POMembers 2VLAN/SVI 100 2000VLANperFEX 10FPVLAN 50MAC 7K 1K vPC 120VLANpervPC 1,10,16HSRPv2 100 2000OSPFNeighbor 16BGPNeighbor 2 18 PIMNeighbor 256 2000 PBRSequence 20 200

43

UnicastRoute 10K 22KMulticastRoute 500 3K MulticastSource 20MulticastOIF 10 MulticastGroup 100 390FabricExtender 3FPIS‐ISadjacencies 6 FPNumberofSwitchID’s 20PrimaryVLAN 20 140SecondaryVLAN 25 280Phy.PortsusedforPVLAN 468Port‐channel 125 388Port‐channelinPVLAN 18 388VPCPOwithPVLAN 10 256Hostmode 18 32PromiscuousAccess 2 64PromiscuousTrunk 2 186TrunkSecondary 7 186PVLAN 50 60vPCConfig‐Sync 120Netflow 84 WCCP 58

4.13 ISSU Matrix for DC1, ENT and M1 vPC Image ISSU/COLDBOOT DC1 ENT M1vPC7.2.0>7.2.0.upg ISSU PASS PASS PASS6.2.12>7.2.0 ISSU PASS PASS6.2.10>7.2.0 ISSU PASS6.2.8a>7.2.0 ISSU PASS 6.2.8b>7.2.0 ISSU PASS7.2.0>7.2.0.upg COLDBOOT PASS 6.2.12>7.2.0 COLDBOOT PASS6.2.10>7.2.0 COLDBOOT PASS 6.2.8a>7.2.0 COLDBOOT PASS 6.2.8b>7.2.0 COLDBOOT PASS PASS PASS7.2.0>6.2.12 COLDBOOT PASS PASS 7.2.0>6.2.10 COLDBOOT PASS7.2.0>6.2.8a COLDBOOT PASS7.2.0>6.2.8b COLDBOOT PASS PASS

44

5 NVT Findings/Conclusion/Recommendations Assigned/NewStillworkingonfixesandmaybeseeninCCOimageUnreproducibleNotseeninCCOimagemayhavefixedbyothercodefixes.Verified/ResolvedFixedinCCOimageClosedSystemlimitationandbehaviorwillremainthesameCSCus00094Symptom:IPFIBprocessmightcrashuponenablingLDPautoconfigfeature.Conditions: whilerunningthesystemimage6.2(10)andinpresenceofF3moduleswithMPLSfeaturesetenabled(configurationnotsupportedcurrently),theIPFIBprocessmightcrashuponenablingLDPautoconfig. Workaround:None,deploy7.2(0)instead.Severity:SevereStatus:ClosedPlatformSeen:Nexus7700ResolvedReleases:ApplicableReleases:6.2(10)CSCus00165Symptom: InaN7700systemrunning6.2(10),F3modulesmightfailtocomeonlinedueto“SAC_INTR:XbarASICinterruptoccurred:SACChicoInterruptDetected”Conditions: UnderthesameconditionsofCSCus00094, upontheIPFIBprocesscrash,alltheF3modulesmightfailtocomeonline. Workaround:None,deploy7.2(0)instead.Severity:SevereStatus:ClosedPlatformSeen:Nexus7700ResolvedReleases:ApplicableReleases:6.2(10)CSCus02115Symptom:M1failureuponupgradingthesystemimagefrom6.2(10)toa7.2(0)dailyimage.Conditions: whileupgradingthesystemimagefrom6.2(10)toaninterim7.2(0)build,oneoftheM1modulesfailedthebootupsequence. Workaround:reloadfailedmodule.Severity:SevereStatus:UnreproduciblePlatformSeen:Nexus7000ResolvedReleases:ApplicableReleases:7.2(0)CSCus22635Symptom:Sometime,theconfigurationoftheIPv6VIPmightnotsucceed.Conditions:Theproblemhasoccurredonlyonesanditcouldnotberecreatedanymore.Workaround:reloadthesystemSeverity:SevereStatus:UnreproduciblePlatformSeen:Nexus7000ResolvedReleases:ApplicableReleases:7.2(0)

45

CSCus31113Symptom:Sometimes,inascaledenvironment,MP‐BGProutesmightfailtobeproperlyinstalledintheRIBforoneormoreVRFs.Conditions:Sometimes,uponsystemreloadinascaledenvironment,someroutesorsomemroutesmightnotgetproperlyinstalledinthesharedmemory.Workaround: changethesettingforthe“limit‐resourcesu4route‐memminimumNNNmaximumNNN”andthenperformasystemreloadorasystemswitchoverinordertoensurethenewvalueisproperlyparsed.Toestimatethepropervalueforthememoryallocation,thefollowingcommandcanbeused“showrouting<ip|ipv4|ipv6|multicast>estimatememoryestimateroute<#ofroutes|mroutes>” Severity:MinorStatus:AssignedPlatformSeen:Nexus7000ResolvedReleases:ApplicableReleases:CSCus50951Symptom:Sometimes,inascaledenvironment,MP‐BGProutesmightfailtobeproperlyinstalledintheRIBforoneormoreVRFs.Conditions:Sometimes,uponsystemreloadinascaledenvironmentwithmanyhostroutes,someroutes(orsomemroutes)mightnotgetproperlyinstalledinthesharedmemory.Workaround: RestartBGPprocessorreloadthesystem

Severity:SevereStatus:UnreproduciblePlatformSeen:Nexus7700ResolvedReleases:ApplicableReleases:CSCut60771Symptom:InaMVPNsite,multicasttrafficsourcedlocallymightgetcontinuouslyduplicatedforthereceiversattachedtotheprimaryvPCorphanport.Conditions:InagivenMVPNsitewithbothlocalsourcesandreceivers,multicasttrafficmightgetcontinuouslyduplicatedforallthelocalgroupsforthereceiversattachedtothevPCprimaryorphanport.Workaround: None Severity:SevereStatus:Unreproducible PlatformSeen:Nexus7000ResolvedReleases:ApplicableReleases:7.2(0)CSCuu02810:Symptom:vlan_mgrtimedoutduringsystembringupwithcopyrsandreloadConditions:Aftercopyr&s,reloadofsystem,vdc,resultsinvlan_mgrtimeouterrorsWorkaround:noneSeverity:SevereStatus:ResolvedPlatform:Nexus7000ResolvedReleases:7.2.0ApplicableReleases:7.2.0CSCuu27998:

46

Symptom:("InvalidVLANID")communicatingwithMTS_SAP_VLAN_MGRConditions:Invalidnativevlanallocationduringavdcreload.Workaround:noneSeverity:ModerateStatus:AssignedPlatform:Nexus7000ResolvedReleases:ApplicableReleases:7.2.0CSCuu38691:Symptom:ISSUfaileddueto/nxos/tmpspaceexhaustionwithloggingConditions:ISSUfailsbeforepre‐checkandabortswithoutdoingthepre‐checkWorkaround:noneSeverity:ModerateStatus:AssignedPlatform:Nexus7000ResolvedReleases:ApplicableReleases:7.2.0CSCus21952Symptom:PVLANtrafficblock‐holeduringCOLDBOOTfrom7.2.0.512.binto6.2.10.binCondition: L3egresstrafficfromTrunksecondaryportwasnotworkingduetodummybdconfigurationforprimaryvlan.Workaround:Use6.2.12insteadof6.2.10Severity:Status:Resolvedandverifiedin6.2.12(fixnotavailablein6.2.10)Platform:Nexus7000ResolvedRelease:6.2.12ApplicableRelease:CSCuq47581Symptom:VPCPeer‐linkflapuponISSUto7.2.0Condition: ObjecttrackingfeatureinplaceWorkaround:RemoveobjecttrackingwhileperformISSUandadditbackSeverity:Status:CLOSEDPlatform:Nexus7000ResolvedRelease:ApplicableRelease:7.2.0CSCus71454Symptom:VPCprimarylegflapupononVPCpeerlinkflapCondition: VPC leg configured as Private VLAN host modeWorkaround:NoSeverity:2Status:APlatform:Nexus7000ResolvedRelease:ApplicableRelease:7.2.0

47

Symptom:FEXHIFconfigurationmisswhileCOLDBOOTfrom7.2.0to6.2.xCondition: COLD BOOT from 7.2.0 to 6.2.xWorkaround:Re‐applyHIFconfigurationafterFEXcomeonlineSeverity:Status:Platform:Nexus7000ResolvedRelease:ApplicableRelease:CSCuu39696Symptom:Removingpvlanhost‐associationfromoneVPCPOflapsallVPCPOCondition: VPC Po configured as community hostWorkaround:NoSeverity:2Status:APlatform:Nexus7000ResolvedRelease:ApplicableRelease:6.2.12&7.2.0

6 Appendix

6.1 VxLAN Solution Configuration Guide Thefollowingconfigurationsaredividedasbelow

‐ Spine‐ Leaf‐ LISPMapserver‐ LISPBranchrouter

Spine configuration guide N77K

o Featuresetrequiremento Installfeature‐setfabric#Installedfeature‐setfabrico Feature‐setfabric#Enable/Disablefabrico featurebgp#Enable/DisableBorderGatewayProtocolo featurelisp#Enable/DisableLocator/IDSeparationProtocolo featurepim#Enable/DisableProtocolIndependentMulticasto featureospf#Enable/DisableOpenShortestPathFirstProtocolo featureinterface‐vlan#Enable/Disableinterfacevlano featurelacp#Enable/DisableLACPo featurebfd#Enable/DisableBFDo featurenvoverlay#Enable/DisableNVOverlayo featurevni #Enable/DisableVirtualNetworkSegment(VNI)

o Globalconfigurationcommando nvoverlayevpn#Enable/DisableEthernetVPN(EVPN)o systembridge‐domain#ConfigureBridge‐domainIDo vni<range> #configureVNIrangeo ippimrp‐address<>#ConfigurestaticRPforgrouprangeforBUM

Traffico bridge‐domain<>#configureBridge‐domainID

o VRFConfiguration

48

o vrfcontext<>#CreateVRFandenterVRFmodeo Vni<> #configureL3VNIforVXLANo rdauto #GenerateRDautomaticallyo address‐familyipv4unicast#ConfigureIPv4addressfamilyo route‐targetimport<> #ImportTarget‐VPNcommunityo route‐targetimport<>evpn#SpecifyTargetforEVPNrouteso route‐targetexport<>#ExportTarget‐VPNcommunityo route‐targetexport<>evpn#SpecifyTargetforEVPNrouteso route‐targetbothauto#ExportAndImportTarget‐VPNcommunityo route‐targetbothautoevpn#SpecifyTargetforEVPNroutes

o MapthebridgedomaintoL3VNIo bridge‐domain<> #Bridge‐domainIDo membervni<> #configureL3VNIassociatedwithVRF

o Bridge‐domaininterface(eachVRFmusthave1bridge‐domain)o interfacebdi<> #Bdiinterfaceo noshutdown#Enable/disableaninterfaceo vrfmember<> #ConfigureVRFparameterso ipforward#Enableipforwardingoninterface

o Port‐channelinterface(forallL3links)o interfaceport‐channel1 #PortChannelinterfaceo mtu9216 #Configuremtufortheporto bfdinterval<>min_rx<>multiplier<>#BFDintervalcommandso noipredirects #disableIPdirected‐broadcasto ipaddress<> #ConfigureIPaddressoninterfaceo ippimsparse‐mode #onfiguressparse‐modePIMoninterfaceo Interfaceethernet<> #EthernetIEEE802.3zo mtu9216 #Configuremtufortheporto channel‐group1modeactive #Configureportchannelparameterso noshutdown #Enable/disableaninterface

o NVEinterface

o interfacenve1 #NVEinterfaceo noshutdown #Enable/disableaninterfaceo source‐interface<> #NVESource‐Interfaceo host‐reachabilityprotocolbgp #Configurehostreachabilityadvertisemento membervni<>associate‐vrf #Associatevniwithavrf

o BGPGlobalconfigurationo routerbgp1 #BorderGatewayProtocol(BGP)o address‐familyipv4unicast #Configureunicastaddress‐familyo network<loopback> #ConfigureanIPprefixtoadvertiseo redistributedirectroute‐map<passall> #Directlyconnectedo maximum‐paths<> #Numberofparallelpaths(EBGP)o maximum‐pathsibgp<> #Numberofparallelpaths(IBGP)

49

o address‐familyl2vpnevpn #ConfigureL2VPNEVPNaddress‐familyo route‐map<passall>permit10#Permitanyo vrf<> #ProductionVRFo address‐familyipv4unicast#Configureunicastaddress‐familyo network0.0.0.0/0 #Defaultrouteo advertisel2vpnevpn #advertisel2vpnevpnrouteso redistributedirectroute‐mappassall#Directlyconnected

o IBGP(SpineandLeafconfiguration)–SuggestiontoconfigureIPv4overinterfaceand

EVPNpeeringoverloopback

o templatepeerIBGP‐IPv4 #Templateconfigurationforpeerparameterso bfd #BidirectionalFastDetectionfortheneighboro remote‐as1 #SpecifyASNumberoftheneighboro password3<> #Configureapasswordforneighboro address‐familyipv4unicast #Configureunicastaddress‐familyo send‐community #SendCommunityattributetothisneighboro route‐reflector‐client #ConfigureaneighborasRoutereflectorcliento route‐mapSELFout #Applyroute‐maptoneighboro route‐mapSELFpermit10 #route‐mapnameo setipnext‐hoppeer‐address #Usepeeraddress(forBGPonly)o neighbor<>remote‐as1 #neighborIBGPpeeringo inheritpeerIBGP‐IPv4 #Inheritatemplateo update‐source<> #SpecifysourceofBGPsessionandupdates

o templatepeerIBGP‐EVPN #Templateconfigurationforpeerparameterso bfd #BidirectionalFastDetectionfortheneighboro remote‐as1 #SpecifyASNumberoftheneighboro password3<> #Configureapasswordforneighbor o update‐sourceloopback0 #SpecifysourceofBGPsessionandupdateso address‐familyl2vpnevpn #ConfigureL2VPNEVPNaddress‐familyo send‐communityextended #SendStandardandExtendedCommunity

attributeso route‐reflector‐client #ConfigureaneighborasRoutereflectorcliento neighbor<>remote‐as1 #neighborIBGPpeeringo inheritpeerIBGP‐EVPN #Inheritatemplate

o EBGP(Inter‐PODSpine)–SuggestiontoconfigureIPv4overinterfaceandEVPNpeeringoverloopback

o templatepeerEBGP‐IPv4 #Templateconfigurationforpeerparameterso bfd #BidirectionalFastDetectionfortheneighbor

o remote‐as10 #SpecifyASNumberoftheneighboro password3<> #Configureapasswordforneighbor o address‐familyipv4unicast #Configureunicastaddress‐familyo send‐communityextended #SendCommunityattributetothisneighboro neighbor<>remote‐as10 #neighborEBGPpeering

50

o inheritpeerEBGP‐IPv4 #Inheritatemplate

o templatepeerEBGP‐EVPN #Templateconfigurationforpeerparameterso bfd #BidirectionalFastDetectionfortheneighboro remote‐as10 #SpecifyASNumberoftheneighboro password3<> #Configureapasswordforneighboro update‐sourceloopback0 #SpecifysourceofBGPsessionandupdateso ebgp‐multihop2 #SpecifymultihopTTLforremotepeero address‐familyl2vpnevpn #ConfigureL2VPNEVPNaddress‐familyo send‐communityextended #SendStandardandExtendedCommunity

attributeso route‐mapUNCHGDout #Applyroute‐maptoneighboro route‐mapUNCHGDpermit10#route‐mapnameo setipnext‐hopunchanged #Useunchangedaddress(foreBGPsessiononly)o neighbor<>remote‐as10 #neighborEBGPpeeringo inheritpeerEBGP‐EVPN #Inheritatemplate

o IBGP (Spine and Legacy TOR configuration) – Legacy TORs eBGP peer‐to‐spine through "fake iBGP" by manipulating "local AS" in as‐path. So BGP peering is iBGP, but the local AS are

different, so the route gets redistributed to MP‐iBGP L2VPN EVPN.

o Routerbgp1 #BorderGatewayProtocol(BGP)o VRF<> #ProductionVRFo neighbor<>remote‐as201 #neighborEBGPpeeringo local‐as202no‐prependreplace‐as #Specifythelocal‐asnumberforthe

eBGPneighbor,Donotprependthelocal‐asnumbertoupdatesfromtheeBGPneighbor,Prependonlythelocal‐asnumbertoupdatestoeBGPneighbor

o address‐familyipv4unicast #Configureunicastaddress‐family

o IBGP(Intra‐PODSpineoverRinglinks)–OSPFoverringlinksanIPv4andEVPNoverloopbacko routerospf1 #OpenShortestPathFirst(OSPF)o router‐id<loopback> #SetOSPFprocessrouter‐id

o interfaceEthernet1/22o mtu9216 #Configuremtufortheporto bfdinterval<>min_rx<>multiplier3#BFDintervalo nobfdecho #disableBFDEchoo noipredirects #disableIPdirected‐broadcasto ipaddress<> #ConfigureIPaddressoninterfaceo iprouterospf1area0.0.0.0 #enableospfontheinterfaceo noshutdown #Enable/disableaninterface

51

o neighbor<>remote‐as1 #neighborIBGPpeeringo update‐sourceloopback0 #SpecifysourceofBGPsessionandupdateso address‐familyipv4unicast #Configureunicastaddress‐familyo send‐communityextended #SendStandardandExtendedCommunity

attributeso route‐mapLOCAL_PREFin #Applyroute‐maptoneighboro address‐familyl2vpnevpn #ConfigureL2VPNEVPNaddress‐familyo send‐communityextended #SendStandardandExtendedCommunity

attributeso route‐mapLOCAL_PREFin #Applyroute‐maptoneighbor

o route‐mapLOCAL_PREFpermit10#route‐mapnameo setlocal‐preference50 #BGPlocalpreferencepathattribute(configure

lessthandefault100)

o EBGP(corelink)–setuproute‐mapwhichwilladvertiseonlynon‐mobilitysubnets

o Routerbgp1 #BorderGatewayProtocol(BGP)o VRF<> #ProductionVRFo neighbor<>remote‐as11 #neighborIBGPpeeringo address‐familyipv4unicast #Configureunicastaddress‐familyo route‐mapWAN‐mapout #Applyroute‐maptoneighbor

o route‐mapWAN‐mappermit10 #route‐mapnameo matchipaddressprefix‐listwan‐list #Matchentriesofprefix‐lists

o ipprefix‐listwan‐listseq5permit<subnet>#onlyallownon‐mobilityprefixes

towardcoreandLISPbranch

LISP Configuration on Spine (N7K)

o vrfcontext<> #ProductionVRF o iplispetr #ConfiguresLISPEgressTunnelRouter

(ETR)o lispinstance‐id1000 #ConfiguresInstance‐IDforglobaldata‐

mappingso iplispitrmap‐resolver<ipadd>#TointeractwithMap‐Resolver

(primary)o iplispitrmap‐resolver<ipadd> #TointeractwithMap‐Resolver

(secondary)o iplispetrmap‐server<ipadd>key<>#AuthenticationkeyusedwithMap‐

Server(Primary)o iplispetrmap‐server<ipadd>key<>#AuthenticationkeyusedwithMap‐

Server(secondary)

o lispdynamic‐eid101 #Configuredynamic‐EIDsforroamingo database‐mapping<Mobilitynetwork><corelinkaddress>priority1

weight

52

#configureEID‐prefixandlocator‐setfordynamic‐EIDo register‐route‐notificationstag1#Registermore‐specificroutesofthe

database‐mappingEID‐prefixtoMap‐Server,Include

onlyrouteswiththisBGPtag

LISP configuration on Map server ( CSR1kv)

o routerlisp #Locator/IDSeparationProtocolo ddtauthoritativeinstance‐id1000<Subnet>#ddtauthoritativetoallow

(VXLANMobilitysubnet

o map‐server‐peer<ipaddress>#IPv4Peermap‐serverlocatoraddress(secondary)

o map‐server‐peer<ipaddress>#IPv4Peermap‐serverlocatoraddress(Primary)

o ddt #Enableddto site<name> #LISPsitenameo authentication‐key<key>#passwordo eid‐prefixinstance‐id1000<subnet>accept‐more‐specifics#allowVXLAN

Mobilitysubnet

o eid‐prefixinstance‐id1000<subnet>accept‐more‐specifics#allowBranchsubnet

o ipv4map‐server#ConfiguresaLISPMapServero ipv4map‐resolver#ConfiguresaLISPMapResolver(MR)

LISP configuration on Branch ( ASR1K) o routerlisp #Locator/IDSeparationProtocolo eid‐tabledefaultinstance‐id1000#ConfiguresInstance‐IDforglobaldata‐

mappings

o database‐mapping<Mobilitynetwork><corelinkaddress>priority1weight#configureEID‐prefixandlocator‐setfordynamic‐EID

o ipv4itrmap‐resolver<ipaddress>#TointeractwithMap‐Resolver(primary)o ipv4itrmap‐resolver<ipaddress>#TointeractwithMap‐Resolver(secondary)o ipv4itr#ConfiguresaLISPIngressTunnelRouter(ITR)o ipv4etrmap‐server<ipaddress>key<key>#AuthenticationkeyusedwithMap‐

Server(Primary)o ipv4etrmap‐server<ipaddress>key<key>#AuthenticationkeyusedwithMap‐

Server(secondary)

53

o ipv4etr #ConfiguresaLISPEgressTunnelRouter(ETR)

o ipv4map‐cache‐limit<> #Addressfamilyspecificmapcacheconfiguration

Leaf configuration guide N9K o Featuresetrequirement

o featurebgp#Enable/DisableBorderGatewayProtocolo featurepim#Enable/DisableProtocolIndependentMulticasto featureinterface‐vlan#Enable/Disableinterfacevlano featurelacp#Enable/DisableLACPo featurebfd#Enable/DisableBFDo featurenvoverlay#Enable/DisableNVOverlayo featurevni #Enable/DisableVirtualNetworkSegment(VNI)o featurevn‐segment‐vlan‐based#Enable/DisableVLANbasedVNsegmento featurevpc #Enable/DisableVPC(VirtualPortChannel)

o Globalconfigurationcommando nvoverlayevpn#Enable/DisableEthernetVPN(EVPN)o ippimrp‐address<>#ConfigurestaticRPforgrouprangeforBUM

Traffico fabricforwardinganycast‐gateway‐mac<>#AnycastGatewayMACoftheSwitch

o Vlantovn‐segmentmappingconfigurationcommand(NoteitisrequireforeachL2VLANandL3VLANperVRF)o vlan<> #Vlancommandso vn‐segment<> #VNSegmentidoftheVLAN

o VRFConfigurationo vrfcontext<>#CreateVRFandenterVRFmodeo Vni<> #configureL3VNIforVXLANo rdauto #GenerateRDautomaticallyo address‐familyipv4unicast#ConfigureIPv4addressfamilyo route‐targetimport<> #ImportTarget‐VPNcommunityo route‐targetimport<>evpn#SpecifyTargetforEVPNrouteso route‐targetexport<>#ExportTarget‐VPNcommunityo route‐targetexport<>evpn#SpecifyTargetforEVPNrouteso route‐targetbothauto#ExportAndImportTarget‐VPNcommunityo route‐targetbothautoevpn#SpecifyTargetforEVPNroutes

o MapthebridgedomaintoL3VNIo bridge‐domain<> #Bridge‐domainIDo membervni<> #configureL3VNIassociatedwithVRF

o VPCdomain<>

o VPNdomain #Specifydomaino peer‐switch #EnablepeerswitchonvPCpairswitches

54

o rolepriority<>#ConfigureprioritytobeusedduringvPCrole(primary/secondary)election

o system‐priority<>#Configuresystempriorityo peer‐keepalivedestination<>>source<>vrf<>#Keepalive/Hellowithpeer

switcho peer‐gateway #EnableL3forwardingforpacketsdestinedtopeer's

gatewaymac‐addresso ipv6ndsynchronize #DisplayNeighborDiscoveryinterfaceinformationo iparpsynchronize #CFSsynchronize

o VPCKeepalivelink

o interfaceport‐channel<> #PortChannelinterfaceo mtu9216 #Configuremtufortheporto vrfmemberVPC #Setinterface'sVRFmembershipo ipaddress<> #ConfigureIPaddressoninterface

o VPCpeer‐linko interfaceport‐channel<> #PortChannelinterfaceo switchportmodetrunk #Portmodetrunko switchporttrunkallowedvlan<>#SetallowedVLANcharacteristicswhen

interfaceintrunkingmode

o spanning‐treeporttypenetwork#Considertheinterfaceasinter‐switchlinko vpcpeer‐link #Specifyifthislinkisusedforpeer

communication

o VPVlegport‐channel(hostconnectedlink)

o interfaceport‐channel<> #PortChannelinterfaceo switchportmodeaccess #Portmodeaccesso switchportaccessvlan<> #Setaccessmodecharacteristicsoftheinterfaceo spanning‐treebpdufilterenable #EnableBPDUfilteringforthisinterfaceo mtu9216 #Configuremtufortheporto vpc<> #VirtualPortChannelconfiguration

o Vlaninterfaceconfiguration

o interfaceVlan101 #Vlaninterfaceo noshutdown #Enable/disableaninterfaceo mtu9216 #Configuremtufortheporto vrfmember<> #ConfigureVRFparameterso noipredirects #DisableSendICMPRedirectmessageso ipaddress<> #ConfigureIPaddressoninterfaceo noipv6redirects #DisablesendingICMPv6Redirect

messageso fabricforwardingmodeanycast‐gateway#AnycastGatewayForwardingMode

55

o VlaninterfaceconfigurationmappedtoL3VNI

o interfacevlan<> #Vlaninterfaceo noshutdown #Enable/disableaninterfaceo mtu9216 #Configuremtufortheporto vrfmember<> #ConfigureVRFparameterso noipredirects #ConfigureIPaddressoninterfaceo ipforward #Enableipforwardingoninterface

o Port‐channelinterface(forallL3links)

o interfaceport‐channel1 #PortChannelinterfaceo mtu9216 #Configuremtufortheporto bfdinterval<>min_rx<>multiplier<>#BFDintervalcommandso noipredirects #disableIPdirected‐broadcasto ipaddress<> #ConfigureIPaddressoninterfaceo ippimsparse‐mode #Configuressparse‐modePIMoninterfaceo Interfaceethernet<> #EthernetIEEE802.3zo mtu9216 #Configuremtufortheporto channel‐group1modeactive #Configureportchannelparameterso noshutdown #Enable/disableaninterface

o NVEinterface

o interfacenve1 #NVEinterfaceo noshutdown #Enable/disableaninterfaceo source‐interface<> #NVESource‐Interfaceo host‐reachabilityprotocolbgp #Configurehostreachabilityadvertisemento membervni<L3VNI>associate‐vrf #AssociateL3vniwithavrfo membervni<L2VNI> #NVEVN‐SegmentMembershipo mcast‐group<> #NVEMulticastGroup

o BGPGlobalconfigurationo routerbgp1 #BorderGatewayProtocol(BGP)o address‐familyipv4unicast #Configureunicastaddress‐familyo network<loopback> #ConfigureanIPprefixtoadvertiseo maximum‐pathsibgp<> #Numberofparallelpaths(IBGP)o address‐familyl2vpnevpn #ConfigureL2VPNEVPNaddress‐familyo route‐map<passall>permit10#Permitanyo vrf<> #ProductionVRFo address‐familyipv4unicast#Configureunicastaddress‐familyo advertisel2vpnevpn #advertisel2vpnevpnrouteso redistributedirectroute‐mappassall#Directlyconnected

o IBGP(SpineandLeafconfiguration)–SuggestiontoconfigureIPv4overinterfaceand

EVPNpeeringoverloopback

o templatepeerIBGP‐IPv4 #Templateconfigurationforpeerparameters

56

o bfd #BidirectionalFastDetectionfortheneighboro remote‐as1 #SpecifyASNumberoftheneighboro password3<> #Configureapasswordforneighboro address‐familyipv4unicast #Configureunicastaddress‐familyo send‐community #SendCommunityattributetothisneighboro o neighbor<>remote‐as1 #neighborIBGPpeeringo inheritpeerIBGP‐IPv4 #Inheritatemplateo update‐source<> #SpecifysourceofBGPsessionandupdates

o templatepeerIBGP‐EVPN #Templateconfigurationforpeerparameterso bfd #BidirectionalFastDetectionfortheneighboro remote‐as1 #SpecifyASNumberoftheneighboro password3<> #Configureapasswordforneighbor o update‐sourceloopback0 #SpecifysourceofBGPsessionandupdateso address‐familyl2vpnevpn #ConfigureL2VPNEVPNaddress‐familyo send‐communityextended #SendStandardandExtendedCommunity

attributeso neighbor<>remote‐as1 #neighborIBGPpeeringo inheritpeerIBGP‐EVPN #Inheritatemplate

o IBGPintervPCSVIlinkoverpeer‐link(OnlyforIPv4peering)andaddvlanidtopeerlinkallowlist

o interfaceVlan3901 #Vlaninterfaceo noshutdown #Enable/disableaninterfaceo bfdinterval<>min_rx<>multiplier<> #BFDintervalcommandso nobfdecho #DisableEchofunctionforalladdressfamilieso noipredirects #disableIPdirected‐broadcasto ipaddress<> #ConfigureIPaddressoninterfaceo noipv6redirects #DisablesendingICMPv6Redirectmessageso ippimsparse‐mode #Configuressparse‐modePIMoninterface

o routerbgp<> #BorderGatewayProtocol(BGP)o address‐familyipv4unicast #Configureunicastaddress‐familyo network<> #AddSVIinterfaceaddresso neighbor<>remote‐as1 #neighborIBGPpeeringo bfd #BidirectionalFastDetectionfortheneighboro password3<> #Configureapasswordforneighbor

o update‐source<sviid) #SpecifysourceofBGPsessionandupdateso address‐familyipv4unicast #Configureunicastaddress‐familyo send‐communityextended #SendStandardandExtendedCommunity

attributeso route‐reflector‐client #ConfigureaneighborasRoutereflectorcliento route‐mapNHSout #Applyroute‐maptoneighbor

o route‐mapNHSpermit10 #route‐mapnameo setlocal‐preference50 #BGPlocalpreferencepathattribute(configure

lessthandefault100)

57

o setipnext‐hoppeer‐address #Usepeeraddress(forBGPonly)

o EVPNConfigurationforeachL2VNI

o Evpn #EnterEVPNconfigurationmodeo vni<>l2 #ConfigureEthernetVPNIDo rdauto #VPNRouteDistinguisherAutoo route‐targetimportauto #ImportTarget‐VPNcommunityandGenerate

RTautomaticallyo route‐targetimport<> #RTextcommunityinaa:nnformato route‐targetexportauto #ExportTarget‐VPNcommunityandGenerate

RTautomaticallyo route‐targetexport<> #RTextcommunityinaa:nnformat

6.2 Service Provider MPLS Configuration Guideline Thefollowingconfigurationsareappliedtothetestnetwork:

Commonsystemcontrol,managementandaccounting:CommonsystemfeatureslikeSSH,TACACS+,Syslog,SNMP,NTP,SPAN,DNSandManagementVRFareconfigured.

o featuretacacs+ #enablingthetacacsfeatureo tacacsserverhost<ipaddress>key<0/7> #configurethetacacsserverto

authenticateuserso aaagroupservertacacs+<groupname> #enableservergroupsfor

redundancy server<ipaddress> use‐vrf<vrf_name> #use‐vrfbasedonserver

reachabilityo snmp‐serveruser<user‐name><group‐name>authmd5<pass‐phrase>priv

<pass‐phrase>localizedkey #snmpv3userwithauthenticationenabled

o ntpserver<ipaddress> #enablentpwithserveripaddresso ipdomain‐name<domainname> #enabledomain‐nameo interfacemgmt0 #configuremgmt0

vrfmembermanagement ipaddress<ip_address>

o powerredundancy‐modeps‐redundant #setthepowerinredundancymode

o nosystemadmin‐vdc #disablesystemadmin‐vdco installfeature‐setfabricpath #enablefeature‐setforfabricpatho installfeature‐setmpls #enablefeature‐setformplso vdc<VDC_name>id1 #spawnsanewnamedVDC

limit‐resourcemodule‐type<module‐type> #enable<module‐type>modulesforthegivenVDC

allowfeature‐setfabricpath #allowsfabricpathfeature‐settobeenabledonthegivenVDC

allowfeature‐setmpls #allowsmplsfeature‐settobeenabledonthegivenVDC

allocateinterface<interface‐ranges> #allocateoneofmoreinterfacerangestothegivenVDC.Whenassigninganyrangeofinterfacesthemaximumgranularityapplicableforthe“allocateinterface”commandislimitedtothelayoutoftheport‐groupforeachspecifictypeofmodule.Attachingtothemoduleandusing“showhardware

58

internaldev‐port‐map”allowstodisplaythemappingbetweenfrontpanelportsandASICinstances.

limit‐resourceu4route‐memminimum200maximum200 #allowstoallocatemoresharedmemoryfortheIPv4routes.ToestimatetheamountofmemorytobeallocatedforbothIPv4orIPv6routes,thefollowingtwocommandscanbeused:“showiproutesum”oneachVRFand“showrouting<ip|ipv4|ipv6>memoryestimateroutes<total‐number‐of‐routes>next‐hop<N>”.Likewise,thesamemethodcanbeappliedformulticastallocation.

o control‐plane service‐policyinputtest‐‐copp‐policy‐strict #enablesthetest—copp‐

policy‐strictforthecontrol‐planecoppcopyprofilestrictprefixmvpnclass‐maptypecontrol‐planematch‐anymvpn‐copp‐class‐normalmatchexceptionipmulticastrpf‐failurematchexceptionipv6multicastrpf‐failurecontrol‐plane #applythemodifiedmvpnclasstothecontrol‐planetoreducetheeffectsofPIMassertissue(CSCut68318)service‐policyinputmvpn‐copp‐policy‐strict

o snmp‐serveruseradminnetwork‐adminauthmd50x213ea412ed9e450340d412f4c9741a25 #enablesuseradminwithnetwork‐adminprivilegesandmd5authentication

o ippimauto‐rpforwardlisten #enablestoforwardauto‐rp

messages”o ipmsdporiginator‐id<interface> #enablestheoriginator‐idfor

MSDPmessageso ipmsdppeer<remote‐peer‐address>connect‐source<interface> #establishMSDP

peeringwithagivenremotepeer

o spanning‐treevlan<vlan‐ranges>priority<priority>#enablesspanningtreeforthespecifiedlistofvlanranges

o vrfcontext<vrf‐name> #createanewvrf ippimrp‐address<rp‐address>group‐list<mcast‐groups/mask> #

configurestaticRPforthespecifiedsetofmulticastgroupswithinthegivenvrf rd<N:M> #configuretheroute‐

discriminatorforthegivenvrf.WhendeployedinavPC/vPC+scenariositisrecommendedtousedifferentRDsonthetwovPCpeers.

mdtdefault<mcast‐group‐address> #configuretheMDTdefaulttunnelforthegivenvrf

mdtdata<mcast‐group‐address/mask>threshold<rate>#configureasetofMDTdatatunnelsforthegivenvrfdependingonthesizeoftheconfiguredmask

address‐familyipv4unicast #configureIPv4unicastAF

route‐targetimport<NN:M> route‐targetexport<NN:M>

59

o vpcdomain<N> #configurevPCdomainnumberN

peer‐switch #enablespeer‐switchonthetwovPCpeerstoactasasingledevicefortheSTP

peer‐keepalivedestination<remote‐peer>source<local‐peer>vrfkeepalive#configurethepeer‐keepaliveonthespecifiedvrf.

delayrestore180 #delaysthelegbringupfor3minutestoguaranteetheconvergenceoftheroutingprotocolstominimizeNStrafficdisruption

peer‐gateway #toallowthevPCpeerdevicetoactastheactivegatewayforpacketsaddressedtotheotherpeerdevicerouterMAC

fabricpathmulticastload‐balance #enablesload‐balancingofmulticasttrafficbetweenthetwovPC+peerdevices.

fabricpathswitch‐id1 #definestheemulatedswitch‐idforthefabricpathvPCpair

config‐sync #enableconfigsynchronizationbetweenthetwovPCpeers

iparpsynchronize #enableARPsynchronizationbetweenthetwovPCpeers

o SwitchedVirtualInterfaces(SVIs):

interfaceVlanN mtu9216 #enablejumboframes vrfmember<vrf‐name> #assigntheSVItoagivenvrf noipredirects #preventtheroutertosend

redirectsmessagestotheclients(ICMP) ipaddress<ip‐address>/<24bitmask> ipv6address<ipv4‐address>/<64bitmask> ipospfpassive‐interface #disabletheroutingupdateson

thespeficiedOSPFinterfacepreventingtheformationofanyOSPFadjacency iprouterospf1area0.0.0.0 #enableOSPFonthespecified

interfaceandplaceitinarea0 ippimsparse‐mode #enablePIMsparsemode hsrpversion2 #settheHSRPversiontwo hsrpG #definestheHSRPgroupGforthe

specifiedinterface preemptdelayminimum120 #delaysthepreemptionfortwo

minutesuponHSRPswitchovertominimizenetworkdisruptions priority101forwarding‐thresholdlower1upper101 #definethepriority

forthisHSRPpeer ip<hsrp‐ip‐address> #HSRPvirtualIPaddress hsrpGv6ipv6 #definestheHSRPv6groupGv6for

thespecifiedinterface preemptdelayminimum120 priority101forwarding‐thresholdlower1upper101 ip<hsrpv6‐ip‐address>

o LACP:LACPisusedforlinkaggregationtoformport‐channelsacrossthenetwork:

interfaceport‐channelN #createaport‐channelN mtu9216 #enablejumboframesontheport‐

channelN ipaddress<ip‐address>/<24bitmask>

60

ipospfnetworkpoint‐to‐point #definetheOSPFnetworktypeasp2ptominimizethetimetoformtheOSPFadjacency

iprouterospf1area0.0.0.0 #enableOSPFonthespecifiedinterfaceandplaceitinarea0

ippimsparse‐mode #enablePIMsparsemode interfaceport‐channelM #createthevPCpeer‐linkasa

switchedport‐channelcarryingallthenecessaryVLANs. switchport switchportmodetrunk switchporttrunkallowedvlan11‐510,2001‐2500 spanning‐treeporttypenetwork #definesthenetworktypeas

networkforthevPCpeer‐link mtu9216 #enablejumboframes vpcpeer‐link #definestheport‐channelMasthe

vPCpeer‐link interfaceport‐channelR switchport switchportmodetrunk switchporttrunkallowedvlan11‐60 mtu9216 #enablejumboframes vpcR #definestheport‐channelRasa

vPCleg

o MPLSLDPconfiguration: mplsldpconfiguration router‐idLo0force #definestheloopbackinterface0

asrouter‐idforalltheLDPupdates

o OSPF/LDPconfigurations:

routerospf1 #settheOSPFprocessidasone router‐id<router‐id> #settheOSPFrouter‐id mplsldpautoconfigarea0.0.0.0 #turnonLDPonalltheOSPF

interfacesconfiguredinthebackbonearea(refertosection5forfurtherinformation)

o BGP:iBGPisconfiguredbetweenthecoreswitchesandthetworoute‐reflectors.iBGPis

alsoconfiguredbetweeneachPEoneachsiteandthetwoRRs: routerbgp1 #definestheBGPASasone graceful‐restart‐helper #enablesthegraceful‐restart‐

helper address‐familyipv4unicast maximum‐pathsibgp2 #enableICMPfortheAFIPv4

unicast templatepeerRR‐CLIENT #definesthetemplatefortheRR

iBGPpeering remote‐as1 #iBGPsession update‐sourceloopback0 #usestheloopbackinterfacezero

asthesourceoftheBGPupdatesandmessages address‐familyipv4unicast route‐reflector‐client #setthetemplatetobeaclientfor

theroute‐reflectorclusterintheAFIPv4unicast

61

address‐familyvpnv4unicast send‐communityextended address‐familyvpnv6unicast send‐communityextended #allowstheextendedattributesfor

bothAFVPNv4andVPNv6 address‐familyipv4mdt #enablesMVPNservices neighbor<RR1‐loopback‐interface‐0> inheritpeerRR‐CLIENT neighbor<RR2‐loopback‐interface‐0> inheritpeerRR‐CLIENT #enableiBGPpeeringwiththetwo

route‐reflectorsconfiguredinacluster vrfvrf1 #definestheVPNv4parameters

suchasrouteinjections,routeredistributions,ECMP,etc.etc. address‐familyipv4unicast network<loopbackinterface>/<32bitmask> #injecttheloopback

interfacetobeadvertisedtotheotheriBGPpeers.ThisinterfaceisusedascustomerRPinterface

network<SVI1‐ip‐address>/<24bitmask> … network<SVIN‐ip‐address>/<24bitmask> #injecttheSVIsubnets

thatbelongtothespecifiedvrfintoiBGP maximum‐pathsibgp2 #enableECMP address‐familyipv4unicast

o IGMPv2:IGMPisusedbyhoststojoinmulticastgroupsofinterest.IGMPsnoopingis

enabledonallswitchesintheaggregation‐accessblockstopreventfloodingofmulticastdatatraffic. ipigmpsnooping #bydefaultenabledonNexus

o FP:FabricPathisdeployedintheaggregationblockDC1‐Dist‐N7k‐102.The

spinelayeriscomprisedofNexus7000switchesandtheleafswitchesaredeployedusingNexus5000switches:

feature‐setfabricpath #proceduretoinstallandenablefabricpathfeatureset

fabricpathtopology12 membervlan2001‐2500 #definesthesetofVLANsusedfor

thevPC+accessswitch(CE12) fabricpathtopology6 membervlan11‐260 fabricpathtopology7 membervlan261‐510 vlan11‐510,2001‐2500 modefabricpath #definesthe2fabricpath

topologiesdeployedforthe2CEdevices(CE6andCE7) fabricpathswitch‐id106 #definesthelocalFPswitch‐id vpcdomain100 #entersthevPC/vPC+parameters fabricpathmulticastload‐balance fabricpathswitch‐id1 #definestheemulatedFPswitch‐

id.ThisiscommonbetweenthetwovPCpeers interfaceport‐channel7 #ThisinterfaceisusedasvPC

peer‐link.IthastocarrybothFPandvPC+VLANsandithastobeconfiguredinFPmode

switchportmodefabricpath

62

fabricpathisismetric800 #sinceinsteadystateitisnotdesiredtoforwardtrafficthroughthevPCpeer‐link,itisrecommendedtosettheISISmetricthehighestintheFPdomain

fabricpathtopology‐member6 fabricpathtopology‐member7 fabricpathtopology‐member12 #FPtopologydefinitions interfaceport‐channel100 switchportmodefabricpath

fabricpathisismetric10 #inordertominimizetherootchangesformulticasttraffic,itisrecommendedtosettheISISmetricinalltheFPinterfaces.Thiswillminimizetheimpactonthetrafficconvergenceuponnetworkdisruptions

6.3 DC1 Configuration Guideline Thefollowingconfigurationsareappliedtothetestnetwork:

Commonsystemcontrol,managementandaccounting:CommonsystemfeatureslikeSSH,TACACS+,Syslog,SNMP,NTP,SPAN,DNSandManagementVRFareconfigured.

o featuretacacs+ #enablingthetacacsfeatureo tacacsserverhost<ipaddress>key<0/7> #configurethetacacsserverto


redundancy server<ipaddress> use‐vrf<vrf_name> #use‐vrfbasedonserver

reachabilityo snmp‐serveruser<user‐name><group‐name>authmd5<pass‐phrase>priv





o featurebgp #enablebgpo routerbgp<autonomous‐id> #bgpautonomous‐ido router‐id<router‐id>o graceful‐restartstalepath‐time<120>o log‐neighbor‐changeso address‐familyipv4unicasto redistributedirectroute‐map<acl‐name> #route‐mapusedforredistribution

directlyconnectedsubnetso redistributeospf1route‐map<acl‐name> #route‐mapusedforredistribution

OSPFrouteso maximum‐paths<8>o maximum‐pathsibgp<8>o neighbor<neighboripaddres>remote‐as100090 #BGPpeero address‐familyipv4unicasto prefix‐listNO_SELFin #aclconfiguredtorestrict

prefiximport

63


o featureospf #enableospfforIPv4o featureospfv3 #enableospfforIPv6o routerospf<instance‐tag>o router‐id<ipaddress>o redistributebgp<as_no>route‐map<acl‐name> #route‐mapusedfor

redistributionforbgprouteso log‐adjacency‐changeso timersthrottlespf100200500o timersthrottlelsa50100300o auto‐costreference‐bandwidth1000000o default‐metric<1>

PIM‐SM:PIMSparseMode/PIMAnySourceMulticastisdeployedacrossthenetworkto

supportmulticast.Eachaggregation‐accessblockisconfiguredwiththeRPforthelocallysourcedgroups.

o featurepim #enablepimo ippimrp‐address<rp‐address>group‐list<multicast‐groups>#configurestatic

RPforamulticastgrouprangeo ippimsend‐rp‐announceloopback2prefix‐list<multicast‐groups>#configure

candidateauto‐rpo ippimsend‐rp‐discoveryloopback2 #configureauto‐rp

mapping‐agento ippimssmrange<> #configurepimssmfor

defaultrange o ippimauto‐rpforwardlisten #enableauto‐rpmessages

forwarding


o featuremsdp #enablemsdpo ipmsdporiginator‐id<interface> #configuresource

interfaceformsdppeering,generallyloopbackinterfaceo ipmsdppeer<ipaddress>connect‐source<interface> #configurepeer

address

vPC:ThevPCtechnologyisdeployedintheaggregation‐accessblockDC1‐Dist‐N7k‐101asshowninFigure1.Inaddition,dual‐sidedvPCisconfiguredbetweentheNexus7000andNexus5000switches

o featurevpc #enablevpco vpcdomain<domain‐id> #configurevpcdomain‐id

peer‐switch #enablepeer‐switchforfasterSTPconvergence

rolepriority200 #configurepriority peer‐keepalivedestination<ipaddress>source<ipaddress>vrfvpc‐

keepalive #configurekeep‐alivelink peer‐gateway #enablepeer‐gatewayto

avoidvPCloop track<id> #tracktheL3core

connectivitytoavoidblack‐hole iparpsynchronize #configurearp

synchronizeforfasterconvergenceofaddresstables

64

FP:FabricPathisdeployedintheaggregationblockDC1‐Dist‐N7k‐102.Thespinelayer

comprisesNexus7000switchesandtheleafswitchesaredeployedusingNexus5000switches.

o feature‐setfabricpath #configurefeature‐setfabricpath

o vlan<vlan‐range> modefabricpath #configurevlan‐rangein

fabricpatho fabricpathswitch‐id<switch‐id> #configureswitch‐ido vpcdomain<domain‐id> #configurevpcdomain‐id

fabricpathswitch‐id<vpc+_switch‐id> #configurevirtualswitchforpeerspresentonthenetwork

o interfaceport‐channel<po> #configurefabricpatinterface switchportmodefabricpath #configurefabricpath


accessblocks.Thespanningtreerootisplacedontheaggregationlevel.BPDUFilterandPortFastEdgeareconfiguredontheaccessportstowardsthehosts.

o interfaceport‐channel<po> #configureport‐channel switchport switchportaccessvlan<vlan> spanning‐treeporttypeedge #enablehost spanning‐treebpdufilterenable #configurebpdufilter

HSRP:HSRPisusedasthefirsthopgatewayprotocolforhosts.

o interfaceVlan<id> #configuresvio ipaccess‐group<acl>in #enableaccess‐listo ipaccess‐group<acl>outo noipredirectso ipaddress<ipaddress>o hsrpversion2o hsrp1

authenticationmd5key‐stringcisco #enableauthentication preemptdelayminimum200 priority200 ip<ipaddress> #HSRPIPaddress

FEX:FabricExtenders(Nexus2000)aredeployedonNexus7000

IGMP:IGMPisusedbyhoststojoinmulticastgroupsofinterest.IGMPsnoopingis

enabledonallswitchesintheaggregation‐accessblockstopreventfloodingofmulticastdatatraffic.

o ipigmpsnooping #bydefaultenabledonNexus

LACP:LACPisusedforlinkaggregationtoformport‐channelsacrossthenetwork.o featurelacp #enableLACP,bydefaultLACPis

usedonallport‐channel

UDLD:UDLDaggressivemodeisconfiguredacrossthenetworktodetectandpreventunidirectionallinks

o featureudld #enablefeatureudld

65

o udldaggressive #udldaggressivemodeisenabledtore‐establishtheconnectionwiththeneighbor

PVLAN:PrivateVLANconfiguredatDC101betweenNexus7000VPCpeersto: Nexus5000 CAT6500

FollowingPrivateVLANmodesconfigured: Promiscuous Isolated(host) Isolated(trunk) SinglePVLANassociationinaport‐channel(hostmode) MultiplePVLANassociationinaport‐channel(trunkmode) PVLANPromiscuousinhostmode PVLANPromiscuousintrunkmode

o featureprivate‐vlan #enablefeatureprivate‐vlano vlan<vlan‐id> #configureprimaryvlan

private‐vlanprimaryo vlan<vlan‐id>

private‐vlan<isolated/community> #configuresecondaryvlan private‐vlanassociation<vlan‐id> #configureassociationwith

primaryvlano interfaceport‐channel<port‐channel #configureport‐channel

switchport switchportmodeprivate‐vlantrunksecondary#PLANtrunkmode switchportprivate‐vlantrunkallowedvlan1 #configurenativevlan switchportprivate‐vlanassociationtrunk<primary><secondary>

o interfaceport‐channel<>

switchportswitchportmodeprivate‐vlanpromiscuous#PVLANPromiscuousswitchportprivate‐vlanmapping12011211‐1213#PVLANmappingvpc71#assignVPC

o interfaceport‐channel<>

switchportswitchportmodeprivate‐vlanhost#PVLANhostmodeswitchportprivate‐vlanhost‐association12011213#PVLANAssociationvpc81

6.4 ENT1 Configuration Guide Thefollowingconfigurationsareappliedtothetestnetwork:

Commonsystemcontrol,managementandaccounting:Commonsystemfeatureslike

SSH,TACACS+,Syslog,SNMP,NTP,SPAN,DNSandManagementVRFareconfigured.o featuretacacs+ #enablingthetacacsfeatureo tacacsserverhost<ipaddress>key<0/7> #configurethetacacsserverto


redundancy server<ipaddress>

66

use‐vrf<vrf_name> #use‐vrfbasedonserverreachability

o snmp‐serveruser<user‐name><group‐name>authmd5<pass‐phrase>priv<pass‐phrase>localizedkey #snmpv3userwithauthenticationenabled




o featurebgp #enablebgpo routerbgp<autonomous‐id> #bgpautonomous‐ido router‐id<router‐id>o graceful‐restartstalepath‐time<120>o log‐neighbor‐changeso address‐familyipv4unicasto redistributedirectroute‐map<acl‐name> #route‐mapusedforredistribution

directlyconnectedsubnetso redistributeospf1route‐map<acl‐name> #route‐mapusedforredistribution

OSPFrouteso maximum‐paths<8>o maximum‐pathsibgp<8>o neighbor<neighboripaddres>remote‐as100090 #BGPpeero address‐familyipv4unicasto prefix‐listNO_SELFin #aclconfiguredtorestrict

prefiximport


o featureospf #enableospfforIPv4o featureospfv3 #enableospfforIPv6o routerospf<instance‐tag>o router‐id<ipaddress>o redistributebgp<as_no>route‐map<acl‐name> #route‐mapusedfor

redistributionforbgprouteso log‐adjacency‐changeso timersthrottlespf100200500o timersthrottlelsa50100300o auto‐costreference‐bandwidth1000000o default‐metric<1>

PIM‐SM:PIMSparseMode/PIMAnySourceMulticastisdeployedacrossthenetworkto

supportmulticast.Eachaggregation‐accessblockisconfiguredwiththeRPforthelocallysourcedgroups.

o featurepim #enablepimo ippimrp‐address<rp‐address>group‐list<multicast‐groups>#configurestatic

RPforamulticastgrouprangeo ippimsend‐rp‐announceloopback2prefix‐list<multicast‐groups>#configure

candidateauto‐rpo ippimsend‐rp‐discoveryloopback2 #configureauto‐rp

mapping‐agent

67

o ippimssmrange<> #configurepimssmfordefaultrange

o ippimauto‐rpforwardlisten #enableauto‐rpmessagesforwarding


o featuremsdp #enablemsdpo ipmsdporiginator‐id<interface> #configuresource

interfaceformsdppeering,generallyloopbackinterfaceo ipmsdppeer<ipaddress>connect‐source<interface> #configurepeer

address

vPC:vPCtechnologyisdeployedintheaggregation‐accessblockDC2‐Dist‐N7k‐201.Inaddition,dual‐sidedvPCisconfiguredbetweentheNexus7000andNexus5000switches.




keepalive #configurekeep‐alivelink

peer‐gateway #enablepeer‐gatewaytoavoidvPCloop

track<id> #tracktheL3coreconnectivitytoavoidblack‐hole

iparpsynchronize #configurearpsynchronizeforfasterconvergenceofaddresstables


accessblockDC‐Dist‐N7K‐201.MSTPisenabledonDC‐Dist‐N7K‐202forthesamepurposewhereverapplicable.Thespanningtreerootisplacedontheaggregationlevel.BPDUFilterandPortFastEdgeareconfiguredontheaccessportstowardshosts.


SNMP:SNMPtrapsareenabledandSNMPscriptsareusedtocollectsysteminformation

andtomonitorpotentialmemoryleaks. HSRP:HSRPisusedasthefirsthopgatewayprotocolforhosts.

o interfaceVlan<id> #configuresvio ipaccess‐group<acl>in #enableaccess‐listo ipaccess‐group<acl>outo noipredirectso ipaddress<ipaddress>o hsrpversion2o hsrp1

authenticationmd5key‐stringcisco #enableauthentication

68

preemptdelayminimum200 priority<priority> ip<ipaddress> #HSRPIPaddress

FEX:MultipletypesofFabricExtendersaredeployedonNexus5000parentswitches. IGMP:IGMPisusedbyhoststojoinmulticastgroupsofinterest.IGMPsnoopingis

enabledonallswitchesintheaggregation‐accessblockstopreventfloodingofmulticastdatatraffic.

o ipigmpsnooping #bydefaultenabledonNexus

LACP:LACPisusedforlinkaggregationtoformport‐channelsacrossthenetwork.o featurelacp #enableLACP,bydefaultLACPis

usedonallport‐channel

UDLD:UDLDaggressivemodeisconfiguredacrossthenetworktodetectandpreventunidirectionallinks

o featureudld #enablefeatureudldo udldaggressive #udldaggressivemodeisenabled

tore‐establishtheconnectionwiththeneighbor

RouteMAPforInter‐VRFPBR

o featurepbr #enablefeaturepbrroute‐mapGLOBAL‐to‐VRFpermit10 #defineroute‐map

matchipaddressPBR‐GLOBAL‐VRF#matchACLfortheroute‐map setvrfA#defineVRF

ipaccess‐listPBR‐VRF‐GLOBAL#defineACLfortheroute‐map 10permitip151.15.1.0/24any interfaceVlan1501#CreateSVIinterface vrfmemberA noipredirects ipaddress151.15.1.152/24 noipv6redirects iprouterospf1area0.0.0.151 ippimsparse‐mode ippimdr‐priority100 ippolicyroute‐mapVRF‐to‐global hsrpversion2 hsrp1501 authenticationtexteCATS priority100forwarding‐thresholdlower1upper100 timers13 ip151.15.1.1 ipdhcprelayaddress172.28.92.48 ipdhcprelayaddress172.28.92.49 noshutdown mtu9000

6.5 M1 vPC Scale Configuration Guideline Thefollowingconfigurationsareappliedtothetestnetwork:

69

Commonsystemcontrol,managementandaccounting:Commonsystemfeatureslike

SSH,Syslog,SNMP,NTPandManagementVRFareconfigured.o snmp‐serveruser<user‐name><group‐name>authmd5<pass‐phrase>priv




vPC:vPCtechnologyisdeployedinthenetworkbetweentheN7kandtheCatalystVSS

switchesasshowninthefigure3.




keepalive #configurekeep‐alivelink

peer‐gateway #enablepeer‐gatewaytoavoidvPCloop

iparpsynchronize #configurearpsynchronizeforfasterconvergenceofaddresstables


accessblocks.Thespanningtreerootisplacedontheaggregationlevel.RootGuardisconfiguredontheaggregationleveltoenforcerootplacement.BPDUFilter,BPDUGuardandPortFastEdgeareconfiguredontheaccessportstowardshosts.


LACP:LACPisusedforlinkaggregationtoformport‐channelsacrossthenetwork

o featurelacp #enableLACP,bydefaultLACPisusedonallport‐channel

PVLAN:PVLANisconfiguredinthenetworkandisthemainfocusoftesting.The

followingPVLANcomponentsarecoveredinthenetwork:o PVLANprimaryandsecondaryVLAN(Isolated)o SecondaryTrunkandpromiscuoustrunkonvPCo PrivateVLANpromiscuousaccessandPrivatevlanhostonclassisport‐channel.

70

o PrivateVLANpromiscuoustrunk,secondarytrunkonclassicPort‐channel

o featureprivate‐vlan #enablefeatureprivate‐vlano vlan<vlan‐id> #configureprimaryvlan

private‐vlanprimaryo vlan<vlan‐id>

private‐vlan<isolated/community> #configuresecondaryvlan private‐vlanassociation<vlan‐id> #configureassociationwith

primaryvlano interfaceport‐channel<port‐channel> #configureport‐channel

switchport switchportmodeprivate‐vlantrunksecondary switchportprivate‐vlantrunkallowedvlan1 #configurenativevlan switchportprivate‐vlanassociationtrunk<primary><secondary>

o interfaceport‐channel<port‐channel> #configureport‐channel switchport switchportmodeprivate‐vlantrunkpromiscuous switchportprivate‐vlanmappingtrunk<primarysecondary1,

secondary2,…> #configureprimarytosecondarymapping

o interfaceport‐channel<port‐channel> #configureport‐channel switchport switchportmodeprivate‐vlanpromiscuous #configurepromiscuous

access switchportprivate‐vlanmapping<primarysecondary>

#configureprimarytosecondarymappingo interfaceport‐channel<port‐channel> #configureport‐channel

switchport switchportmodeprivate‐vlanhost #configurehost

switchportprivate‐vlanhost‐association<primarysecondary> #configureprimarytosecondaryhost‐association

nexus validation test phase 4 - cisco · 4 service provider mpls: the topology validates the inter...

Documents