unified mpls: advanced scaling for core and edge...

© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public

Unified MPLS: Advanced Scaling for Core and Edge Networks BRKSPG-2405

1

Rajiv Asati Distinguished Engineer


Abstract

Service Providers (SPs) are striving towards becoming 'Experience Providers' while offering many residential and/or commercial services. Many SPs have to build an agile Next Gen Networks (NGN) that can optimally deliver the 'Any Play' promise. However, as the Networks continue to get are getting bigger, fatter and richer, some of the conventional wisdom of designing IP/MPLS networks is no longer sufficient. This session introduces a 'Cisco Validated Design' for building Next-Gen Networks' Core and Edge. It briefly discusses the technologies integral to such a design and focus on their implementation using IOS-XR platforms (CRS-1/3 and ASR 9000). The session looks at the scaling designs and properties of IP, MPLS, the IGP and BGP as well as the protection mechanisms IP/LDP FRR and MPLS-TE FRR.

This session is intended to cover - - Unicast routing + MPLS design - Fast Restoration - Topology Dependency - Test Results - Case Study


Agenda

Introduction

Solution Overview ‒ Unicast Routing + MPLS Design

‒ Fast Restoration

‒ Topology Dependency

‒ Test Results

Case Study

Conclusion

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSPG-2405 4

Introduction




‒ Results

Case Study

Conclusion

Agenda


Introduction Trend Networks becoming larger

‒ Quad-play (Video, Voice, Data & Mobility) ‒ Merger & Acquisition ‒ Growth

Exponential bandwidth consumption ‒ Business Services ‒ Mobile

MPLS in the Access ‒ Seamless MPLS ‒ MPLS-TP

BGP ASN consolidation ‒ Single ASN offering to customers


Introduction NGN Requirements Large Network

‒ 2000+ routers, say

Multi-Play Services Anywhere in network ‒ Service Instantiation happens anywhere

End-to-End Visibility ‒ v4/v6 Uni/Multicast based Services

Fast Convergence or Restoration ‒ Closer to Zero loss, the better.

Scale & Performance


Introduction




‒ Results

Case Study

Conclusion

Agenda


Solution Overview

Unicast Routing + MPLS - Divide & Conquer 1. Isolate IGP domains

2. Connect IGP domains using BGP

Fast Restoration – Leverage FRR 1. IP FRR (IGP LFA & BGP PIC)

2. MPLS FRR (LDP FRR & TE FRR)

Topological Consideration – Choose it right 1. PoP Design

2. ECMP vs. Link-Bundling

Services – Scale


Introduction




‒ Results

Case Study

Conclusion

Agenda


Routing + MPLS Design Must Provide…. PE-to-PE Routes (and Label Switched Paths)

‒ PE needs /32 routes to other PEs

‒ PE placement shouldn’t matter

Single BGP ASN

Backbone

Aggregation

.

Access Region 2

.

PE31

R

PE21

Access

.

Region1

.

Aggregation

PE11

PE21

LSP


Routing + MPLS Design Conventional Wisdom Says… Advertise infrastructure (e.g. PE) routes in IGP

Advertise infrastructure (e.g. PE) labels in LDP

Segment IGP domains (i.e. ISIS L1/L2 or OSPF Areas)

Aggregation

.

Access Region 2

.

PE31

R

PE21

Access .

Region1

.

Aggregation

PE11

PE21

Backbone

ISIS Level 2 Or

OSPF Area 0

ISIS Level 1 Or

OSPF Area X

ISIS Level 1 Or

OSPF Area Y

BGP for Services End-to-End IGP & LDP for Infrastructure


Routing + MPLS Design Conventional Wisdom Not Good Enough… Large IGP database size a concern

‒ For fast(er) convergence

Large IGP domain a concern ‒ For Network Stability.

Large LDP database a concern


Routing + MPLS Design ‘Divide & Conquer’ – Game Plan

Disconnect & Isolate IGP domains ‒ No more end-to-end IGP view

Leverage BGP for infrastructure (i.e. PE) routes ‒ Also for infrastructure (i.e. PE) labels

Backbone Aggregation

.

Access Region 2

.

PE31

R

PE21

Access .

Region1

.

Aggregation

PE11

PE21

ISIS Level 2 Or

OSPF Area 0

ISIS Level 1 Or

OSPF Area X

ISIS Level 1 Or

OSPF Area Y

Isolated IGP & LDP Isolated IGP & LDP Isolated IGP & LDP BGP for Infrastructure

BGP for Services


Routing + MPLS Design Divide & Conquer – End Result

Example - ‘PE31’ Reachability

Control Plane Flow – RIB/FIB Table View

Data Plane Flow – PE11 to PE31 Traffic View


.

Access Region 2

.

R

PE21

Access .

Region1

.

Aggregation

PE21

ISIS Level 2 Or

OSPF Area 0

ISIS Level 1 Or

OSPF Area X

ISIS Level 1 Or

OSPF Area Y

PE31 :: Next-Hop = P1; BGP; Label = L100; BGP P1 :: Next-Hop = P11; IGP Label = L200; LDP

PE31 :: Next-Hop = P2; BGP Label = L101; BGP P2:: Next-Hop = P100; IGP Label = L201; LDP

PE11

P1

P11

P2

PE31 :: Next-Hop = P31; IGP Label = L110; LDP

PE31

IP L200 L100

IP L110

IP IP L201 L101

P100


Routing + MPLS Design Divide & Conquer – Summary 1. IGP is restricted to carry only internal routes

‒ Non-zero or L1 area carries only routes for that area

‒ Backbone carries only backbone routes *

2. PE redistributes its loopback into IGP as well as iBGP+Label 3. PE peers with its local ABRs using iBGP

‒ ABRs act as Route-reflectors

‒ ABRs reflect _only_ Infrastructure (i.e. PE) routes

4. ABR, as RR, changes the BGP Next-hop to itself ‒ On every BGP advertised routes

5. PEs separately peer for Services (VPN, say)

* ISIS L1->L2 (or L1->L1) Redistribution Cannot Be Avoided Yet, but OSPF Non-Zero<->Zero Area Redistribution Can Be.


Routing + MPLS Design Divide & Conquer

1. IGP is restricted to carry only the internal routes ‒ Non-zero or L1 area carries only routes for that area

‒ Backbone carries only backbone routes *


.

Access Region 2

.

PE31

R

PE21

Access .

Region1

.

Aggregation

PE11

PE21

ISIS Level 2 Or

OSPF Area 0

ISIS Level 1 Or

OSPF Area X

ISIS Level 1 Or

OSPF Area Y

1

* Unlike OSPF, ISIS Backbone Would Carry Both L1 and L2 Routes Since L1->L2 (or L1->L1) Redistribution Cannot Be Avoided Yet

ABR ABR

Isolated IGP Isolated IGP Isolated IGP



1. PE redistributes its loopback into IGP as well as iBGP+Label


.

Access Region 2

.

PE31

R

PE21

Access .

Region1

.

Aggregation

PE11

PE21

ISIS Level 2 Or

OSPF Area 0

ISIS Level 1 Or

OSPF Area X

ISIS Level 1 Or

OSPF Area Y

2

Loopback Int Redistributed into IGP and BGP+Label

ABR ABR



1. PE peers with its local ABRs using iBGP+label ‒ ABRs act as Route-reflectors

‒ ABRs reflect _only_ Infrastructure (i.e. PE) routes

‒ RRs also in the backbone


.

Access Region 2

.

PE31

R

PE21

Access .

Region1

.

Aggregation

PE11

PE21

ISIS Level 2 Or

OSPF Area 0

ISIS Level 1 Or

OSPF Area X

ISIS Level 1 Or

OSPF Area Y

iBGP+Label Peering

3

ABR ABR



1. ABR, as RR, changes the BGP Next-hop to itself ‒ On each BGP advertised routes


.

Access Region 2

.

PE31

R

PE21

Access .

Region1

.

Aggregation

PE11

PE21

ISIS Level 2 Or

OSPF Area 0

ISIS Level 1 Or

OSPF Area X

ISIS Level 1 Or

OSPF Area Y

ABR Sets BGP NH to Itself ABR Sets BGP NH to Itself

4

ABR ABR

BGP Prefix PE31: Next-Hop = P1; Label=L100

BGP Prefix PE31: Next-Hop = P2; Label=L101

BGP Prefix PE31: Next-Hop = PE31; Label=Null

P1 P2



1. PEs separately peer using iBGP for Services ‒ Dedicated RRs for IPv4/6, VPNv4/6, L2VPN, etc.

‒ More Details on BGP Scale for Services Later…


.

Access Region 2

.

PE31

R

PE21

Access .

Region1

.

Aggregation

PE11

PE21

ISIS Level 2 Or

OSPF Area 0 . .

5

ABR ABR

RRs RRs RRs


Routing + MPLS Design Divide & Conquer – End Result

Example - ‘L3VPN Services’

PE11 sends L3VPN traffic for an L3VPN prefix “A” to PE31


.

Access Region 2

.

R P31

Access .

Region1

.

Aggregation

PE21

ISIS Level 2 Or

OSPF Area 0

ISIS Level 1 Or

OSPF Area X

ISIS Level 1 Or

OSPF Area Y

PE31 :: Next-Hop = P2; BGP Label = L101; BGP P2:: Next-Hop = P100; IGP Label = L201; LDP

PE11

P11

P2

PE31 :: Next-Hop = P31; IGP Label = L110; LDP

PE31

IP L200 L100

IP L30

IP IP L101 L30

P100

L30

P1

L3VPN “A”:: next-Hop = CE31; IGP Label = Unlabel

IP

IP L100 L30

L201 L110

L30

PE31 :: next-hop = P1; BGP; label = L100; BGP P1 :: Next-hop = P11; IGP label = L200; LDP

L3VPN “A” Next-Hop = PE31; BGP Label = L30 ; BGP PE31 :: Next-Hop = P1; BGP; Label = L100; BGP P1 :: Next-Hop = P11; IGP Label = L200; LDP


Routing + MPLS Design Take-Away

Higher Network scale is attainable ‒ 1000s of routers

BGP and MPLS Label Stacking are key


Introduction




‒ Results

Case Study

Conclusion

Agenda


Fast Restoration

Business Services demanding faster restoration ‒ Against link or node failures

“Service Differentiator” for many operators

Faster Restoration is driving towards 0 loss ‒ ~50ms restoration may be good enough for many

‒ Requirements influence Complexity and Cost

Fast Restoration is optimal with “Local Protection” ‒ pre-compute and pre-install alternate path

‒ no need for remote nodes to know about the failure


Fast Restoration

Fast Restoration of Services i.e. BGP Prefixes ‒ BGP Prefix Independent Convergence (PIC)

Fast Restoration of BGP next-hops i.e. IGP Prefixes ‒ IP FRR (LFA) with LDP FRR (or RSVP-TE FRR)

Fast Convergence (FC) of IP routing protocols is key and still required


Fast Restoration vs. Fast Convergence

Detection (link or node aliveness, routing updates

received) State propagation

(routing updates send)

Walkthrough routing

DB’s Compute primary path & label Download

to HW FIB

Switch to newer path


Fast Restoration vs. Fast Convergence

Detection (link or node aliveness, routing updates

received) State propagation

(routing updates send)

Walkthrough routing

DB’s Compute primary path & label Download

to HW FIB

Switch to newer path

Switch to Repair Path

Pre-Compute Repair path Download

to HW FIB

Offline Calculation


Remember that FRR is intended for temporary restoration Fast Convergence (FC) is key for IP routing protocols Faster the routing convergence, faster the permanent

restoration ‒ <1sec restoration is possible

Routing convergence happens at the process level, hence, depends on the platform processor ‒ Restoration time can not be guaranteed

Edge FC

Edge FRR

POP FRR

Core FRR

POP FC Core FC

Fast Convergence IGP Prefixes


Detect Link/node down event as fast as possible ‒ BFD, Layer2 protocol keep-alives, Alarms, IGP fast hellos, Proactive

Protection

Generate the link state event—LSP/LSA generation is optimized Propagate the changes in the network as soon as possible—

Flooding and passing is optimized Recalculate the paths (run SPF) as soon as possible—Support

of incremental SPF and optimized for full SPF Install the new routes in the routing/forwarding table with Prefix

Prioritization CRITICAL: IPTV SSM sources

HIGH: Most Important PE’s

MEDIUM: All other PE’s

LOW: All other prefixes

Fast Convergence IGP Prefixes

MUST for

FRR

MUST for FC

Edge FC

Edge FRR

POP FRR

Core FRR

POP FC Core FC


Fast Convergence IGP Tuning for FC

OSPF Event Propagation

‒ timers pacing flood value

‒ timers pacing retransmission value

‒ default values are 33 msec/66 msec

OSPF Subsecond Hellos Configuration:

‒ ipospf dead-interval minimal hello-multiplier value

‒ Value—range 3–20

OSPF LSA Generation Exponential Backoff

timers throttle lsa all lsa-start lsa-hold lsa-max

‒ timers lsa arrival timer

OSPF SPF ExponentialBackoff

‒ Timers throttle spfspf-start spf-hold spf-max

‒ All LSA/SPF values are in ms

• IS-IS hello interval/ Hello Multiplier

isis hello-interval { seconds | minimal }

isis hello-multiplier value ------- Value—range 3–20

• IS-IS LSP-Generation Exponential Backoff

lsp-gen-interval lsp-max lsp-start lsp-hold

lsp-max—(sec) lsp-hold—(msec) lsp-start—(msec)

• IS-IS Event Propagation lsp-interval value

Default rate - one LSP every 33 ms

• Fast LSP Flooding fast-flood lsp-number (Previously ip fast-

convergence)

• IS-IS SPF Exponential Backoff spf-interval spf-max spf-start spf-hold

<spf-max>- (sec) <spf-start> - (msec) <spf-hold> - (msec)

prc-interval prc-max prc-start prc-hold <prc-max>- (sec) <prc-start> - (msec) <prc-hold> - (msec)

OSPF Tuning IS-IS Tuning

Note: MinLSArrival Must Be <= lsa-Hold

Edge FC

Edge FRR

POP FRR

Core FRR

POP FC Core FC

Reference


Fast Restoration IGP Prefixes MPLS FRR and IP FRR are viable options

‒ Both pre-compute and pre-install alternate path

IP FRR (LFA) is simpler than RSVP-TE based MPLS FRR ‒ Easy to configure and manage ‒ Does not require network-wide support ‒ Has topological dependencies

IP FRR (LFA), with LDP LSP, provides simpler MPLS FRR ‒ Easy to configure and manage ‒ Does not require network-wide support ‒ Removes most of topological dependencies

Use IP FRR & LDP FRR (RSVP-TE FRR only if one have to)

‒ RSVP-TE for bandwidth engineering as usual

FRR – Fast Reroute LFA – Loop Free Alternates LSP – Label Switched Path


IP/LDP FRR: Apply it as an intra PoP and inter PoP FRR solution

RSVP-TE FRR: Apply it as an inter PoP FRR solution, if IP/LDP FRR doesn’t give enough coverage

PoP

PoP

PoP

PoP

PE

P

P

PoP

Edge FC

Edge FRR

POP FRR

Core FRR

POP FC Core FC Fast Restoration IGP Prefixes

Intra PoP Inter PoP


Fast Restoration IGP – IP FRR

IP FRR (Loop Free Alternates) provides a pre-computed backup (aka repair path) per destination prefix

IP FRR (LFA) can be deployed in two ways : ‒ Per-Link LFA – Protects all the destinations reachable via the

protected link

‒ Per-Prefix LFA – Protects a destination against the next-hop link or node failure

IP FRR (LFA) well applies to most SP topologies ‒ http://tools.ietf.org/html/draft-ietf-rtgwg-lfa-applicability-00

Note: SPF calculations for LFAs are performed in background and pre-empted in case of any convergence events

Reference


A backup path for all prefixes reachable via next-hop node (F) over the protected link (S-F) ‒ 1 SPF per protected link

No node protection possible

Sub-optimal forwarding during FRR

Fast Restoration – IGP IP FRR : Per-Link LFA

S F

D

Primary link Backup link

Route D Primary Next Hop: F Backup Next Hop: R1

R1

Protecting Node Next-hop Node


S F

R1

D

Route D NH: F, LFA: R1

Route D NH:F

R2

Edge FC

Edge FRR

POP FRR

Core FRR

POP FC Core FC

R3

Availability of the backup NH is dependent on the topology and link metric assignments

All depends on metric assignment

10

10

10

10

10 10

Route D NH: F

LFA: no

Route D NH: S

RP/0/0/CPU0:ospf-3-2(config)#router ospf 1 RP/0/0/CPU0:ospf-3-2(config-ospf)#area 0 RP/0/0/CPU0:ospf-3-2(config-ospf-ar)#int pos 0/3/0/0 RP/0/0/CPU0:ospf-3-2(config-ospf-ar-if)#fast-reroute per-link enable

Fast Restoration – IGP IP FRR : Per-Link LFA


A backup path for a prefix (e.g. D) reachable via next-hop node (F) ‒ 1 SPF per neighbor

No node protection possible

Sub-optimal forwarding during FRR

Fast Restoration – IGP IP FRR : Per-Prefix LFA

S F

D

Backup path1 (link protection) Backup path2 (node protection)

Route D Primary Next Hop: F Backup Next Hop: R1

R1

Protecting Node Next-hop Node


By default, LFA computation is disabled

LFA needs to be enabled only on protecting router

Fast Restoration – IGP IP FRR : Per-Prefix LFA (Configuration)

! router isis fast-reroute per-prefix {level-1 | level-2} {all | route-map <route-map-name>} ! router ospf 1 fast-reroute per-prefix enable prefix-priority low !

S

router isis <instance-id> interface <type> <instance> address-family ipv4 [unicast] fast-reroute per-prefix level <1|2>

IOS

IOS-XR


10.0.0.0/8

20.0.0.0/8


IGP pre-computes a backup path per IGP prefix FIB pre-installs the backup path in dataplane

2

6 5

5 1

1

2

4

6

10.0.0.0/8, NH = D, cost= 10 20.0.0.0/8, NH = D, cost= 7

A

F

B

D E

C


10.0.0.0/8

20.0.0.0/8


10.0.0.0/8, NH = D, cost= 10 20.0.0.0/8, NH = D, cost= 7

10.0.0.0/8, NH = C, cost=11 20.0.0.0/8, NH = A, cost=9

10.0.0.0/8, NH = A, cost=14 20.0.0.0/8, NH = direct, cost=6

• IGP pre-computes a backup path per IGP prefix • FIB pre-installs the backup path in dataplane

A

F

B

D E

C

2

6 5

5 1

1

2

4

6


10.0.0.0/8

20.0.0.0/8


10.0.0.0/8, NH = D, cost= 10 20.0.0.0/8, NH = D, cost= 7 10.0.0.0/8, NH = D, cost=10 –

LFA: B 20.0.0.0/8, NH = D, cost=7 – LFA: F

• IGP pre-computes a backup path per IGP prefix • FIB pre-installs the backup path in dataplane

A

F

B

D E

C

2

6 5

5 1

1

2

4

6


Fast Restoration – IGP LFA with LDP

The link between A and B failed.

A sends packets to C instead by swapping labelA with labelC distributed by C.

LDP requirement: Downstream Unsolicited; Liberal Retention

The backup path for destination P/p must contain the label bound by the backup neighbor

This is why, whether the IGP computes per-prefix or per-link, the RIB and FIB representation is always per-prefix ‒ this allows to store the per-path dependent backup label

Protecting Node

Link Failure

A

C

B

Primary Path Repair Path

P/p packet labelA

packet labelB


R2

R4

R6 R7

R5

R3

R1

Access Region

Backbone

Fast Restoration – IGP Remote LFA (aka PQ) Any node which meets the P

and Q properties

‒ P: the set of nodes reachable from R2 without traversing [R2-R4]

‒ Q: the set of nodes which can reach R4 without traversing [R2-R4]

Best PQ node

‒ The closest from R2: R5

Establish a directed LDP session with the selected PQ node

Backbone


Fast Restoration – IGP Remote LFA (aka PQ) R2’s LIB

‒ R4’s label for FEC R6 = 408



R2’s FIB for destination R6 ‒ Primary: out-label = 408, oif = R4

‒ Backup: out-label = 502

oif = [push 103, oif = R1] R2

R4

R6 R7

R5

R3

R1

Access Region

Backbone

103

408

502


Fast Restoration RSVP-TE FRR

RSVP-TE FRR link protection (and prefix independent): <50ms

Easy to operate with auto-tunnel

RSVP-TE FRR node protection (and prefix independent):

<100ms (depends on time to detect the node failure)

RSVP-TE FRR path protection (and prefix independent):

Time depends of time to signal the path error to the head end (not a local mechanism)

Challenging to operate (due to due to its end-end / 1:1 protection)

Appropriated to specific scenario

Edge FC

Edge FRR

POP FRR

Core FRR

POP FC Core FC

Note: RSVP-TE Provides FRR Mechanism as well as: • Bandwidth Management • Traffic Engineering • Is not Topology Dependent like IP LFA


Fast Restoration RSVP-TE FRR – Link Protection

Router C

Router D Router A Router B Router E

interface Tunnel0 tunnel destination Router D … explicit-path R2-R3-R4 notunnel mpls traffic-eng autoroute announce

interface POS0/0 mpls traffic-eng backup-path Tunnel0

interface Tunnel0 tunnel destination Router E .. etc ... tunnel mpls traffic-eng fast-reroute

x

Edge FC

Edge FRR

POP FRR

Core FRR

POP FC Core FC


What if Router D failed?

Link protection would not help as the backup tunnel terminates on Router D (which is the NHop of the protected link)

Protect tunnel to the next hop PAST the protected link (NNhop)

Router D

Router C

Router A Router B Router E

Fast ReRoute Backup Tunnel

NHop

Protected Link

Router F

Fast ReRoute Backup Tunnel

NNHop

Edge FC

Edge FRR

POP FRR

Core FRR

POP FC Core FC Fast Restoration RSVP-TE FRR – Node Protection


IP FRR RSVP-TE/MPLS FRR

1 Repair Path Least Cost

Constraints Based with Bandwidth Guarantee and Path Control

2 SRLG Capable Capable 3 Link Protection Capable Capable 4 Node Protection Capable Capable 5 Path Protection Not Capable Capable 6 Control Plane Requirement None with LFA RSVP-TE

7 Load Distribution over Multiple Repair Paths

Capable Not Capable

8 Provisioning Complexity Minimal, If Any Significant 9 Topology Dependency Yes No

Fast Restoration IP FRR (LFA) vs. RSVP-TE FRR


Fast Restoration BGP PIC (Prefix Independent Convergence)

Edge FC

Edge FRR

POP FRR

Core FRR

POP FC Core FC


What Is PIC or BGP FRR?

PIC provides a fast convergence functionality upon failure to cutover to any backup path within sub-seconds independent of the number of prefixes

BGP Fast Reroute (BGP FRR)—enables BGP to use alternate paths within sub-seconds after a failure of the primary or active paths

PIC or FRR dependent routing protocols (e.g. BGP) install backup paths

Without backup paths

‒ Convergence is driven from the routing protocols updating the RIB and FIB one prefix at a time - Convergence times directly proportional to the number of affected prefixes

With backup paths

‒ Paths in RIB/FIB available for immediate use

‒ Predictable and constant convergence time independent of number of prefixes


Site2

Site1

1

3

1. PIC Core – When IGP Path Changes 2. PIC Edge – When Remote PE Node (or Its Reachability)

Fails 3. PIC Edge – When PE-CE Link Fails

PIC Edge vs. PIC Core

PE3

PE1

PE2 2

CE1 CE2

PIC Core CLI on 7600 - cef table output-chain build favor convergence-speed


BGP PIC Edge PE-CE Link Protection

PE1 and PE2 pre-compute bgp backup paths using bgp best-external approach

When primary link PE1 - CE1 fails: ‒PE1 holds on to the bgp local labels and re-routes CE1’s traffic to PE2 using labels advertised by PE2

‒ Uses fixed timer to clean up stale local labels

‒ PE3 is expected to converge and start using PE2’s label to send traffic to CE1

CE2

PE1

PE2

CE1 PE3

MPLS-VPN

Edge FC

Edge FRR

POP FRR

Core FRR

POP FC Core FC

Normal Path Backup Path

router bgp 100 address-family ipv4 vrf V1 bgpadvertise-best-external

router bgp 100 address-family ipv4 vrf V1 bgp additional-paths install


PE1, PE2 and PE3 precomputes bgp backup

When node PE1 fails: ‒ IGP notification on PE3 invalidates active path

‒ Switches to backup path

‒ PE3 is expected to converge to start using PE2’s label to send traffic toCE1

CE2

PE1

PE2

CE1 PE3

MPLS-VPN

Edge FC

Edge FRR

POP FRR

Core FRR

POP FC Core FC

Normal Path Backup Path

BGP PIC Edge PE-Node Protection


Convergence With and Without PIC BGP PIC Core and PIC Edge

Without PIC : Convergence is a function of number of affected prefixes during failure

With PIC : Convergence is predictable and remains constant independent of the number of prefixes

Core

1

10

100

1000

10000

100000

1

2500

0

5000

0

7500

0

1000

00

1250

00

1500

00

1750

00

2000

00

2250

00

2500

00

2750

00

3000

00

3250

00

3500

00

Prefix

Lo

C (

ms) PIC

no PIC

1

10

100

1000

10000

100000

1000000

0

5000

0

1000

00

1500

00

2000

00

2500

00

3000

00

3500

00

4000

00

4500

00

5000

00

Prefix

msec

250k PIC

250k no PIC

500k PIC

500k no PIC

PIC Core PIC Edge


Fast Restoration Design Take-Away

Leverage IP FRR (LFA) with MPLS / LDP wherever possible ‒ LFA is simpler, local (requires no

interoperability)

Leverage TE FRR, if we must have to. ‒ Bandwidth Engineering, for example.

Leverage BGP PIC for faster BGP convergence ‒ PIC is local (requires no interoperability)


Introduction



‒ Topology Consideration

‒ Results

Case Study

Conclusion

Agenda


Topological Consideration

What topology is chosen makes a big difference ‒ convergence, traffic engineering, capacity planning, routing table, stability..

Topological Options may vary ‒ Flat vs. Hierarchical

‒ Hub & Spoke vs. Ring (Square)

Also, the evergreen question about ECMP vs. LAG


Flat PoP Topology LFA Applicability

Reference


Hierarchical PoP Topology LFA Applicability

Reference


Topological Consideration ECMP vs. Link-Bundle

Factors E-LinkBundle

ECMP

1 Member Links’ Speed Must Be Same Yes No

2 Member Links on Any LC Yes Yes

3 Routing Adjacency One Many

4 Routing Table Impact No Yes

5 Max Number of Member Links 64 16 (32*)

6 Line-Rate Multicast (Members on Any LCs) Yes? Yes

7 Port Failure Localized to the Router Yes No

8 BFD on Each Member Link Yes** Yes

9 Video Monitoring – Better Accounting No Yes

10 Non-Stop Routing (NSR), Forwarding (NSF) Yes Yes


Topological Consideration Take-Away

Triangle topology (i.e. Hub & Spoke) for PE connectivity is advantageous ‒ Naturally benefits from IP FRR

Linkbundling gaining more traction


Introduction




‒ Test Results

Case Study

Conclusion

Agenda


Results

The solution discussed here is a part of a complete end-to-end architecture for delivery of residential, business, and RAN backhaul services ‒ It is thoroughly validated for each service in the areas of:

- Functionality, Scalability, Performance / SLA, QoS, High Availability, Network Management, OAM

The solution (results) is well documented ‒ Design & Implementation Guide (DIG) available through your

account team


ASR-9000

CRS-1

7600

ASR-1000

3400E / 4948 MWR-2941 / ISR

Internet Video Headend/DC SEF Infrastructure

PoP A Hub & Spoke Aggregation Topology

PoP B Ring Aggregation Topology

PoP C Business MSE (Ethernet + TDM)

10GE 1GE

NGN Testbed


NGN Testbed – Platforms

Role Platform Version Aggregation Node

ASR-9000 IOS-XR 4.0.1

Core Node CRS-1/3 IOS-XR 4.0.1

Access Node ME-3400E 12.2(55)SE

Access Node 4500, 4948 15.0(2)SG

Access Node MWR-2941-A CSR 3.3

Service Edge Node

ASR-1000, Cisco 7600

15.1(1)S

Active Network Ab t ti

3.7.2


Test Area Results

1 Topology Scalability PoPs – 100 (3 Real + 97 Simulated) Infrastructure BGP Routes – 100K; Infrastructure ISIS Routes – 12K;

2 Service Scalability Residential – 120K Triple Play Subscribers; Business L2VPN – 16K E-Line, 4K E-LAN (20K MACs); Business L3VPN – 4K VPNs (1M Routes); IP RAN/ TDM – 4K AToM PWE3;

3 Service (High) Availability Link & Node Failure and Recovery: <50 msec (Hub & Spoke Topology) <500 msec (Ring Topology)

Results – Summary

* A Few Exceptions During Node Recovery with High Scale


Results (Just an Example) Service Convergence During Link Failure

0

100

200

300

400

500

600

700

800

NNI Failure (H&S)

NNI Failure (ring)

UNI Failure (Ethernet)

UNI Failure (uWave)

50

500

200

750

RAN Backhaul Service

RAN Backhaul Service

msec


Introduction




‒ Results

Case Study

Conclusion

Agenda


Case Study

SPs are fast embracing Cisco NGN reference

The next two slides illustrate the actual deployed networks -


Case Study #1 APAC Mobile Operator / SP

PE PE

U-PE U-PE

Star

Regional Data Center

CSR CSR

mx40GE

SR SR

CR CR

PE PE

U-PE U-PE

CSR CSR

CR CR

mx40GE

nx10GE nx10GE

Mini-Core

Aggregation

Access

CSR - Cell Site Router SR – Service Router CR – Core Router BR – Backbone Router

300 Mbps per CSR (Radio)

9 Gbps per U-PE <10 CSR (Radios) per U-PE

3x40GE’s per SR Pair 378Gbps per SR Pair

Backbone


Tier 1 Hub

Tier 2 Hub

SDC1 SDC 2

7600/ASR9k

Backbone

Distribution

Aggregation

Hub Agg

Hub Router

7600/ASR9k

ASR9k

Service Routers

Legend 1 GE Link

10 GE Ring Link

10 GE Point to Point Link

Video EQAM

CMTS

7600/ASR9k

Redundant SDC May Not Be Present

Case Study #2 US Cable Operator / SP

Tier 1 Hub ASR9k

CPEs


Introduction




‒ Results

Case Study

Conclusion

Agenda


Conclusion

Learned design options for large networks ‒ How to scale Routing (+MPLS) !

‒ What Fast Restoration technique is suitable! Where!

‒ Which Topology makes sense !

‒ Services Consideration !

Got the proof points ‒ Deployed case studies


Additional Slides


LFA Roadmap – IPv4

MPLS TE-FRR 1-hop link

7600 (IOS)

ASR1000 (IOS-XE)

ASR9k (IOS-XR)

CRS-1 (IOS-XR)

Per Link LFA FRR Not Available Not Available 4.0.1 3.5.0

OSPF LFA FRR (per prefix)

15.1(3)S 3.4S 4.2.0 4.2.0

ISIS LFA FRR (per prefix)

15.1(2)S 3.4S 4.0.1 4.0.1

EIGRP FRR (per prefix) 15.2(4)S* 3.7S*

OSPF Remote LFA 15.2(2)S 3.6S 4.3.1* 4.3.1*

ISIS Remote LFA 15.2(2)S 3.6S 4.3.1* 4.3.1*

BGP PIC Core for IP/MPLS

12.2(33)SRC 2.5S 3.7.0 3.4

BGP PIC Edge 12.2(33)SRE 2.5S 4.0.0 4.0.0

*Future


LFA Roadmap – IPv6

MPLS TE-FRR 1-hop link

7600 (IOS)

ASR1000 (IOS-XE)

ASR9k (IOS-XR)

CRS-1 (IOS-XR)

Per Link LFA FRR Not Available Not Available 4.3.1* 4.3.1*

OSPF LFA FRR (per prefix)

Radar Radar 4.3.1* 4.3.1*

ISIS LFA FRR (per prefix) Radar Radar 4.3.1* 4.3.1*

EIGRP FRR (per prefix) Radar Radar Radar Radar

OSPF Remote LFA Radar Radar Radar Radar

ISIS Remote LFA Radar Radar Radar Radar

BGP PIC Core 3.5S 3.5S 3.7.0 3.4

BGP PIC Edge 3.5S 3.5S 4.0.0 4.0.0

*Future


BGP Next-Hop Tracking

Makes the next-hop failure detection event-driven instead of timer-driven

Next-hop tracking (NHT) feature allows to track BGP next-hops in the RIB

If the RIB entry changes, then the client such as BGP is notified

Allows for new path selection for BGP routes as soon as the notification is received

On/off knob as well as configuration option on how long to wait before starting new path selection

Edge FC

Edge FRR

POP FRR

Core FRR

POP FC Core FC


Behavior Without NHT

Site2

RR1

P2

RR2

P4

P1 P3

PE1# show ip route 192.168.1.3 % Subnet Not in Table

PE1

PE3

PE4

pe1#sh ip bgp vpnv4 vrf vpna 10.1.2.0/24 BGP routing table entry for 100:1:10.1.2.0/24, version 42 Paths: (1 available, best #1, table vpna) Advertised to update-groups: 1 Local 192.168.1.3 (metric 145) from 192.168.1.2 (192.168.1.2) Origin incomplete, metric 0, localpref 100, valid, internal, best Extended Community: RT:100:1 Originator: 192.168.1.3, Cluster list: 192.168.1.2, mpls labels in/out nolabel/28

Tic…Tic…60sec

Wait 180 Seconds?? No!!!

Traffic Loss for Up to 60 Secs Due to BGP Scanner Interval

10.1.2.0/24

Site1 10.1.1.0/24

CE1

Edge FC

Edge FRR

POP FRR

Core FRR

POP FC Core FC


Site1 10.1.1.0/24

RR1 RR2

P2 P4

P1 P3

CE1 PE1

PE3

PE4

wg2pe1#sh ip bgp vpnv4 all 10.1.2.0 BGP routing table entry for 100:1:10.1.2.0/24, version 51 Paths: (1 available, best #1, table vpna) Flag: 0x820 Advertised to update-groups: 1 Local 192.168.1.4 (metric 193) from 192.168.1.2 (192.168.1.2) Origin incomplete, metric 0, localpref 100, valid, internal, best Extended Community: RT:100:1 Originator: 192.168.1.4, Cluster list: 192.168.1.2, mpls labels in/out nolabel/32

The Time Period Determines How Long BGP Will Wait Before Running the Best Path Algorithm After Notification Is Received.

router bgp 100 address-family ipv4 unicast bgp nexthop trigger enable bgp nexthop trigger delay 5

Potential Time Saving Is Up to 60 Secs

Site2

10.1.2.0/24

Edge FC

Edge FRR

POP FRR

Core FRR

POP FC Core FC Behavior with NHT-Enabled

unified mpls: advanced scaling for core and edge...

Documents