Lecture 14: Inter-domain Routing Stability

CS 268 class

March 8th, 2004

(slides from Timothy Griffin’s tutorial and Craig Labovitz’s NANOG talk)

Outline of the Today’s class

• An Introduction to BGP

• BGP and the Stable Paths problem

• Convergence of BGP in the real world

• Conclusions and Open Issues

Inter-domain Routing basics

• Internet is composed of over 16000 autonomous

systems

• BGP = Border Gateway Protocol

– Is a Policy-Based routing protocol

– Is the de facto inter-domain routing protocol of

today’s global Internet

• Relatively simple protocol, but configuration is

complex and the entire world can see, and be

impacted by, your mistakes.

BGP Operations (Simplified)

Establish session on TCP port 179

Exchange all active routes

Exchange incremental updates

AS1

AS2

While connection is ALIVE exchangeroute UPDATE messages

BGP session

Four Types of BGP Messages

• Open : Establish a peering session.

• Keep Alive : Handshake at regular intervals.

• Notification : Shuts down a peering session.

• Update : Announcing new routes or withdrawing

previously announced routes.

announcement = prefix + attributes values

Two Types of BGP Neighbor Relationships

• External Neighbor (eBGP) in a different Autonomous Systems

• Internal Neighbor (iBGP) in the same Autonomous System AS1

AS2

eBGP

iBGP

iBGP is routed (using IGP!)

iBGP Peers Must be Fully Meshed

iBGP neighbors do not announce routes received via iBGP to other iBGPneighbors.

eBGP update

iBGP updates

• iBGP is needed to avoid routing loops within an AS

• Injecting external routes into IGP does not scale and causes BGP policy information to be lost

• BGP does not provide “shortest path” routing

• Is iBGP an IGP? NO!

Important BGP attributes

• LocalPREF– Local preference policy to choose “most”

preferred route

• Multi-exit Discriminator– Which peering point to choose?

• Import Rules– What route advertisements do I accept?

• Export Rules– Which routes do I forward to whom?

Route Selection Summary

Highest Local Preference

Shortest ASPATH

Lowest MED

i-BGP < e-BGP

Lowest IGP cost to BGP egress

Lowest router ID

traffic engineering

Enforce relationships

Throw up hands andbreak ties

Implementing Customer/Provider and Peer/Peer relationships

• Enforce transit relationships – Outbound route filtering

• Enforce order of route preference– provider < peer < customer

Two parts:

Import Routes

Frompeer

Frompeer

Fromprovider

Fromprovider

From customer

From customer

provider route customer routepeer route ISP route

Export Routes

Topeer

Topeer

Tocustomer

Tocustomer

Toprovider

From provider

provider route customer routepeer route ISP route

filtersblock

Outline of the Today’s class

• An Introduction to BGP

• BGP and the Stable Paths problem

• Convergence of BGP in the real world

• Conclusions and Open Issues

This talk

What Problem is BGP solving?

Underlying problem

Shortest Paths

Distributed means of computing a solution.

X?

RIP, OSPF, IS-IS

BGP

• aid in the design of policy analysis algorithms and heuristics, • aid in the analysis and design of BGP and extensions,• help explain some BGP routing anomalies, • provide a fun way of thinking about the protocol

Having an X can

1

Q : How simple can X get? A: The Stable Paths Problem (SPP)

2 5 5 2 1 0

0

2 1 02 0

1 3 01 0

3 0

4 2 04 3 0

3

4

2

1

• A graph of nodes and edges, • Node 0, called the origin, • For each non-zero node, a

set or permitted paths to the origin. This set always contains the “null path”.

• A ranking of permitted paths at each node. Null path is always least preferred. (Not shown in diagram)

An instance of the SPP :

When modeling BGP : nodes represent BGP speaking border routers, and 0 represents a node originating some address block

most preferred…least preferred (not null)

Yes, the translation gets messy!

5 5 2 1 0

1

A Solution to a Stable Paths Problem

2

0

2 1 02 0

1 3 01 0

3 0

4 2 04 3 0

3

4

2

1

• node u’s assigned path is either the null path

or is a path uwP, where wP is assigned to

node w and {u,w} is an edge in the graph,

• each node is assigned the highest ranked

path among those consistent with the paths

assigned to its neighbors.

A Solution need not represent a shortest path tree, or a spanning tree.

A solution is an assignment of permitted paths to each node such that

A Stable Paths Problem may have multiple solutions

First solution

1

0

2

1 2 01 0

1

0

2

1

0

2

2 1 02 0

1 2 01 0

2 1 02 0

1 2 01 0

2 1 02 0

Second solutionDISAGREE

Multiple sets of BGP routing policies can map down to the same Stable Paths Problem : DISAGREE in RPSL (Version I)

1

0

2

1 2 01 0

2 1 02 0

import : from AS1 action pref = 0; accept ANY; from AS0 action pref = 10; accept ANY; export : to AS2 announce ANY;

import : from AS2 action pref = 0; accept ANY; from AS0 action pref = 10; accept ANY; export : to AS1 announce ANY;

export : to AS1, AS2 announce AS0;

DISAGREE in RPSL (Version II)

1

0

2

1 2 01 0

2 1 02 0

import : from AS-ANY action pref = 0; accept community.contains(1:1); from AS-ANY action pref = 10; accept ANY; export : to AS2 announce ANY;

export : to AS1 set community.append(2:1); announce AS0; to AS2 set community.append(1:1); announce AS0

import : from AS-ANY action pref = 0; accept community.contains(2:1); from AS-ANY action pref = 10; accept ANY; export : to AS1 announce ANY;

Assume AS1 and AS2 use “neighbor send-community” command ….

DISAGREE in RPSL (Version III)

1

0

2

1 2 01 0

2 1 02 0

import : from AS-ANY accept ANY; export : to AS2 announce ANY;

export : to AS1 action aspath.prepend(AS0, AS0, AS0); announce AS0; to AS2 announce AS0

import : from AS1 action pref = 0; accept ANY; from AS0 action pref = 10; accept ANY; export : to AS1 announce ANY;

The interaction of all BGP policies is directly represented in SPP

Multiple solutions can result in “Route Triggering”

1

02

3

1 01 2 3 0

2 3 02 1 0

3 2 1 03 0

1

02

3

1

02

3

Remove primary link Restore primary link

1 01 2 3 0

2 3 03 1 0

3 2 1 03 0

primary link

backup link

SPP helps explain possibility of BGP divergence

• BGP is not guaranteed to converge to a stable routing. Policy inconsistencies can lead to “livelock” protocol oscillations.

• See “Persistent Route Oscillations in Inter-domain Routing” by K. Varadhan, R. Govindan, and D. Estrin. ISI report, 1996

Solvable Can Diverge

must convergemust diverge

The SPP view :

2

0

31

2 1 02 0

1 3 01 0

3 4 2 03 0

4

4 2 04 3 0

BAD GADGET : No SolutionWith a BGP-likeprotocol, each nodewill do the best it can, so at least onenode will always have the opportunity toimprove its path. Result : persistent oscillation.

SURPRISE : Beware of Backup Policies

2

0

31

2 1 02 0

1 3 01 0

3 4 2 03 0

4

4 04 2 04 3 0

Becomes BAD GADGET if link (4, 0) goes down.

BGP is not robust : it is not guaranteed to recover from network failures.

PRECARIOUS

1

0

2

1 2 01 0

2 1 02 0

3

4

5 6

5 3 1 05 6 3 1 2 05 3 1 2 0

6 3 1 06 4 3 1 2 06 3 1 2 0

4 3 1 04 5 3 1 2 04 3 1 2 0

3 1 03 1 2 0

As with DISAGREE, this part has two distinct solutions

This part has a solution only when node 1 is assigned the direct path (1 0).

Has a solution, but can get “trapped”

What is to be done?

StaticApproach

Inter-AScoordination

Automated Analysis of Routing Policies(This is very hard).

Dynamic Approach

Extend BGP with a dynamic means of detecting and suppressingpolicy-based oscillations?

These approaches are complementary

Research papers on SPP

“An Analysis of BGP Convergence Properties”Timothy G. Griffin, Gordon Wilfong

“Policy Disputes in Path Vector Protocols”Timothy G. Griffin, F. Bruce Shepherd, Gordon Wilfong

SIGCOMM’99

INFOCOM’00

ICNP ‘99

Model BGP, show static analysis is hard

Define Stable Paths Problem and develop sufficient condition for “sanity”

Dynamic solution based on histories

“A Safe Path Vector Protocol”Timothy G. Griffin, Gordon Wilfong

“Stable Internet Routing without Global Coordination”Lixin Gao, Jennifer Rexford

SIGMETRICS’00

Show that if certain guidelines are followed, then all is well.Rule: Do not forward route advertisements from peers or Providers to other peers or providers.

Outline of the Today’s class

• An Introduction to BGP

• BGP and the Stable Paths problem

• Convergence of BGP in the real world

• Conclusions and Open Issues

Convergence in the real-world?

• [Labovitz99] Experimental results from two year study which measured 150,000 BGP faults injected into peering sessions at several IXPs

• Found– Internet averages 3 minutes to converge after

failover– Some multihomed failovers (short to long

ASPath) require 15 minutes

Problems with Distance Vector

• Distance vector protocols (e.g. RIP) suffer routing table loops – Counting-to-infinity– Routing table loops– Bouncing problem

• BGP uses path vector to “solve” problems seen with RIP and other Bellman-Ford derived protocols

Counting to Infinity

A B2

B 2R 1+2=3

A 2R 1

R

1

R 5+2=7

2+3=5

R 7+2=9

Taming Infinity• Routing Information Protocol (RIP) solved

counting to infinity problem by re-defining infinity.– Added speedups: poison reverse, split horizon,

triggered updates.– Strictly increasing O(N)

• ASPath limits “infinity” to the width of the Internet (an ASPath through all your neighbors)– Monotonically increasing– Upper bound?

BGP Convergence ExampleR

AS0 AS1

AS2AS3

*B R via AS3 B R via AS1,AS3 B R via AS2,AS3

*B R via AS3 B R via AS0,AS3 B R via AS2,AS3

*B R via AS3 B R via AS1,AS3 B R via AS2,AS3

AS0 AS1 AS2

** **B R via 203

*B R via 031 B R via 103

N > 4?

AS1673

AS237

AS5696

AS2497

AS1239

AS6453

AS701

AS2914

AS6461

AS5000

AS6113

AS1

2914 237

1 5696 237

1239 5696 237

2497 5696 237

701 6461 5696 237

6461 5696 237

237

5696 237

5000 237

6113 2914 237

1673 5696 237

6453 1239 5696 237

The Problem with BGP

• If we assume 1. unbounded delay on BGP processing and propagation

2. Full BGP mesh BGP peers

3. Constrained shortest path first selection algorithm

• BGP is O(N!), where N number of default-free BGP speakers

There exists possible ordering of messages There exists possible ordering of messages such that BGP will explore all possible such that BGP will explore all possible ASPaths of all possible lengthsASPaths of all possible lengths

BGP and RIP• RIP precisely monotonically increasing. Can explore

metrics (1…N)• BGP monotonically increasing. Multiple (N!) ways to

represent a path metric of N.

• BGP “solved” RIP routing table loop problem by making it exponentially worse…

2117 5696 2129

2117 1 5696 2129

2117 2041 3508 3508 4540 7037 1239 5696 2129

2117 1 2041 3508 3508 4540 7037 1239 5696 2129

2117 2041 3508 3508 4540 7037 1239 6113 5696 2129

2117 1 2041 3508 3508 4540 7037 1239 6113 5696 2129

BGP Best Case

What is the best we can expect from BGP?What is the best we can expect from BGP? Implementation of MinRouteAdver timer Implementation of MinRouteAdver timer

leads to 30 second roundsleads to 30 second rounds• Time complexity is O(n-3)*30 secondsTime complexity is O(n-3)*30 seconds• State/Computational complexity O(n)State/Computational complexity O(n)• At its best, BGP performs as well as RIP2 At its best, BGP performs as well as RIP2

(but uses exponentially more memory in (but uses exponentially more memory in the process)the process)

MinRouteAdver

• Minimum interval between successive updates sent to a peer for a given prefix– Allow for greater efficiency/packing of updates– Rate throttle

• Applied only to announcements (at least according to BGP RFC)

• Applied on (prefix destination, peer) basis, but implemented on (peer) basis

MinRouteAdver

• 30*(N-3) delay due to creation mutual dependencies. Provide proof that N-3 rounds necessarily created during bounded BGP MinRouteAdver convergence

• Rounds due to– Ambiguity in the BGP RFC and lack receiver loop

detection– Inclusion of BGP withdrawals with

MinRouteAdver (in violation of RFC)

Conclusions

• Internet routing has serious convergence problems– Result 1 [Griffin et al.]: BGP does not satisfy the

stable paths problem.– Result 2 [Rexford et al.]: If every AS follows a set

of guidelines then Internet routing should not have convergence problems.

– Result 3 [Labovitz et al.]: An extensive measurement study shows that Internet convergence can be in the order of several minutes.

Open issues?

• Convergence analysis (lower,upper) bounds are very weak are “worst-worst” case scenarios.

• Can we design a cleaner protocol that has provably good convergence properties?– What about link-state routing?

• Should we really care about convergence?– Routes to popular prefixes are stable [IMC03]