hitesh ballani, paul francis, tuan cao and jia wang cornell …ballani.azurewebsites.net › talks...

Making Routers Last Longer with ViAggre

Hitesh Ballani, Paul Francis, Tuan Cao and JiaWang

Cornell University and AT&T Labs–Research

NSDI 2009

Motivation: Rapid Routing Table Growth

0

50000

100000

150000

200000

250000

300000

0806040200989694929088

Inte

rnet

Rou

ting

Tab

le S

ize

Year

282,000 prefixes (Sep’08)

0

50000

100000

150000

200000

250000

300000

0806040200989694929088

Inte

rnet

Rou

ting

Tab

le S

ize

Year

282,000 prefixes (Sep’08)

[Data Credit: Geoff Huston]

Motivation: Rapid Routing Table Growth

0

100000

200000

300000

400000

500000

140806040200989694929088

Inte

rnet

Rou

ting

Tab

le S

ize

Year

0

100000

200000

300000

400000

500000

140806040200989694929088

Inte

rnet

Rou

ting

Tab

le S

ize

Year

??

Rapid future growth

I IPv4 exhaustionI IPv6 deployment

Routing Table stored in Forwarding InformationBase (FIB) on Routers

Large Routing Table ⇒ More FIB space on Routers

Does FIB Size Matter?

The problem is Scaling Properties of FIB memory(low volume, off-chip SRAM)

Technical concernsI Power and Heat dissipation problems

Business concerns

I Low-volume, off-chip SRAM does not trackMoore’s law

I Larger routing table ⇒ Less cost-effectivenetworks

I Price per byte forwarded increases

I Cost of router memory upgrades


Anecdotal evidence shows ISPs are willing toundergo some pain to extend the lifetime of their

routers

Virtual Aggregation (ViAggre)

A “configuration-only” approach to shrinking routerFIBs

I Applies to legacy routersI Can be adopted independently by any ISP

Real World Impact

I IETF Standards effortI Huawei implementing ViAggre into routers

Key Insight: Divide the routing burden

A router only needs to keep routes for a fraction ofthe address space

Talk Outline

I Motivation[]y

I Router Innards[]y

I Big Picture[]y

I ViAggre Design[]y

I Design Concerns[]y

I Evaluation[]y

I Deployment[]y

Router Innards

Route Processor

Line Card

ASIC FIB

Line Card

Line Card

Line Card

RIB

Router

Routing Protocol

(RP)

Switch Fabric

RP

Router

RP

Router

Router Innards

Route Processor

Line Card

ASIC FIB

Line Card

Line Card

Line Card

RIB

Router

Routing Protocol

(RP)

Switch Fabric

RP

Router

RP

Router

Control PlaneParticipates in routing protocol

Router Innards

Route Processor

Line Card

ASIC FIB

Line Card

Line Card

Line Card

RIB

Router

Routing Protocol

(RP)

Switch Fabric

RP

Router

RP

Router

RoutingInformation

(DRAM $)Base

Control PlaneRIB is a table of routes and is stored on slow

memory

Router Innards

Route Processor

Line Card

ASIC FIB

Line Card

Line Card

Line Card

RIB

Router

Routing Protocol

(RP)

Switch Fabric

RP

Router

RP

Router

Forwarding Information Base(SRAM $$$)

Data PlaneResponsible for sending packets based on FIB (stored

in fast memory)

Routing Scalability Problem Space

[MapEncap’96]

[GSE, ID’97]

[Atoms, ’04]

[CRIO, ICNP’06]

[LISP, ID’07]

[SIRA, ID’07]

[TRRP, ’07]

[APT, ID’07]

[Six/One,

MobiArch’08]

[Francis, CNIS’94]

[Deering, ID’00]

[Hain, ID’02]

[Krioukov, Arxiv’05]

[Shim6, ID’07]

[Multipath, ’08]

A few problems afflict Internet routing scalabilityLots of work to address these problems

FIB growthRIB growth

Routing Convergence,Update Churn, ....


[MapEncap’96]

[GSE, ID’97]

[Atoms, ’04]

[CRIO, ICNP’06]

[LISP, ID’07]

[SIRA, ID’07]

[TRRP, ’07]

[APT, ID’07]

[Six/One,

MobiArch’08]


[Deering, ID’00]

[Hain, ID’02]


[Shim6, ID’07]

[Multipath, ’08]

Separate edge from the core




[MapEncap’96]

[GSE, ID’97]

[Atoms, ’04]

[CRIO, ICNP’06]

[LISP, ID’07]

[SIRA, ID’07]

[TRRP, ’07]

[APT, ID’07]

[Six/One,

MobiArch’08]


[Deering, ID’00]

[Hain, ID’02]


[Shim6, ID’07]

[Multipath, ’08]

Geographical routing




[MapEncap’96]

[GSE, ID’97]

[Atoms, ’04]

[CRIO, ICNP’06]

[LISP, ID’07]

[SIRA, ID’07]

[TRRP, ’07]

[APT, ID’07]

[Six/One,

MobiArch’08]


[Deering, ID’00]

[Hain, ID’02]


[Shim6, ID’07]

[Multipath, ’08]

Compact routing




[MapEncap’96]

[GSE, ID’97]

[Atoms, ’04]

[CRIO, ICNP’06]

[LISP, ID’07]

[SIRA, ID’07]

[TRRP, ’07]

[APT, ID’07]

[Six/One,

MobiArch’08]


[Deering, ID’00]

[Hain, ID’02]


[Shim6, ID’07]

[Multipath, ’08]

Elimination Approaches




[MapEncap’96]

[GSE, ID’97]

[Atoms, ’04]

[CRIO, ICNP’06]

[LISP, ID’07]

[SIRA, ID’07]

[TRRP, ’07]

[APT, ID’07]

[Six/One,

MobiArch’08]


[Deering, ID’00]

[Hain, ID’02]


[Shim6, ID’07]

[Multipath, ’08]

All require architectural changeSo many good ideas, so little impact!




[MapEncap’96]

[GSE, ID’97]

[Atoms, ’04]

[CRIO, ICNP’06]

[LISP, ID’07]

[SIRA, ID’07]

[TRRP, ’07]

[APT, ID’07]

[Six/One,

MobiArch’08]


[Deering, ID’00]

[Hain, ID’02]


[Shim6, ID’07]

[Multipath, ’08]

Can we devise an incremental solution by focusing ona subset of the problem space?




[MapEncap’96]

[GSE, ID’97]

[Atoms, ’04]

[CRIO, ICNP’06]

[LISP, ID’07]

[SIRA, ID’07]

[TRRP, ’07]

[APT, ID’07]

[Six/One,

MobiArch’08]


[Deering, ID’00]

[Hain, ID’02]


[Shim6, ID’07]

[Multipath, ’08]

This Talk: Focuses on reducing FIB size



Talk Outline

I Motivation[]y

I Router Innards[]y

I Big Picture[]y

I ViAggre Design[]y


I Evaluation[]y

I Deployment[]y

ViAggre: Basic Idea

PoP A PoP C

PoP B

ISP

IPv4AddressSpace

0.0.0.0

255.255.255.255

External Router External

Router

Today: All routers have routes to all destinations

ViAggre: Basic Idea

PoP A PoP C

PoP B

ISP

0.0.0.0

255.255.255.255


Router

0/2

64/2

128/2

192/2

VirtualPrefixes

Divide address space into Virtual Prefixes (VPs)Notation: “/2” implies that the first two bits are used to group IP

addresses. “0/2” represents addresses starting with 00.

i.e. 0/2 ⇒ 0.0.0.0/2 ⇒ [0.0.0.0 to 63.255.255.255]

ViAggre: Basic Idea0.0.0.0

255.255.255.255


Router

0/2

64/2

128/2

192/2

VirtualPrefixes

Aggregation Pointsfor Green VP

Assign Virtual Prefixes to the routersGreen Aggregation Points maintain routes to green

prefixes


255.255.255.255


Router

0/2

64/2

128/2

192/2

VirtualPrefixes


Routers only have routes to a fraction of the addressspace


255.255.255.255


Router

0/2

64/2

128/2

192/2

VirtualPrefixes


1. How to achieve such division of the routing tablewithout changes to routers and external cooperation?

2. How do packets traverse even though routers havepartial routing tables?

ViAggre Control-Plane0.0.0.0

255.255.255.255


Router

0/2

64/2

128/2

192/2

Only Blue Routesshould go into FIB

Control-plane needs to ensure that a router’s FIBonly contains routes that the router is aggregating


255.255.255.255


Router

0/2

64/2

128/2

192/2 Full Routing Table

External BGP Peers may advertise full routing table


255.255.255.255


Router

0/2

64/2

128/2

192/2 Full Routing Table

RIB

FIB

Load full routingtable into RIB

Supress all butblue routes from FIB

Simple Approach: FIB SuppressionRouters can load a subset of the RIB into their FIB

High Performance Overhead


255.255.255.255


Router

0/2

64/2

128/2

192/2

Full RoutingTable

Route Reflector

RIB

FIB

Practical Approach: Route-reflector SuppressionExternal router peers with a route-reflector

Blue router receives only blue routes


255.255.255.255


Router

0/2

64/2

128/2

192/2

Full RoutingTable

Route Reflector

Practical Approach: Route-reflector SuppressionRoute-reflectors exchange routes with each other

Data-Plane paths0.0.0.0

255.255.255.255


Router

0/2

64/2

128/2

192/2 Virtual Prefix

Packets destined to a prefix in Red

Consider packets destined to a prefix in the red VP

Data-Plane paths0.0.0.0

255.255.255.255


Router

0/2

64/2

128/2

192/2 I

A

E X

A2

12

ViAggre pathIngress (I) → Aggregation Pt (A) → Egress (E)

Ingress → Aggregation Point0.0.0.0

255.255.255.255


Router

0/2

64/2

128/2

192/2 I

A

E X

A2

1

Router I doesn’t have a route for destination prefix


255.255.255.255


Router

0/2

64/2

128/2

192/2 I

A

E X

A2

1

Advertise Red VP

Aggregation Points advertise corresponding VirtualPrefixes


255.255.255.255


Router

0/2

64/2

128/2

192/2 I

A

E X

A2

1

Advertise Red VP

Prefix Next-HopP1P2

........

0/2 A128/2192/2 ....

....

Blue router has a route for the red Virtual Prefix

Aggregation Point → Egress0.0.0.0

255.255.255.255


Router

0/2

64/2

128/2

192/2 I

A

E X

A2 2

Prefix Next-HopP3P4

X....

64/2 ....

128/2 192/2 ....

....

Aggregation Pt. A has a route for destination prefix


255.255.255.255


Router

0/2

64/2

128/2

192/2 I

A

E X

A2 2

Non-red routers don’t have a route

for dst. prefix

Router A tunnels packet to external router asintermediate routers don’t have route to dst. prefixOriginal packet is encapsulated in tunnel header with X as dst.


255.255.255.255


Router

0/2

64/2

128/2

192/2 I

A

E X

A2 2

......

XATunnel Header

Dst

Original Packet

Src

Router A tunnels packet to external router asintermediate routers don’t have route to dst. prefixOriginal packet is encapsulated in tunnel header with X as dst.


255.255.255.255


Router

0/2

64/2

128/2

192/2 I

A

E X

A2

1

Strip tunnel headerfrom outgoing pkts

Egress Router strips the tunnel header off outgoingpackets

Talk Outline

I Motivation[]y

I Router Innards[]y

I Big Picture[]y

I ViAggre Design[]y


I Evaluation[]y

I Deployment[]y

Failure of Aggregation Point0.0.0.0

255.255.255.255


Router

0/2

64/2

128/2

192/2 I

A

E X

A2

1

What if Aggregation Pt. A fails?


255.255.255.255


Router

0/2

64/2

128/2

192/2 I

A

E X

A2

Prefix Next-HopP1P2

........

0/2 A2128/2192/2 ....

....

Router I installs the route advertised by A2


255.255.255.255


Router

0/2

64/2

128/2

192/2 I

A

E X

A2

Prefix Next-HopP1P2

........

0/2 A2128/2192/2 ....

....

Packets are re-routed appropriately

ViAggre’s impact on ISP’s traffic0.0.0.0

255.255.255.255


Router

0/2

64/2

128/2

192/2 I

A

E X

A2

1

ViAggre paths can be longer than native pathsTraffic stretch, increased router and link load, etc.

Popular Prefixes

Traffic volume follows power-law distribution

I 95% of the traffic goes to 5% of prefixesI Has held up for years

Install “Popular Prefixes” in routers

I Stable over weeksI Mitigates ViAggre’s impact on the ISP’s traffic

Talk Outline

I Motivation[]y

I Router Innards[]y

I Big Picture[]y

I ViAggre Design[]y


I Evaluation[]y

I Deployment[]y

ViAggre’s impact on adopting ISP

Positive Negative

Increase in path length(Stretch in msec)

Reduction in FIB Size

(% of global Load Increase

routing table) (Increase in trafficcarried by routers)

ViAggre’s impact on adopting ISP

Positive Negative

Increase in path length(Stretch in msec)

Reduction in FIB Size

(% of global Load Increase

routing table) (Increase in trafficcarried by routers)

ViAggre deployment optionsI Choosing Virtual Prefixes

I Choosing Aggregation Points

I Choosing Popular Prefixes

ISP can make these choices to tune +ves Vs -ves

Choosing Aggregation Points

Assigning more routers to aggregate a virtual prefix

I Reduces Stretch imposed on Traffic (as there is a

close-by aggregation point to send traffic to)

I Increases FIB size (as more cumulative FIB space isused)

ISP can choose aggregation points to trade-off

FIB Size Vs Stretch

Aggregation Point Assignment Problem

min Worst FIB Sizes.t. Worst Stretch ≤ Constraint

Constraint on Worst Stretch ensuresI ISP’s Service Level Agreements not breached

I Latency-sensitive traffic not hurt too much

Worst FIB SizeI Important for provisioning routers

Aforementioned Constraint ProblemI Can be mapped to MultiCommodity Facility Location

I NP-hard problem

I Logarithmic approximation algorithm [Ravi, Sinha, SODA’04]

Tier-1 ISP Study

We implemented a greedy approximation algorithm

Algorithm Input: Data from tier-1 ISP

I Topology, Routing tables, Traffic matrix

Used our algorithm with varying stretch constraints

FIB Size Vs Stretch

0

5

10

15

20

25

30

0 2 4 6 8 10

FIB

Siz

e (

% o

f Int

erne

t rou

ting

tabl

e)

Worst-case Stretch Constraint (msec)

Worst-case FIB SizeAverage FIB Size

0

5

10

15

20

25

30

0 2 4 6 8 10

FIB

Siz

e (

% o

f Int

erne

t rou

ting

tabl

e)



FIB Size reduces as Stretch constraint is relaxed

FIB Size Vs Stretch

0

5

10

15

20

25

30

0 2 4 6 8 10

FIB

Siz

e (

% o

f Int

erne

t rou

ting

tabl

e)



0

5

10

15

20

25

30

0 2 4 6 8 10

FIB

Siz

e (

% o

f Int

erne

t rou

ting

tabl

e)



Worst-case FIB Size10K prefixes

(4% of global routing table)

FIB Size reduces as Stretch constraint is relaxed

FIB Size Vs Stretch

0

5

10

15

20

25

30

0 2 4 6 8 10 0

1

2

3

4

5

6

7F

IB S

ize

(%

of g

loba

l rou

ting

tabl

e)

Act

ual S

tret

ch (

mse

c)



Worst StretchAverage Stretch

0

5

10

15

20

25

30

0 2 4 6 8 10 0

1

2

3

4

5

6

7F

IB S

ize

(%

of g

loba

l rou

ting

tabl

e)

Act

ual S

tret

ch (

mse

c)




Average Stretch is negligible

FIB Size Vs Stretch

0

5

10

15

20

25

30

0 2 4 6 8 10 0

1

2

3

4

5

6

7F

IB S

ize

(%

of g

loba

l rou

ting

tabl

e)

Act

ual S

tret

ch (

mse

c)




0

5

10

15

20

25

30

0 2 4 6 8 10 0

1

2

3

4

5

6

7F

IB S

ize

(%

of g

loba

l rou

ting

tabl

e)

Act

ual S

tret

ch (

mse

c)




Average Stretch0.2 msec

Average Stretch is negligible

FIB Size Vs Stretch

0

5

10

15

20

25

30

0 2 4 6 8 10 0

1

2

3

4

5

6

7F

IB S

ize

(%

of g

loba

l rou

ting

tabl

e)

Act

ual S

tret

ch (

mse

c)




0

5

10

15

20

25

30

0 2 4 6 8 10 0

1

2

3

4

5

6

7F

IB S

ize

(%

of g

loba

l rou

ting

tabl

e)

Act

ual S

tret

ch (

mse

c)




ViAggre can extend lifetime of outdatedrouters by 7-10 years while imposing no stretch

(Worst-case Stretch Constraint = 0ms)

Router Load

Naıve ViAggre deployment

I Traffic routed through aggregation points

I Can lead to substantial load increase acrossrouters

I Alleviative: Use of Popular Prefixes

Router Load

Naıve ViAggre deployment

I Traffic routed through aggregation points

I Can lead to substantial load increase acrossrouters

I Alleviative: Use of Popular Prefixes

A lot of traffic destined to popular prefixes

60

70

80

90

100

2015105321% o

f Tra

ffic

Car

ried

% of Popular Prefixes

60

70

80

90

100

2015105321% o

f Tra

ffic

Car

ried


Router Load

100

39

10

1

0.120151053210

24.519.514.59.56.54.5Lo

ad In

crea

se(%

of n

ativ

e lo

ad)


Worst FIB size (% of DFZ routing table)

100

39

10

1

0.120151053210

24.519.514.59.56.54.5Lo

ad In

crea

se(%

of n

ativ

e lo

ad)


Worst FIB size (% of DFZ routing table)

Popular prefixes populated in all routers5% Popular prefixes ⇒ Max. Load Increase= 1.38%

ViAggre Pros

10x reduction in FIB Size

I Negligible Traffic Stretch (<0.2 msec)I Negligible Increase in Load (<1.5%)

Advantages

I Can be incrementally deployed

I Can be deployed on a limited-scale

I Incentive for deploymentI No change to ISP’s routing setup

I Does not affect routes advertised toneighbors

I Does not restrict routing policies

ViAggre Cons

Control-plane hacks can impactI Installation Time

I Convergence Time

I Failover Time

Planning OverheadI Choosing virtual prefixes

I Assigning aggregation pointsI Assuring network robustness

Configuration overhead of a configuration-onlysolution

ViAggre Deployment on WAIL

ISP with ViAggre

PoP1 PoP2

AS2 AS3

Routes propagated using

I Status Quo

I ViAggre (prefix lists for selective advertisement)


ISP with ViAggre

PoP1 PoP2

AS2 AS3


I Status Quo


Routes propagated using mesh of internal BGP peerings


ISP with ViAggre

PoP1 PoP2

AS2 AS3

RR1 RR2


I Status Quo


Prefix List size depends on # of popular prefixes


ISP with ViAggre

PoP1 PoP2

AS2 AS3

X

Measuring Control-Plane Overhead

Restart external peeringMeasure Installation Time

Installation Time on WAIL

50

100

150

200

250

300

0 50 100 150 200 250

Inst

alla

tion

Tim

e (s

ec)

Number of Prefixes Advertised (thousands)

Status QuoViAggre, 2% PopularPrefixesViAggre, 5% PopularPrefixes

ViAggre reduces Installation Time

Full Routing Table Installation TimeStatus Quo=273sec, ViAggre (2% Popular Prefixes)=124sec

ViAggre management overhead

Developed Configuration Tool

I ∼330 line python script

I Extracts information from existing configuration files

I Generates ViAggre configuration files

I Planning component in the works

Working with a router vendor (Huawei)

I Implement ViAggre natively

I IETF Draft

ViAggre Conclusion

ViAggre shrinks the FIB on routers

I Can be used by ISPs today!

I 10x reduction in FIB Size

I Negligible traffic stretchI Negligible load increase

ISPs can extend lifetime of their routersI Outdated routers can be used for 7-10 years

Is this a “complete” solution? No

I A simple and effective first step

Thank You!


YesTony Li [IAB Workshop’06]

Vince Fuller [APRICOT’07]IAB Workshop [RFC 4984]

. . .

NoDefaultOff [HotNets’05]

AIP [SIGCOMM’08]

. . .




. . .

MaybeMe


AIP [SIGCOMM’08]

. . .




. . .

MaybeMe


AIP [SIGCOMM’08]

. . .

Other reasons to reduce FIB Size

I Rapid future multihomingI To facilitate commodification of ISP business

Anecdotal evidence shows ISPs are willing toundergo some pain to extend the lifetime of their

routers

Rapid Routing Table Growth

BigISPs

LittleISPs

Sites

A B C

a1 a2 b1 b2 c1 c2

a11 a12 a21 a22 b11 b12 b21 b22 c11 c12 c21 c22

Internet Routing Scalability is based on hierarchyRequires addressing to be aligned with topology


BigISPs

LittleISPs

Sites

A B C

a1 a2 b1 b2 c1 c2

a11 a12 a21 a22 b11 b12 b21 b22 c11 c12 c21 c22

Address � Topology MatchSites a11 and a12 are addressed from the address block of a1

which is addressed from the address block of A{a11, a12} ⊂ a1 ⊂ A


BigISPs

LittleISPs

Sites

A B C

a1 a2 b1 b2 c1 c2

a11 a12 a21 a22 b11 b12 b21 b22 c11 c12 c21 c22

A

Routing should scale by:Number of top-level ISPs and Fan-out

Routing state on A: {B, C, a1, a2}


BigISPs

LittleISPs

Sites

A B C

a1 a2 b1 b2 c1 c2

a11 a12 a21 a22 b11 b12 b21 b22 c11 c12 c21 c22

Address � Topology MismatchMultihoming, Load Balancing, Address

Fragmentation, Bad Operational Practices

hitesh ballani, paul francis, tuan cao and jia wang cornell …ballani.azurewebsites.net › talks...

Documents