Sizing Router BuffersSizing Router Buffers

Isaac Keslassy (Technion)

Guido Appenzeller & Nick McKeown (Stanford)

2

Routers Need Packet BuffersRouters Need Packet Buffers

It’s well known that routers need packet buffers It’s less clear why and how much

Goal of this work is to answer the question:

How much buffering do routers need?

Given that queueing delay is the only variable part of packet delay in the Internet, you’d think we’d know the answer already!

3

How Much Buffer Does a Router How Much Buffer Does a Router Need?Need?

Universally applied rule-of-thumb: A router needs a buffer size:

2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link

Context Mandated in backbone and edge routers. Appears in RFPs and IETF architectural guidelines. Usually referenced to Villamizar and Song: “High Performance

TCP in ANSNET”, CCR, 1994. Already known by inventors of TCP [Van Jacobson, 1988]. Has major consequences for router design.

CTB 2

CRouterSource Destination

2T

4

ExampleExample

10Gb/s linecard Requires 300Mbytes of buffering. Read and write 40 byte packet every 32ns.

Memory technologies DRAM: require 4 devices, but too slow. SRAM: require 80 devices, 1kW, $2000.

Problem gets harder at 40Gb/s Hence RLDRAM, FCRAM, etc.

5

Main Result in This TalkMain Result in This Talk

The rule of thumb is wrong for a core router today

Required buffer is instead of CT 2n

CT 2

6

Outline of this TalkOutline of this Talk The “Rule-of-Thumb” on Buffer Sizing is incorrect

Where the rule of thumb comes from Why it is incorrect for a core router in the Internet today

Real Buffer Requirements in case of Congestion Real Buffer Requirements without Congestion Experimental results from real Networks

7

TCPTCP

CC’ > C

Only W=2 packets may be outstanding

TCP Congestion Window controls the sending rate Sender sends packets, receiver sends ACKs Sending rate is controlled by Window W, At any time, only W unacknowledged packets may be outstanding

The sending rate of TCP is

RouterSource

RTT

WR

Dest

8

Single TCP FlowSingle TCP FlowRouter with large enough buffers for full link utilizationRouter with large enough buffers for full link utilization

B

DestCC’ > C

Source

maxW

2maxW

t

Window size RTT

For every W ACKs received, send W+1 packets

RTT

WR

9

Required buffer is height of Required buffer is height of sawtoothsawtooth

t

B

0

10

Origin of rule-of-thumbOrigin of rule-of-thumb Before and after reducing window size, the sending rate of the

TCP sender is the same

Inserting the rate equation we get

The RTT is part transmission delay T and part queueing delay B/C . We know that after reducing the window, the queueing delay is zero.

newold RR

new

new

old

old

RTT

W

RTT

W

T

W

CBT

W oldold

2

2/

/2

CTB 2

11

Rule-of-thumbRule-of-thumb

Rule-of-thumb makes sense for one flow Typical backbone link has > 20,000 flows Does the rule-of-thumb still hold?

Answer: If flows are perfectly synchronized, then Yes. If flows are desynchronized then No.

12

Outline of this TalkOutline of this Talk The “Rule-of-Thumb” on Buffer Sizing is incorrect Real Buffer Requirements in case of Congestion

Correct buffer requirements for a congested router Result:

Real Buffer Requirements without Congestion Experimental results from real Networks

nCTB /2

13

If flows are synchronizedIf flows are synchronized

Aggregate window has same dynamics Therefore buffer occupancy has same dynamics Rule-of-thumb still holds.

2maxW

t

max

2

W

maxW

maxW

14

When are Flows When are Flows Synchronized?Synchronized?

Small numbers of flows tend to synchronize

Large aggregates of flows are not synchronized For > 200 flows, synchronization disappears Measurements in the core give no indication

of synchronization

15

If flows are not If flows are not synchronizedsynchronized

ProbabilityDistribution

B

0

Buffer Size

W

16

Central Limit TheoremCentral Limit Theorem

CLT tells us that the more variables (congestion windows of flows) we have, the narrower the Gaussian (fluctuation of sum of windows)

Width of Gaussian decreases with

Buffer size should also decrease with

n

CT

n

BB n

21

n

1

n

1

17

Required buffer sizeRequired buffer size

2T C

n

Simulation

18

SummarySummary Flows in the core are desynchronized

For desynchronized flows, routers need only buffers of

n

CTB

2

19

Outline of this TalkOutline of this Talk The “Rule-of-Thumb” on Buffer Sizing is incorrect Real Buffer Requirements in case of Congestion Real Buffer Requirements without Congestion

Correct buffer requirements for an over-provisioned network Result: Even smaller buffers

Experimental results from real Networks

20

Short FlowsShort Flows

So far we were assuming a congested router with long flows in congestion avoidance mode. What about flows in slow start? Do buffer requirements differ?

Answer: Yes, however: Required buffer in such cases is independent of line

speed and RTT (same for 1Mbit/s or 40 Gbit/s) In mixes of flows, long flows drive buffer requirements

Short flow result relevant for uncongested routers

21

A single, short-lived TCP flowA single, short-lived TCP flowFlow length 62 packets, RTT ~140 msFlow length 62 packets, RTT ~140 ms

2

4

8

16

32

RTTsynfin ack

received

Flow Completion Time (FCT)

22

Average Queue lengthAverage Queue length

capacity :C 40Mbit sload : 0.8

ρ)E[S]-2(1]E[Sρ 22

(S is burst distribution of flows)

23

Queue DistributionQueue Distribution

We derived closed-form estimates of the queue distribution using Effective Bandwidth Gives very good closed form approximation

Buffer requirements for short flows Small & independent of line speed and RTT

In mixes of flows, long flows dominate buffer requirements

]E[S

E[S]

ρ

ρ)(κeb)P(Q bκ

2

12

24

Outline of this TalkOutline of this Talk The “Rule-of-Thumb” on Buffer Sizing is incorrect Real Buffer Requirements in case of Congestion Real Buffer Requirements without Congestion Results from Real Networks

Lab results with a physical router Experiments on production networks with real traffic

27

Long Flows – Utilization (II)Long Flows – Utilization (II) Model vs. ns2 vs. Physical Model vs. ns2 vs. Physical

RouterRouterGSR 12000, OC3 Line CardGSR 12000, OC3 Line Card

TCP

Flows

Router Buffer Link Utilization

Pkts RAM Model Sim Exp

100 0.5 x

1 x

2 x

3 x

64

129

258

387

1Mb

2Mb

4Mb

8Mb

96.9%

99.9%

100%

100%

94.7%

99.3%

99.9%

99.8%

94.9%

98.1%

99.8%

99.7%

400 0.5 x

1 x

2 x

3 x

32

64

128

192

512kb

1Mb

2Mb

4Mb

99.7%

100%

100%

100%

99.2%

99.8%

100%

100%

99.5%

100%

100%

99.9%

2T C

n

28

Short Flows – Queue Short Flows – Queue DistributionDistribution

Model vs. Physical Router, OC3 Line CardModel vs. Physical Router, OC3 Line Card

29

Experiments with live traffic (I)Experiments with live traffic (I)

Stanford University Gateway Link from internet to student dormitories Estimated 400 concurrent flows, 25 Mb/s 7200 VXR (shared memory router)

TCP

Flows

Router Buffer Link Utilization

Pkts Model Exp

400 0.8 x

1.2 x

1.5 x

>>2 x

46

65

85

500

95.9%

99.5%

99.9%

100%

97.4%

97.6%

98.5%

99.9%

2T C

n

Thanks to Sunia Yang, Wayne Sung and the Stanford Backbone Team

30

Experiment with live traffic (II)Experiment with live traffic (II)Internet2 link Indianapolis to Kansas CityInternet2 link Indianapolis to Kansas City

Link Setup 10Gb/s link, T640 Default Buffer: ~1000 ms Flows of 1 Gb/s Loss requirement < 10-8

Experiment Reduced buffer to 10 ms (1%) - nothing happened Reduced buffer to 5 ms (0.5%) - nothing happened Next: buffer of 2ms (0.2%) Experiment ongoing…

Thanks to Stanislav Shalunov of Internet2 and Guy Almes (now at NSF)

31

OutlineOutline

The Rule of Thumb The buffer requirements for a congested router Buffer requirements for short flows (slow-start) Experimental Verification Conclusion

32

Impact on Router DesignImpact on Router Design 10Gb/s linecard with 200,000 x 56kb/s flows

Rule-of-thumb: Buffer = 2.5Gbits Requires external, slow DRAM

Becomes: Buffer = 6Mbits Can use on-chip, fast SRAM Completion time halved for short-flows

40Gb/s linecard with 40,000 x 1Mb/s flows Rule-of-thumb: Buffer = 10Gbits Becomes: Buffer = 50Mbits

For more details… “Sizing Router Buffers – Guido Appenzeller, Isaac Keslassy and

Nick McKeown, to appear at SIGCOMM 2004

33

Open QuestionsOpen Questions

Since buffers can be made much smaller than the rule-of-thumb, can we make all-optical buffers?

How small can buffers be? What is the congestion control algorithm that

minimizes the buffer size?