![Page 1: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/1.jpg)
Sizing Router BuffersSizing Router Buffers
Guido AppenzellerIsaac KeslassyNick McKeown
Stanford University
http://yuba.stanford.edu/~appenz
![Page 2: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/2.jpg)
2
How much Buffer does a Router need?How much Buffer does a Router need?
� Universally applied rule-of-thumb:� A router needs a buffer size:
� 2T is the two-way propagation delay (or just 250ms)� C is capacity of bottleneck link
� Context� Mandated in backbone and edge routers.� Appears in RFPs and IETF architectural guidelines.� Usually referenced to Villamizar and Song: “High Performance
TCP in ANSNET”, CCR, 1994. � Already known by inventors of TCP [Van Jacobson, 1988]� Has major consequences for router design
CTB ×= 2
CRouterSource Destination
2T
![Page 3: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/3.jpg)
3
ExampleExample
� 10Gb/s linecard� Requires 300Mbytes of buffering.
� Read and write 40 byte packet every 32ns.
� Memory technologies� DRAM: require 4 devices, but too slow. � SRAM: require 80 devices, 1kW, $2000.
� Problem gets harder at 40Gb/s� Hence RLDRAM, FCRAM, etc.
![Page 4: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/4.jpg)
4
Outline of this TalkOutline of this Talk
� The “Rule-of-Thumb” on Buffer Sizing is incorrect� Where the rule of thumb comes from� Why it is incorrect for a core router in the Internet today
� Real Buffer Requirements in case of Congestion� Real Buffer Requirements without Congestion� Experimental results from real Networks
![Page 5: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/5.jpg)
5
TCP Congestion ControlTCP Congestion Control
Only W packets may be outstanding
Rule for adjusting W� If an ACK is received: W W+1/W� If a packet is lost: W W/2
Source Dest
maxW
2maxW
t
Window size
![Page 6: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/6.jpg)
6
Buffer = rule of thumbBuffer = rule of thumb
![Page 7: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/7.jpg)
7
OverOver--buffered Linkbuffered Link
![Page 8: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/8.jpg)
8
UnderUnder--buffered Linkbuffered Link
![Page 9: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/9.jpg)
9
Summary Summary UnbufferedUnbuffered LinkLink
t
W Minimum window for full utilization
� The router can’t fully utilize the link� If the window is too small, link is not full� If the link is full, next window increase causes drop� With no buffer it still achieves 75% utilization
![Page 10: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/10.jpg)
10
Summary Buffered LinkSummary Buffered Link
t
W
Minimum window for full utilization
� With sufficient buffering we achieve full link utilization� The window is always above the critical threshold� Buffer absorbs changes in window size
� Buffer Size = Height of TCP Sawtooth� Minimum buffer size needed is 2T*C
� This is the origin of the rule-of-thumb
Buffer
![Page 11: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/11.jpg)
11
RuleRule--ofof--thumbthumb
� Rule-of-thumb makes sense for one flow
� Typical backbone link has > 20,000 flows� Does the rule-of-thumb still hold?
� Answer: � If flows are perfectly synchronized, then Yes.� If flows are desynchronized then No.
![Page 12: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/12.jpg)
12
Outline of this TalkOutline of this Talk
� The “Rule-of-Thumb” on Buffer Sizing is incorrect� Real Buffer Requirements in case of Congestion
� Correct buffer requirements for a congested router� Result:
� Real Buffer Requirements without Congestion� Experimental results from real Networks
nCTB /2 ×=
![Page 13: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/13.jpg)
13
If flows are synchronizedIf flows are synchronized
� Aggregate window has same dynamics
� Therefore buffer occupancy has same dynamics� Rule-of-thumb still holds.
2maxW
t
max
2
W∑
maxW∑
maxW
![Page 14: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/14.jpg)
14
If flows are not synchronizedIf flows are not synchronized
ProbabilityDistribution
B
0
Buffer Size
∑W
![Page 15: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/15.jpg)
15
Central Limit TheoremCentral Limit Theorem
� CLT tells us that the more variables (Congestion Windows of Flows) we have, the narrower the Gaussian (Fluctuation of sum of windows)
� Width of Gaussian decreases with
� Buffer size should also decreases with
n
CT
n
BB n ×=→ = 21
n
1
n
1
![Page 16: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/16.jpg)
16
Required buffer sizeRequired buffer size
2T C
n
×
Simulation
![Page 17: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/17.jpg)
17
Summary Congested RouterSummary Congested Router
� Flows in the core are desynchronized� Substantial experimental evidence� Supported by ns2 simulations
� For desynchronized, long-lived flows you need only buffers of
n
CTB
×= 2
![Page 18: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/18.jpg)
18
Outline of this TalkOutline of this Talk
� The “Rule-of-Thumb” on Buffer Sizing is incorrect� Real Buffer Requirements in case of Congestion� Real Buffer Requirements without Congestion
� Correct buffer requirements for an over-provisioned network� Result: Even smaller buffers
� Experimental results from real Networks
![Page 19: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/19.jpg)
19
UncongestedUncongested RouterRouter
� So far we were assuming long flows in congestion avoidance mode.� What about flows in slow-start?� Do buffer requirements differ?
� Answer: Yes, you need even fewer buffersCaveat: In mixes of long and short flows, long flow effects dominate
Therefore:� Short flow effects are only of interest on uncongested routers� Only useful if you have an overprovisioned network and that
never is congested
![Page 20: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/20.jpg)
20
A single, shortA single, short--lived TCP flowlived TCP flowFlow length 62 packets, RTT ~140 msFlow length 62 packets, RTT ~140 ms
2
4
8
16
32
RTTsynfin ack
received
Flow Completion Time (FCT)
![Page 21: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/21.jpg)
21
Short Flow ModelShort Flow Model
� Basic Idea: Model queue distributionAllows to calculate:
� loss rate� average queue length
� flow completion time
� Complete model is in the paper
![Page 22: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/22.jpg)
22
Average Queue lengthAverage Queue length
capacity :C 40Mbit sload : 0.8
ρ)E[S]-2(1
]E[Sρ 22
Queue Model
Experimental Data
![Page 23: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/23.jpg)
23
Buffers for Short Flows on an Buffers for Short Flows on an
uncongesteduncongested RouterRouter
� Results from Short-Flow Model � Buffer required only depends on lengths of bursts and load � Example - for bursts of up to size 16 at load 80%
� For 1% loss probability B = 115 Packets� For 0.01% loss probability B = 230 packets etc.� Bursts of size 12 is maximum for Windows XP
� This is independent of line speed and RTT� Same for a 1 Mb/s router and a 40 Gb/s router!
� In mixes of flows, long flow effects dominate� Also holds for length distributions, e.g. Pareto
![Page 24: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/24.jpg)
24
Outline of this TalkOutline of this Talk
� The “Rule-of-Thumb” on Buffer Sizing is incorrect� Real Buffer Requirements in case of Congestion� Real Buffer Requirements without Congestion � Results from real Networks
� Lab results with a physical router� Experiments on production networks with real traffic
![Page 25: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/25.jpg)
25
Long Flows Long Flows –– UtilizationUtilization
Model vs. ns2 vs. Physical RouterModel vs. ns2 vs. Physical RouterGSR 12000, OC3 Line CardGSR 12000, OC3 Line Card
Link UtilizationRouter Buffer
99.5%100%100%99.9%
99.2%99.8%100%100%
99.7%100%100%100%
512kb1Mb2Mb4Mb
3264128192
0.5 x1 x2 x3 x
400
94.9%98.1%99.8%99.7%
94.7%99.3%99.9%99.8%
96.9%99.9%100%100%
1Mb2Mb4Mb8Mb
64129258387
0.5 x1 x2 x3 x
100
ExpSimModelRAMPktsTCP
Flows2T C
n
×
Thanks to Joel Sommers and Paul Barford of University of Wisconsin-Madison
![Page 26: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/26.jpg)
26
Short Flows Short Flows –– Queue DistributionQueue DistributionModel vs. Physical Router, OC3 Line CardModel vs. Physical Router, OC3 Line Card
![Page 27: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/27.jpg)
27
Experiments with live traffic (I)Experiments with live traffic (I)
� Stanford University Gateway� Link from internet to student dormitories� Estimated 400 concurrent flows, 25 Mb/s� 7200 VXR (shared memory router)
Link UtilizationRouter Buffer
97.4%97.6%98.5%99.9%
95.9%99.5%99.9%100%
466585500
0.8 x1.2 x1.5 x>>2 x
400
ExpModelPkts
TCPFlows 2T C
n
×
Thanks to Sunia Yang, Wayne Sung and the Stanford Backbone Team
![Page 28: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/28.jpg)
28
Experiment with live traffic (II)Experiment with live traffic (II)
Internet2 link Indianapolis to Kansas CityInternet2 link Indianapolis to Kansas City
� Link Setup� 10Gb/s link, T640� Default Buffer: ~1000 ms� Flows of 1 Gb/s� Loss requirement < 10-8
� Experiment� Reduced buffer to 10 ms (1%) - nothing happened� Reduced buffer to 5 ms (0.5%) - nothing happened� Next: buffer of 2ms (0.2%)� Experiment ongoing…
Thanks to Stanislav Shalunov of Internet2 and Guy Almes (now at NSF)
![Page 29: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/29.jpg)
29
How much buffer does a router need?How much buffer does a router need?
Still applicable in a few select cases (e.g. I2 speed records)
�Single flow saturates router
�Few synchronized flows with congestion
CommentsBufferScenario
CT ×2
n
CT ×2
Works if there is never any congestion (optimists only)100’s of pkts
�One or many flows
�Not Congested, << 1
Applicable for the core and edge of the internet today.
�Many flows
�Congestion
CommentsBufferScenario
The old “Rule-of-Thumb”
The new “Rule-of-Thumb”
![Page 30: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/30.jpg)
30
Impact on Router DesignImpact on Router Design
� 10Gb/s linecard with 200,000 x 56kb/s flows� Rule-of-thumb: Buffer = 2.5Gbits
� Requires external, slow DRAM
� Becomes: Buffer = 6Mbits� Can use on-chip, fast SRAM
� 40Gb/s linecard with 40,000 x 1Mb/s flows� Rule-of-thumb: Buffer = 10Gbits� Becomes: Buffer = 50Mbits
![Page 31: Sizing Router Buffers€¦ · A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge](https://reader030.vdocument.in/reader030/viewer/2022041006/5eab861eb531a031c60a8f02/html5/thumbnails/31.jpg)
Thanks!Thanks!