Improving Individual Flow Performance withMultiple Queue Fair QueuingManfred Georg, Christoph Jechlitschek, and Sergey Gorinsky

Applied Research LaboratoryDepartment of Computer Science and Engineering

Washington University in St. LouisSt. Louis, Missouri 63130-4899, USA

Email: {mgeorg, chrisj, gorinsky}@arl.wustl.edu

Abstract-Fair Queuing (FQ) algorithms provide isola- In this paper, we consider link scheduling with con-tion between packet flows, allowing max-min fair sharing stant memory requirements regardless of the numberof a link even when flows misbehave. However, fairness of flows. Such scheduling disciplines include Stochas-comes at the expense of per-flow state. To keep the memoryrequirement independent of the flow count, the router can tic Fair Queuing (SFQ) [11], Stochastic Fair Blueisolate aggregates of flows, rather than individual flows. (SFB) [12], and Random Early Detection with Pref-We investigate the feasibility of protecting individual flows erential Dropping (RED-PD) [13]. When the numberunder such aggregate isolation in the context of Multiple of flows becomes large, these schemes generally treatQueue Fair Queuing (MQFQ), where the router maintains multiple flows as a single aggregate. For example, SFQa fixed number of queues and allows each flow to accessmultiple queues. MQFQ places packets into the shortest uses a fixed number of ueues and serves mulgtile flowsqueue associated with their flow. The extra queues protect from the same queue. Flows within an aggregate are notthe flow against congestion caused by a misbehaving flow isolated from one another and hence share queuing delayin a shared queue. However, multiple per-flow queues and loss characteristics.also enable the misbehaving flow to increase its unfairly We develop Multiple Queue Fair Queuing (MQFQ),acquired fraction of the link capacity. We discuss avoidanceof packet reordering within a flow and compare MQFQ an SF extensIon where the router also mantans a fixedwith prior schemes for aggregate scheduling. number of FIFO queues but allows a flow to access

more than one queue. Upon arrival of a packet fromI. INTRODUCTION a flow, the router places the packet into the shortest of

the queues associated with the flow. MQFQ intends toFair Queuing (FQ) algorithms [1]-[7] dedicate a sep- serve different flows from different sets of queues so

arate queue to each flow and schedule packets for trans- that a misbehaving flow is unable to congest all queuesmission over a congested link so that every flow receives of another flow. Our investigation shows that two queuesan average service rate that approximates its max-min per flow are optimal because a higher number of per-flowfair share of the link capacity [8]. In comparison to queues permits a greedy flow to grab a larger fractionthe traditional First-In First-Out (FIFO) scheduling of of the link capacity. We also experimentally comparepackets, FQ provides a significant degree of isolation MQFQ with SFQ and SFB.between flows and therefore exhibits superior resilienceagainst misbehaving flows. For example, if a User Data- II. RELATED WORKgram Protocol (UDP) flow transmits at an unfairly high SFQ maps each flow into one of a fixed number ofrate, the excessive transmission does not disrupt well- FIFO queues. When the number of flows is high, abehaving flows; instead, FQ penalizes the aggressive flow shares its queue with other flows. SFQ strives toflow through accumulation and eventual discard of its distribute flows evenly among all queues and therebypackets at the router. Fair queuing also improves the support statistically fair sharing in networks with well-fairness properties of end-to-end congestion control. behaving traffic. Furthermore, SFQ offers some pro-While the sending rate of Transmission Control Protocol tection against greedy flows by limiting the impact of(TCP) in a network of FIFO routers is inversely propor- excessive transmission on the flows in other queues. Intional to round-trip time [9], [10] and hence not max- particular, if all k queues of the link are backlogged, themin fair, using FQ with sufficient buffers at bottleneck service rate for any flow is capped at the fair share oflinks enables TCP to transmit at max-mmn fair rates. one queue, i.e., 1/k of the link capacity.Unfortunately, the fairness benefits of FQ come at the SFB extends Blue [14], which itself is an enhancementexpense of maintaining per-flow state at the router. of Random Early Detection (RED) [15]. Blue maintains

1-4244-1 185-8/07/$25.00 © 2007 IEEE 141

a single FIFO queue but might discard an incoming 1.000packet even if the link buffer is not full. The discard 0.500. MQFQ partial interferenceprobability depends on current and past queuing: while SFQ complete interferencea buffer overflow increases the discard probability, the - - MQFQ complete interferencediscard probability decreases whenever the queue emp- 0 100-'ties. SFB extends Blue by adding a Bloom filter to take - Xover calculation of the discard probability. The Bloom @-filter uses multiple hash functions to assign each flow to axa fixed number of bins. Every bin has a fixed size and avariable discard probability. When a packet arrives from 001 0 "a flow, SFB increments a counter for each bin associated 0.005_with the flow. If a bin overflows, SFB discards the packetand increases the discard probability of the bin. SFBdiscards the incoming packet with probability equal to 0.001t 2 30 10 20 30 40 50 60the minimum among the discard probabilities of the bins Number of queuesassociated with the flow. If a bin empties, its discard Fig. 1. Probabilities of flow interference under MQFQ and SFQ.probability decreases.A different way to deal with a misbehaving flow we report the derived probabilities of flow interference.

is to detect it and limit its rate explicitly [16]-[18]. In SFQ with k queues, two flows interfere when hashedHowever, the identify-and-limit approach suffers from into the same queue:the following drawbacks: 1) its effectiveness depends on 1the traffic pattern: e.g., during coordinated attacks, the PSFQ k (1)number of misbehaving flows that need to be identifiedand rate-limited might exceed the maximum supported In MQFQ, two flows interfere either completely whenby the router; 2) since identification takes time, rate- flow x shares all its queues with flow y:limiting kicks in only after some delay; 3) mild cheaters omp 4 3that inflate transmission modestly might evade detec- PMQFQ = k2 (2)tion. To ameliorate these problems, identify-and-limit or partially when flow x shares at least one of its queuesschemes can adopt the technique proposed in this paper. with flow y:In particular, one can identify and rate-limit greedyflows first and apply our multiple-queue technique to ppart 4 6 3the remaining flows. MQFQk k2 +

Figure 1 shows that all three probabilities decrease whenIII*MULTIPLE QUEUE FAIR QUEUING the number of queues grows. While interference under

Multiple Queue Fair Queuing (MQFQ) is an SFQ en- SFQ and partial interference under MQFQ diminishhancement that allows a flow to utilize multiple queues. similarly as 0(l), complete interference under MQFQInstead of a single hash function as in SFQ, MQFQ decreases much faster as 0(k2).uses multiple hash functions to determine a set of FIFO Although MQFQ might place packets of a flow intoqueues for a flow. When a packet arrives, MQFQ applies different queues, no packet reordering occurs if pack-all hash functions to the packet header to compute ets have the same size. If packet sizes are different,potential queues. MQFQ puts the packet into the queue reordering is possible but contained within one roundwith the soonest service. If one queue associated with a of queue traversal. One remedy is to buffer packets forflow grows large, the flow uses another of its queues and one round to restore their order before sending them intothereby bypasses the congestion. Since a flow can flood the link. Alternatively, if the router fragments incomingmultiple queues, there exists a trade-off between the packets into equally-sized cells, applies MQFQ to thedegree of extra capacity surrendered to a misbehaving cells, and reassembles the packets before sending themflow and the number of flows starved by the misbehaving into the link, then reordering affects neither the cells norflow. As we show later, using two queues per flow is the reassembled packets.most beneficial. Unless explicitly stated otherwise, oursubsequent references to MQFQ denote its instance with IV. EVALUATIONtwo hash functions. As in SFQ, MQFQ serves all queues We conduct simulations in ns-2 version 2.29 [20] toin the round-robin order. compare MQFQ with SFQ and SFB experimentally. For

In its extended version [19], our paper analyzes inter- SFQ, we modify the ns-2 default implementation byference between flows under SFQ and MQFQ. Below, improving the hash functions to avoid their frequent

1-4244-1 185-8/07/$25.00 (C 2007 IEEE 142

350 700SF0 SF0

300 SMFB 600 SFB

250 500

200 400

< 150 300

100 200

50 100

0 00 10 20 30 40 50 0 10 20 30 40 50

Flow number Flow numberFig. 2. Responsive traffic from 50 well-behaving TCP flows. Fig. 3. An unfair CBR flow against 49 well-behaving TCP flows.

collisions. We implement MQFQ by extending our im- more than 300 kbps, i.e., about a one-queue share of theplementation of SFQ. While all queues share the link link capacity. Due to uneven distribution of flows amongbuffer space, we allow a queue to grow beyond its fair queues, SFQ also gives out unfairly high rates to somememory share if free space is at least the queue size TCP flows at the expense of other TCP flows. Whereasplus two packets. The code of our implementations is SFB restricts the CBR flow most successfully, MQFQpublicly available [21]. is not as effective and moreover surrenders almost aWe experiment in a single-bottleneck dumbbell topol- two-queue share of link capacity to the misbehaver.

ogy. The core bottleneck link has capacity 5 Mbps. The MQFQ lives up to its intention to protect individualcapacity of each access link is 100 Mbps. Round-trip flows: slowest TCP flows receive highest throughputpropagation delays are fixed at 60 ms for constant-bitrate under MQFQ.(CBR) flows but distributed uniformly between 60 and The benefits from MQFQ are most apparent when the80 ms for TCP flows. All experiments use 1,000-byte number of misbehaving flows is large. Figure 4 reportspackets and 100-packet link buffers. We schedule the throughputs in experiments where fifty CBR flows trans-bottleneck link using one of the evaluated schemes. All mit at an unfairly high rate about 150 kbps each andother links adhere to FIFO queuing and discard packets thereby overload the 5-Mbps link by 50%. The CBRonly upon buffer overflow. Unless stated otherwise, SFQ transmission is randomized, allowing a sending rate toand MQFQ use 16 queues. SFB employs two levels of deviate slightly from the 150-kbps average. Throughputsbins with 23 bins on each level. We measure steady- under MQFQ vary the least: from 74 to 138 kbps.state throughputs between 10 and 50 seconds into each The throughput range under SFQ is wider: from 56experiment and repeat the experiment 10 times. Plotted to 152 kbps. SFB falters dramatically: the multitude ofresults order flows by throughput and depict one standard misbehaving flows overwhelms the Bloom filter, inflatesdeviation with error bars. We shift the plots for MQFQ the discard probabilities, and prevents SFB from utilizingand SFB by 0.2 and 0.4 respectively in order to reduce the bottleneck link fully. SFB limits individual rates foroverlap and improve readability of our graphs. 27 flows to about 20 kbps. Large error bars for the other

First, we examine well-behaving settings where the flows indicate that SFB limits each of these flows inbottleneck link carries data from fifty TCP flows. Fig- some but not all of our ten experiments.ure 2 shows that SFQ, SFB, and MQFQ provide flows Finally, we change the number of queues per flow andwith similar distributions of throughput. Because SFQ denote the respective version ofMQFQ by appending theyields the widest range of individual throughputs, fair- number to its name: SFQ is MQFQ1, regular MQFQ isness is the worst under SFQ. While slowest flows achieve MQFQ2, etc. As Figure 5 shows for fifty well-behavinghighest throughput under MQFQ, the graph confirms the TCP flows, 2 queues versus 1 queue per flow yieldintended fairness benefits of MQFQ. significant improvement but increasing the number ofWe modify the above scenario by replacing one of the queues above 2 provides only marginal extra benefits.

TCP flows with a CBR flow that persistently transmits Furthermore, our other experiments [19] confirm that aat an unfairly high rate of 2.5 Mbps. Since the CBR flow misbehaving CBR flow is able to grab almost the entirealways acquires the highest throughput under each of the rate allocated to the queues that the flow can access.examined schemes, this flow always appears in Figure 3 Although the misbehaver acquires the extra capacityas number 50. Under SFQ, the misbehaving flow forces primary at the expense of fast flows, slow flows do notall TCP competitors out from its queue and acquires benefit from enriching the CBR flow either. Hence, we

1-4244-1 185-8/07/$25.00 (C 2007 IEEE 143

180 300SFQ H-*---X SFQ

160 -MQFQ - MQFQSFB 250 MQFQ3

140 -MQFQ6

U 0120 - 200

0 ~ ~ ~~~ ~ ~ ~ ~ ~ ~ ~ ~~~~~~5

-r 60 100

I I ~~~~~0Fig. 4. Link overload by fifty CBR flows. Fig. 5. Fifty well-behaving TCP flows under MQFQ with different

numbers of queues per flow.

conclude that two queues per flow constitute the bestconfiguration for MQFQ. [5] M. Shreedhar and G. Varghese, "Efficient Fair Queueing using

Deficit Round Robin," in Proceedings ofACM SIGCOMM, 1995,

V. CONCLUSION pp. 231-242.[6] J. C. Bennett and H. Zhang, "WF2Q: Worst-case Fair Weighted

In this paper, we explored how to protect individual Fair Queueing," in Proceedings ofIEEE INFOCOM, March 1996,flowsfrom nfaircrosstrafic ina rouer tht proides

pp. 120-128.flows from unfair cross traffic in a router that provides [7] s. Ramabhadran and J. Pasquale, "Stratified Round Robin: Aisolation only between flow aggregates. In particular, we Low Complexity Packet Scheduler with Bandwidth Fairness andproposed and evaluated MQFQ, a discipline that employs Bounded Delay," in Proceedings ofACM SIGCOMM, 2003, pp.proposed

n m e of u u s a m sh239-249.a fixed number of FIFO queues and multiple hash [8] D. Bertsekas and R. Gallager, Data Networks. Prentice-Hall,functions. By applying the hash functions to headers 1987.of incoming packets, MQFQ strives to associate each [9] J. Mahdavi and S. Floyd, "TCP-Friendly Unicast Rate-Based

Flow Control," January 1997, end2end-interest mailing list.flow with a different subset of the FIFO queues. When a [10] J. Padhye, V. Firoiu, D. Towsley, and J. Kurose, "Modeling TCPpacket arrives from a flow, the router places the packet Throughput: A Simple Model and its Empirical Validation," ininto the shortest queue associated with the flow. Our in- Proceedings ofACM SIGCOMM, September 1998.

vestigation showed that two queues per flow are optimal. [11] P. E. McKenney, "Stochastic Fairness Queueing," in Proceedingsof IEEE INFOCOM, June 1990, pp. 733-740.

While packet reordering within a flow might severely [12] W. Feng, D. D. Kandlur, D. Saha, and K. G. Shin, "Stochas-undermine TCP performance, we discussed avoidance tic Fair Blue: A Queue Management Algorithm for Enforcing

of such reordering.WealsocomparedMQFQ wFairness," in Proceedings of IEEE INFOCOM, 2001, pp. 22-26.Of such reordering. We also compared MQFQ with [13] R. Mahajan, S. Floyd, and D. Wetherall, "Controlling High-SFQ and SFB experimentally in both well-behaving and Bandwidth Flows at the Congested Router," in Proceedings IEEEmisbehaving settings. Our experiments confirmed that ICNP 2001, November 2001.

[14] W. Feng, K. Shin, D. Kandlur, and D. Saha, "The BLUE Activeslowest flows achieve highest throughput under MQFQ. Queue Management Algorithms," in Proceedings of IEEE/ACMWith a single aggressive CBR flow, the protection of Transactions on Networking, August 2002, pp. 513-528.weakest flows comes at the price of surrendering more [15] S. Floyd and V. Jacobson, "Random Early Detection Gateways

for Congestion Avoidance," in Proceedings ofIEEE/ACM Trans-capacity to the misbehaver. However, when the number actions on Networking, August 1993, pp. 397-413.of misbehaving flows is large, MQFQ balances all in- [16] R. Pan, B. Prabhakar, and K. Psounis, "CHOKe, A Statelessdividual flow throughputs much more fairly than under Active Queue Management Scheme for Approximating Fair

SFQ or SFB. Bandwidth Allocation," in Proceedings of IEEE INFOCOM,SFQ or SFB. ~~~~~~~~~~~~March2000.[17] C. Estan and G. Varghese, "New Directions in Traffic Mea-

REFERENCES surement and Accounting," in Proceedings ofACM SIGCOMM,August 2002.

[1] J. Nagle, "On Packet Switches With Infinite Storage, IEEE [18] R. Pan, L. Breslau, B. Prabhakar, and S. Shenker, "ApproximateTransactions on Communications, vol. 35, no. 4, pp. 435438, Fairness Through Differential Dropping," ACM SIGCOMM CCR,April 1987. vol. 33, no. 2, pp. 23-39, 2003.

[2] A. Demers, S. Keshav, and S. Shenker, "Analysis and S Imulation [19] M. Georg, C. Jechlitschek, and S. Gorinsky, "Improving Indi-of a Fair Queueing Algorithm," in Proceedings of ACM SIG vidual Flow Performance with Multiple Queue Fair Queuing,"COMM, 1989, pp. 1-12. Washington University in St. Louis, Tech. Rep. WUCSE-2007-

[3] A. K. Parekh and R. G. Gallager, "A Generalized Processor Shar- 30, May 2007.ing Approach to Flow Control in Integrated Services Networks [20] "The Network Simulator - ns-2," http://www.isi.edu/nsnamlns/.- The Multiple Node Case," IEEE Transactions on Networking, [21] C. Jechlitschek, "4MQFQ Implementation in ns-2 and Supportingvol. 2, no. 2, pp. 137-150, 1994. Scripts," February 2007,

[4] 5. Golestani, "A Self-Clocked Fair Queuing Scheme for Broad- http://www.cse.wustl.edu/rmgeorg/mqfq/ .band Applications," in Proceedings of IEEE INFOCOM, 1994,pp. 636-646.

1-4244-1 185-8/07/$25.00 (C 2007 IEEE 144