documentnw
DESCRIPTION
nwTRANSCRIPT
![Page 1: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/1.jpg)
1
DRFQ: Multi-Resource Fair Queueing for Packet Processing
Ali Ghodsi1,3, Vyas Sekar2, Matei Zaharia1, Ion Stoica1
1UC Berkeley, 2Intel ISTC/Stony Brook, 3KTH
![Page 2: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/2.jpg)
2
Increasing Network Complexity
• Packet processing becoming evermore sophisticated– Software Defined Networking (SDN)– Middleboxes– Software Routers (e.g. RouteBricks)– Hardware Acceleration (e.g. SSLShader)
• Data plane no longer merely forwarding– WAN optimization– Caching– IDS– VPN
![Page 3: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/3.jpg)
3
Motivation
• Flows increasingly haveheterogeneous resource consumption– Intrusion detection bottlenecking on CPU– Small packets bottleneck memory-
bandwidth – Unprocessed large packets bottleneck
on link bwScheduling based on a single resource
insufficient
![Page 4: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/4.jpg)
4
Problem
How to schedule packets from different flows,
when packets consume multiple resources?
How to generalize fair queueing to multiple resources?
![Page 5: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/5.jpg)
5
Contribution
Allocation in
Space
Allocation in Time
Single-Resource Fairness
Max-Min Fairness
Fair Queuein
gMulti-Resource Fairness
DRF DRFQ
Generalize Virtual Time to Multiple Resources
![Page 6: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/6.jpg)
6
Outline
• Analysis of Natural Policies• DRF allocations in Space• DRFQ: DRF allocations in Time• Implementation/Evaluation
![Page 7: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/7.jpg)
7
Desirable Multi-Resource Properties
• Share guarantee:– Each flow can get 1/n of at least one
resource
• Strategy-proofness:– A flow shouldn’t be able to finish faster
by increasing the resources required to process it.
![Page 8: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/8.jpg)
flow 1CPU
NIC
time 0 1 32 4 5 6 7 98 10 11
flow 2
Violation of Share Guarantee• Example of FQ applied to a
– Two resources CPU and NIC, used serially– Two flows with profiles <2 μs,1 μs> and <1 μs,1 μs>– FQ based on NIC alternates one packet from each flow
Share Guarantee Violated by Single Resource FQ
flow 2flow 1100%
50%
0%CPU NIC
66%
33%
33%
33%
8
p1 p1
p1 p1
p2 p2
p2 p2
p3 p3
p3 p3
p4
![Page 9: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/9.jpg)
9
Violation of Strategy-Proofness
• Bottleneck fairness– Determine which resource is bottlenecked– Apply FQ to that resource
• Example with Bottleneck Fairness – 2 resources (CPU, NIC), 3 flows <10,1>, <10,14>, <10,14>– CPU bottlenecked and split equally
– Flow 1 changes to <10,7>. NIC bottlenecked and split equallyBottleneck Fairness Violates Strategy-
Proofness
CPU NIC0%
100%
50%
48%
CPU NIC0%
100%
50%
33%
flow 1
flow 2
flow 333%
![Page 10: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/10.jpg)
11
Natural Policy
• Per-Resource Fairness (PRF)– Have a buffer between each resource– Apply fair queueing to each resource
• Per-Resource Fairness not strategy-proof– 2 resources, 2 flows <4,1>, <1,2>
– Flow 1 changes demand to <4,2>
CPU NIC0%
100%
50%
86%Flow 1
Flow 243%
14%57%
CPU NIC0%
100%
50%
33% 66%
66% 33%
![Page 11: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/11.jpg)
12
Problems with Per-Resource Fairness
• PRF violates strategy-proofness– Can be manipulated by wasting
resources
• PRF requires per-resource queues– Problematic for parallel resource
consumptione.g. CPU and memory consumption in a module
![Page 12: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/12.jpg)
13
Why care about strategy-proofness?
• Lack of strategy-proofness encourages wastage– Decreasing goodput of the system
• Networking applications especially savvy– Peer-to-peer apps manipulate to get more resources
• Trivially guaranteed for single resource fairness– But not for multi-resource fairness
![Page 13: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/13.jpg)
14
Summary of Policies
Policy Share Guarantee
Strategy-Proofness
Fair Queueing a Single Resource
Bottleneck Fairness
Per-Resource Fairness X
Dominant Resource Fairness
X X
![Page 14: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/14.jpg)
15
Outline
• Analysis of Natural Policies• DRF allocations in Space• DRFQ: DRF allocations in Time• Implementation/Evaluation
![Page 15: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/15.jpg)
16
Dominant Resource Fairness
• DRF originally in the cloud computing context– Satisfies share guarantee– Satisfies strategy-proofness
![Page 16: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/16.jpg)
17
DRF Allocations• Dominant resource of a user is the resource she is
allocated most of – Dominant share is the user’s share of her dominant resource
• DRF: apply max-min fairness to dominant shares– ”Equalize” the dominant share of all users
Total resources: <16 CPUs, 16 GB mem>User 1 demand: <3 CPU, 1 GB mem> dom res: CPUUser 2 demand: <1 CPU, 4 GB mem> dom res: mem
User 2User 1100%
50%
0%CPU mem
3 CPUs 12 GB
12 CPUs
4 GB
66%
66%
![Page 17: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/17.jpg)
18
Allocations in Space vs Time
• DRF provides allocations in space– Given 1000 CPUs and 1 TB mem, how
much to allocate to each user
• DRFQ provides DRF allocations in time–Multiplex packets to achieve DRF
allocations over time
![Page 18: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/18.jpg)
19
Outline
• Analysis of Natural Policies• DRF allocations in Space• DRFQ: DRF allocations in Time• Implementation/Evaluation
![Page 19: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/19.jpg)
20
Determining Packet Resource Consumption
• A-priori packet link usage known in FQ– Packet size divided by throughput of link
• Packet processing time a-priori unknown for multi-resources– Depends on the modules that process it
• Leverage Start-time Fair Queueing (SFQ)– Schedules based on virtual start time of packets– Start time of packet p independent of resource
consumption of packet p
![Page 20: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/20.jpg)
21
Memoryless Requirement
• Virtual Clock simulates flows with dedicated 1/n link– Attach start and finish tags according to dedicated link– Serve packet with smallest finish tag (work conserving)
• Problem– During light load a flow might get more than 1/n– That flow experiences long delays when new flows start
• Requirement: memoryless scheduling– A flow’s share of resources should be independent of its
share in the past
![Page 21: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/21.jpg)
slop
e 2
Memoryless through Virtual Time
• Virtual time to track amount service received– A unit of virtual time
always corresponds to sameamount of service
• Example with 2 flows– Time 0: one backlogged flow– Time 20: two backlogged flows
• Schedule the packets according to V(t)– Assign virtual start/finish time when packet
arrives
Real Time
Vir
tual Ti
me
V(t
)
20
40 60 80
40
60
20
slope
1
![Page 22: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/22.jpg)
23
Dove-tailing Requirement
• FQ: flow size should determine service, not packet size– Flow with 10 1kb packets gets same service as 5 2kb
packets
• Want flow processing time, not packet processing time– Example: give same service to these flows:
Flow 1: p1 <1,2>, p2 <2,1>, p3 <1,2>, p4 <2,1>, …Flow 2: p1 <3,3>, p2 <3,3>, p3<3,3>, p4 <3,3>, …
• Requirement: dove-tailing– Packet processing times should be independent of how
resource consumption is distributed in a flow
![Page 23: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/23.jpg)
24
Tradeoff
• Dovetailing and memoryless property at odds– Dovetailing needs to remember past
consumption
• DRFQ developed in three steps–Memoryless DRFQ: uses a single virtual time– Dovetailing DRFQ : use virtual time per resource– DRFQ: generalizes both, tradeoff between
memoryless and dovetailing
![Page 24: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/24.jpg)
25
Memoryless DRFQ
• Attach a virtual start and finish time to every packet– S(p) and F(p) of packet p
• Computing virtual finish time F(P)1. F(p) = S(p) + maxi{ p_time(p, i) }
p_time(p, i) = { processing time of p on resource i }
• Computing virtual start time, S(p)2. S(p) = max( F(p-1), C(t) )
C(t) = { max start time of currently serviced packet }
• Service the packet with minimum virtual start time
![Page 25: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/25.jpg)
Memoryless DRFQ example• Two flows become backlogged at time 0– Flow 1 alternates <1,2> and <2,1> packet processing– Flow 2 uses <3,3> packet processing time
1. F(p) = S(p) + maxi{ p_time(p, i) }
2. S(p) = max( F(p-1), C(t) )
27
Flow 1 P1S: 0 F: 2
Flow 1 P2S: 2 F: 4
Flow 1 P3S: 4 F: 6
Flow 1 P4S: 6 F: 8
Flow 1 P5S: 8 F: 10
Flow 2 P1S: 0 F: 3
Flow 2 P2S: 3 F: 6
Flow 2 P3S: 6 F: 9
Flow 1 gets worse service than Flow 2
![Page 26: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/26.jpg)
28
Dovetailing DRFQ
• Keep track of start and finish time per resource– Use max start time per resource for
scheduling– Dovetail by keeping track of all resource
usage
![Page 27: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/27.jpg)
Dovetailing DRFQ example
29
Flow 1S1: 0 F1: 1S2: 0 F2: 2
Flow 1S1: 1 F1: 3S2: 2 F2: 3
Flow 1S1: 3 F1: 4S2: 3 F2: 5
Flow 1S1: 4 F1: 6S2: 5 F2: 6
Flow 1S1: 6 F1: 7S2: 6 F2: 8
Flow 2S1: 0 F1: 3S2: 0 F2: 3
Flow 2S1: 3 F1: 6S2: 3 F2: 6
Flow 2S1: 6 F1: 9S2: 6 F2: 9
Dovetailing ensures both flows get same service
• Two flows become backlogged at time 0– Flow 1 alternates <1,2> and <2,1> packet
processing– Flow 2 uses <3,3> per packet
![Page 28: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/28.jpg)
30
DRFQ algorithm
• DRFQ bounds dovetailing to Δ processing time– Dovetail up to Δ processing time units– Memoryless beyond Δ
• DRFQ is a generalization– When Δ=0 then DRFQ=memoryless DRFQ– When Δ=∞ then DRFQ=dovetailing DRFQ
• Set Δ to a few packets worth of processing
![Page 29: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/29.jpg)
31
Outline
• Analysis of Natural Policies• DRF allocations in Space• DRFQ: DRF allocations in Time• Implementation/Evaluation
![Page 30: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/30.jpg)
32
Isolation Experiment• DRFQ Implementation in Click– 2 elephants: 40K/sec basic, 40K/sec
IPSec– 2 mice: 1/sec basic, 0.5/sec basic
Non-backlogged flows isolated from backlogged flows
![Page 31: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/31.jpg)
33
Simulating Bottleneck Fairness
• 2 flows and 2 res. <CPU, NIC> – Demands <1,6> and <7,1>
bottleneck unclear
• Especially bad for TCP and video/audio traffic
![Page 32: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/32.jpg)
34
Summary
• Packet processing becoming evermore sophisticated– Consume multiple resources
• Natural policies not suitable– Per-Resource Fairness (PRF) not strategy-proof– Bottleneck Fairness doesn’t provide isolation
• Proposed Dominant Resource Fair Queueing (DRFQ)– Generalization of FQ to multiple resources– Generalizes virtual time to multiple resources– Provides tradeoff between memoryless and dovetailing– Provides share-guarantee (isolation) and strategy-
proofness
![Page 33: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/33.jpg)
35
![Page 34: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/34.jpg)
36
![Page 35: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/35.jpg)
37
Overhead
• 350 MB trace run through our Click implementation
• Evaluate overhead of two modules– Intrusion Detection, 2% overhead– Flow monitoring, 4% overhead
![Page 36: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/36.jpg)
38
Determining Resource Consumption
• Resource consumption obvious in routers– Packet size divided by link rate
• Generalize consumption to processing time– Normalized time a resource takes to process
packet
• Normalized processing time– e.g. 1 core takes 20μs to service a packet,
on a quad-core the packet processing time is 5μs– Packet processing time ≠ packet service time
![Page 37: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/37.jpg)
39
Module Consumption Estimation
• Linear estimation of processing time– For module m and resource r as function of
packet size
• R2 > 0.90 for most modules
![Page 38: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/38.jpg)
40
Simulating Bottleneck Fairness
• 2 flows and 2 res. <CPU, NIC> – Demands <1,6> and <7,1> bottleneck
unclear
– CPU bottleneck: 7×<1,6> + <7,1> = <14,43>
– NIC bottleneck:<1,6> + 6×<7,1> = <43, 12>– Periodically oscillates the bottleneck
![Page 39: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/39.jpg)
41
TCP and oscillations
• Implemented Bottleneck Fairness in Click– 20 ms artificial link delay added to simulate WAN– Bottleneck determined every 300 ms– 1 BW-bound flow and 1 CPU-bound flow
Oscillations in Bottleneck degrade performance of TCP
![Page 40: Documentnw](https://reader033.vdocument.in/reader033/viewer/2022061110/54540a12b1af9f880a8b482c/html5/thumbnails/40.jpg)
42
Multi-Resource Consumption Contexts
• Different modules within a middlebox– E.g. Bro modules for HTTP, FTP, telnet
• Different apps on a consolidated middlebox– Different applications consume different
resources
• Other contexts– VM scheduling in hypervisors– Requests to a shared service (e.g. HDFS)