1

FCP: A Flexible Transport Framework for Accommodating Diversity

Dongsu Han (KAIST)Robert Grandl† , Aditya Akella†,

Srinivasan Seshan*

† University of Wisconsin-Madison*Carnegie Mellon University

2

Evolution of Transport Protocols

Network (IP)Transport

{ Reliability/Loss recoveryIn-order deliveryFlow controlCongestion control

3

Evolution of Transport Protocols

Multipath

Transport

Quality of Experience(QoE)

4

Evolution of Transport Protocols

Multipath

Transport

Quality of Experience(QoE)

Support for diversity:Multi-homing, multiple paths,

multiple sub-streams,multimedia streaming,

Web applications,Data transfer with a deadline

5

Evolution of Transport Protocols

Transport

{ Reliability/Loss recoveryIn-order deliveryFlow controlCongestion controlCongestion control

6

Evolution of Transport Protocols

Congestion control

Transport

Resource Allocation

Fairness and efficiency

Coordination

7

Problems in Supporting Diversity in Congestion Control

Cannot ensure coexistence.

RCP

TCPD3

[sigcomm]

XCP [sigcomm]

8

Evolution of TCPSACK (1996)NewReno (1999)

MulTCPExplicit Congestion Notification

TCP NiceFastTCP

TCP window-scaling, 1988

TCP BIC, CUBIC 2008

MPTCP2010

TCP Friendly Rate Control

1. End-point flexibility: Purely end-point based2. Coexistence: Invariant for fairness (TCP friendliness)

MPTCP

TFRC

NewReno

CUBIC

9

End-point based vs. Router-AssistedHighFlexibility,Diversity

End-point based [TCP]

High Efficiency

Router-Assisted [XCP, RCP]

Feedback on network’s state

Can we achieve the best of both worlds?

10

Our Approach

1. Decouple coexistence issues (fairness and efficiency) from end-point control

Flex

ible

Con

trol

Fairn

ess

Effici

ency

End-host Network

2. Introduce generic abstractions for resource allocation.

Network’s state (feedback)

End-point’s future behavior (feed-forward)

11

Flow 1

Budget W ($/sec)

Flow i’s price: Pi ($/Byte)

Sender ReceiverNetwork

...Flow 2

Flow nSender can distribute its budget to its flows. Flow i’s budget wi,

subject to W ≥ Σwi

Flow i’s rate: Ri = budget/price = wi /Pi (Byte/sec)

PRICE

Decoupling for Flexibility1

12

Flow 1

Budget W ($/sec)

Sender ReceiverNetwork

...Flow 2

Flow n

Flow i’s budget wi,

subject to W ≥ Σwi

PRICE

Decoupling for Flexibility1

1. Flexibility at the end-points in how its budget is used.

2. Flexibility in network price generation.

Invariant for fairness

13

• Feedback: “congestion price” reflecting the “cost” of sending data across the link [Kelly]

P1P2P3

Feedback: Pricing2a

Price = p3

Round Trip Time

Price = p3+p2

Round Trip Time

Price = p3+p2+p1

Round Trip Time

Congestion Header Congestion Header Congestion Header

14

• Feedback: “congestion price” reflecting the “cost” of sending data across the link [Kelly]

P1P2P3

Feedback: Pricing2a

Price feedback = p3+p2+p1

PriceEchoback

Sender updatesthe rate.Rate = budget/price

This implements proportional fairness [Kelly].

15

Price Calculation

Router updates the price, upon packet reception.

Time

Load

Averaging window = 2 x AvgRTT

Feedback: Pricing2a

Recent load

f(Average recent load)Price ($/byte) =

16

Price CalculationRouter updates the price, upon packet reception.

Time

Instantaneous Incoming budget ($)= Price ($/byte) x Bytes Received (bytes)

Averaging window = 2 x AvgRTT

Feedback: Pricing2a

Incoming budget

Average Incoming BudgetRemaining Link Capacity

Price ($/byte) =

17

Price Calculation

Time

Instantaneous Incoming budget ($)

Average Incoming BudgetRemaining Link Capacity

Price ($/byte) =

Feedback: Pricing2a

New Price

Round Trip Time (rtt)

Incoming budget

I(t)=

Router stores recent history of price,

18

Feed-forward: Preloading2b

1 ($/sec) 10 ($/sec)Sender

PRICE

Price: p $/byteRate = 1/p Bps

Rate = 10/p Bps

19

Feed-forward: Preloading2b

1 ($/sec) 10 ($/sec)Sender

Rate = 1/p

Rate = 10/p’

Price

Round Trip Time

Preload=9

Price

Round Trip Time

Preload=9

(1+preload) x price(t-rtt) x bytes

Inco

min

g bu

dget

($)

Price goes up (from p to p’)

Congestion Header

20

Evaluation

Flow 1

Budget W ($/sec)

...Flow 2

Flow n

PRICE

3. Flexibility in network price generation.

2. End-point budget allocation.

1. Basic performance of FCP

21

Fast Convergence/Accurate Feedback Se

ndin

g Ra

te (M

bps)

Time

RTT = {25ms, 50ms, 125ms, 250ms, 625ms }

22

Fast Convergence/Accurate Feedback Se

ndin

g Ra

te (M

bps)

Time

Ideal RTT = {25ms, 50ms, 125ms, 250ms, 625ms }

23

Fast Convergence/Accurate Feedback Se

ndin

g Ra

te (M

bps)

Time

Ideal RTT = {25ms, 50ms, 125ms, 250ms, 625ms }

24

Fast Convergence/Accurate Feedback Se

ndin

g Ra

te (M

bps)

Time

Ideal RTT = {25ms, 50ms, 125ms, 250ms, 625ms }

25

Fast Convergence/Accurate Feedback Se

ndin

g Ra

te (M

bps)

Time

Ideal RTT = {25ms, 50ms, 125ms, 250ms, 625ms }

26

Fast Convergence/Accurate Feedback Se

ndin

g Ra

te (M

bps)

Time

Ideal RTT = {25ms, 50ms, 125ms, 250ms, 625ms }

27

Fast Convergence/Accurate Feedback

Time

FCP

Ideal

Send

ing

Rate

(Mbp

s)Se

ndin

g Ra

te (M

bps)

28

Fast Convergence/Accurate Feedback

Time

FCP

RCPIdeal

Send

ing

Rate

(Mbp

s)Se

ndin

g Ra

te (M

bps)

Send

ing

Rate

(Mbp

s)

Overloaded when new flows arrive

29

Fast Convergence/Accurate Feedback

Fairness AIMD

Time

FCP

RCP

XCP

Ideal

Send

ing

Rate

(Mbp

s)Se

ndin

g Ra

te (M

bps)

Send

ing

Rate

(Mbp

s)Se

ndin

g Ra

te (M

bps)

Overloaded when new flows arrive

30

Local Stability of FCP

• Random perturbation of [-30,30]% introduced every second for a duration of 100 ms.

31

Faster Convergence/Accurate Feedback

FCP’s converges faster and the feedback is more accurate.

Fan-

out =

5

…

100Mbps

Slower, inaccurate

Slowest. Fairness AIMD

More results in the paper:- Fairness, efficiency- Local stability- Effectiveness of preloading- Overhead of price calculation

32

Budget Determines the Fair-share.

Fan-out =50…

Host A Host B

100Mbps

1 ($/sec) 1 ($/sec)

33

Local Stability of FCP

• Random perturbation of [-30,30]% introduced every second for a duration of 100 ms.

3434

Accurate feedback with preloading• Double the load every RTT from t= 0 to 1 sec.• RCP: double the # of flows• FCP (our design): double the budget

35

Accurate feedback with preloading• Double the load every RTT from t= 0 to 1 sec.• RCP: double the # of flows• FCP (our design): double the budget

36

End-point Flexibility: Topology

Sender0 (S0)Budget 1

Sender1 (S1)Budget 2

Sender2 (S2)Budget 1

Receiver0

Receiver1

Round-trip latency : 12 ms

250 Mbps link

50 Mbps link200 Mbps links

Top flow

Btm flow

37

Budget allocation is up to end-points

Budget of S1’s top flow ($/sec)

S1’s

Thro

ughp

ut (M

bps)

Total

38

Budget allocation is up to end-points

Budget of S1’s top flow ($/sec)

S1’s

Thro

ughp

ut (M

bps)

Total Top flow Bottom flow

Max-throughput

Equal-throughput

Equal-budget

39

End-point Diversity: Background flows

Non-bottleneckLink capacity: 200Mbps

100Mbps50Mbps

Flow1 Flow2 (Background)Flow3

Flow0

Link latency: 3ms

Flow2 (Background)

40

Diversity in Network Pricing

• FCP can also support diverse behaviors in network price generation.

• Examples (in the paper)– Deadline support [D3 SIGCOMM 11]– Aggregate resource allocation in a multi-tenant

data-center– Stable bandwidth allocation for streaming– Multicast congestion control

41

Deadline Support [D3 SIGCOMM 11]

Deadline Queue

Best Effort Queue

Price

Time

Constant Pricing

Preload

Price

Time

Variable Pricing

Deadline flows preload to specify the desired rate

Average Incoming BudgetRemaining Link Capacity

Price ($/byte) =

42

Differential Pricing for Deadline Support

Constant Pricing for Deadline Flows

Pric

ePr

ice

Constant Pricing for Deadline Flows

Variable Pricingfor BE flows

43

Differential Pricing for Deadline Support

Dead flows

Rejected/BE flows

Best effort (BE)

Pric

e

Constant Pricing for Deadline Flows

Variable Pricingfor BE flows

Price increase

44

Packet processing performance

Per-core FCP throughput : 2.6Mpps (1.25 Gbps @ 64 bytes)Per-core Click baseline: 6.1 Mpps (3.1 Gbps @64bytes)

45

Packet processing performance

About 1/3 spent on actual price calculation1/3 spent on clock_gettime (can be offloaded to h/w)

46

Summary

• FCP accommodates diverse behaviors in resource allocation while utilizing explicit feedback.

• FCP maximizes end-point’s flexibility by simplifying the mechanism of coexistence.

• FCP’s explicit feed-back and feed-forward provides a generic interface for efficient resource allocation.

47

Research Theme

• Cooperative model for resource allocation.• Network knows information that end-points

do not know.• End-points know information that the network

does not know. • Can we benefit from a new interface between

the network and the end-points?

48

Any more examples?

49

Mobile Networks

• Unique properties:– Under a single administrative domain– A complete ecosystem.

50

51

• Prof. Song

52

What is research?

• People have their own style: “diversity”• What is your “uniqueness”?

Understanding Tradeoffs in Incremental Deployment of New Network Architectures CoNEXT 2013FCP: A Flexible Transport Framework for Accommodating Diversity Sigcomm 2013CAMEO: A Middleware for Mobile Advertisement Delivery MobiSys 2013Supporting Long Term Evolution in an Internet Architecture Ph.D. ThesisRPT: Re-architecting Loss Protection for Content-Aware Networks NSDI 2012XIA: Efficient Support for Evolvable Internetworking NSDI 2012

53

Conclusion

54

55

Budget allocation is up to end-points

Budget of S1’s top flow ($/sec)

S1’s

Thro

ughp

ut (M

bps)

Total Top flow Bottom flow

56

Equal-throughput

Budget allocation is up to end-points

Budget of S1’s top flow ($/sec)

S1’s

Thro

ughp

ut (M

bps)

Total Top flow Bottom flow

57

Budget Determines the Fair-share.

Fan-out =50…

Host A Host B

100Mbps

1 ($/sec) 1 ($/sec)

58

Misbehaving Hosts

Detection of misbehaving hosts: Expected incoming budget ≠ incoming budgetIdentification: Need per-host state.

59

Packet processing performance

Per-core FCP throughput : 2.6Mpps (1.25 Gbps @ 64 bytes)Per-core Click baseline: 6.1 Mpps (3.1 Gbps @64bytes)

60

Packet processing performance

About 1/3 spent on actual price calculation1/3 spent on clock_gettime (can be offloaded to h/w)

61

Memory Overhead

• Implementation stores 500 msec of history at 10 usec granularity = 50K entries

Time

Price ($/byte), Incoming Budget($)

More recent

62

Local Stability of FCP

• Random perturbation of [-30,30]% introduced every second for a duration of 100 ms.

63

Incremental Deployment

TCP Queue

Legacy TCP link

FCP link

TCP

FCP

FCP segment

1. Assign budget on aggregate TCP flows.2. TCP flows are treated as a single FCP flow.3. The mapped FCP flow’s rate determines the TCP queue’s drain rate.

FCP feedback

TCP

64

FCP and RCP

65

Short Flow’s Completion Time

6666

Accurate feedback with preloading• Double the load every RTT from t= 0 to 1 sec.• RCP: double the # of flows• FCP (our design): double the budget

67

Accurate feedback with preloading• Double the load every RTT from t= 0 to 1 sec.• RCP: double the # of flows• FCP (our design): double the budget

68

Budget Determines the Fair-share.

Fan-out =50…

Host A Host B

100Mbps

1 ($/sec) 1 ($/sec)

69

Requirements for CC Diversity (1)1) Flexible control at the end-points to handle

new requirements.

Maximize throughput with multiple pathsSupport deadlines [sigcomm 2011,2012]Support background flowsOptimize for Quality of Experience [sigcomm2013]

High-level Application Objectives

High-level Application Objectives

70

High-level Application Objectives

Requirements for CC Diversity (2)

1) Flexible control at the end-points2) Ensure coexistence (fairness and effciency)

Fairness and efficiency

71

Aggregate Control in a Multi-tenant Data-center

ToR Switch

20 Servers/Rack

VM1 VM2

VM3 VM4

VM39 VM40

…

Gro

up 0

Gro

up 1

Two tenants with the same weight sharing a rack. Each of their flow has equal budget. The network dynamically translates the value.

A rack with 20 servers

1¥/sec 20 $/sec

20 $/sec

1 ¥/secLink price: 1 €/bit

Weight 1

Weight 1

Price feedback (¥): 1 €/bit x exchange ¥ /€

Price feedback ($): 1 €/bit x exchange $/€

72

Aggregate Control in a Multi-tenant Data-center

ToR Switch

20 Servers/Rack

VM1 VM2

VM3 VM4

VM39 VM40

…

73

Feed-forward: Preloading2b 1 ($/sec)

PRICE

10 ($/sec)Sender

PRICE Budget = $10/sec

Preload = PRELOAD PRELOAD PRELOAD

PRELOAD PRELOAD PRELOAD

PRICE PRICE

PRICE PRICE

74

• Feedback: “congestion price” reflecting the “cost” of sending data across the link [Kelly]

P1P2P3

Total Price = p1+p2+p3PRICE DATA

PRICE ACK

PriceEchoback

Sender updatesthe rate.Rate = budget/price

This implements proportional fairness [Kelly].

Feedback: Pricing2a

75

Deadline Support [D3 SIGCOMM 11]

1) Flows specify the desired rate by preloading.2) The network admits a flow if it is within its capacity, otherwise the flow gets a best effort service. – Two class queuing at the network (deadline, BE)– Deadline (BE) flows get fixed (variable) pricing.

Admitted deadline flows

Rejected deadline flows

Best effort

76

Faster Convergence/Accurate Feedback

FCP’s converges faster and the feedback is more accurate.

Fan-

out =

5

…

100Mbps

Overloaded when new flows arrive

Fairness AIMD

77

Price CalculationRouter updates the price, upon packet reception.

Time

Instantaneous Incoming budget ($)= Price ($/byte) x Bytes Received (bytes)

Averaging window = 2 x AvgRTT

Average Incoming BudgetRemaining Link Capacity

Price ($/byte) =

Feedback: Pricing2a

Incoming budget

I(t)=