fast tcp speaker: ray veune: room 1026 date: 25 th october, 2003 time:10:00am

FAST TCPFAST TCP

Speaker: Speaker: RayRayVeune: Veune: Room 1026Room 1026Date: Date: 2525thth October, 2003 October, 2003Time:Time: 10:00am10:00am

MotivationMotivation

Demand for ultrascale networkingDemand for ultrascale networking HENP (High Energy and Nuclear Physics)HENP (High Energy and Nuclear Physics) Data volumes of tens of Petabytes (10Data volumes of tens of Petabytes (101515) to ) to

Exabytes (10Exabytes (101818)) Require Terabit/sec (10Require Terabit/sec (1015 15 bit/sec or 1000Gbbit/sec or 1000Gb

it/sec)it/sec) Scalability problem of TCPScalability problem of TCP

Losses must be extremely rareLosses must be extremely rare TCP must induce lossTCP must induce loss Underutilization and oscillationUnderutilization and oscillation

Scalability problem of TCP Scalability problem of TCP extremely loss packet loss extremely loss packet loss

possibilitypossibility Rate = 1.3 * MTU / (RTT * sqrt(Loss))Rate = 1.3 * MTU / (RTT * sqrt(Loss)) MTU = 1500bytes, RTT = 10msMTU = 1500bytes, RTT = 10ms

BandwidthBandwidth Loss rateLoss rate

10Mbps10Mbps 2.43 x 102.43 x 10-2-2

100Mbps100Mbps 2.43 x 102.43 x 10-4-4

1Gbps1Gbps 2.43 x 102.43 x 10-6-6

10Gbps10Gbps 2.43 x 102.43 x 10-8-8

1Tbps1Tbps 2.43 x 102.43 x 10-12-12

1Tbps (RTT:100ms)1Tbps (RTT:100ms) 3.80 x 103.80 x 10-14-14

Scalability problem of TCP Scalability problem of TCP inevitable packet lossinevitable packet loss

TCP needs to create lossesTCP needs to create losses Single bit network feedback signalSingle bit network feedback signal

Scalability problem of TCP Scalability problem of TCP Underutilization and oscillationUnderutilization and oscillation

AIMD (1, 0.5)AIMD (1, 0.5) At large window size (in excess of 10,0At large window size (in excess of 10,0

00 pkts):00 pkts): Halving window on loss event is too drastiHalving window on loss event is too drasti

cc Increasing window by one packet per RTT Increasing window by one packet per RTT

is too conservativeis too conservative

FAST TCPFAST TCPAchievementAchievement

CERN (European Organization for NucCERN (European Organization for Nuclear Research) sent 1.1 Terabytes of dalear Research) sent 1.1 Terabytes of data at 5.44 Gbpsta at 5.44 Gbps

Full-length DVD film in 7 seconds !!Full-length DVD film in 7 seconds !!

FAST TCPFAST TCP

Flow based vs Packet basedFlow based vs Packet based Network delay vs Packet lossNetwork delay vs Packet loss TCP-Vegas vs TCP-RenoTCP-Vegas vs TCP-Reno Stabilized VegasStabilized Vegas

TCP VegasTCP VegasTechniquesTechniques

New Retransmission MechanismNew Retransmission Mechanism Congestion Avoidance MechanismCongestion Avoidance Mechanism Modified Slow-Start MechanismModified Slow-Start Mechanism

TCP VegasTCP VegasNew Retransmission MechanismNew Retransmission Mechanism

TimeoutTimeout nn duplicate ACKs duplicate ACKs

TCP VegasTCP VegasNew Retransmission MechanismNew Retransmission Mechanism

Check time record fCheck time record for the first duplicate or the first duplicate packetpacket

non-duplicate ACKs non-duplicate ACKs first or second after first or second after retransmissionretransmission Catch other segment Catch other segment

lost previous to retralost previous to retransmissionnsmission

TCP VegasTCP VegasCongestion Avoidance MechanismCongestion Avoidance Mechanism

Detect network delay by monitoring RDetect network delay by monitoring RTTTT

BaseRTT and ActualRTTBaseRTT and ActualRTT

Source Dest.Router

Expected = WindowSize / BaseRTTExpected = WindowSize / BaseRTT

Diff = Expected - ActualDiff = Expected - Actual Diff >> 0, Diff >> 0, decrease sending ratedecrease sending rate Diff = 0, Diff = 0, increase sending rateincrease sending rate

αα < Diff < < Diff < ββ


αα ββ DiffDiff


Extra buffers occupiedExtra buffers occupied BaseRTT: 100ms, segment size: 1KBBaseRTT: 100ms, segment size: 1KB

Expected = WindowSize / BaseRTTExpected = WindowSize / BaseRTTαα = 30KB/s, = 30KB/s, ββ=60KB/s=60KB/s

αα => 30KBps * 100ms / 1KB = 3=> 30KBps * 100ms / 1KB = 3 ββ=> 60KBps * 100ms / 1KB = 6=> 60KBps * 100ms / 1KB = 6


αα==ββ Diff <Diff <αα

increase one segment per RTTincrease one segment per RTT Diff =Diff =αα

no change in windows sizeno change in windows size Diff >Diff >αα

decrease one segment per RTTdecrease one segment per RTT

TCP VegasTCP VegasSlow-Start MechanismSlow-Start Mechanism

TCP-renoTCP-reno Send two segment for each ACK receivedSend two segment for each ACK received Exponential growth every RTTExponential growth every RTT

TCP-VegasTCP-Vegas Exponential growth every alternative RTTExponential growth every alternative RTT γγthresholdthreshold

Diff >Diff >γγ Changes from slow-start mode to linear I/D Changes from slow-start mode to linear I/D

modemode

Stability of TCP Vegas Stability of TCP Vegas Network modelNetwork model

Set of Set of LL links with finite capacities links with finite capacities cc c = c = ((ccl l , l , l L L))

NN sources indexed by sources indexed by rr Each source r uses a set of link defined Each source r uses a set of link defined

by the by the LL NN routing matrix routing matrix

RRlrlr = { = { 1 if source r uses link l0 otherwise

Stability of TCP VegasStability of TCP VegasNetwork modelNetwork model

For each link For each link ll, the congestion , the congestion measure measure ppll((tt) is call price) is call price

For each source For each source rr, it maintains a , it maintains a rate rate xxrr((tt) in packets/sec) in packets/sec

Equilibrium forward delay from Equilibrium forward delay from source source rr to link to link ll : :

Equilibrium backward delay from Equilibrium backward delay from link link ll to source to source rr : :

lr

lr


Aggregate price source Aggregate price source rr observes in its pat observes in its pathh qqrr((tt) = ) = RRlr lr ppl l ((t t - - ) )

Aggregate source rate link Aggregate source rate link ll observes observes yyll ((tt) = ) = RRlrlr x xrr ((t -t - ) )

x1(t)

x2(t)

p1(t)

p2(t)

p3(t) p4(t)

llr

rlr


Tr denote equilibrium RTTTr denote equilibrium RTT + + = = TrTr,, ll LL

Dynamical system of TCP VegasDynamical system of TCP Vegas

ppll ( (tt) =) =( ( yyl l ( ( t t ) – ) – ccll ) / ) / ccll if if ppll ( (tt) > 0) > 0( ( yyl l ( ( t t ) – ) – ccll ) / ) / ccl l ) )++ if if ppll ( (tt) = 0) = 0

xxrr ( (tt) = 1/) = 1/TrTr 22(t) sgn( 1 – (t) sgn( 1 – xxrr((tt))qqrr((tt) / ) / rrddrr ) )

TrTr ( (tt) = ) = ddrr + + q qrr( ( tt ) )

WhereWhere sgn(z) = 1 if z > 0, -1 if z < 0 and 0 if z = 0sgn(z) = 1 if z > 0, -1 if z < 0 and 0 if z = 0(z)(z)++ = max { 0 , z } = max { 0 , z }

lr lr

Stability of TCP Vegas Stability of TCP Vegas Approximate modelApproximate model

xxrr ( (tt) = 1/) = 1/TrTr 22(t) sgn( 1 – (t) sgn( 1 – xxrr((tt))qqrr((tt) / ) / rrddrr ) ) sgn(z) sgn(z) 2/ 2/ tan tan-1-1 ( (z)z)

xxrr ( (tt) = ) = ((2/2/TrTr 22(t)(t)))tantan-1-1 (1 – (1 – xxrr((tt))qqrr((tt) / ) / rrddrr ) )

Stability of TCP Vegas Stability of TCP Vegas Approximate modelApproximate model

In equilibrium, the source rate In equilibrium, the source rate xxrr** and and

aggregate price aggregate price qqrr** satisfy satisfy

xxrr** q qrr

** = = r r ddrr

Stability of TCP Vegas Stability of TCP Vegas Theorem 1Theorem 1

Suppose for all Suppose for all rr, , kk00TrTr max maxrr TrTr for some for some kk00.. Let M be an upper bound on the number of linkLet M be an upper bound on the number of link

s in the path of any source, M s in the path of any source, M max maxrr ll RRlrlr.. The Vegas model is locally asymptotically stable The Vegas model is locally asymptotically stable

around the equilibrium point (around the equilibrium point (xxrr** , , yyll

* * , , ppll** , , qqrr

** ) if ) if

maxmaxr r xxrr** Tr sinc Tr sinc (ň / x (ň / xrr

** Tr ) < Tr ) < / Mk / Mk0022

ň = 2ň = 2// Let Let (a) be the unique solution in ( 0, (a) be the unique solution in ( 0, /2) of/2) of tan tan = a as a strictly i = a as a strictly i

ncreasing function of ancreasing function of a ssinc inc = sin = sin / /

Stability of TCP Vegas Stability of TCP Vegas Theorem 1Theorem 1


** Tr ) < Tr ) < / Mk / Mk0022

() is strictly increasing() is strictly increasing sincsinc() is strictly decreasing() is strictly decreasing LHS is strictly increasing in windows sLHS is strictly increasing in windows s

ize ize xxrr** Tr Tr

Theorem 1: Stability condition impose Theorem 1: Stability condition impose a limit on max windows sizea limit on max windows size

Stability of TCP Vegas Stability of TCP Vegas Corollary 2Corollary 2


** Tr ) < Tr ) < / Mk / Mk0022

All source has the same target queue length, All source has the same target queue length, r r ddr r = = for all for all rr

Corollary 2: LHS is strictly increasing in Corollary 2: LHS is strictly increasing in qqrr** / / TT

r r , implying a lower bound on queueing delay, implying a lower bound on queueing delay

qr*qr* / /Tr Tr

sinc sinc ( ) ( )TrTrqr*qr*ňň

..

minminr r > Mk> Mk0022

Stability of TCP Vegas Stability of TCP Vegas Corollary 3Corollary 3

SinceSince () < () < / 2, sinc / 2, sinc () > 2 / () > 2 / , , kk0 0 1 1 Corollary 3: minCorollary 3: minrr qqrr

** / / Tr > 2M / Tr > 2M / If MIf M 2, then RHS bigger than 1, since 2, then RHS bigger than 1, since Tr = dTr = drr + +

qqrr**

M = 1M = 1 The stability condition cannot be satisfied if a soThe stability condition cannot be satisfied if a so

urce has more than one linkurce has more than one link Sufficient in multilink caseSufficient in multilink case Necessary and sufficient in single-link-homogenNecessary and sufficient in single-link-homogen

eous-source caseeous-source case

Stability of TCP Vegas Stability of TCP Vegas Single link with homogeneous sourceSingle link with homogeneous source

A single link of capacity A single link of capacity cc,, Shared by N homogeneous source,Shared by N homogeneous source, with round trip propagation delay with round trip propagation delay dd

Stability of TCP Vegas Stability of TCP Vegas Single link with homogeneous sourceSingle link with homogeneous source

From corollary 3: From corollary 3: qqrr** / / Tr > 2 / Tr > 2 / for all for all rr

Tr / qTr / qrr** < < /2, /2, since since Tr = d + qTr = d + qrr

* *

d / qd / qrr* * < (< (/2 – 1) => /2 – 1) => d < (d < (/2 – 1) /2 – 1) qqrr

**

Since Since qqrr* * = = / / xxrr

** => => (( N)/c N)/c cd < (cd < (/2 – 1) /2 – 1) N N Conclusion: bandwidth delay product sConclusion: bandwidth delay product s

hould be smallhould be small

Stabilized VegasStabilized Vegas

xxrr ( (tt) = ) = ((2/2/TrTr 22(t)(t)))tantan-1-1 (1 – (1 – xxrr((tt))qqrr((tt) / ) / rrddrr ) ) xxrr ( (tt) = ) = ((ww//TrTr 22(t)(t)))tantan-1-1 rr((tt))((1 – 1 – xxrr((tt))qqrr((tt)/)/rrddrr - -rr((tt) ) qqrr

((tt))))

1 – 1 – xxrr( ( t t ) ) qqrr( ( t t ) /) /r r ddrr

1 – 1 – xxrr( ( t t ) ) qqrr( ( t t ) /) /r r ddrr - -rr( ( t t ) ) qqrr( ( t )t )

rr( ( t t ) = (1 / ) = (1 / ) ) (( Tr( t ) / q Tr( t ) / qrr( ( t ) t ) )) rr( ( t t ) = ( ) = ( / w ) / w ) (( x xrr( ( t t ) ) Tr( t ) Tr( t ) ))


The gain The gain rr( ( t t ) serves as a normalization ) serves as a normalization to to qqrr( ( t )t )

Additional differential term Additional differential term rr( ( t t ) ) qqrr( ( t t )) anticipates the future of anticipates the future of qqrr( ( t )t )

Without: xWithout: xrr( ( t t ) will increase if ) will increase if xxrr((tt))qqrr((tt)< )< rrddrr

Even Even xxrr((tt))qqrr((tt)/)/rrddrr is small, is small, xxrr( ( t t ) may de) may decrease if prices are rapidly growingcrease if prices are rapidly growing


Stability condition for stabilized VegasStability condition for stabilized Vegas

wherewhere = tan = tan-1-1 (( (2 (2)/(1-)/(1-) ) )) Stabilized Vegas can choose a small Stabilized Vegas can choose a small

( a>0, ( a>0, (0,1) (0,1) )) such that RHS can be larger f such that RHS can be larger for better stability of the original Vegas or better stability of the original Vegas cd < cd < ((/2 – 1) /2 – 1) N N

22 + + 22aa22

22 + a + a22cd < ( - 1 ) N

Simulation ResultsSimulation Results

Reno/Reno/RenoReno

Reno/Reno/VegasVegas

Regas/ReRegas/Renono

Vegas/Vegas/VegasVegas

Throughput Throughput (KB/s)(KB/s)

60/10960/109 61/12361/123 66/11966/119 74/13174/131

Throughput Throughput RatioRatio

1.00/1.01.00/1.000

1.02/1.131.02/1.13 1.10/1.091.10/1.09 1.23/1.201.23/1.20

Retranmissions Retranmissions (KB)(KB)

30/2230/22 43/1.843/1.8 1.5/1.81.5/1.8 0.3/0.10.3/0.1

Retransmit Retransmit RatiosRatios

1.00/1.01.00/1.000

1.43/0.081.43/0.08 0.05/0.820.05/0.82 0.01/0.010.01/0.01One-on-One (300KB and 1MB) TransfersOne-on-One (300KB and 1MB) Transfers

cc = 200KB/s = 200KB/s50ms delay50ms delay


= 20= 20 N = 100 flowsN = 100 flows Fixed packet size of 1KBFixed packet size of 1KB FIFO /w Droptail, queue capacity = 200FIFO /w Droptail, queue capacity = 200

0000 ( a , ( a , ) = ( 0.5 , 0.015 ) ) = ( 0.5 , 0.015 )

Simulation ResultsSimulation Resultscc = 100 pkts/s and = 100 pkts/s and dd = 10ms = 10ms

Experimental ResultsExperimental Results

FAST TCP was first demonstrated publicly in during the SuperComputing Conference (SC2002) in Baltimore, MD, in November 16–22 2002

Caltech-SLAC research team CERN DataTAG StarLight TeraGrid Cisco Level(3).

Experimental ResultsExperimental Results

Experimental ResultsExperimental ResultsThroughput and utilizationThroughput and utilization

#flow#flow ThroughThroughputput

MbpsMbps

utilizatiutilizationon

DelaDelayy

msms

DistanDistancece

kmkm

DuratioDurationn

ss

Bmps Bmps 10101515

TransfTransfer GBer GB

11 925 925 (266)(266)

95% 95% (27%)(27%)

180180 10,03710,037 3,6003,600 9.28 9.28 (2.67(2.67

))

387 387 (111)(111)

22 1,797 1,797 (931)(931)

92% 92% (48%)(48%)

180180 10,03710,037 3,6003,600 18.03 18.03 (9.35(9.35

))

753 753 (390)(390)

77 6,1236,123 90%90% 8585 3,9483,948 21,60021,600 24.1724.17 15,39615,396

99 7,9407,940 90%90% 8585 3,9483,948 4,0304,030 31.3531.35 3,7253,725

1010 8,6098,609 88%88% 8585 3,9483,948 21,60021,600 33.9933.99 21,64721,647 SC2002 FAST experimental result Current TCP implementation in Linux v2.4.18

on Jan 27-28, 2003

FAST TCPFAST TCPConclusionConclusion

Problem of current TCP Problem of current TCP EquilibriumEquilibrium Dynamic problemDynamic problem

FAST TCPFAST TCP Equation-based control with queuing Equation-based control with queuing

delaydelay TCP VegasTCP Vegas Stabilized VegasStabilized Vegas

Q&AQ&A

Thank youThank you

fast tcp speaker: ray veune: room 1026 date: 25 th october, 2003 time:10:00am

Documents