Smoothing the XCP Sender

Abstract—It is known fact that modern IP networks are adopting eXplicit Control Protocol (XCP) as the de-facto standard. In the implementation of XCP, the XCP sender outputs packets in bursts such that all the packets inside the congestion window are outputted without any interval between them. The burst not only increases the queuing delay, but also makes XCP with heterogeneous delays to be unstable. In the present paper, a Smooth XCP (SXCP) is proposed, where the interval of outputting packets at the XCP sender is no less than the round trip time divided by the congestion window. The simulations show that SXCP not only decreases the queuing delay, but also improves utilization and stability.

Keywords: Congestion Control; XCP; RCP; SXCP; Stability

I. INTRODUCTION To share network resources among competitive flows, the

implemented solutions relied mostly on pure end-to-end mechanisms, in particular TCP Congestion Control [1]. As an active queue management algorithm, Random Early Detection (RED) [2] regulates the behavior of total traffic to avoid TCP global synchronization and keep the average queue size low. Along with high-bandwidth optical links and long-delay satellite links, mission critical applications, such as cluster-based storage systems and Grid Computing [3], are in great demand for more efficient bandwidth allocation.

The eXplicit Control Protocol (XCP) [4] introduced a new concept of decoupling utilization control from fairness control and became a de-facto standard in modern IP networks. From the point of view of utilization control, XCP is proved to be stable and efficient regardless of the link capacity and of the round trip time (RTT). From the point of view of fairness control, XCP can implement differential bandwidth allocation algorithms, such as max-min fairness and the shadow prices model [4][5][6].

When new flows arrive randomly and have few packets to send, the Rate Control Protocol (RCP) [7] behaves much closer to the emulation of processor sharing (PS) [8] than XCP. However, the paper [9] demonstrates that RCP demands large buffers at the bottleneck link. On one way, XCP is a type of the ready-and-go scheme. The bottleneck router of XCP allocates parts of the spare bandwidth to source hosts and the source hosts increase their output rate such that the capacity of the bottleneck link is not exceeded. So, the XCP based routers

only require small buffers to absorb the burst traffic. On the contrary, RCP is a type of the go-and-ready scheme. The source hosts output with the fairness rate and the bottleneck link of RCP will become congested when many new flows arrive at a short time interval. Thus a large number of buffers, proportional to the average RTT, are required at the bottleneck router to accommodate the extra traffic due to rate mismatch. As RCP is vulnerable to burst traffic such as Incast [3], XCP should still play an irreplaceable role in explicit congestion control.

In the implementation of XCP [4][10], the sender outputs packets inside the congestion window without any interval between them. The burst not only increases the queuing delay, but also makes it unstable in heterogeneous delays environments [11]. In this paper, a smooth XCP sender, which outputs packet in a reasonable interval is introduced and its design explained.

This paper is organized as follows. In the next section the algorithm and implementation of XCP are introduced, and the unstable cause of XCP with heterogeneous delay will be revealed. Section III will propose the new implementation of XCP, which is called the Smooth XCP (SXCP). The transient processes of XCP, RCP and SXCP are compared. More packet-level simulations are given in section IV, which show that SXCP achieves lower queuing delay, higher utilization and stability compared with XCP.

II. THE INSTABILITY OF XCP IN HETEROGENEOUS DELAYS In the XCP system, there are the XCP sender, the XCP

receiver and the XCP router. The sender inserts the congestion header on the output packets, which includes the current congestion window H_cwnd, the round trip time H_rtt and the desired window increase H_feedback. The XCP router collects the load characteristic from the congestion header and updates the field H_feedback according to its spare bandwidth. The XCP receiver returns the allowed window increase H_feedback. back to the sender with its acknowledgement.

In the implementation, the efficiency controller at the XCP router calculates the aggregate feedback � in each control interval according to (1).

� � � � � � � � � (1)

Zhiqiang Shi Dan Ionescu Dongli Zhang szq@iscas.ac.cn ionescu@site.uottawa.ca dzhang@site.uottawa.ca

Institute of Software SITE SITE Chinese Academy of Sciences University of Ottawa University of Ottawa

Beijing, China Ottawa, Canada Ottawa, Canada

95978-1-4577-1826-7/11/$26.00 ©2011 IEEE ICON 2011

where � and are constant parameter and set to 0.4 and 0.226. The term d is the average round trip time, S is the spare bandwidth and Q is the persistent queue size. The fairness controller at the XCP router allocates the aggregate feedback to the individual flow according to the rule of Additive-Increase Multiplicative- Decrease (AIMD).

When the increase feedback arrives, the XCP sender outputs all the packets inside the expanded congestion window without any interval between them, which is called the burst of the XCP sender in this paper. Although XCP is stable in many cases, it certainly falls into instability in the network with heterogeneous delays [11], as a result of the above burst.

Figure 1: Multiple flows with heterogeneous delays

Simulation 1: A simulation of heterogeneous delays in a network whose topology is depicted in Figure 1 was setup in ns-2 [12]. The ten flows were set to send packets from the sources Si to the destination D from 0 to 140s, which guarantee the stable systems have enough time to reach stable state. We use �� ��� to represent the upper simulations, where bw is the capacity of the bottleneck link measured in Mbps. Figure 2 shows the arrival ratios at the bottleneck link in last 2 seconds, which gives the variation detail. The arrival ratio is the ratio of the actual arrival rate over the theoretical arrival rate. The arrival ratios in (a) and (b) are averaged over 220ms, which show that �� ��� and �� ����� are stable while �� ��� and �� ����� are unstable. The above results suggest that the low or high capacity is not the cause of instability in heterogeneous delays. The arrival ratios in (c) are averaged over 40ms, which reveal that the cause of instability is that the flow with longer RTT outputs packets is so burst that the output duration lasts not more than 80ms in one RTT.

The bottleneck link has two states: the congestion state and the idleness state�� When it is in the congestion state, the queuing delay will expand the average interval between packets of the longer RTT flow, shrink non-traffic duration in its round trip time and reduce its traffic burst. After that, the congestion window of the XCP sender will be shrunk. This will keep the average interval constant but expand the non-traffic duration in its round trip time. On the other hand, the idleness state will increase the congestion window of the XCP sender, which will decrease the average interval of packets and enhance its traffic burst because the XCP host outputs all the packets inside the expanded congestion window without any interval. When the link capacities are 50Mbps and 1500Mbps, the effect of the queuing delay is stronger than the burst of the XCP sender, the traffic of the longer delay flow is evenly

distributed on the RTT at last. On the contrary, the traffic keeps being sent in burst such that the XCP system becomes unstable. The reason also confirms that XCP is locally stable but globally unstable in heterogeneous delay networks [11].

Figure 2: The arrival ratios of �� ���, �� ���, �� �����, and �� ����� with heterogeneous delays

III. THE SMOOTH XCP SENDER To overcome the instability problem, it was proposed in [11]

to extend the control period to the maximum RTT of all the flows. However, the author also admits that this method will take more time to converge to the stable state. According to the above analysis of section II, the cause of instability is the burst of the XCP sender. As such, in this paper the Smooth XCP (SXCP) is proposed which regulates the interval of packets such that it should be no less than �.

� � ��������������

������� � ���������� �!"#$

In the upper equation, the term H_apsize is the average packet size of the XCP sender.

Simulation 2: A simulation of single bottleneck link was setup in ns-2 as depicted in Figure 3, while the capacity of the bottleneck link was set to 10Mbps and the propagation delay of each link to 25m. In this simulation, N flows were set to send packets from the sources Si to the destination D, where " � �%&% ' %(. The first flow starts at 0.19ms, and the other flows start at 3.19ms. The queue sizes at the bottleneck link were set to be large enough to accommodate all the burst packets and no packet should be lost.

138 138.2 138.4 138.6 138.8 139 139.2 139.4 139.6 139.8 14070

80

90

100

110

(a)

The

Tot

al A

rriv

al R

atio

in la

st t

wo

seco

nds

(%)

eH(10) eH(50) eH(1000) eH(1500)

138 138.2 138.4 138.6 138.8 139 139.2 139.4 139.6 139.8 1400

20

40

60

80

100

(b)

The

Arr

ival

Rat

io o

f F

low

10

in la

st t

wo

seco

nds

(%)

138 138.2 138.4 138.6 138.8 139 139.2 139.4 139.6 139.8 1400

100

200

300

400

500

(c)

The

Arr

ival

Rat

io o

f F

low

10

in la

st t

wo

seco

nds

(%)

Time (sec)

96

Figure 3: The topology of single bottleneck link

Figure 4: Two flows share the bottleneck link

Figure 5: Three flows share the bottleneck link

Figure 4 shows the variation in time of the arrival rates and the queue lengths of XCP, RCP and SXCP When N is 2. It is obvious that the queue length of SXCP is less than that of XCP, and both of them are much less than RCP.

Figure 5 shows the arrival rates and the queue lengths of XCP, RCP and SXCP When N is 3. The peak queue length of RCP in Figure 4 is 186KB which is larger the bandwidth-delay production 125KB, while the peak queue length of RCP in Figure 5 is 349KB. It is obvious that if more flows arrive at the same time, it will result in a longer queue. This is the problem of the go-and-ready scheme.

Simulation 3: A simulation of SXCP with heterogeneous delays was setup in ns-2 where a network topology was implemented as depicted in Figure 1, while the capacity of the bottleneck link was set to 1000Mbps. Figure 6 shows the arrival rates of XCP and SXCP, both being averaged over 40ms. One can observe that SXCP is stable in the case of heterogeneous delays and it converges to the link capacity with the same speed as in the case of uniform delays.

Figure 6: The arrival rate of SXCP with heterogeneous delays

IV. PERFORMANCE As SXCP postpones outputting packets at the sender site, it

is important to investigate the utilization and service time of SXCP. The simulations in [4] demonstrate that XCP has high utility and there is no packet loss as compared with RED, CSFQ, REM and AVQ. As such, packet loss will be out of our considerations.

A. The Simulations of Long-Lived Flows The network topology for this simulation case is given in

Figure 3. In this case, the capacities of the links are kept in the [2Mbps, 1Gbps] range, the RTT in [40ms, 1.2 sec] range, and the number of sources were in the [10, 1000] interval. The performance of SXCP and XCP are displayed considering the bottleneck utility from R to D and the average queue length (AQL). The AQL is measured in packet numbers, and represents the traffic burst in the steady state.

Impact of Capacity: It is shown that an increase in link capacity (with the resulting increase of per-flow bandwidth) will cause a linear increase of the queue length in XCP and SXCP systems. )*+,-./*01� 23 In this simulation, 50 long-live FTP flows were created to share a bottleneck link from R to D. Additionally, there were 50 flows traversing the reverse path. They were used merely to create a 2-way traffic environment. The delay of each link was set to 20ms. Figure 7 shows that when the capacity increases from 2Mbps to 1Gbps, the utilities of the two systems are over 93%, while the AQL of XCP is more than the double of the SXCP one.

0 1 2 3 4 5 6 70

2

4

6

8

10

12

(a)

Arr

ival

Rat

e (M

bps)

XCP RCP SXCP

0 1 2 3 4 5 6 70

50

100

150

200

(b)

Qle

n (K

B)

Time (sec)

0 1 2 3 4 5 6 70

2

4

6

8

10

12

(a)

Arr

ival

Rat

e (M

bps)

XCP RCP SXCP

0 1 2 3 4 5 6 70

100

200

300

400

(b)

Qle

n (K

B)

Time (sec)

0 1 2 3 4 5 6 70

200

400

600

800

1000

1200

(a)

The

tot

al a

rriv

al r

ate

in f

irst

seve

n se

cond

s (M

bps)

XCP SXCP

0 1 2 3 4 5 6 70

100

200

300

400

500

(b)

the

arriv

al r

ate

of f

low

10

in f

irst

seve

n se

cond

s (M

bps)

Time (sec)

97

Figure 7: The performances as a function of capacity

Figure 8: The performances as a function of the flow number

Figure 9: The performances as a function of the RTT

Impact of Number of Flows: Fixing the bottleneck capacity at 150Mbps and the round trip time at 80ms, and repeating the same experiment with a varying number of FTP sources the performance of the SXCP as compared with XCP is explored as a function of the number of sources of traffic. Other

parameters have the same values as in the previous experiment.

)*+,-./*01� �3 Figure 8 demonstrates that when the number of long-live FTP flows ranges from 10 to 1000, the utilities of the two systems are over 93%. The trends of the average queue length of the two systems are steady before 500 flows. When the number of flows is larger than 500, the queue increases to absorb the rounding error [4].

Impact of the Feedback Delay: Using a bottleneck capacity at 150Mbps, the impact of increased delay on the performance of congestion control was explored. All other parameters have the same values as they were used in the previous experiment. Simulation 6: In this simulation, 50 long-live FTP flows were created to share the same bottleneck link. Additionally, there were 50 flows traversing the reverse path which were used merely to create a 2-way traffic environment. All the links had the same delay, which was increased from 10ms to 300ms. Figure 9 shows that the utilities of the two systems are over 97%. When the round trip time ranges from 40ms to 1.2sec, the average queue length of XCP increases faster than SXCP.

According to the long-lived flow simulations, it is obvious that the utilities of the two systems are at the same level, but the average queue length of SXCP is much lower than XCP.

B. The Simulations of Short Web-Like Flows Since a large number of flows in the Internet are short web-

like flows, it is important to investigate the impact of such dynamic flows on congestion control.

Simulation 7:� The network topology for this simulation case is given in Figure 3. In this experiment ten long-lived FTP flows traversing the bottleneck link were generated, while another ten long-lived FTP flows traversing the reverse path were also considered. All link capacities were 30 Mbps and all the link propagation delays were set to 25ms. Short flows from R to D arrived following the distribution of Poisson processes. Their transfer size was derived from a Pareto distribution with an average of 30 packet (ns implementation with shape_= 1.35), which complies with real web traffic [13]. The average service time (AST), which is the average duration of the short web flows, is computed as follows.

4�5 � �6 � 78 �9

:

9;<= � 78 !9��9

:

9;<=

in which K is the number of short web flows, si is the service time of the flow i, and ni is the packet number of the flow i.

Figure 10 shows the variation of the bottleneck utilization, and the AST and AQL as functions of the load created by the short web-like flows. It shows that XCP and SXCP have the same performance in regards to the average service time and utilization, while the average queue length of SXCP is much lower than the XCP one. Figure 10 (c) shows the AQL of XCP decrease as the load of the web traffic increases. The reason is that the traffic load becomes steady as the load of the web traffic increases.

0 100 200 300 400 500 600 700 800 900 1000

94

96

98

100

(a)

Util

ity(%

)

XCP SXCP

0 100 200 300 400 500 600 700 800 900 10000

0.5

1

1.5

(b)

AQ

L (K

Pac

kets

)

Bottleneck Capacity (Mbps)

0 100 200 300 400 500 600 700 800 900 100094

96

98

100

(a)

Util

ity(%

)

XCP SXCP

0 100 200 300 400 500 600 700 800 900 10000

0.2

0.4

0.6

0.8

1

(b)

AQ

L (K

Pac

kets

)

Number of FTP Flows

0 0.2 0.4 0.6 0.8 1 1.297

98

99

100

(a)

Util

ity(%

)

XCP SXCP

0 0.2 0.4 0.6 0.8 1 1.20

1

2

3

4

5

(b)

AQ

L (K

Pac

kets

)

Round Trip Time (sec)

98

Figure 10: the performances as a function of the load of web-like flows

C. The Simulations of Short Web-Like Flows with Heteroge-neous Delays

In long-lived flows with heterogeneous delays, XCP may become unstable and the utilization will be low. The following simulation shows that the conclusion is also correct in regards to the short web-like flows with heterogeneous delays.

Simulation 8: In this experiment, the network topology was kept the same as the one used in Simulation 7 except that the link propagation delay from S10 to R was set as 500ms. In Figure 11 the graphs of the bottleneck utilization, AST and AQL in heterogeneous delays are given. Comparing the results displayed in Figure 10 and Figure 11, one can see that SXCP has a similar performance in both cases, while XCP shows the apparent degradation in all the three aspects, the bottleneck utilization, the AST and the AQL.

V. CONCLUSION In this paper, the instability of XCP related to

heterogeneous delays was investigated, and the traffic burst of the XCP sender was found to be the cause. To overcome the problem, the Smooth XCP called SXCP was proposed. The SXCP regulates the traffic at the sender site which not only improves the stability, but also increases the link utility and decreases the queue length and the queuing delay of long-lived flows and short web-liked flows.

ACKNOWLEDGMENT This work is supported by National Natural Science

Foundation of China under Grants No. 60873258, and by Chinese National 973 Fundamental Research Program under Grants No.2007CB307100 and No.2007CB307106.

Figure 11: The performances as a function of the load of web-like flows in heterogeneous delays

REFERENCES [1] V. Jacobson, Congestion avoidance and control, ACM Computer

Communication Review, Proceedings of the SIGCOMM’88 Symposium, 18, 4:314–329, Aug. 1988.

[2] S. Floyd and V. Jacobson, Random early detection gateways for congestion avoidance, In IEEE/ACM Transactions on Networking, 1(4):397–413, Aug. 1993.

[3] A. Phanishayee, E. Krevat, Measurement and Analysis of TCP Throughput Collapse in Cluster-based Storage Systems, USENIX Conference on File and Storage Technologies, 2008.

[4] D. Katabi, M. Handley, and C. Rohrs, Congestion control for high bandwidth-delay product networks. In Proceedings of ACM SIGCOMM 2002, August 2002.

[5] F. Kelly, A. Maulloo, and D. Tan, Rate control for communication networks: shadow prices, proportional fairness and stability, Journal of the Operational Research Society, Vol. 49, Issue: 3, Publisher: Macmillan Press, Pages: 237-252, 1998.

[6] S.H. Low, L.L.H. Andrew, and B.P. Wydrowski. Understanding XCP: Equilibrium and fairness. In Proceedings of IEEE INFOCOM 2005, 2005.

[7] I. Dukkipati, M. Kobayashi, R. Zhang-shen, Nick Mckeown, Processor sharing flows in the internet, Thirteenth International Workshop on Quality of Service, IWQoS 2005.

[8] A. K. Parekh, R. G. Gallager, A generalized processor sharing approach to flow control in integrated services networks: The single node case. Proceedings of IEEE Infocom, 1992.

[9] F. Kelly, G. Raina, Thomas Voice, Stability and fairness of explicit congestion control with small buffers, ACM SIGCOMM Computer Communication Review, Vol. 38 Issue 3, July 2008.

[10] Y. Zhang, T.R. Henderson, An Implementation and Experimental Study of the explicit control protocol (XCP), Proceedings IEEE Infocom 2005.

[11] L. L. H. Andrew, B. P. Wydrowski, and S. H. Low, An example of instability in XCP, http://netlab.caltech.edu/ lachlan/abstract/ xcpInstability.pdf.

[12] The network simulator ns-2. http://www.isi.edu/ nsnam/ns. [13] M. E. Crovella and A. Bestavros. Self-similarity in world wide web

traffic: Evidence and possible causes. In IEEE/ACM Transactions on Networking, (6):835–846, Dec. 1997.

10 20 30 40 50 60 70 8070

80

90

100

(a)

Util

ity(%

)

XCP SXCP

10 20 30 40 50 60 70 80

800

1000

1200

1400

1600

(b)

AS

T (

ms)

10 20 30 40 50 60 70 800

10

20

30

40

(c)

AQ

L (P

acke

ts)

The Load of the Web Traffic (%)

10 20 30 40 50 60 70 8070

80

90

100

(a)

Util

ity(%

)

XCP SXCP

10 20 30 40 50 60 70 80

800

1000

1200

1400

1600

(b)

AS

T (

ms)

10 20 30 40 50 60 70 800

50

100

(c)

AQ

L (P

acke

ts)

The Load of the Web Traffic (%)

99