This paper is a draft that will appear in IEEE/ACM Transactions on Networking.

Final editing is still expected. Please replace it with the final version when published.

Modeling the Performance of HTTP

Over Several Transport Protocols�

John Heidemann Katia Obraczka Joe Touch

DRAFT June 6, 1997

Abstract

This paper considers the interaction of HTTP withseveral transport protocols, including TCP, Transac-tion TCP, a UDP-based request{response protocol,and HTTP with persistent TCP connections. Wepresent an analytic model for each of these proto-cols and use that model to evaluate network overheadcarrying HTTP tra�c across a variety of networkcharacteristics. This model includes an analysis ofthe transient e�ects of TCP slow-start. We validatethis model by comparing it to network packet tracesmeasured with two protocols (HTTP and persistentHTTP) over local and wide-area networks. We showthat the model is accurate within 5% of measuredperformance for wide-area networks, but can underes-timate latency when the bandwidth is high and delayis low. We use the model to compare the connection-setup costs of these protocols, bounding the possibleperformance improvement. We evaluate these costsfor a range of network characteristics, �nding thatsetup optimizations are relatively unimportant forcurrent modem, ISDN, and LAN users but can pro-

�This research is supported by the Defense AdvancedResearch Projects Agency (DARPA) through FBI contracts#J-FBI-95-185 entitled \Cities Online", and #J-FBI-95-204,\Global Operating Systems Technologies". The views and con-clusions contained in this document are those of the authorsand should not be interpreted as necessarily representing theo�cial policies, either expressed or implied, of the Departmentof the Army, DARPA, or the U.S. Government.The authors can be contacted at 4676 Admiralty Way, Ma-

rina del Rey, CA, 90292-6695, or by electronic mail to johnh,katia or touch, each @isi.edu.This paper is copyright c 1997 by the Institute of Electrical

and Electronics Engineers.

vide moderate to substantial performance improve-ment over high-speed WANs. We also use the modelto predict performance over future network charac-teristics.

1 Introduction

The World Wide Web [1] has rapidly become one ofthe most popular Internet services [2]. The popular-ity of the web has resulted in a corresponding popu-larity for HTTP, the standard Hyper-Text TransportProtocol [3, 4]. HTTP is layered over TCP.

The strengths of TCP are well known. TCPis a well understood protocol with carefully tuned ow control and congestion avoidance algorithms [5].These characteristics make TCP an excellent proto-col for bulk data transport in congested networks.

Web tra�c is not ideally matched to TCP, how-ever. In practice, web access is request{response ori-ented with bursts of numerous requests and small,unidirectional responses. Retrieval of a complete webpage requires separate requests for text and each em-bedded image, thus making tra�c inherently bursty.Responses are small to keep transmission times down;several studies have documented typical response sizeas less than 1KB [6], 6KB [7], or 21KB [8]. Finally,web users often bounce rapidly from site to site, asveri�ed by both client [6] and server side traces [8].1

1For example, from the 1995-96 Boston University sur-vey [6] we can deduce that an upper bound on the mean num-ber of unique URLs read from each site is 15.5. The NCSAserver-side traces suggest that clients read a mean of 2.92 textpages at their site per-browsing session.

Unfortunately, TCP is poorly suited to frequent,short, request{response-style tra�c. Frequent con-nection setup and tear-down costs burden serverswith many connections left in TIME WAIT state [9,10]. Short connections can interact poorly withTCP's slow-start algorithm for congestion avoid-ance [10]. Finally, TCP's initial three-way handshakeadds latency to each transaction [10].These mismatches between the needs of HTTP and

the services provided by TCP contribute to increasedlatency for most web users. Fundamentally, TCP isoptimized for large-scale bulk data transport, whileHTTP often needs a light-weight, request{responseprotocol. Other request{response-style services, in-cluding transfer of short e-mail messages or �les andRPC protocols, would also bene�t from protocol im-provements.Prior work in this �eld (reviewed in the next sec-

tion) has identi�ed and proposed solutions to each ofthese problems. This paper builds upon that work inseveral ways. First, we present analytic models forHTTP tra�c over several transport protocols. Thesemodels allow us to compare current and future pro-tocols in a common framework. We can also predictprotocol performance on proposed networks. We vali-date our models by comparing them to measurementsfrom an actual system. Finally, we use the insightprovided by the models to re ect on the needs ofprotocols for HTTP tra�c.This paper makes several contributions to the �eld.

First, we provide a detailed examination of the tran-sient e�ects of TCP connection start-up. We showthat the start-up behavior of TCP's slow-start al-gorithm depends on the acknowledgment policy ofthe receiver and that this fact often results in thecongestion window opening signi�cantly slower thanhas often been described. These e�ects are par-ticularly important for HTTP (and similar short,request{response protocols) where transient e�ectsoften dominate performance. Second, we provide ananalytic model for web transport over several pro-tocols and use this model to compare these proto-cols and to validate the experimental results of priorresearchers. Third, we apply our model to predictprotocol performance in a broad range of networkcharacteristics including future networks characteris-

tics. Finally, we apply these predictions to evaluatethe performance of proposed HTTP enhancements.We �nd that, while proposed enhancements such aspersistent HTTP are e�ective with high-bandwidthnetwork characteristics, they o�er much more mod-est gains in the medium- and low-bandwidth networkcharacteristics common to most Internet users today.

2 Related Work

This section summarizes previous and current workin the areas of HTTP performance evaluation as wellas transport protocols that have been proposed asalternatives to TCP.

A simpli�ed version of the HTTP over TCP andcaching TCP models of this paper can be found else-where [11]. That paper focuses on comparison ofHTTP with and without persistent connections; thispaper more accurately models slow-start and work-loads and analyzes additional protocols in more de-tail.

2.1 Persistent-Connection HTTP

Padmanabhan and Mogul conducted experiments toquantify the cost of using TCP as HTTP's trans-port mechanism [10]. Their examination of a typicalHTTP request{response demonstrated throughputsfor short responses as small as 10% of the through-put obtainable by bulk data transfers under similarnetwork conditions. They attribute these costs toTCP's connection setup and slow-start mechanisms.

To amortize TCP's connection overhead over mul-tiple HTTP interactions, Padmanabhan and Mogulpropose a \persistent-connection" HTTP, or P-HTTP, a variant of HTTP that uses one TCP con-nection to carry multiple HTTP requests [10]. Mogulalso investigates trace-driven simulations of HTTPand P-HTTP, demonstrating that P-HTTP can avoidthese setup costs and achieve signi�cantly better per-formance than HTTP when there is temporal localityin web accesses [12]. By requiring fewer TCP connec-tions than HTTP, P-HTTP also conserves server andnetwork resources.

2

Padmanabhan and Mogul's results have beencorroborated by Simon Spero in an unpublishedstudy [13]. A version of P-HTTP is part of the spec-i�cation of HTTP/1.1 [4].

Both Padmanabhan and Mogul's and Spero's anal-yses of HTTP overhead were founded on mea-surements between relatively well-connected Inter-net hosts (bandwidth about 1Mb/s, round-trip time70ms). We show that our analytic model of perfor-mance allows us to extend these results to other net-works. We validate their results for well-connectedhosts; in such cases P-HTTP will improve perfor-mance. We also show that when either bandwidthor delay degrade (perhaps due to wide-area conges-tion, bottleneck links such as a modem or ISDN, orco-location of hosts), then P-HTTP performance im-provements are much more modest.

A recent technical note has suggested that useof pipelining is important to get good performancefrom the HTTP/1.1 implementation of P-HTTP [14].Pipelining reduces the number of packets transmittedand supports request independence (as discussed inSection 4.1). We discuss the performance implica-tions of HTTP/1.1 with pipelining in Section 5.5.

2.2 Transaction TCP

Transaction TCP, or T/TCP [9, 15], was proposed tobridge the gap between the services provided by UDPand TCP for request{response applications.2 T/TCPimproves TCP performance by caching per-host in-formation su�cient to bypass the TCP's three-wayhandshake and avoid slow start.3 T/TCP also short-ens TCP's TIME WAIT period from 240 to 12 sec-onds, reducing the duration that per-connection stateis retained.

Stevens compares the time to complete a client-server transaction using TCP, UDP, and T/TCP fordi�erent sizes of the request and reply over Pentium-based hardware on a 10Mb/s Ethernet [16]. As ex-

2A T/TCP \transaction" is a request{response exchange,not a database-style transaction.

3While the functional speci�cations for T/TCP suggestthat congestion window be cached, the reference implemen-tation (at ftp://ftp.isi.edu/pub/braden/TTCP.tar.Z) does notcache this value.

pected, the UDP-based client-server yields the small-est latency (11 to 81ms, depending on packet size),and the TCP-based interaction takes the longest tocomplete (36 to 105ms). In Stevens' experiments,T/TCP is about 5ms more expensive than UDP fora given packet size (and therefore 55{18% faster thanTCP).We extend Steven's results by modeling HTTP

tra�c over T/TCP. We also show that, with re-spect to connection establishment costs, HTTP traf-�c over T/TCP and persistent-connection TCP (P-HTTP over TCP) behave identically.

2.3 UDP-Based Request{Response

Protocols

The Asynchronous Reliable Delivery Protocol(ARDP) is one example of a reliable message passingprotocol built atop UDP for request{response-styleinteractions between a client and a server. ARDPwas proposed and implemented as the transportmechanism for the Prospero information discoverytool [17].ARDP's main design goal is to provide a reli-

able yet light-weight communication mechanism totransport requests and responses between clients andservers. The current version of ARDP (in develop-ment) borrows TCP-style ow-control, congestion-avoidance, and retransmission algorithms.4 ARDPavoids TCP's three-way handshake, instead ran-domly selecting connection identi�ers. This approachtrades connection setup overhead for a chance of ac-cidental (or intentional) connection identi�er reuse.We will show that although avoiding the three-way

handshake is helpful, caching congestion-control in-formation is important for optimal performance.

3 Network and Tra�c Model

To understand the performance of HTTP over dif-ferent transport protocols we must characterize thenetwork and the tra�c we expect to send. We con-sider each of these in turn.

4Older versions of ARDP have a �xed window size of 161250-byte packets and do not do slow-start.

3

rtt round-trip timebw bandwidthmss maximum segment sizestt segment-transmission timemuws maximum useful window size

Table 1: Network characteristics a�ectingHTTP performance.

3.1 Network Model

Figure 1 shows the beginning of a typical TCP packetexchange. Several parameters are needed to charac-terize this exchange; we list these in Table 1.The �rst three parameters listed in Table 1, round-

trip time, bandwidth, and maximum segment size5

are all properties of a given network path (althoughobserved round-trip time and bandwidth may changedue to variations in network and server load). Theremaining two parameters can be derived from theothers. Segment-transmission time, the time it takesto send the largest possible packet, is directly relatedto bandwidth and segment size:

stt = mss=bw

Maximum useful window size is the bandwidth{delay product expressed in an integral number ofpackets. This �nal parameter represents the num-ber of segments which must be in ight to keep thenetwork \pipe" full. When the current window sizeis less than muws there will be a delay while acknowl-edgements return to the sender; when window size isat least muws segments then there will be continuous ow of data. Analytically, muws is:

muws = drtt=stte

Note that muws is the bandwidth{delay productexpressed in an integral number of packets.A �nal network characteristic not considered here

is transmission error rate. We discuss the e�ects ofpacket loss on our results when we validate our modelin Section 5.3. The primary goal of this paper is to ex-amine startup e�ects of transport protocols. A com-plete discussion of the e�ects of error on transport

5Our segment sizes already account for link-level headers.

HTTPClient

HTTPServer

setup

initialrequest

reply

round−triptime (RTT)

server"think time"

stall

segmenttransmissiontime

slowstart

sustainedtransfer

Figure 1: Packets exchanged in an HTTP overTCP connection not limited by bandwidth.Bold arrows indicate data transfer while thin arrowsshow SYN- or ACK-only packets.

performance are beyond the scope of this paper, sofor the remainder of this paper we assume error-freetransmission

Having de�ned these parameters, we can nowquantify them for the networks in which we are in-terested. These values are given in Table 2.

Our models will use these parameters to predictperformance across a range of networks. Wherepossible we have directly measured these parame-ters on actual systems. For Ethernet, we use ob-

4

network rtt bw mss stt muws

Ethernet 0:7ms 8:72Mb/s 1460 B 1:28ms 1 pktsFast-Ethernet 0:7 100 1460 0:111 7Slow-Internet 161 0:102 512 38:5 5Fast-Internet 89 1:02 512 3:85 24Modem 250 0:0275 512 142 2ISDN 30 0:122 512 32 1WAN-Modem 350 0:0275 512 142 3WAN-ISDN 130 0:122 512 32 5ADSL 30 6 512 0:651 47DirecPC 500 1 512 3:91 128N-Ethernet 0:7 8:72 1460 1:28 1N-Fast-Internet 80 1:17 1460 9:52 9N-Modem 150 0:0275 1460 396 1

Table 2: Network characteristics for several existing networks. N-Ethernet, N-Fast-Internet,and N-Modem are discussed in Section 5.5.

served bandwidth and latency measured betweentwo Sun SPARC 20/71 hosts connected by a ded-icated 10Mb/s Ethernet. For the Internet we em-ploy two di�erent values, \fast" and \slow" Inter-net, which corresponds to measured communicationsspeeds between well-connected hosts on the same anddi�erent continents, respectively.6 Actual Internetperformance represents a continuum between thesepoints. The fast-Internet case corresponds roughly tocharacteristics present in Mogul's and Spero's stud-ies [12, 13]. DirecPC also presents measured valuesfrom a system with satellite down-link and a modemback-channel [18]. (We assume that the back-channelis not a factor limiting performance.)

For several other networks we have had to estimatethese parameters. Modem and ISDN �gures employmeasured latencies and theoretical bandwidths. Fast-Ethernet and ADSL use theoretical bandwidths andlatencies suggested by similar systems (10Mb/s Eth-ernet and ISDN, respectively). We look forward toreplacing these estimates with actual measurementsas these systems become available.

Finally, only the Slow- and Fast-Internet �guresconsider wide-area limitations on latency and band-

6For the fast-Internet case we measured communicationsbetween darkstar.isi.edu and prep.ai.mit.edu, for the slow case,between darkstar.isi.edu and ftp.connect.org.uk. These mea-surements were taken on a Saturday afternoon (7 hours westof UTC), 11 May 1996.

width; other cases assume that the client is directlyconnected to the server by the given networkingtechnology. We can re ect wide-area limitations byadding latency and capping bandwidth to that ob-served in the Slow- and Fast-Internet cases. For Mo-dem and ISDN we therefore show WAN version with100ms additional latency. For faster technologies(ADSL, DirecPC, Slow- and Fast-Ethernets) band-width and latency can be approximated by the Slow-to Fast-Internet cases.We can already observe that muws is fairly low

in many current networks, with the exception of thefast-Internet case. Once the transmission windowhas opened up past this value, acknowledgments ofoutstanding packets are returned as fast as packetsare transmitted. muws is directly related to protocoloverhead; we will show later that when it is small,connection setup optimizations have little to opti-mize and so provide performance similar to HTTPover TCP.

3.2 Tra�c Model

Performance of transport protocols also dependsupon the tra�c characteristics of HTTP. We con-sider several potential HTTP workloads:

small page A single 5KB web page.

medium page A single 25KB web page.

5

large page A single 100KB web page.

small cluster A single 6,651B page with embedded3,883B and 1,866B images.7

medium cluster A single 3,220B page with threeembedded images, sizes 57,613B, 2,344B, and14,190B.8

large cluster A single 100KB page with 10 embed-ded 25KB images.

Notice that at each change in size (from smallto medium to large), the total amount of data ex-changed is about �ve times larger.Each of the cluster workloads requires multiple

HTTP requests, one per page or image. In multi-request exchanges we assume that all requests areindependent. Requests A and B are independent ifrequest B can be initiated before request A has com-pleted. Although a simple client program would se-quentially request each item, modern multi-threadedbrowsers initiate multiple parallel image requests asthe basic HTML page is received, thus allowing somedegree of request independence.Choice of tra�c will in uence our protocol evalua-

tion. The small and medium cases are representativeof typical web pages today. We consider the largecases representative of what may become common inthe future as networks with higher bandwidth becomemore widely available.Finally, we also need to model the size of HTTP

requests and the time it takes a server to parse andinterpret the request (server processing time). Tosimplify our model we assume a constant requestsize of 256B and zero processing-time. A more com-plex model of request size is not warranted sincerequests almost always �t in one segment and soperformance is dominated by response size.9 Thezero processing-time assumption is clearly incorrect;request-handling overhead depends strongly on server

7The front page (http://www.yahoo.com) at Yahoo on May1, 1996.

8The front page (http://www.gnn.com) at GNN on May 1,1996.

9Some older browsers had substantially longer requests [13].Performance concerns are one reason modern browsers keeprequests short.

hardware, software, and load. We remove this e�ectfrom our computations to focus instead on the net-work protocol aspects of HTTP instead of server im-plementation. We reconsider processing-time whenwe validate our model in Section 5.

4 Protocol Analysis

We next examine protocol performance for the net-works and tra�c patterns we are considering. Weexamine the interactions between HTTP and severalclasses of protocols:

TCP HTTP currently runs over TCP, opening anew connection for each transaction.

Connection caching protocols To avoid connec-tion setup overheads, P-HTTP and T/TCPcache connections or connection informationacross multiple requests.

UDP-based request{response protocolsARDP employs UDP to avoid TCP setup costs.

To examine these protocols against a consistentbaseline we �rst consider a lower bound on trans-action time.

4.1 Minimum Transmit Times

Logically, the minimum possible transaction time isthe one round-trip time delay inherent in communi-cation, plus the time it takes to send the request andreceive the reply, and any time spent at the server:

Tmin = rtt (1)

+reqmin

+processing

+replymin

reqmin = reqsize=bw

replymin = replysize=bw

This equation is straightforward and can be re-duced to data size divided by bandwidth. We present

6

it in this detail to illustrate the di�erences and simi-larities between di�erent transport protocols.A series of n independent requests will incur only

one round-trip latency because they are pipelined.The total required time will therefore be:

Smin = rtt+nX

i=1

(Tmin(i)� rtt) (2)

The assumption of independent requests implies abrowser which is multi-threaded or which pipelinesrequests and that the subsequent requests are sentimmediately without waiting for the prior response.(In particular, we assume that the second request canbe made before the complete results of the �rst re-quest have returned.) If we were to assume a single-threaded browser making n sequential requests wewould then add an additional (n � 1) � rtt delay toEquation 2 while the client determines and requeststhe next page. If we assume that no additional re-quests could be made until the �rst was completed(as would be the case in a multi-threaded browserwhere all image references are in the last segment ofthe �rst page), we would add one additional rtt.Because of our assumptions about request size and

processing-time, the primary factor in uencing min-imal transmission times will be replymin. Table 3summarizes the minimumpossible transmission timesfor the networks and workloads we consider.

4.2 Simple Model

We can construct a very simple estimate of whentransport protocol overhead will be signi�cant bycomparing the ratio of the bandwidth{delay productto the transaction size. In any streaming transportprotocol several round-trip exchanges are requiredto reach steady-state performance (assuming net-working conditions are not known a priori). Whenthe o�ered transaction is too small, stability is notachieved, and transient e�ects are ampli�ed.We can approximate the minimum amount of time

a connection would have to stabilize by comparingthe ratio of transmitted data to pipe size. For anHTTP transaction, assuming that reqsize is zero, thisratio is:

replysizebw� rtt

(3)

When this ratio is small we would expect protocolsetup costs to dominate performance; when it is largesetup costs would be amortized.An alternate view of the same concept inverts this

ratio to get the pipe size in reply-sized units.

rtt

replysize=bw(4)

This equation is a good approximation for oneround-trip overhead per reply (the exact value wouldbe rtt=Tmin). We can use this equation to providea �rst approximation for setup overheads. To esti-mate overhead for single page retrievals we apply thisequation directly. For clusters of retrievals we use theharmonic mean10 of ratios for each retrieval if eachretrieval requires a separate connection. If connec-tion overhead is amortized across all of a cluster'sreplies (as it would be if retrievals were batched overa single connection) we treat all replies as a singlelarge response.Table 4 shows these ratios, highlighting exchanges

where the overhead approximation exceeds 25%.These simple equations provide a good predictor ofwhere transient e�ects will be high, but they failto accurately capture these e�ects when the band-width/delay product rises. To accurately estimateperformance of actual network protocols in thesecases we next consider the e�ects of congestion avoid-ance protocols.

4.3 HTTP over TCP

We next consider the overhead present in TCP whencompared to the minimum transaction time. TCPadds several sources of overhead: protocol headers,the three-way handshake at connection setup, theslow-start algorithm, and retransmissions and con-gestion control delay due to packet loss. Packet lossrates depend on a number of factors beyond the scope

10The harmonic mean of n values is n=(P

1=xi).

7

network small-page medium-page large-page small-cluster medium-cluster large-cluster

Ethernet 5:4ms 23:3ms 90:5ms 12:2ms 69:3ms 317msFast-Ethernet 1:11 2:67 8:53 1:7 6:68 28:3Slow-Internet 565 2100 7870 1150 6050 27300Fast-Internet 129 283 860 188 678 2800Modem 1740 7430 28800 3910 22000 101000ISDN 366 1650 6450 853 4930 22600WAN-Modem 1840 7530 28900 4010 22100 101000WAN-ISDN 466 1750 6550 953 5030 22700ADSL 36:8 62:9 161 46:7 130 489DirecPC 541 697 1280 600 1100 3260

Table 3: Minimal theoretical times to send di�erent workloads across di�erent networks.

small- medium- large- small- medium- large-

network page page page cluster cluster cluster

Ethernet 0.16 0.03 0.01 0.19 / 0.06 0.04 / 0.01 0.02 / 0.00Fast-Ethernet 1.79* 0.36* 0.09 2.22*/ 0.74* 0.47*/ 0.12 0.28*/ 0.03Slow-Internet 0.42* 0.08 0.02 0.52*/ 0.17 0.11 / 0.03 0.07 / 0.01Fast-Internet 2.31* 0.46* 0.12 2.87*/ 0.96* 0.61*/ 0.15 0.36*/ 0.03Modem 0.18 0.04 0.01 0.22 / 0.07 0.05 / 0.01 0.03 / 0.00ISDN 0.09 0.02 0.00 0.12 / 0.04 0.02 / 0.01 0.01 / 0.00WAN-Modem 0.25 0.05 0.01 0.30*/ 0.10 0.07 / 0.02 0.04 / 0.00WAN-ISDN 0.41* 0.08 0.02 0.50*/ 0.17 0.11 / 0.03 0.06 / 0.01ADSL 4.61* 0.92* 0.23 5.71*/ 1.90* 1.22*/ 0.30* 0.72*/ 0.07DirecPC 12.80* 2.56* 0.64* 15.86*/ 5.29* 3.39*/ 0.85* 2.01*/ 0.18

Table 4: Approximation of one round-trip overhead per transaction (Equation 4). Highlighted val-ues indicate ratios more than 0.25 where transient e�ects may dominate performance. For cluster workloadsratios are given assuming separate and single connections.

8

of this paper; we therefore consider only the �rstthree sources of overhead. We discuss how packetloss would impact our model in Section 5.3.An idealized packet trace for a request{response

transaction over TCP is shown in Figure 1. In thispacket trace we can see how the three-way handshake(labeled setup) adds one round-trip time overhead.We next consider the e�ects of TCP's slow-start al-gorithm.

4.3.1 TCP slow-start

TCP's slow-start algorithm limits transmission bya congestion window (cwnd) which is initialized toone segment and increases each time an ACK is re-ceived [5]. The size of increase changes: initiallycwnd grows in one segment increments (the slow-startphase), then later by 1=cwnd (congestion avoidancephase). TCP is thus \clocked" by the ow of ac-knowledgments.For high bandwidth/delay-product paths, TCP ini-

tially alternates between segment transmission andstalls waiting for ACKs from these segments to re-turn. The number of segments sent per stall in-creases exponentially during slow-start until enoughsegments are in ight that ACKs return continuously.To model slow-start behavior we therefore need toknow how many segments are sent between stalls,how much time each stall wastes, and how many stallsoccur until steady state is reached or transmissionends.We originally expected that the number of packets

between each stall would follow a simple exponen-tial pattern: 1, 2, 4, 8, and so on. Modern TCPimplementations deviate from this behavior for tworeasons. First, in BSD-derived TCP implementa-tions the ACK of the SYN packet on the HTTPserver opens the congestion window, so the cwnd

for the reply begins at 2. Second, TCP's delayed-acknowledgment algorithm normally causes the clientto ACK every other segment, not each segment [19].Because the congestion window opens per ACK re-ceived rather than per segment acknowledged, theslow-start window opens much slower than is usu-ally assumed. We develop the exact relationship andreview the details of slow-start and delayed acknowl-

no delayed delayed ACK every

stall ACKs ACKs segment

1 2 (2) 2 (2) 2 (2)2 3 (5) 3 (5) 4 (6)3 3 (8) 5 (10) 8 (14)4 6 (14) 8 (18) 16 (30)5 9 (23) 12 (30) 32 (62)6 12 (35) 18 (48) 64 (126)7 18 (53) 27 (75) 128 (254)8 27 (80) 41 (116) 256 (510)9 42 (122) 62 (178) 512 (1022)10 63 (185) 93 (271) 1024 (2046)

Table 5: Number of segments between slow-start stalls for di�erent acknowledgment poli-cies. The cumulative number of segments sentis given in parentheses. These columns representsegsnda(i) (csegsnda(i)), segsda(i) (csegsda(i)), andsegsae(i) (csegsae(i)) from Appendix A, where i isthe stall number.

edgments in Appendix A. Table 5 summarizes our�ndings for three di�erent acknowledgment policies.The rightmost column illustrates our original expec-tations, the left column shows a lower-bound on slow-start performance.Not all the time of each stall is completely wasted:

an increasing part of each stall is spent sending pack-ets, until cwnd opens past muws. Appendix A quan-ti�es this cost with the formula slowstartTCP. In thenext section we use this result to develop HTTP overTCP transaction time.

4.3.2 Performance and discussion

We can summarize the cost of accessing an object viaHTTP over TCP by including the extra round-trip ofthe setup and the slow-start costs:

TTCP = 2 � rtt (5)

+reqmin

+processing

+replyTCP

replyTCP = slowstartTCP+replymin

9

small- medium- large- small- medium- large-

network model page page page cluster cluster cluster

Ethernet TCP 1.13 1.03 1.01 1.17 1.04 1.02caching 1.13 1.03 1.01 1.06 1.01 1.00ARDP 1.00 1.00 1.00 1.00 1.00 1.00

Fast-Ethernet TCP 2.16* 1.62* 1.19 2.92* 1.79* 1.64*caching 2.16* 1.62* 1.19 1.76* 1.19 1.06ARDP 1.53* 1.36* 1.11 1.69* 1.37* 1.37*

Slow-Internet TCP 1.50* 1.13 1.04 1.74* 1.19 1.11caching 1.50* 1.13 1.04 1.25 1.05 1.01ARDP 1.22 1.06 1.02 1.32* 1.08 1.05

Fast-Internet TCP 2.94* 2.11* 1.36* 4.60* 2.55* 2.23*caching 2.94* 2.11* 1.36* 2.34* 1.37* 1.11ARDP 2.26* 1.79* 1.26* 3.18* 2.03* 1.88*

Modem TCP 1.14 1.03 1.01 1.19 1.05 1.03caching 1.14 1.03 1.01 1.06 1.01 1.00ARDP 1.00 1.00 1.00 1.00 1.00 1.00

ISDN TCP 1.08 1.02 1.00 1.11 1.02 1.01caching 1.08 1.02 1.00 1.04 1.01 1.00ARDP 1.00 1.00 1.00 1.00 1.00 1.00

WAN-Modem TCP 1.30* 1.07 1.02 1.42* 1.10 1.06caching 1.30* 1.07 1.02 1.14 1.03 1.01ARDP 1.11 1.03 1.01 1.16 1.04 1.02

WAN-ISDN TCP 1.49* 1.13 1.03 1.72* 1.18 1.11caching 1.49* 1.13 1.03 1.24 1.05 1.01ARDP 1.21 1.06 1.01 1.31* 1.08 1.05

ADSL TCP 3.37* 3.12* 1.83* 6.01* 4.03* 3.99*caching 3.37* 3.12* 1.83* 2.87* 1.67* 1.27*ARDP 2.56* 2.64* 1.64* 4.08* 3.10* 3.32*

DirecPC TCP 3.74* 4.44* 3.36* 7.60* 7.57* 9.30*caching 3.74* 4.44* 3.36* 3.47* 2.35* 1.93*ARDP 2.82* 3.72* 2.97* 5.10* 5.75* 7.61*

Table 6: Ratios of predicted protocol transaction times to minimum transaction time fordi�erent protocol models, workloads, and network characteristics. Ratios are for the HTTP over TCP model(STCP=Smin); HTTP over TCP with connection caching, assuming no cache hit (Scache-miss=Smin, withcache hits the performance ratio is always 1); and HTTP over ARDP (SARDP=Smin). Highlighted valuesindicate overheads larger than 0.25.

10

The cost of a series of independent requests is then:

STCP = rtt+

nXi=1

(TTCP(i)� rtt) (6)

We can now de�ne the overhead of HTTP overTCP as:

overheadTCP = STCP=Smin (7)

The TCP lines of Table 6 show overheadTCP(given by Equation 7) for a variety of workloads andnetwork characteristics (we describe the other lines ofthis table in the following sections). To highlight sig-ni�cant overhead we indicate ratios higher than 1.25with italics.We can draw several conclusions from these results.

First, overhead for the Ethernet, modem, and ISDNnetworks is reasonable (less than 25% overhead) forall workloads (although adding WAN delays raisessmall-page and small-cluster overheads to signi�cantlevels). These networks have amuws at most 2 and sodo not pay any slow-start penalty. The extra round-trip cost of the three-way handshake is proportionallylargest for short responses, but even there, overheadis less than 25%.On the other hand, network such as Fast-Ethernet,

Fast-Internet, ADSL, and DirecPC have substantiallyhigher overheads because they have much highermuws sizes. The high bandwidth{delay product ofthese networks requires 7{128 segments in-transit to�ll the communications \pipe"; for small �les, sub-stantial time is spent waiting for packet acknowledg-ments. These overheads are most pronounced whensmaller workloads are processed, since larger work-loads are able to expand the congestion window andamortize the connection setup cost over the durationof the connection.Local-area networks behave di�erently than wide-

area networks in two ways that a�ect performance.First, many TCP implementations disable slow-starton LANs where congestion control is done at the linklayer. Second, most LANs allow TCP segments muchlarger than are typical in WANs (1460 bytes insteadof 512 or 536 bytes), although wide deployment of

0100

200300

400

10K

100K

1M

10M

100M

2

4

6

8

10

RTT (milliseconds)Bandwidth (bits/second)

Ove

rhea

d (r

atio

to m

inim

um tr

ansa

ctio

n tim

e)

Figure 2: TCP congestion avoidance over-head for various bandwidth{delay products(STCP=Smin), for a 512B segment size and thesmall-cluster workload.

path-MTU discovery supports larger WAN segmentsize [20]. An interesting observation is that employ-ing slow-start does not substantially a�ect perfor-mance over 10Mb/s Ethernet because large segment-size and low round-trip time result in a small muws.

To explore the e�ects of these overheads for a widervariety of network characteristics, Figure 2 examinesa single workload (the small-cluster workload) with a�xed segment size (512B) and a range of bandwidthsand round-trip times.

Again, we observe in this graph that overhead isfairly low when the bandwidth{delay product (and somuws) is small, either because of small bandwidths orsmall round-trip-times. Overhead rises sharply whenthe product of these values grows. Finally, overheadtops out at about 9 times the minimal transfer time.

This workload-dependent limit indicates networkcharacteristics where all transmission time for alltransactions is spent in the slow-start realm. (Forthe medium-cluster case the maximum possible over-head is about 16 times minimal, for large-cluster it isabout 55 times minimal.)

Note that these high overheads occur only when

11

the bandwidth{delay product is very large, as in asatellite connection. It is well known that TCP isnot the best protocol for such situations. When eitherbandwidth is low or delay is very low, TCP performsmuch better.

4.4 HTTP over TCP with connection

caching

P-HTTP [10] has been proposed as a solution toseveral problems resulting from running HTTP overTCP, and persistent connections are part of theHTTP/1.1 standard [4]. P-HTTP reuses a singleTCP connection to amortize connection setup andcongestion avoidance costs across several HTTP re-quests. When a transaction is completed, the TCPconnection is left open for subsequent transactions.This connection is reused for subsequent requests; itwill be closed if server or client demand is high, orwhen idle for a given length of time.

Stevens has suggested the use of T/TCP for HTTPtra�c [16]. T/TCP [9] is an extension of TCPenhanced to support transactions. T/TCP cachesTCP connection setup information so that subse-quent TCP connections avoid the three-way hand-shake and reduce slow-start cost. Thus, like P-HTTP, T/TCP pays the full cost of TCP setup forthe �rst connection and avoids setup costs for subse-quent connections. Cached T/TCP information willbe ushed using algorithms similar to those for break-ing P-HTTP connections (for example, using a least-recently used or a periodic time-out).

Although P-HTTP provides connection caching atthe user level and T/TCP does so in the kernel, aseries of requests in either protocol performs iden-tically (at the level of detail we consider here). Ineach protocol the �rst request pays the full cost toopen a standard TCP connection, but subsequentrequests avoid the three-way-handshake and con-tinue to develop the slow-start window. We there-fore model these protocols together as \HTTP overcaching TCP" protocols.

There are two possible costs for caching protocols,with and without cache hits:

Tcache-miss = TTCP (8)

Tcache-hit = Tmin (9)

In a series of requests to a new server the �rst willalways be a miss and subsequent requests will usuallybe hits.To quantify a series of requests we make the sim-

plifying assumption that after the �rst exchangethe congestion window will be completely opened.For the purpose of our model, the congestion win-dow is fully opened when cwnd � muws, sinceopening the window further will not impact perfor-mance. Therefore, for our assumption to be true,cwnd(stalls(dreplysize=msse)) � muws after the �rstexchange, where replysize is the size of the �rst replyfrom the server. (We analyze cwnd and stalls in Ap-pendix A.) Our workloads satisfy this assumption forthe networks listed in Table 2 satisfy this assumption.Given this assumption, all requests after the �rst

will be cache hits under conditions of normal load(under high load the client or server may terminateconnections causing additional cache misses). The�rst request may or may not be a cache hit dependingon the user's browsing pattern and server load. Wetherefore de�ne two formulas for series of independentaccesses, one assuming that the �rst request is a cachehit and one assuming that it's not:

S�rst-miss = Tcache-miss(1) (10)

+

nXi=2

(Tcache-hit(i)� rtt)

S�rst-hit = rtt+

nXi=1

(Tcache-hit(i)� rtt) (11)

= Smin

Finally, overhead is again:

overhead�rst-miss = S�rst-miss=Smin (12)

12

overhead�rst-hit = S�rst-hit=Smin (13)

= 1

The caching lines of Table 6 show the perfor-mance results of our workloads assuming that the�rst web page access does not use a cached connec-tion (overhead�rst-miss in Equation 12). Note that

for the cluster workloads, accesses after the �rst arecache hits.Several observations about HTTP over connection-

caching TCP protocols are apparent from this table.First, HTTP over caching TCP performance is thesame as standard HTTP over TCP performance forsingle page queries. Second, caching-TCP perfor-mance is somewhat better than standard TCP forthe cluster cases because connections after the �rstneed not pay for three-way handshake or slow-start.We explore this relationship further in Section 6. Fi-nally, overhead is still high for the Fast-Ethernet andFast-Internet cases with cluster workloads. In thesecases, the large bandwidth{delay product results insigni�cant overhead while the congestion window isopened even when amortized over an entire series ofconnections.Finally, if we assume that the �rst transaction

results in a cache-hit for a caching-TCP protocol(Equation 13), then caching-TCP has no overhead.Thus, when caches last long enough to encompass ac-cess of multiple clusters, caching protocols are veryhelpful. Implementation issues can limit this bene�t,however [21].

4.5 HTTP over Multiple, Concurrent

TCP Connections

Many web browsers open multiple concurrent con-nections to mitigate TCP start-up costs (HTTP overparallel connections). We can bound their perfor-mance by HTTP over TCP with and without con-nection caching. Our HTTP-over-TCP model over-estimates transmission time by not considering paral-lelism in the concurrent slow-start of each connection.HTTP over connection caching underestimates trans-mission time by assuming that there is no penalty

for slow-starts of later requests in a cluster. A betterapproximation might be obtained by treating rtt asif it were rtt=n, for n concurrent connections. Com-pletely specifying behavior with multiple parallel con-nections is an area of continuing work.

4.6 HTTP over UDP-Based

Protocols

Since web access is typically request{response ori-ented, we examine the performance of HTTP overa request{response style protocol such as ARDP.ARDP avoids TCP's three-way handshake, whileit keeps TCP's slow-start algorithm for congestionavoidance. Therefore, the time to complete a HTTPtransaction over ARDP is:

TARDP = rtt (14)

+reqARDP

+processing

+replyARDP

reqARDP = reqmin

replyARDP = slowstartTCP+replysize=bw

The total time to complete a series of independentHTTP requests is given by:

SARDP = rtt+

nXi=1

(TARDP(i)� rtt) (15)

and the overhead by

overheadARDP = SARDP=Smin (16)

The ARDP lines of Table 6 show overheadARDP(Equation 16 for the di�erent workloads and networkcharacteristics). Note that for the Ethernet, modem,and ISDN networks, HTTP transactions over ARDPresult in minimal transaction times. This con�rmsthat because of their small maximum useful windowsize (muws), these networks do not pay any slow-startpenalty.

13

On the other hand, ARDP's overhead becomes no-ticeable in the higher bandwidth{delay-product cases(Fast-Ethernet, both Internets, ADSL and DirecPC).ARDP also incurs higher overhead than TCP withconnection caching for the cluster workloads. Thisoverhead is due to the fact that ARDP always slow-starts, while caching the connection setup parametersallows the caching protocols to avoid slow-start everytime.Avoiding the three-way handshake is especially

helpful for single, brief request{response interac-tions. For a series of requests to the same server,though, ARDP performance su�ers because we donot cache congestion information between calls. Asa result of this observation we plan to provide simplecongestion-information caching in a future version ofARDP.

5 Validation

To relate the analytic results of the prior section toreal-world performance we next validate them usingtraces of actual HTTP tra�c. This validation hasthree goals: �rst, to show that we model aspectsof the protocol relevant to performance. Second, toshow that comparisons between the modeled proto-cols are valid. Finally, we will look for areas whereimplementations can be improved.Since it would be impractical to validate each of

the 150 combinations of workload, network, and theprotocols described in this paper, we instead consideronly four cases: HTTP over simple and caching TCPtransport protocols with the small-cluster workloadand Ethernet and Fast-Internet networks.

5.1 Methodology

Our experiments consisted of the four combinationsof Ethernet and Fast-Internet networks and HTTPover simple and caching TCP protocols. In all casesour server computer was a Sun SPARC model 20/71running SunOS 4.1.3 with some TCP/IP modi�-cations (IP-multicast support and a 16KB defaultTCP window size). We describe server software andclient hardware con�gurations below. In all cases our

HTTP client was a custom-written Perl script retriev-ing the small-cluster workload.11 We also logged allrelevant TCP tra�c on the server's network.

For experiments over Ethernet, the client computerwas a Sun4-20/71 identical to the server. These com-puters were connected by a dedicated 10Mb/s Ether-net. Note that SunOS bypasses TCP slow-start whenboth computers are on the same physical network.We wanted to measure the e�ects of standard TCPin a high-bandwidth, low-latency environment ratherthan that of a particular TCP implementation, so weremoved this optimization for our experiments.

For the Fast-Internet experiments the client com-puter was a Sun SPARC-20 running unmodi�edSunOS 4.1.3. Measurements between the server (lo-cated in Los Angeles) and the client (in Washing-ton, D.C.) were taken over the Internet with evening(U.S. West-coast time) background tra�c present.Average round-trip time was 133ms and bandwidthwas 0.734Mb/s (as measured by repeated FTP of a1.5MB �le) over the 11 hops between the client andour server at ISI.

Our implementation of HTTP over simple TCPwas HTTP/1.0 with an Apache 1.0.5 server. Theclient made HTTP/1.0-style requests.

For HTTP over caching TCP protocols we usedthe fourth beta version of Apache 1.1 with somemodi�cations. This server implements \keep-alive"HTTP connections, an experimental implementa-tion of persistent connection HTTP (abbreviatedHTTP/1.0+KA) similar in character to persistentconnections recently standardized as HTTP/1.1 [4].This server was slightly modi�ed to avoid two in-teractions between P-HTTP and TCP which sub-stantially reduce performance [21]. Our clientmade HTTP/1.0-style requests with the \Connec-tion: Keep-Alive" header; the server returned MIME-style headers and page contents with page size deter-mined by the Content-Length header.

11Although the program is interpreted, we have veri�ed thatit can saturate our Ethernet and so does not pose a perfor-mance bottleneck.

14

5.2 Slow-Start Validation

We have observed that a key aspect of HTTP overTCP performance is slow-start behavior. Slow-startperformance is dependent upon the client's ACKrate; when a client acknowledges every other segmentwith delayed acknowledgments the congestion win-dow opens much more slowly than if every segmentis explicitly acknowledged. Table 5 summarizes thesee�ects based upon bounds of the slow-start rate de-veloped in Appendix A.

To validate that our bounds on the slow-start rateare accurate, we examined a number of packet tracesof HTTP over caching TCP (HTTP/1.0+KA) andFTP tra�c between Los Angeles and Washington,D.C., hosts. Figure 3 shows a representative plotof the packets exchanged for the HTTP/1.0+KA re-quests for the small-cluster workload. As can be seenfrom this graph, the round-trip time is about 133msand the client acknowledges every other packet. Fromthe pattern of ACKs we can infer that no timeout-induced delayed acknowledgments occurred in thesetransactions.

To validate our slow-start model we will exam-ine two parts of the total exchange, the �rst request(from time 0 to 0.7s) and the second (from 0.7 to0.9s).

In the �rst request we see the pattern predicted bythe no-delayed-acknowledgment analysis of Table 5:2, 3, 3, and 6 segments, each with an � 1 rtt de-lay stall between them. (Note that in the 6-segmentstall the sixth segment advances the sequence num-ber by only 19 bytes and so is plotted nearly on top ofthe �fth segment, and that the ack for this segmentwas piggybacked on the next request.) From this weconclude that, in the absence of delayed acknowledg-ments, we correctly predict segments per stall.

In BSD implementations of TCP the delayed ACKtimer �res every 200ms, independent of packet ar-rival. We expect that delayed ACKs will speed upopening of the slow-start window by causing occa-sional single-segment ACKs instead of delays untiltwo packets have been received. In practice we ob-serve that delayed ACKs sometimes alter both thepattern of packet transmission between stalls and thestall delay. For the small-cluster workload, delayed

ACKs seem to have little e�ect on overall perfor-mance.

In the second request we see back-to-back trans-mission of all nine segments (again, the �nal segmentis short and is obscured by the previous segment onthe plot). This behavior is consistent with our modelof the congestion window; the window started at twoand was opened by one for each of the seven acknowl-edgments received.

Based on analysis of packet traces from whichthese examples are drawn we conclude that our lowerbound for segments-per-stall is accurate assumingthat no delayed ACKs are triggered.

5.3 Model Adjustments

Our model focuses on transport-layer issues andtherefore makes several simplifying assumptionsabout client and server software. To validate ourresults we must account for these assumptions. Inparticular:

server processing-time Our basic model assumeszero processing-time. In our experiments we ob-serve an average 3.7ms server processing-time,so we must add 3.7ms per page to our adjustedmodel.

request independence Our basic model assumesthat all requests are independent. Unlike mod-ern browsers, our simple client is not multi-threaded and so issues dependent requests (asdescribed in Section 3.2). We can see two suchstalls (marked \dependent delay") in Figure 3.We correct for this artifact by adding 1 rtt delayper page after the �rst.

inexact bandwidths Our basic model assumedthat Fast-Internet bandwidth was 1Mb/s with89ms rtt. We selected these characteristics toemulate experimental characteristics of other re-searchers. Our validation experiments insteadobserved a bandwidth of 0.734 Mb/s and a rtt

of 133ms; we correct for this using observed net-work characteristics in our revised estimate.

15

0

2000

4000

6000

8000

10000

12000

0 0.6 0.8 1

sequ

ence

num

ber

(in b

ytes

)

time since first SYN (in seconds)

dependent delays

stall 1

stall 2

stall 3

ack

request

data

first request secondrequest

thirdrequest

handshake

0.2 0.40.1 0.3 0.5 0.7 0.9

Figure 3: A packet trace of the HTTP over caching TCP (HTTP/1.0+KA) requests for thesmall-cluster workload.

Table 7 shows our original predictions and our pre-dictions adjusted for these factors. We interpret theseresults in the next section.Finally, our model assumes no packet loss. We be-

lieve that we experienced no packet loss in our Eth-ernet experiments and loss in only one of the fast-Internet transactions. Packet loss depends heavily oncongestion. Because packet loss is detected by time-out and causes re-evaluation of congestion control,even low packet loss rates can cause measured val-ues substantially longer than predicted by our model.In such cases, connection setup costs would likely beoverwhelmed by loss-recovery, and therefore, the ben-e�ts of connection caching protocols less noticeable.

5.4 Model Validation and Discussion

We undertook this validation with two goals: to com-pare our analysis to reality and to insure that com-parisons among our analytic results are valid. Weevaluate these goals below. In addition, in the pro-cess of validation we found several interactions be-

tween P-HTTP and TCP which substantially reduceperformance.Except for the case of HTTP/1.0 over Ethernet,

our validation suggests that the model, when ad-justed, is accurate within 5% of measured values.The HTTP/1.0 over Ethernet case shows a discrep-ancy of about 40%. A high bandwidth and low de-lay link (like Ethernet) makes modeled network over-head small, so other kinds of overheads (which arenot modeled) can be noticeable. Furthermore, themodeled overhead is very sensitive to latency at highbandwidths as is shown in Figures 2 and Figure 4.Another source of error in our model results from

interactions between the application-level behaviorand the underlying TCP implementation. In thecourse of validation we found two such interactionsthat crippled HTTP/1.0+KA performance [21]. Inboth cases, packets shorter than maximum-segment-size caused our TCP connection to wait for a de-layed acknowledgment, stalling data transfer for upto 200ms. We worked around both these prob-lems with small application-level changes, eliminat-

16

prediction ratioprotocol implement. network basic adjusted measured m:aTCP HTTP/1.0 Ethernet 12:8ms 26:8ms 36:8ms(10ms, �2.0ms) 1.37caching-TCP HTTP/1.0+KA Ethernet 11:4 25:4 26:6 (8.8, �1.7) 1.05TCP HTTP/1.0 Fast-Internet 977 1730 1716 (101, �20.1) 0.99caching-TCP HTTP/1.0+KA Fast-Internet 536 1070 1103 (48, �9.5) 1.03

Table 7: Validation experiments for our models. All experiments used the small-cluster workload.Basic indicates our basic (unadjusted) model, adjusted is the model corrected as described in Section 5.3,measured indicates the average over 100 trials with the standard deviation and 95% con�dence intervalsgiven in parentheses, ratio m:a shows the ratio of measured to prediction/adjusted times.

ing these sources of error. Other interactions be-tween the application-level interface and our TCPimplementation result in the transmission of shortsegments. We do not believe that this interactioncauses the catastrophic performance loss observed inthe other interactions, but it is a source of somemodel error.

We believe that our second goal has also been met:valid comparisons of what is modeled can be madebetween the protocols. The Ethernet case suggeststhat care must be taken when transaction time issmall (say, less than 50ms), but the performance ofwide-area HTTP exchanges is dominated by networkprotocol behavior described in our model. Since themodels capture the essence of performance in suchnetworks, comparisons between the models shouldcorrespond to comparisons between the protocols op-erating in actual networks.

5.5 Additional Validation

A recent technical note by the World-Wide Web Con-sortium has suggested that pipelining substantiallyreduces packet counts for HTTP/1.1 [14]. We callthe resulting protocol persistent-connection HTTPwith pipelining, abbreviated HTTP/1.1+P. A com-parison of their results with our model's predictionsis particularly interesting both because their observa-tions are made with di�erent client and server soft-ware and because they have optimized the bu�eringof their system to improve performance. This com-parison provides an additional level of validation ofour model.

Their experiments compare �rst-fetch and thecache validation of a 42KB web page with 41 em-bedded images totalling 125KB. They examined per-formance for three networks: high bandwidth, lowlatency; high bandwidth, high latency; and low band-width, high latency. These nearly match our Ether-net, Fast-Internet, and Modem results and are shownin Table 2 as N-Ethernet, N-Fast-Internet, and N-Modem, although each case used a 1460B mss. (Weestimated bandwidth for N-Fast-Internet based onexamination of their traces; they presented band-widths and rtts for the other cases.) They consid-ered four protocols: HTTP/1.0 with multiple parallelconnections, HTTP/1.1, HTTP/1.1 with pipelining,and HTTP/1.1 with pipelining and compression. Weconsider only the case of HTTP/1.1 with pipelining,designating it HTTP/1.1+P. (We did not considerHTTP/1.0 because we do not model parallel connec-tions, HTTP/1.1 because of the bu�ering problemsthey experienced, and HTTP/1.1 with pipelining andcompression because we do not model compression.)Their client software was either a custom robot in thecases we consider. Their server software was eitherApache or Jigsaw. A complete description of theirmethodology can be found in their technical note [14].

Table 8 summarizes the results of their measure-ments and our predictions. The adjusted portionof the prediction corresponds to addition of a 3:7msserver processing time.

The N-Ethernet and N-Fast-Internet cases showsubstantial discrepancy from our predicted values.We do not have enough information about theirtraces to understand the discrepancy for the N-

17

prediction ratioprotocol implement. network server basic adjusted measurement m:acaching-TCP HTTP/1.1+P N-Ethernet Jigsaw 160ms 316ms 690ms 2.18caching-TCP HTTP/1.1+P N-Ethernet Apache 160 316 520 1.64caching-TCP HTTP/1.1+P N-Fast-Internet Jigsaw 1470 1620 2860 / 1860 1.77 / 1.15caching-TCP HTTP/1.1+P N-Fast-Internet Apache 1470 1620 3500 / 2419 2.16 / 1.49caching-TCP HTTP/1.1+P N-Modem Jigsaw 49600 49800 52810 1.06caching-TCP HTTP/1.1+P N-Modem Apache 49600 49800 52360 1.05

Table 8: Additional validation experiments for our models. These experiments use the workloaddescribed in Section 5.5. Basic indicates our basic (unadjusted) model, adjusted is the model adjusted forprocessing time, measurement indicates performance as measured in [14] with two values for N-Fast-Internetas described in Section 5.5, ratio m:a shows the ratio of the measurement to prediction/adjusted.

Ethernet case at this time, although, as described inSection 5.4, in LANs, per-packet processing (which isnot considered in our model) can overwhelm connec-tion startup costs.

For the N-Fast-Internet case we also found sub-stantial discrepancy (1.77{2.16 times slower perfor-mance than predicted). Examination of their tracesfor this network con�guration shows a consistent stallof about 1 second following the third segment of thereply. We believe that this stall is due to an in-teraction between a short TCP segment and TCPsilly-window avoidance [22] similar to the odd/short-�nal-segment problem we encountered in our experi-ments [21]. If so, this interaction can be avoided byappropriate bu�ering. We correct for it by subtract-ing 1 second from the measured times. With thiscorrection our model is much closer to the measuredvalues which are 1.15{1.49 times slower.

For the N-Modem case the prediction correspondsclosely to observed performance. These experimentscorroborate our validation, suggesting that althoughour models can be inaccurate when applied to LANs,they can provide guidance to protocol designers forwide-area and low-bandwidth network conditions.We also note that models can provide a useful \san-ity check" against observed performance and led us toinvestigate the anomaly in the N-Fast-Internet case.

6 Protocol Discussion

We have presented analytic models for HTTP overseveral transport protocols and demonstrated that,with care, these models can be applied to currentas well as future network characteristics. From thiswork we can draw several conclusions about the inter-actions between HTTP and di�erent transport pro-tocols.

First, HTTP over TCP overhead is fairly low un-der networking characteristics today. Figure 4 showsa contour plot of TCP overhead for various networkcharacteristics for the small-cluster workload. In thistwo-dimensional representation of the graph of Fig-ure 2 we solve for overhead (STCP=Smin) for a setof representative points and employ linear interpola-tion between them. We show contour lines at every50% increase in overhead. Of networking technolo-gies deployed today, only the Fast-Internet case showssubstantial overhead. Modem and ISDN technologiesused for the \last mile" of the Internet today showmoderate overhead when coupled with wide-area la-tency, but little overhead if the server is nearby.

Second, TCP overhead becomes signi�cant whenthe bandwidth{delay product rises. Again referringto Figure 4, the fast-Internet performance shows sub-stantial room for improvement (current performanceis 5.20 times slower than the theoretically minimaltransfer time), as do developing last-mile technolo-gies such as ADSL and DirecPC12.

12The DirecPC region falls out of scale and is not shown in

18

0

0.05

0.1

0.15

0.2

0.25

0.3

10000 100000 1e+06 1e+07 1e+08

late

ncy

(sec

)

bandwidth (bps)

1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9 9.5

Ethernet Fast-Ethernet

Slow-Internet

Fast-Internet

Modem

ISDN

WAN-ISDN

ADSL

Figure 4: Predicted overheads of HTTP over TCP relative to minimum possible costs. Thetwo axes show primary network parameters bw and rtt; segment size and workload are �xed at 512 andsmall-cluster, respectively. Contour lines show the surface corresponding to TCP overhead (STCP=Smin);solid lines are a factor of 1, dashed lines 0.5 times minimum. Label centers indicate bandwidth and latencypoints corresponding to the sample networks described in the text.

19

In these cases HTTP optimizations become impor-tant. Figure 5 shows the advantage of connection-caching protocols in di�erent network con�gurations.In this graph, the long dashed line shows whenstandard HTTP takes 1.5 times as long as cachingprotocols, while dotted lines show intervals of 0.1.As can be seen, performance is marginally betterin many cases (Modem, ISDN, and Slow-Internet);caching protocols are 80% faster only when presentedwith network characteristics similar to Fast-Internet(moderate bandwidth and latency), ADSL (high-bandwidth, low-latency), or DirecPC (high band-width and latency). The performance improvementalways approaches a workload-dependent limit asthe bandwidth{delay product rises; in this case theasymptote is 2, the ratio of 8:4 (non-caching:caching)round-trip delays.

A recent technical note by W3C has suggestedthat pipelining substantially reduces packet countsfor persistent-connection HTTP [14]. Although theysubstantially reduce packet counts, their measure-ments of elapsed times support the conclusion thatHTTP over caching-TCP protocols o�er compara-tively modest performance improvements over low-bandwidth{delay connections today but can providesubstantial improvement when conditions approachthe Fast-Internet case.

We note that our model can be used to predictHTTP performance for network technologies onlynow being deployed such as ADSL and DirecPC. Theability to vary workload and network characteristicsis important here.

Finally, our protocol analysis has in uenced de-sign of UDP-based protocols at ISI. We are cur-rently in the process of adapting ARDP to use TCP-like congestion avoidance algorithms. As a result ofthis study we have concluded that ARDP must cacheinformation about recent congestion window behav-ior to provide good performance for large request{response exchanges.

the graph.

7 Conclusions and Future

Work

This work makes three contributions to the study ofHTTP. First, we have developed a simple analyticmodel for HTTP performance over di�erent networksand transport protocols. Second, we have used thismodel to compare the relative performance of exist-ing protocols for various network characteristics andworkloads. Finally, this model has given us insightinto the needs of request{response-style protocols.

Our analytic model of request{response perfor-mance is important both because it allows compari-son of existing protocols under current network char-acteristics and because it allows prediction of proto-col performance under future networking and work-load characteristics. Our model predicts web perfor-mance within 5% of measured values for wide-areatra�c. For networks with high bandwidth and lowdelay it becomes less accurate as non-modeled costsbecome noticeable. With this caveat, we believe thatthe model can be an e�ective means of comparingdi�erent protocols at a given network con�gurationand across di�erent network characteristics.

In addition to providing a model useful for HTTP,our analysis of slow-start behavior applies to otheruses of TCP where transient behavior cannot be ig-nored. Applicationsmight include RPC-systems, andtransfer of short e-mail messages or FTP of short �les.

By applying our model to existing protocols andnetworks we were able to draw several conclusionsabout their behavior. We con�rmed that TCP over-head is low when the bandwidth{delay product islow. In the Ethernet, modem, and ISDN cases over-head was consistently less than 25% for our work-loads. Even when latency rose to WAN levels, mo-dem and ISDN overhead was only moderate for cer-tain workloads. We demonstrated that overhead wasvery signi�cant when the bandwidth{delay productwas large.

Connection caching protocols reduce overhead forthe cluster cases (where a cluster represents the textand images that make up a single web page); wetherefore conclude that these protocols will be usefuleven if users visit only single \pages" on sites before

20

0

0.05

0.1

0.15

0.2

0.25

0.3

10000 100000 1e+06 1e+07 1e+08

late

ncy

(sec

)

bandwidth (bps)

1.1 1.21.3 1.4 1.51.6 1.7 1.8 1.9

Ethernet Fast-Ethernet

Slow-Internet

Fast-Internet

Modem

ISDN

WAN-ISDN

ADSL

Figure 5: Predicted ratio of HTTP over TCP to HTTP over caching TCP, assuming noinitial connection caching. The two axes show primary network parameters bw and rtt; segment sizeand workload are �xed at 512 and small-cluster, respectively. Contour lines show the surface correspondingto the ratio STCP=S�rst-miss; long dashed lines are a factor of 0.5; �ne dashed lines, 0.1. Label centers

indicate bandwidth and latency points corresponding to the sample networks described in the text.

21

changing servers.

Finally, validation of our model has led to in-sight into request{response protocol design and sug-gested several areas for future work. Validation ofthese experiments have detected interactions betweenapplication- and kernel-level networking that sub-stantially reduce performance [21].

A broader question is how to optimize TCP forbrief, request{response-style tra�c. We are currentlyexploring two approaches to this problem. We areexamining how TCP congestion-control informationshould be initialized for multiple connections sepa-rated by space or time [23]; this work investigatesalternatives to divide bandwidth among existing andnew connections and for reusing cached congestioninformation. Given a large initial window, we are in-vestigating how a rate-limited addition to slow-startcan prevent overloading intermediate routers [21].

We have generalized our experiences with TCPto other transport protocols. We have also foundthat the performance of protocols that fail tocache congestion-control information su�ers in high-bandwidth{delay conditions, and have modi�ed ourdesign for ARDP accordingly.

Acknowledgments

We would like to thank Kedar Jog for his early workon our HTTP benchmarking scripts. The authorswould like to thank Ted Faber for his discussionsabout web performance analysis, and B. Cli�ord Neu-man, Rod Van Meter, Steven Augart, Brian Tung,Geo� Kuenning, Joseph Bannister, Jon Postel, andthe anonymous referees for comments about the pa-per. Finally, we are grateful to Allison Mankin,Dante DeLucia, and B. Cli�ord Neuman for accessto computer facilities for our cross-Internet measure-ments.

Software Availability

We plan to make the software used in the validationof these measurements available, including the HTTPbenchmarking programs and the programs used to

evaluate the models. Please contact the authors fordetails, or watch hhttp://www.isi.edu/lsam/i.

References

[1] T. Berners-Lee, R. Cailliae, A. Luotonen, H. F.Nielsen, and A. Secret, \The World-Wide Web,"Communications of the ACM, vol. 37, pp. 76{82,Aug. 1994.

[2] V. Paxson, \Empirically-derived analytic models ofwide-area TCP connections," ACM/IEEE Transac-tions on Networking, vol. 2, pp. 316{336, Aug. 1994.

[3] T. Berners-Lee, R. Fielding, and H. Frystyk, \Hy-pertext transfer protocol|HTTP/1.0," RFC 1945,Internet Request For Comments, May 1995.

[4] R. Fielding, J. Gettys, J. Mogul, H. Frystyk,and T. Berners-Lee, \Hypertext transfer protocol|HTTP/1.1," RFC 2068, Internet Request For Com-ments, Jan. 1997.

[5] V. Jacobson, \Congestion avoidance and control,"in Proceedings of the SIGCOMM '88, pp. 314{329,ACM, Aug. 1988.

[6] C. Cunha, A. Bestavros, and M. Crovella, \Charac-teristics of WWW client-based traces," Tech. Rep.95-010, Boston University, Apr. 1995.

[7] J. Touch, \De�ning `high speed' protocols : Fivechallenges and an example that survives the chal-lenges," IEEE Journal of Selected Areas in Commu-nication, vol. 13, pp. 828{835, June 1995.

[8] M. F. Arlitt and C. L. Williamson, \Web serverworkload characterization: The search for invari-ants," in Proceedings of the ACM SIGMETRICS,pp. 126{137, ACM, May 1996.

[9] R. Braden, \Extending TCP for transactions|concepts," RFC 1379, Internet Request For Com-ments, Nov. 1992.

[10] V. N. Padmanabhan and J. C. Mogul, \ImprovingHTTP latency," in Proceedings of the Second Inter-national World Wide Web Conference, Oct. 1994.

[11] J. Touch, J. Heidemann, and K. Obraczka, \Anal-ysis of HTTP performance." Released as webpage http://www.isi.edu/lsam/publications-

/http-perf/, Currently submitted for publication,June 1996.

22

[12] J. C. Mogul, \The case for persistent-connectionHTTP," in Proceedings of the SIGCOMM '95,pp. 299{313, ACM, Aug. 1995.

[13] S. E. Spero, \Analysis of HTTP performance prob-lems." http://sunsite.unc.edu/mdma-release/http-prob.html, 1995.

[14] H. F. Nielsen, J. Gettys, A. Baird-Smith,E. Prud'hommeaux, H. W. Lie, and C. Lilley,\Network performance e�ects of HTTP/1.1, CSS1,and PNG." NOTE-pipelining-970207, availableas web page http://www.w3.org/pub/WWW/-

Protocols/HTTP/Performance/Pipeline.html, 7February 1997.

[15] R. Braden, \T/TCP|TCP extensions for transac-tions functional speci�cation," RFC 1644, InternetRequest For Comments, July 1994.

[16] W. R. Stevens, TCP/IP Illustrated, vol. 3. Addison-Wesley, 1996.

[17] B. C. Neuman, The Virtual System Model: A Scal-able Approach to Organizing Large Systems. Ph.D.dissertation, University of Washington, 1992.

[18] D. DeLucia, \Direcpc performance." Personal com-munication., Oct. 1996.

[19] R. Braden, \Requirements for Internet hosts|communication layers," RFC 1122, Internet RequestFor Comments, Oct. 1989.

[20] J. Mogul and S. Deering, \Path MTU discovery,"RFC 1191, Internet Request For Comments, Nov.1990.

[21] J. Heidemann, \Performance interactions betweenP-HTTP and TCP implementations," ACM Com-puter Communication Review, vol. 27, pp. 65{73,Apr. 1997.

[22] D. D. Clark, \Window and acknowlegement strategyin TCP," RFC 813, Internet Request For Comments,July 1982.

[23] J. Touch, \TCP control block interdependence,"RFC 2140, Internet Request For Comments, Apr.1997.

[24] S. Shenker, L. Zhang, and D. D. Clark, \Some obser-vations on the dynamics of a congestion control al-gorithm," ACM Computer Communication Review,vol. 20, pp. 30{39, Oct. 1990.

A The TCP Slow-Start

Algorithm in Detail

As described in Section 4.3.1, the TCP slow-startalgorithm limits transmission by congestion window(cwnd) when a connection begins. Table 5 summa-rizes our analysis of slow-start performance. Thisappendix looks at the details behind this table, boththe rate at which the congestion window opens andthe amount of time spent waiting.

A.1 The slow-start rate

The basic slow-start algorithm (as presented in Ja-cobson [5]) is that the cwnd begins at one segmentworth of data and then is increased by an additionalsegment for each ACK received. This algorithm re-sults in an exponential increase in cwnd; when cwnd

reaches a threshold (ssthresh, initialized to 64KB),this increase is slowed to linear (1=cwnd per ACKreceived). The exact rate of this exponential is de-pendent on the receiver's acknowledgment rate andwill be bounded by muws. In this appendix we as-sume in�nite muws and ssthresh and examine the ef-fect of di�erent acknowledgment rates. We also con-tinue with the assumptions used in the rest of thepaper: connection bandwidth and rtt are stable overthe length of the connection and packet loss does notoccur. In a real system, cwnd growth will be limitedby packet loss due to congestion, bu�er over ow, orconnection window size.We can therefore divide TCP behavior into a pe-

riod consisting of a series of transmitted segmentsfollowed by a stall.13 Formally, we de�ne segs(i) to

be the number of segments sent in the ith period.To derive segs(i) we will use cwnd(i), the congestionwindow at the beginning of the period (measured insegments), acks(i), the number of acknowledgmentmessages sent in response to segs(i), and unacked(i),the number of unacknowledged segments in period i.The number of segments sent in period i is given bythe following recurrence relation:

13Since segments are sent only in response to an ACK, seg-ments tend to be sent back-to-back (this behavior was �rstnoted by Shenker, Zhang and Clark in simulation experi-ments [24]).

23

segs(i) = cwnd(i)� unacked(i) (17)

where,

cwnd(i) = cwnd(i� 1) + acks(i� 1) (18)

cwnd(1) = 2

Our goal is to determine how many stalls occurwhen sending a given number of packets. The cumu-lative number of segments sent is helpful:

csegs(i) =

nXi=1

segs(i)

These formulae specify the sender-side aspects ofslow-start. The receiver in uences slow-start by itsACK rate. We will indicate the client's ACK policywith subscripts. For a client that acknowledges eachpacket,

acksae(i) = segsae(i)

unackedae(i) = 0

so from Equations 17 and 18,

segsae(i) = cwndae(i)

cwndae(i) = cwndae(i� 1) + acksae(i� 1)

cwndae(1) = 2

The recurrence relation for segsae(i) simpli�es tothe familiar exponential:

segsae(i) = 2i

The fourth column of Table 5 shows sample valuesof segsae(i) and csegsae(i), the cumulative number

of segments sent when clients acknowledge every seg-ment.These equations describe TCP behavior for older

(4.3BSD-Tahoe) implementations; modern imple-mentations implement delayed acknowledgments [19].

A.2 Delayed Acknowledgments

TCP implementations with delayed ACKs sendACKs only after receipt of two full-size segments ora delay of up to a half-second. (Most BSD-derivedimplementations limit this delay to 200ms.) This ap-proach avoids ACKs for many short segments whilepreserving TCP ACK-clocking. It risks transient ef-fects in data-stream start-up [21], and it also reducesthe rate of cwnd growth.We can place a lower-bound on the ACK rate by

assuming that delayed ACKs never occur (or thatthey occur only when all segments have been ac-knowledged). If we assume that delayed ACKs weretimed from receipt of the last (odd) packet, and if thertt was less than the delay, then delayed acknowledg-ments will never trigger. We adjust for this in ourrecurrence by halving the number of ACKs per stall,rounding and carrying over appropriately:

acksnda(i) =jsegsnda(i) + unackednda(i� 1)

2

k

unackednda(0) = 0

unackednda(i) = segsnda(i) + unackednda(i� 1)

� acksnda(i)� 2

Again, from Equations 17 and 18,

segsnda(i) = cwndnda(i)� unackednda(i)

cwndnda(i) = cwndnda(i� 1) + acksnda(i� 1)

cwndnda(1) = 2

The e�ects of this algorithm on slow-start perfor-mance are illustrated in the second column of Ta-ble 5, with csegsnda(i) shown in parentheses. Al-though both csegsae(i) and csegsnda(i) grow expo-nentially, csegsnda(i) lags substantially.

24

This description is slightly more pessimistic thanactually occurs in Berkeley TCP implementations.In BSD the delayed ACK timer �res independent ofsegment receipt every 200ms, so we expect delayedACKs to be generated occasionally, each time ac-knowledging a single segment.

We can place an upper bound on Reno's perfor-mance by assuming that the delayed-ACK timer al-ways �res immediately for the last odd-numberedsegment of any stall. This means that the re-ceiver acknowledges every other packet and delay-acknowledges odd packets. The revised relations are:

acksda(i) =lsegsda(i)

2

m

unackedda(i) = 0

And, from Equations 17 and 18,

segsda(i) = cwndda(i)

cwndda(i) = cwndda(i� 1) + acksda(i� 1)

cwndda(1) = 2

Both segsda(i) and csegsda(i) are shown in thethird column of Table 5. While csegsda(i) is some-what larger than csegsnda(i), it is still much lowerthan csegsae(i).

A.3 Amount of Wasted Time

We would like to quantify the amount of time wastedduring each stall. An upper bound on wasted time isone rtt per stall: the time an ACK takes to return tothe server and its replacement segment to travel tothe client. A more accurate estimate would considerthat the ACK which triggers the start of the nextperiod is generated by the �rst one or two segmentsof the current period. Following the ACK, therefore,the client spends time usefully receiving any othersegments of the �rst period. We can see this in Fig-ure 1; in the �rst stall, the second segment is received

after the �rst ACK has been sent. (The client heremust implement the ACK-every-segment policy.)We can quantify the amount of useful work ac-

complished during a stall, and from there the exactamount of wasted time:

usefulstalltime(i) = (segs(i)� k)=bw

wastedstalltime(i) = rtt� usefulstalltime(i)

where k is either 1 (if every segment is acknowl-edged) or 2 (if delayed acknowledgments are used).To determine the amount of wasted time for an en-

tire transaction we must know the number of stallsthat occur over transaction. Let ss segs be the num-ber of segments sent while TCP slow-starts. A con-nection will slow-start until it either runs out of datato send, reaches muws (and so is sending data con-tinuously), or reaches ssthresh and therefore beginscongestion avoidance. Thus:

ss segs = min(muws; replysize=mss; ssthresh)

The number of stalls across ss segs, stalls(ss segs)is then the smallest n such that:

ss segs � csegs(i)

For a given ss segs, stalls(ss segs) can be obtainedfrom Table 5's �rst column using the appropriate re-ceiver's ACK algorithm. Finally, we can determinethe slow-start delay for a transaction:

slowstartTCP =

stalls(ss segs)Xi=1

wastedstalltime(i)

25