IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 13, NO. 7, SEPTEMBER 1995 1

Billing Users and Pricing for TCPRichard J. Edell, Nick McKeown and Pravin P. Varaiya

Abstract| This paper presents a system for billing users

for their TCP tra�c. This is achieved by postponing the

establishment of connections while the user is contacted,

verifying in a secure way that they are prepared to pay. By

presenting the user with cost and price information, the sys-

tem can be used for cost recovery and to encourage e�cient

use of network resources. The system requires no changes to

existing protocols or applications and can be used to recover

costs between cooperating sites. Statistics collected from a

four day trace of tra�c between the University of Califor-

nia, Berkeley and the rest of the Internet demonstrate that

such a billing system is practical and introduces acceptable

latency. An implementation based on the BayBridge pro-

totype router is described. Our study also indicates that

pricing schemes may be used to control network congestion

either by rescheduling time-insensitive tra�c to a less ex-

pensive time of the day, or by smoothing packet transfers to

reduce tra�c peaks.

Keywords| Internet economics, network tracing, usage-

accounting

I. Introduction

SINCE its creation in 1986, the NSFNET has sustainedan extremely rapid growth rate. With an estimated

20 million people reachable by electronic mail, the NSFfunded backbone is serving a multitude of academic andcommercial organizations. The NSF subsidy for NSFNETand various regional networks amounts to $20 million peryear, of which about $12 million is for NSFNET. The totalcost of the Internet is estimated to be about $200 millionper year. It is likely that NSFNET will soon be gone, sothat Internet will be run entirely by commercial and non-pro�t carriers, and all costs (and pro�ts) will be recoveredthrough charges and pricing schemes.

If all users within an organization generated similaramounts of tra�c, cost recovery could be equitably andsimply achieved by dividing the cost equally among users.However, as might be guessed, and as we shall demonstrate,di�erent Internet users generate widely di�erent amountsof tra�c. Pricing schemes or charges that are designed torecover costs fairly from this diverse population of usersand to allocate network resources in an e�cient mannerrequire the capability to meter individual user tra�c andto present users with prices and charges in a way that en-courages e�cient network use [1]. In this paper, we presenta billing system that can be used to meter users' wide area

Manuscript receivedNovember 1, 1994; revised April 15, 1995. Thisresearch was funded by grants from Paci�c Bell, the California StateMICRO Program 93-152 and NSF grant IRI-9120074.R. Edell and P. P. Varaiya are with the Department of Electri-

cal Engineering and Computer Sciences, University of California,Berkeley, Berkeley, California 94720 USA. Their email addresses arefedell,varaiyag@eecs.berkeley.edu.N. McKeown is with the Departments of Electrical Engineering and

Computer Science, StanfordUniversity, Stanford, CA 94305 USA. Hisemail address is nickm@ee.stanford.edu.

c 1995 IEEE

TCP tra�c, involving the users in the decision to consumeresources and making them accountable for the tra�c thatthey generate.

A number of network billing systems have been previ-ously proposed [2],[3]; in addition, one billing system hasbeen implemented [4]. An excellent discussion of methods,costs and bene�ts of usage metering as well as a proposedbilling infrastructure is given in [2]. In [3], a ow-based ac-counting mechanism is de�ned based on IP data that owsbetween end hosts. Brownlee reports experiences with im-plementing a network billing system [4]. But none of thesesystems provide a mechanism for identifying and involvingindividual users; i.e. these systems identify hosts machinesbut not the users of those machines. In 1990, Estrin etal. suggest some research topics towards usage billing andfeedback [5]. In particular, they propose the instrumen-tation of current networks to determine the performanceimplications of usage metering, and highlight the problemof identifying and authenticating the end user.

Our billing system is designed to meter users (i.e. thepersons using hosts) of the TCP connection-oriented pro-tocol [6],[7], which represents almost all of the tra�c onthe Internet. It is because a connection is established priorto the transfer of data that we are able to identify reliablythe originating user and verify that they are prepared topay for their tra�c. An important feature of this billingsystem is that this can take place without modi�cation tothe TCP/IP protocols, or applications.A usage-based billing system in which the user is directly

involved will a�ect the way in which the Internet is used.Several schemes have been proposed for controlling conges-tion via pricing which require that tra�c is metered andinvolve the user in the feedback mechanism [8]. However,it has not been previously shown that real-time meteringof tra�c for high-speed networks is feasible, or that it ispractical to involve the user in the feedback.

A billing system that controls the access of individualusers will also a�ect the way in which the Internet grows.Users within an organization who need a higher bandwidthconnection to the commercial data carrier can pay for andget preferential use of the improved service.

Usage billing for network tra�c is a controversial subject[6],[9]. It is not the purpose of this paper to argue for thewidespread adoption of usage billing. Our aim is to demon-strate that it is technically feasible to implement a billingsystem that involves users and makes them accountable fortheir tra�c.

In Section II, we provide a detailed description of theproposed billing system. In Section III, we describe anextensive feasibility study of the system, based on a traceof the tra�c that leaves and enters the Berkeley campus.Section IV shows how the BayBridge, a prototype router,

2 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 13, NO. 7, SEPTEMBER 1995

could be to used to meter tra�c. Section V discusses someother relevant measurements. We draw some conclusionsfrom our study in Section VI.

II. Generic System Description

A number of features are essential to any billing system:� No changes to existing Internet protocols|Because ofthe huge number of installed end systems, bridges androuters, it is important that an Internet billing systemwork with the existing Internet protocols. It meansthat the billing system should not require the use ofany special option �elds by end systems (for example,IP options or TCP options).

� No changes to existing applications | Many applica-tions such as ftp, email, gopher and mosaic are inwidespread use today and collectively contribute tomost of the tra�c on the Internet [10]. A billing sys-tem should be able to meter tra�c for these and otherexisting applications without change.

� User Involvement | If a billing system charges indi-vidual users for their tra�c, it must �rst determine theidentity of the user. It should also obtain the explicitapproval from the user or an authorized agent thatthey will pay for the resources consumed. For securityand credibility, this approval should be authenticated,for example with Kerberos [11]. And �nally, the billingsystem should provide accurate and credible on-linefeedback to the user as they consume resources. Thisimplies that the metering of tra�c should be exact andnot based on tra�c sampling.

It would be desirable for a billing system to have thefollowing additional features:

� Provide on-line reporting of aggregate network usage

| This enables schemes that control network conges-tion based on global tra�c measures. The control maybe implemented using priorities or by a time varyingpricing policy [8].

� Allow continued sharing of information and resources

| The growth of the Internet can be attributed toapplications that encourage the sharing of informationand resources between remote sites. It would be ad-vantageous if billing systems could cooperate to iden-tify the user and bill them for their tra�c.

A. Detailed Description of Billing System

A block diagram of the billing system is shown in Fig-ure 1. This �gure shows a single administrative domainthat is connected to the outside world via a Billing Gate-way (BGW) and a billed link. The administrative domaincontains a collection of networks, users and hosts. Thebilling system controls users' access to the billed link byallowing or disallowing TCP connections and by meteringusers' TCP tra�c once a connection has been established.The basic operation is as follows: when a user attempts

to establish a TCP connection with the outside world, theBGW postpones the establishment of the connection whileit tries to identify the originating user. The system con-tacts the user, verifying that they want to establish the

Billed Link

PurchasingAgent

User-idDaemon

AccessController

BillingGateway

Host A

Host B

NameServer

12

Kerberos

3,4

5,6

2,11

7,10

8,9

Administrative Domain

Server

Router1

BillingRecords

Fig. 1. Components of the billing system showing the major com-munications required to establish a connection.

connection and that they are prepared to pay for it. If theuser accepts the connection, the normal connection estab-lishment is allowed to continue and the BGW will beginmetering this connection's tra�c.

We will describe the operation in detail by way of anexample and by referring to the communications markedon Figure 1 and the corresponding time diagram in Fig-ure 2b. The timeline in Figure 2b is an extension of Fig-ure 2a showing all the communications involved in settingup a connection between the originating local host and theremote host.

In the example, the user sits at Host A but is logged into Host B. The user's application on Host B attempts toestablish a TCP connection to a remote host. Consideringeach communication in turn:

1. Host B initiates the connection by sending a TCPSYN message. The BGW recognizes the TCP SYNmessage as an attempt to establish a new connectionto the outside world. The BGW holds onto the TCPSYN message while it determines whether or not theconnection should be allowed and access granted to thebilled link. This is achieved by communications (2-11)to identify and contact the user, determine whetherthe connection will be allowed and to set up the nec-essary state for metering and billing records. The con-nection is always referred to by its unique identi�er:[(source (address,port), destination (address,port)].

2. The BGW contacts the Access Controller which isresponsible for determining whether the connectionshould be allowed. It achieves this by communicat-ing with each of the components in turn.

3,4. The Access Controller asks the User Identi�cationDaemon (Useridd) to identify the user.

5,6. The Access Controller asks the User Name Serverto locate the user's Purchasing Agent.

7. The Access Controller asks the user's PurchasingAgent to verify that the user wishes to establish andpay for this connection.

8,9. The Purchasing Agent may be con�gured to respond

EDELL, MCKEOWN AND VARAIYA: BILLING USERS AND PRICING FOR TCP 3

Originating Host Destination Host

1st SYN

2nd SYNACK 1st SYN

ACK 2nd SYN

(a) Normal 3-Way handshake to establish connections

Originating Host Destination Host

1st SYN

2nd SYNACK 1st SYN

ACK 2nd SYN

(b) Billing System Communications Involved in Establishing a Connection

1st SYN (delayed)

BGW Access

Useridd

Name-server

Controller

UserPurchasingAgent

1 234567

10

8

911

12

S1S2

S3

Fig. 2. The communications required to establish a connection with-out and with the billing system.

automatically on behalf of the user, or may request theexplicit authorization of the user by means of a dialogbox on the user's screen.

10. The Purchasing Agent responds to the Access Con-troller authenticating its reply using Kerberos.

11. If the user con�rms that they wish to establish andpay for the connection, the Access Controller tells theBGW to allow access to the billed link and meter theconnection.

12. The BGW creates an entry for this connection in itsconnection tables and forwards the TCP SYN messagetowards the remote host.

Now that the BGW knows to allow this connection, all fu-ture messages for this connection in either direction will beforwarded without further delay. This means that when re-mote host responds with a TCP SYN+ACK, the BGW willforward the message, updating its connection table entry.

B. Components

The components in Figure 1 fall into three categories.The �rst category is the Access Controller which was de-scribed in detail above.The second category is user involvement. This contains

three components: User Identi�cation Daemon, User NameServer and Purchasing Agent. Together these componentsidentify the originating user, obtain veri�cation from the

user and provide feedback to the user as resources are used.� The User Identi�cation Daemon (Useridd) is a systemdaemon process that runs on all multi-user machineswithin the administrative domain. Request messagesare sent to the Useridd containing the unique con-nection identi�er. The Useridd examines kernel datastructures to determine and return the name of theuser who originated the connection.

� The User Name Server is a standard network namingservice which when supplied with a user name willreturn the location of the user's Purchasing Agent.

� The Purchasing Agent1 is a user-level process responsi-ble for verifying that the user is prepared to pay for theTCP connection. In its simplest form, the PurchasingAgent could just pop up a dialog box on the user'sscreen asking for a simple accept/reject response. Al-ternatively it may be con�gured to automatically re-spond to some or all of the requests that it receivesfrom the Access Controller. For example, it could becon�gured by the user to: (i) automatically accept allconnections below a certain price, but verify with theuser before accepting more expensive connections; (ii)automatically accept all connections to speci�c well-known ports, but verify with the user before acceptingothers; or (iii) automatically accept all connections tocertain destinations. The Purchasing Agent proves itsauthority to represent the user by including a Kerberosauthenticator in its response.

The third category is metering. The two components inthis category are the BGW and Billing Records.

� The BGW is a specialized router that maintains a tableof established TCP connections for metering tra�c, inaddition to performing the usual IP routing functions.The BGW must understand TCP connections so thatit can determine which connection record to update.New connections are TCP messages for any connec-tion that the BGW has not seen before. This includesconnections that establish via other BGWs.The BGW must determine when connections haveclosed so that the entry in the connection table canbe freed. Connections may close in a number of dif-ferent ways and the TCP FIN messages that close theconnection may travel via a di�erent BGW. So thatconnections can be removed from the tables in a timelymanner, the BGW times out connections that are in-active for a long period. If the connections becomeactive again, they are added back into the table.TheBGW maintains a table of all connections with a sepa-rate entry for each established connection keyed by itsunique connection identi�er. This means that the con-nection tables maintained by the BGW may be muchlarger than IP routing tables. It also means that thekey into the connection table is much larger: for an IProuter the key would normally just be the 32-bit desti-nation network address. For a BGW with TCP/IP thekey is 96-bits. In Section III.D we will consider how

1For security purposes, this process is usually run on the host atwhich the user sits.

4 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 13, NO. 7, SEPTEMBER 1995

large the connection tables need to be and compareseveral schemes for looking up entries in the connec-tion tables.

� The Billing Records are responsible for maintainingrecords of metered connections and for providing on-line feedback to the user as the connection progresses.

C. Simpler Cases

A number of possible simpli�cations could be used toreduce the number of communications described above.Each simpli�cation corresponds to bypassing communica-tions shown in Figure 2b, \S1" - \S3". The simpli�cationsreduce the number of communications involved in estab-lishing a connection, thus reducing the latency seen by theuser.1. No user interaction | Do not explicitly involve theuser in the connection acceptance. This was discussedabove and would be achieved by con�guring the Pur-chasing Agent to accept connections automatically.The advantages are that it would make the connec-tion setup faster and would not burden the user withinvolvement in every connection.

2. No Purchasing Agent interaction|Con�gure the Ac-cess Controller so that connections from some users orfor some applications are accepted automatically. Thiseliminates communications (5-10).

3. By machine billing|Con�gure the Access Controllerso that connections from some machines are acceptedautomatically. This would be used when a machine isused by a single user, or if it has been agreed that thetra�c will always be paid for regardless of the user. Infact, this would be necessary for machines that can-not support the Useridd process (this would includesingle-process personal computers). In this case, weeliminate communications (3-10).

It is important to note that both simpli�cations 2 and3 compromise the security of the billing system as neitherobtains authenticated feedback from the user.

D. Complications & Gotchas

We believe that the billing system described above is fea-sible and implementable. However, there are several thingsthat can make the implementation more di�cult, or makethe billing model less appropriate. We consider some ofthese below.

� IP Fragmentation| IP datagrams may be fragmentedby routers as they progress through the network. TheBGW must interpret TCP messages and so it must beable to reassemble fragmented IP datagrams. In ourfeasibility study, we found that less than 0.01% of ourwide-area datagrams are fragmented2.

� Malicious IP Fragmentation| IP datagrams may alsobe severely fragmented by the end station in an at-tempt to circumvent the billing system. This maybe overcome by refusing to forward fragmented data-grams with too few data bytes.

2All of the machines on the Berkeley campus that fragment theiroutgoing tra�c are Macintoshes.

� Hiding data in option �elds | To avoid being billed,the user may attempt to hide data in IP or TCP op-tions �elds. This may be overcome by metering pro-tocol header bytes as well as data bytes. This way,anomalies can be detected and billed for accordingly.

� Time varying route | The packet-switched Internetallows datagrams within a connection to be deliveredvia di�erent routes. If the change is due to load bal-ancing, we can expect the price for delivery to be un-changed. But if there is a change in topology, the pricemay change and should be communicated back to theuser.

� Connections on behalf of, but not originated by a user

| The main reason for involving the user is to makethem accountable for their tra�c. The basic systemachieves this by billing the originator of a TCP con-nection. However, it would be more appropriate tobill the bene�ciary of the connection. In practice, theoriginator of a connection is not necessarily the bene�-ciary. For example, if a process on one host originatesa connection so that it can transfer data to or from aremote host, who is the bene�ciary of the transfer?We discuss a solution to this problem below when wedescribe how cooperative billing can be achieved.

E. Enhancements to the Basic Billing System

� Cooperative Billing Between Administrative Domains

| If two Administrative Domains wish to bill for aconnection that ows between them, they must �rstagree which domain will pay for the connection. Thatdomain then needs to determine which of its users tobill. The user that is billed should be the bene�ciaryof the connection.The following method could be used to determine thebene�ciary. When the connection is established, theBGW asks the originating user if they will pay for theconnection. If the user is the bene�ciary, they shouldagree to pay. Alternatively, the user may request thatthe remote end pay. In this case the local BGW con-tacts the remote BGW which in turn contacts the re-mote user asking if they are prepared to pay. If neitheruser agrees to pay, the connection is refused. If eitherof the end points of the connection is a service, ratherthan a user, its local Access Controller could be con-�gured to accept connections on its behalf.

� User Speci�ed Limits | The Purchasing Agent couldrespond to the Access Controller with conditions andlimits to its acceptance. For example, the user maybe willing to pay up to a speci�ed amount for a con-nection, or for a speci�ed number of packets or bytes.The user may only want to pay for tra�c as long asthe price for the connection stays below a speci�edthreshold. This would allow the implementation of atime-varying threshold pricing policy agreed upon bythe user and policed by the BGW [8]. If during the lifeof the connection any of the limits or thresholds are ex-ceeded, the BGW contacts the user before forwardingany more messages.

EDELL,MCKEOWNANDVARAIYA:BILLINGUSERSANDPRICINGFORTCP

5

�Meterin

gUDP

Tra�c|

Ourbillin

gsystem

isde-

signed

tometer

TCPtra

�c,which

curren

tlycom

prises

alm

ost

allof

thetra�

contheIntern

et.How

ever,a

growingproportion

oftra

�cis

usin

gtheUDP

pro-

tocol,

e.g.MBONE.In

prin

ciple,

ourbillin

gsystem

couldalso

meter

tra�cfrom

these

applica

tions.There

aretwoprin

cipal

requirem

ents.

First,

thatprior

totheBGW

forward

ingdatagram

sitmust

bepossib

leto

identify

theuser

inanequivalen

tway

totheUser

Identi�

cation

Daem

on.Secon

d,that

itbepossib

lefor

theBGW

torelate

eachdatag

ramto

ametered

ow

.How

practical

thisiswill

dependontheapplication

.

III.FeasibilityStudy

Inthissectio

n,wewill

use

theresu

ltsfro

madetailed

studyto

dem

onstrate

that

ourbillin

gsystem

isfeasib

leeven

forlarge

campusnetw

orks.

Ourresu

ltsare

derived

from

atrace

ofnetw

ork

tra�con

theBerk

eleycam

pus

FDDIback

bonenetw

ork. 3

Most

ofourbillin

gsystem

'scom

plex

ityiscon

nection

re-lated

.Therefore,

ourfeasib

ilitystu

dymeasures

several

keystatistics

aboutTCP

connectio

ns.

These

statisticsare:

connectio

nsetu

ptim

e,con

nection

setupinten

sity,active

connectio

ncount,andcom

plex

ityof

search

ingthecon

nec-

tiontab

le.Under

ourbillin

gsystem

,con

nectio

nestab

-lish

mentisdelayed

while

locatin

gtheuser

andthen

ver-ify

ingthatthey

will

pay;therefore,

wemeasu

redcon

nec-

tionsetu

platencywith

outthebillin

gsystem

andestim

atetheaddition

alcon

nection

setuplaten

cyintro

duced

bythe

billin

gsystem

.Several

components

ofourbillin

gsystem

areactiva

tedonce

forevery

connectio

n;therefore,

wemea-

sured

connectio

nsetu

pinten

sityto

determ

inethereq

uired

throughputfor

these

components.

TheBGW

main

tainsa

tableof

active

connection

s;therefo

re,wemeasu

retheevo-

lution

ofcon

nectio

ncou

ntto

determ

inethereq

uired

size

ofthecon

nection

table.

Thecon

nection

tableisfreq

uently

searched;therefo

re,wemeasure

theperform

ance

ofdi�er-

entsea

rchmeth

odsto

determ

inethecomplexity

ofcon

nec-

tiontablelookup.

TheBGW

processes

every

data

gramfor

meterin

g;there-

fore,

ourfeasib

ilitystu

dymeasures

datag

raminten

sityto

determ

inethereq

uired

datagram

throughputfor

theBGW.

A.AbouttheTrace

TheBerkeley

campuscommunity

iscomposed

of40,000

students,

faculty

andsta

�mem

bers.

Thecampusnetw

orkintercon

nects

22,000hosts

ofwhich

6,312

hosts

partici-

pated

inWANTCPcon

nection

sdurin

gthestu

dyperio

d.

Thecam

pusnetw

ork

isconnected

totheIntern

etbytwo

1.5M

bps\T

-1"lin

ksandasin

gle

boundary

router.

We

consid

erevery

thingontheoth

ersid

eof

thisrouter

tobe

the\ou

tsideworld

."4

3Durin

gthemonth

ofSeptem

ber

1994,Berk

eleycontrib

uted

the

13th

largestamountofWANtra

�ctotheNSFNEToutofover22,000

registe

rednetw

orks[12].

4This

de�nitio

nof\outsid

eworld

"inclu

des

some

geographi-

cally

and

topologica

llynearby

sitessuch

asLawren

ceBerk

eleyLaboratorie

s.

0

10

0

20

0

30

0

40

0

50

0

60

0

70

0

80

0

90

0

10

00

11

00

Intensity (Packets/sec)

01

/17

/95

01

/18

/95

01

/19

/95

01

/20

/95

Avg

Ou

tbo

un

dA

vg In

bo

un

d

Fig.3.

Avera

geinten

sityofdatagramsenterin

gandlea

vingthe

Berk

eleycampus.

Ourtrace

was

obtain

edbyloggin

gall

TCP/IP

head

ersthat

appeared

ontheBerk

eleyback

bonenetw

ork.A50n

stim

estampisattach

edto

eachlog

entry.

Theanaly

sispre-

sented

inthispaper

consid

ersWAN

tra�cover

thefou

rdaysfrom

midnigh

tMonday

January

16throu

ghmidnigh

tFrid

ayJan

uary

20,1995.

Durin

gourstu

dyperio

d,we

measu

red89

gigabytes

ofWANtra�

cin

439million

data-

grams.

OfthisWANtra�

c,92%

ofthebytes

and92%

ofthedatagram

swere

forTCPtra�

c.OfthisWAN

TCP

tra�c,

86%of

thebytes

and84%

ofthedatagram

swere

frommulti-u

serhosts

(seeTableI).

Figu

re3show

show

the

averagerate

ofinboundandoutbounddatagram

schanged

overourstu

dyperio

d.

B.Additio

nalConnectionSetupLatency

Figu

re2a

illustrated

thecom

munication

involv

edin

es-tab

lishingaTCPcon

nection

with

outbillin

gandFigu

re2b

illustrated

theaddition

alcom

munication

required

byour

billin

gsystem

.Obviou

slythisaddition

alcom

munication

increases

connection

setuptim

e.Therelativ

esign

i�can

ceof

this

addition

aldelay

dependson

thedistrib

ution

ofcon

nection

setuptim

es.Figu

re4show

sthecumulative

distrib

ution

ofmeasu

redcon

nection

setuptim

esfor

out-

boundcon

nection

swith

outthebillin

gsystem

.Ouranaly

-sis

show

edthat

80%ofthecon

nection

stook

betw

een24m

sand734m

sto

establish

,with

amedian

timeof

116ms,and

anaverage

timeof

519ms.

6 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 13, NO. 7, SEPTEMBER 1995

TABLE I

Amount of TCP WAN Traffic by Host Type.

Number Per Host Averagesa Percentage in Category

Host Type of Hosts Connections Datagrams Bytes Datagrams Bytes

Multi-user Computers 2,119 1,306 153,226 32,428,047 83.8% 85.7%

Personal Computers 1,527 64 5,447 1,167,732 2.1% 2.2%

Unknown 2,666 237 20,413 3,617,429 14.0% 12.0%

aThese are four day averages.

0.001 0.01 0.1 1 100

10

20

30

40

50

60

70

80

90

100

Connection Setup Time (Seconds)

%

Fig. 4. Cumulative distribution of setup latencies for connectionsthat originate on the Berkeley campus and connect to a hostelsewhere on the Internet.

Our trace experiment included the identi�cation of userson selected machines using the \User Identi�cation Dae-mon" (Useridd) implemented as a user-level process. Ouranalysis5 showed that the average response time was 84ms.

We can use these results to estimate how long it wouldtake for the rest of the connection setup overhead. Let'sassume that each of the other three request/reply pairs willtake approximately the same amount of time. Then theaverage additional delay is 336ms. Therefore, the averageconnection setup time with the billing system would be855ms, an increase of approximately 65%.

5This data is from an earlier trace recorded from midnight Novem-ber 10 through midnight November 14, 1994.

0 10 20 30 40 50 60 70 800

1

2

3

4

5

6

7

8

9

Intensity (Connections/Second)

Pro

babi

lity

Den

sity

(%

)

50 percentile 99.9 percentile

Fig. 5. Rate at which connection are established to and from theBerkeley campus.

C. Connection Throughput Requirements

This section discusses the throughput requirements forthe functional blocks. Most of the billing system functionalblocks are activated once per connection setup. Therefore,the peak intensity of connection setup is an important pa-rameter. Figure 5 shows the distribution of connectionsetup intensities over one second intervals. The peak ratewas 79 connection setups per second, and 99.9% of sec-onds had 41 or fewer connection setups. Therefore, thethroughput requirements for the functional blocks can beeasily achieved.

D. Size of Connection Table

While a connection is active, an entry must be main-tained in the BGW connection table. For the billing system

EDELL,MCKEOWNANDVARAIYA:BILLINGUSERSANDPRICINGFORTCP

7

60

0

80

0

10

00

12

00

14

00

16

00

18

00

20

00

22

00

24

00

26

00

Active Connection Count

01

/17

/95

01

/18

/95

01

/19

/95

01

/20

/95

with

ou

t time

ou

t

Fig.6.

Acountofthenumberofcurren

tlyesta

blish

edconnectio

ns

toandfro

mtheBerk

eleycampus.

tobefea

sible,

these

tables

mustnotbetoolarge.

Wehave

used

ourtra

cedata

toreco

nstru

cttheconnectio

ncou

nt

evolutio

nin

twodi�eren

tways.

The�rst

reconstru

ction

meth

od

removes

connection

sfrom

thetab

leonly

when

thetrace

data

indica

testhat

aconnection

hasclosed

.Figure

6show

sthisrecon

struc-

tion.This�gure

clearlyexhibits

anupward

drift

inthe

connection

count.

Som

eofthisdrift

canbeattrib

uted

tolong-duratio

nconnectio

nsthatstart

butdonot

enddur-

ingthetrace

perio

d.Most

ofthedrift

isfro

mhosts

that

failto

prop

erlyclo

se-dow

ncon

nection

s.Fromthis,

wecon

-clu

dethat

aBGW

cannot

reliablydetect

theclo

singofall

connectio

ns.

Thesecon

drecon

structio

nmeth

odincorp

orates

aninac-

tivity

time-o

ut.

Figure

7show

stheevolution

ofcon

nec-

tioncou

ntifconnectio

nsare

timed-ou

tafter

60minutes.

With

thetim

e-outpolicy

themaximum

connectio

ncou

nt

isbelow

1,350entries

andthere

isnolongeranysign

i�can

tupward

drift.

E.Complexity

ofConnectionLookup

TheBGW

meters

every

TCPmessa

gethat

itforw

ards.

Itistherefore

importan

tthat

connection

tableentries

arelocated

quick

ly.These

entries

arekeyed

bya96-bitcon

nec-

tionidenti�

er.Usin

gthiskey

todirectly

index

a296entry

table

inRAM

orusin

ga96-b

itwideCAM

(conten

tsad-

dressab

lemem

ory,

sometim

escalled

asso

ciativemem

ory)

40

0

50

0

60

0

70

0

80

0

90

0

10

00

11

00

12

00

13

00

14

00

Active Connection Count

01

/17

/95

01

/18

/95

01

/19

/95

01

/20

/95

with

time

ou

t

Fig.7.Acountofthenumber

ofcurren

tlyesta

blish

edconnectio

ns.

Connectio

nsare

timed-outafter

aninactiv

ityperio

dof60min-

utes.

isim

practical.

Wehave

founde�ective

hash

ingfunction

sfor

hard

ware

andsoftw

areim

plem

entation

susin

geith

erCAM

orRAM

based

connection

tables.

Weem

ulate

thebehavior

ofeach

hash

ingfunction

with

ourtrace

data,

measu

ringtheaver-

agenumberoflookupsthat

arereq

uired

to�ndthecorrect

tableentry.

For

CAM

based

tables,

wecon

sidered

three

widthsthat

arecom

monlyavailab

leandused

incom

mercialIP

routers:

48,32

and16

bits

wide.

Ahash

ingfunction

isapplied

tothe96-b

itkey

tored

uce

itto

a48,

32or

16-bitindex

which

isused

to�ndthe�rst

match

intheCAM.Asso

ciatedwith

eachCAM

entry

isashadow

entry

atthesam

eo�set

inRAM.Theshadow

entry

contain

san

orthogon

alportion

ofthekey

that

when

combined

with

theindex

canbeused

torecover

thefull96-b

itkey.

Theorth

ogonalportion

sof

the

keyare

compared

toverify

that

theentries

match

.Ifthey

donot,

then

thesearch

isrep

eatedlookingfor

thenext

match

intheCAM.Figu

re8illu

stratesthisstru

cture.

Figu

re9show

show

CAM

hash

ingfunction

sfCAM;1 ,

fCAM;2 ,

fCAM;3

map

the96-b

itcon

nection

identi�

erto

48,32

or16

bits.

Figu

res10a-c

plot

theaverage

numberof

comparison

spertab

lelookupover

thetrace

perio

d.Wesee

that

function

fCAM;1ismuch

lesse�ective

than

theoth

-ers.

Thisisbecau

setheport

numbers

forftp

andftp

-data

di�er

byexactly

one.

Function

sfCAM;2andfCAM;3both

overcomethisprob

lem.Figu

re10c

show

sthat

theperfor-

8IEEEJOURNALONSELECTEDAREASIN

COMMUNICATIONS,VOL.13,NO.7,SEPTEMBER1995

Shadow

RA

MC

AM

IP A

ddressT

CP

portC

ounters Flags

Match?

48

4896

4848-bit key

HIT

fC

AM

3,

Source

Destination

Host A

ddressP

ortH

ost Address

PortC

onnection Table

Fig.8.

Thestru

cture

oftheCAM-based

connectio

ntables

inthe

BayBrid

geprototype.

CA

M H

ash Function

CAM Width (bits)

483216

A0

A1

⊕P

0P

1⊕

3216

A0

A1

⊕P

0P

1⊕

3216

A0

A1

⊕P

0P

1⊕16

32

A0

A1

P0

P1

⊕⊕

⊕

A0

A1

⊕1

6P

0P

1⊕

⊕

A0

A1

P0

P1

⊕⊕

⊕A

0A

1P

0P

1⊕

⊕⊕

A0

A1

⊕1

6P

0P

1⊕

⊕A

0A

1⊕

16

P0

P1

⊕⊕

fCA

M1,

fCA

M2,

fCA

M3,

RA

MA

0A

1P

0P

1⊕

⊕⊕

ind

ex

Siz

eC

RC

-32

RA

M H

ash Function

fRA

M1,

fRA

M2,

Key to sym

bols:A

0≡

P0

≡

A0

≡

P0

≡

IP address

TC

P port

16-bit swapped

byte swapped

AB

≡

AB

≡⊕

B least significant bits of A

AE

XO

R B

Fig.9.

Hashingfunctio

nsforCAM-basedandRAM-basedconnec-

tiontables.

mance

ofthese

hashingfunction

sissig

ni�can

tlydegrad

edforaCAM

width

ofjust16-bits.

Wenow

turn

ouratten

tionto

theRAM

based

imple-

mentation

.Aconnection

tableisim

plem

ented

inRAM

by

applyingahash

ingfunctio

nto

the96-bitkey

togen

erateamuch

smaller

index.Thisindex

isused

asthedirect

ad-

dress

ofahash

-bucket

inRAM.Ifthebucketcon

tainsmore

than

oneentry,

weassu

methatthebucket

issea

rched

asalin

kedlist.

Weconsid

eredindex

widthsof14throu

gh17

bits,

correspondingto

2kto

128khash

buckets.

Thehash

ingfunctio

nsfRAM;1

and

fRAM;2

are

show

nin

Figu

re9.

Functio

nfRAM;1

issim

ilarto

thefunction

sused

fortheCAM,whereas

fRAM;2uses

thesta

ndard

IEEE

802.X

32-b

itCRCfunctio

n.TheCRCfunctio

nisused

be-

cause

itisasim

pleway

togen

eratepseu

do-ran

dom

keys

inhard

ware.

Figures

11a-b

plottheavera

genumber

ofcomparison

sper

tablelookup.Wesee

thatboth

function

sfRAM;1andfRAM;2are

e�ectiv

eandconclu

dethat

thesim

-plebyte

manipulatio

nsof

fRAM;1are

su�cien

tfor

both

ahard

ware

orsoftw

are

implem

entation

.

0

10

00

20

00

30

00

40

00

50

00

60

00

Intensity (Packets/sec)

01

/17

/95

01

/18

/95

01

/19

/95

01

/20

/95

30

0s A

vera

ge

10

0m

s Pe

ak

Fig.12.

Rate

atwhich

datagramsenter

andlea

vetheBerk

eleycampus.

F.Datagram

ThroughputRequirements

TheBGW

processes

everydatagram

inaway

signi�-

cantly

di�eren

tfrom

norm

alboundary

routers.

Speci�

-cally,

theBGW

must

lookuptheTCPcon

nection

record,

update

thebyte

andpacket

counters,

andstore

theresu

ltsback

inthecon

nection

table.

Figu

re12

show

sthepeak

and

averagedatagram

inten

sities.Ouranaly

sisshow

edthat

the

BGW

mustbecap

ableofprocessin

gdatagram

sat

ratesup

to6,000

packets

persecon

d.Dependingon

thearch

itecture

oftheBGW,thisseem

sto

bequite

feasible.

IV.BayBridgePrototype

Inthissection

weshow

how

theBayBrid

ge,ahigh

per-

formance

prototy

perou

ter,cou

ldbeused

astheBGW

forourbillin

gsystem

.

A.OverviewofBayBrid

ge

TheBayBrid

ge[13],[14]

isatwo-p

ortbrid

geandrou

terdesign

edfor

high

perform

ance

and exibility.

With

both

ports

connected

toFDDIrin

gs,theBayBrid

gehas

been

dem

onstrated

tomake

routin

gdecision

sfor

over200,000

datagram

spersecon

d.Referrin

gtothesystem

architectu

reinFigu

re13,

datagram

sare

processed

attheport

onwhich

they

arrivebyaprogram

mableproto

colcon

verterwhich

consults

aCAM-based

address

table.

Thedatagram

may

beforw

arded

totheoth

erport

orsen

ttothelocalh

ost.The

address

tables

contain

1024entries,

eachentry

consists

of

EDELL,MCKEOWNANDVARAIYA:BILLINGUSERSANDPRICINGFORTCP

9

1

1.0

1

1.0

2

0.9

99

Avg Search Length

01

/19

/95

Ha

shin

g F

un

ction

1H

ash

ing

Fu

nctio

n 2

Ha

shin

g F

un

ction

3

(a)48-bitCAM

1

1.0

1

1.0

2

0.9

99

Avg Search Length

01

/19

/95

Ha

shin

g F

un

ction

1H

ash

ing

Fu

nctio

n 2

Ha

shin

g F

un

ction

3

(b)32-b

itCAM

1

1.0

1

1.0

2

1.0

3

1.0

4

Avg Search Length

01

/19

/95

Ha

shin

g F

un

ction

1H

ash

ing

Fu

nctio

n 2

Ha

shin

g F

un

ction

3

(c)16-b

itCAM

Fig.10.Averagenumberoflookupsforeach

connectio

nin

CAM-based

connectio

ntables.

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.7

3

Avg Search Length

01

/19

/95

2k B

ucke

ts4

k Bu

ckets

8k B

ucke

ts1

6k B

ucke

ts

32

k Bu

ckets

64

k Bu

ckets

12

8k B

ucke

ts

(a)fRAM;1

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.7

3

Avg Search Length

01

/19

/95

2k B

ucke

ts4

k Bu

ckets

8k B

ucke

ts1

6k B

ucke

ts

32

k Bu

ckets

64

k Bu

ckets

12

8k B

ucke

ts

(b)fRAM;2

Fig.11.Averagenumberoflookupsforeach

connectio

nin

RAM-based

connectio

ntables.

10 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 13, NO. 7, SEPTEMBER 1995

Bridge/Router Board

SBus

SBus Card

AddressTables

HostInterface

ProtocolConverter

MUX

ProtocolConverter

MUX

Host SPARCstation

PORTA

PORTB

NetworkInterface

NetworkInterface

Fig. 13. The architecture of the BayBridge prototype.

a 48-bit wide CAM entry, and a shadow entry in RAM.

B. Using the BayBridge as a BGW

The BayBridge may be used as a BGW in the followingway. The CAM-based address tables are implemented asdescribed in Section III.E. The connection tables are shownin Figure 8: hashing function fCAM;3 is applied to the 96-bit connection identi�er to generate a 48-bit index whichis used to �nd a match in the CAM. To verify the match,one of the IP addresses and one of the port numbers aremaintained in the shadow RAM entry. The RAM entry alsocontains counters for bytes and packets in each direction.The next hop routing information is also contained in theshadow RAM entry.When a TCP message arrives, the protocol converter

searches in the connection table for a matching connection.If the entry is not found then this message must be froma new connection. The message is forwarded to the Bay-Bridge's local host which forwards the connection identi�erto the Access Controller. When the connection has beenaccepted, the host accesses the connection tables directlyto insert the new entry.

If the entry is found in the connection table, the counterentries are read, incremented by the protocol converter andwritten back to the table.

C. Feasibility of BayBridge as a BGW

To operate as a BGW, the BayBridge must be able tohandle the intensity of connection setups and have su�-cient space in its connection table for all active connec-tions. It must also be fast enough to process all datagramsas they arrive, �nding the entries quickly in the connectiontables.We found in Section III.C that the intensity of new con-

nections does not exceed 79 per second. This is easilyachievable by the BayBridge: the protocol converter andhost can forward many hundreds of packets per second tothe Access Controller.Section III.D showed that if a 60 minute inactivity time-

out is used, the maximum number of active connections is1,350. The BayBridge has a table size of just 1,024 entries:

too small to hold all active connections. One way to over-come this limitation is to use the hardware CAM-basedaddress tables as a cache for a larger connection table inthe host's memory.In Section III.F we showed that the maximum arrival

rate for datagrams is less than 6,000 packets per second.Assuming that the �rst lookup is successful, the BayBridgecan perform the metering and routing functions in approx-imately 13�s6. This corresponds to a sustainable rate of75,000 packets per second simultaneously in each direction.We conclude that the BayBridge is easily capable of act-

ing as a BGW between the Berkeley campus and the restof the Internet.

V. Other Relevant Measurements

A. Variation in WAN Usage Levels

It is intuitive that di�erent Internet hosts and users gen-erate widely di�erent amounts of tra�c. This intuition isa crucial part of the motivation for usage based cost recov-ery. However, intuition alone should not justify complexcost recovery systems. We now show that there is a verylarge variation in WAN usage by hosts and by users on theBerkeley campus.Table II shows the usage per-host averaged over the du-

ration of the trace for the top-ten most active subdomainswithin Berkeley's network. These statistics indicate thatthe per host average usage varies widely by subdomain.The EECS Department subdomain has the largest num-ber of hosts. The amount of WAN TCP usage by thesehosts averaged 23 megabytes with a standard deviation of56 megabytes. Figures 14a-b show the large variation inusage for these hosts.Figures 15a-b show the distribution of usage per user.7

For this set of users, the average amount of usage was 1.0megabytes with a standard deviation of 3.7 megabytes.

B. Potential for Congestion Pricing

In addition to cost recovery, a billing system can also beused to control congestion by enforcing a pricing scheme.Using our trace data, we consider congestion control attwo di�erent time granularities and measure their e�ect onnetwork tra�c. These results are only intended to givean indication of the possible improvements in network e�-ciency. We do not propose speci�c techniques for achievingthese improvements.In the �rst scheme, we assume that the price to use

the network varies by time of day. For example, traf-�c sent during the day may be more expensive than atnight. Some tra�c is not sensitive to time of delivery andmay be postponed to a less expensive time of the day. Inparticular, we assume that under this scheme email andbulletin-board tra�c8 is \non-interactive" and therefore

6The publishedversion of this paper containeda typographical error(ms instead of �s). We have corrected it here.7This data is from an earlier trace recorded from midnight Novem-

ber 10 through midnight November 14, 1994.8These are the smtp and nntp protocols. They accounted for 9%

and 24%, respectively, of all wide area data bytes during our trace.

EDELL,MCKEOWNANDVARAIYA:BILLINGUSERSANDPRICINGFORTCP

11

TABLEII

AverageUsagebyHostsin

Top-TenSubdomains.

Number

PerHostAveragesa

PercentageinCategoryb

Subdomain

ofHosts

Connections

Datagrams

Bytes

Datagrams

Bytes

cs.berkeley.edu

469

1,393

130,737

23,422,981

26.6%

26.9%

eecs.berkeley.edu

482

594

56,633

9,703,412

11.8%

11.5%

lib.berkeley.edu

395

480

41,645

7,620,778

7.1%

7.4%

math.berkeley.edu

115

677

108,682

24,225,511

5.4%

6.8%

biochem.berkeley.edu

111

803

95,472

23,567,586

4.6%

6.4%

hip.berkeley.edu

1,058

146

13,364

1,935,854

6.1%

5.0%

ssl.berkeley.edu

79

314

77,873

19,349,504

2.7%

3.7%

me.berkeley.edu

235

249

35,406

6,427,425

3.6%

3.7%

cchem.berkeley.edu

295

195

27,295

5,143,642

3.5%

3.7%

geo.berkeley.edu

56

1,656

129,543

26,164,881

3.1%

3.6%

aThese

are

fourdayavera

ges.

bThistableandits

percen

tagecalcu

latio

nsexclu

dehosts

thatprovideserv

icesto

theentire

campus.

10

02

00

30

04

00

14

82

10

10

0

10

00

10

00

0

1e

+0

5

1e

+0

6

1e

+0

7

1e

+0

8

1e

+0

9

1e

+1

0

Bytes

me

an

=9

70

34

12

byte

ssd

ev=

56

35

19

97

byte

s

(a)Scatter

Plot

01

02

03

04

05

06

07

08

09

01

00

0 10

20

30

40

50

60

70

80

90

10

0

% M

achines

% Bytes8

5.3

%

94

.1%

(b)Cumulative

Distrib

ution

Fig.14.Varia

tionofnetw

ork

usagebymachinesin

theEECS.Berk

eley.EDUsubdoamin.

canbedelay

ed.Therem

ainingtra�

cis

assumed

tobe

\interactiv

e."9Wesprea

dthenon-in

teractiv

etra

�cover

timeby\w

ater-pourin

g,"

i.e.,wedistrib

ute

thistra�

cso

asto

level-outthetotal

(interactiv

e+

distrib

uted

non-

9Themostactiv

eofthese

\intera

ctive"

protocolswere

ftp,WWW,

shell,

telnet

and\X."

These

acco

unted

for33%,16%,4%,3%

and

3%,respectiv

ely,

ofallwideareadata

bytes

durin

gourtra

ce.

interactive)

inten

sityas

much

aspossib

le.

Figu

re16

show

stheim

provem

entthat

canbeach

ievedusin

g\w

ater-pourin

g."Thetop

lineshow

stheinten

sityof

allnetw

orktra�

cin

bytes

per

second.Thelow

estlin

eshow

stheinten

sityofjusttheinteractive

tra�c.

Thehori-

zontal

lineshow

stheinten

sityifthenon-in

teractivetra�

cis\poured

"over

theinteractive

tra�c.

Byspread

ingthe

12

IEEEJOURNALONSELECTEDAREASIN

COMMUNICATIONS,VOL.13,NO.7,SEPTEMBER1995

10

02

00

30

01

34

51

0

10

0

10

00

10

00

0

1e

+0

5

1e

+0

6

1e

+0

7

1e

+0

8

Bytes

me

an

=1

04

58

64

byte

ssd

ev=

36

82

39

4 b

ytes

(a)Scatter

Plot

01

02

03

04

05

06

07

08

09

01

00

0 10

20

30

40

50

60

70

80

90

10

0

% U

sers

% Bytes

80

.4%

93

.0%

(b)Cumulative

Distrib

ution

Fig.15.Varia

tionofnetw

ork

usageamongaselected

populatio

nofusers.

non-in

teractivetra

�cinthisway,

thenetw

orkutilization

isalm

ostcon

stantatan

inten

sityof150,00

0bytes

persecon

dwith

apeak

of170,0

00bytes

per

second.Thisrep

resents

a23%

reductio

nin

peak

bandwidth.

Intheseco

ndsch

eme,weassu

methatthenetw

orkiscon

-tro

lledmore

rapidly

inresp

onse

tocon

gestion.Thismay

correspondto

a ow

-control

mech

anism

,tra�

c-shapingus-

ingbu�erin

gor

apolicy

inwhich

theprice

variesquick

lyin

response

tocon

gestion.Toestim

atethepoten

tialgain

ofsuch

asch

eme,Figu

re17show

sthebusiest

0.1

second

and1secon

dinterva

lsdurin

gthetra

ce,measured

usin

gamoving-avera

gebu�er

(forcom

parison

,the�gure

alsoshow

stheavera

ge).

Thepeakinten

sityover

1secon

dis

22%

below

thepeakinten

sityover

0.1seco

nd,suggestin

gthatthepeak

netw

ork

load

canbered

uced

dramatically

by

such

asch

emeusin

gabu�er

ofjust250

,000

bytes.

VI.Conclusion

More

than90percen

tofanestim

ated

$200

million

an-

nual

Intern

etcost

istoday

recovered

fromusers

inthe

formofavariety

oforganiza

tionalcharg

esanduser

prices.

These

charges

andprices

are

design

edfor

administrative

convenien

cerather

thanto

prom

otee�

cientuse

ofnet-

work

resources.

Prices

design

edto

achieve

e�cien

tallo

ca-tio

nor

faircostrecov

ery,how

ever,

require

abillin

gsystem

that

canmeter

theindividual

user's

tra�candthat

canpresen

ttheuser

with

inform

ationthat

encou

ragese�

cient

use.

Thispaper

presen

tsabillin

gsch

emethat

caniden-

tifyandauthenticate

users,

monitor

individualuser

tra�c,

andpresen

ttheuser

with

realtim

epricin

ginform

ation.

Anim

plem

entation

ofthesch

emebased

ontheBayBrid

ge,aprototy

perou

ter,dem

onstrates

that

thebillin

gsch

emeis

practical,

andintro

duces

acceptab

lelaten

cy.

Itisan

open

question

wheth

erthetra�

cdem

andsand

correspondingcosts

ofoperatin

gtheIntern

etwillgrow

suf-

�cien

tlyto

make

pressin

gtheneed

foran

e�cien

tpricin

gandcost

recoverysch

eme.

That

will

dependon

theprolif-

erationofbandwidth

consumingapplication

son

theInter-

net.

Inturn,that

will

dependupon

theability

ofIntern

etproto

cols,lin

ksandrou

tersto

support

such

application

sin

competition

with

other

netw

orkserv

iceprov

iders,

inclu

d-

ingcab

leTVandtelep

honecom

panies.

Statistics

collectedfrom

afou

rday

traceofall

TCPtraf-

�cbetw

eentheUniversity

ofCaliforn

ia,Berkeley,

andthe

\outsid

eworld

"show

both

hom

ogeneities

anddiversities

with

inthecurren

tUniversity

population

.Thedata

reveal

severalfeatu

resthat

arerelevan

tto

thefeasib

ilityof

dif-

ferentpricin

gandcost

recoverysch

emes.

First,

ifpricin

gsch

emes

varybytim

eof

day,

then

upto

33%of

thetraf-

�cwhich

represen

tse-m

ailandbulletin

-board

tra�ccan

be

EDELL,MCKEOWNANDVARAIYA:BILLINGUSERSANDPRICINGFORTCP

13

0 20

40

60

80

10

0

12

0

14

0

16

0

18

0

20

0

22

0Intensity (Bytes/sec) (x 1000)

01

/19

/95

All T

raffic

Inte

ractive

Wa

ter L

eve

l

Fig.16.Reductio

nin

peaknetwork

bandwidth

by"pourin

g"e-m

ail

andbulletin

-board

tra�cover

theday.

shifted

overtim

ewith

littleloss

inbene�tto

theuser,

since

thistra

�cisnotsen

sitiveto

timeofdeliv

ery.Such

ashift

would

reduce

peakutiliza

tion.Seco

nd,on

amuch

smaller

timesca

le,we�ndthat

thepeakinten

sityover

1secon

dis22%

below

thepeakutilization

over0.1

second.Soif

pricin

gsch

emes

canshift

tra�cbyoneseco

nd,thepeak

router

andlin

kcapacity

canbesign

i�cantly

reduced

.Such

shifts

may

beautomatically

accommodated

byincreasin

gbu�er

sizesat

hosts

orin

routers.

Third

,itseem

sfeasib

leto

recovercost

forwidearea

tra�ceith

erfro

mtim

eofday

pricin

gorfrom

short

termpeakpricin

g.Recov

erythrou

ghtim

eofday

pricin

gwouldhave

abroaderbaseandwouldbe

much

simpler

toim

plem

ent.Recov

erythroughshort

termpeak

pricin

gwould

bemore

di�

cultto

implem

ent,butit

would

also

leadto

greater

reductio

nsin

thepeak

capacity

ofrou

tersandlin

ks.Acknowledgments

Theauthors

apprecia

tethegen

erousassista

nce

ofCli�

Frost,

Kevin

Mullally

andKevin

Zim

merm

anin

collectionof

netw

ork

tracedata.

References

[1]

J.Mackie-M

asonandH.Varia

n,\EconomicFAQsAboutthe

Internet",JournalofEconomic

Perspec

tives,vol.8,no.3,pp.

75-96,Summer1994.

[2]

C.Mills,

D.Hirsch

andG.Ruth,\Intern

etAcco

untin

g:Back-

ground",RFC1272,Intern

etEngineerin

gTask

Force

,1991.

0

10

0

20

0

30

0

40

0

50

0

60

0

70

0

80

0

90

0

10

00

11

00

12

00

Peak Bytes/sec (x 1000)

01

/17

/95

01

/18

/95

01

/19

/95

01

/20

/95

Pe

ak 1

00

ms

Pe

ak 1

sA

vera

ge

Fig.17.

Thepeakinten

sity,in

bytes

per

second,oftra

�clea

ving

theBerk

eleycampus,measured

over

0.1

secondand1seco

nd

interv

alscompared

with

theavera

geinten

sity.

[3]

H.W

.Braun,K.C.Cla�yandG.C.Polyzos,\AFramew

ork

for

Flow-Based

Acco

untin

gontheIntern

et",Proc.

oftheSingapore

IntlConferen

ceonNetw

orks,

1993.

[4]

N.Brownlee,

\New

ZealandExperien

ceswith

Netw

ork

Tra�c

Charging",ConneX

ions,

TheIntero

pCompany,vol.8,no.11,

Novem

ber1994.

[5]

D.Estrin

andL.Zhang,\Desig

nConsid

eratio

nsforUsageAc-

countin

gandFeed

back

inIntern

etworks",ACM

Computer

Com-

munica

tionReview

,vol.20,no5,pp.56-66.Octo

ber

1990.

[6]

R.Braden,\Requirem

entsforIntern

etHosts

{Communica

tion

Layers"

,RFC1122,Intern

etEngineerin

gTask

Force,

1989.

[7]

D.E.Comer,

Intern

etworkin

gwith

TCP/IP

|Volume

I,Pren

tice-Hall,2ndEd.,1991.

[8]

J.Mackie-M

asonandH.Varia

n,\Pricin

gtheIntern

et",In

Public

Access

totheIntern

et(K

ahin,B.andKeller,

J.eds.),

Pren

tice-Hall,1994.

[9]

TaxpayerAssets

Project

-Inform

atio

nPolicy

Note

May7,1994.

[10]K.Cla�y,

H.W

.BraunandG.Polyzos,\TrackingLong-Term

Growth

oftheNSFNET",Communica

tionsoftheACM,vol.

37,no.8,pp.34-45,Aug1994.

[11]J.Stein

er,C.NeumanandJ.Schiller,

\Kerb

eros:

AnAuthen-

ticatio

nServ

iceforOpen

Netw

ork

System

s",Proc.

ofUSENIX

Winter

Conferen

ce,pp.191-202,Feb

1988.

[12]Merit

Netw

ork,Inc.,

\NSFNET

BackboneServ

iceTra�cDis-

tributio

nby

Intern

etNetw

ork

Num-

ber,"

ftp://nic.m

erit.edu/nsfn

et/sta

tistics/1994/nsf-9

409.pnets.

Septem

ber1994.

[13]N.McK

eown,R.Edell

andM.T.Le,

\TheBayBrid

ge:

ahigh

speed

brid

ge/router"

.(ProtocolsforHigh-Speed

Netw

orksIII.

IFIP

WG6.1/WG6.4Third

Intern

atio

nalWorkshop,Stockholm

,Sweden,13-15May1992).IFIP

Transactio

nsC

(Communica

-tio

nSystem

s),vol.C

-9:203-18,1993.

[14]N.McK

eown,R.Edell

andM.T.Le,\TheBayBrid

ge:

AHigh

Speed

Brid

ge/Router

betw

eenFDDIandSMDSPart

I|

Sys-

temArch

itectureandPerfo

rmance"

,Submitted

.

14 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 13, NO. 7, SEPTEMBER 1995

Richard Edell (S'94) received the B.S. andM.S. degrees in electrical engineering from theUniversity of California, Berkeley, in 1991 and1994, respectively.

He is currently working toward the Ph.D. de-gree in electrical engineering at the Universityof California, Berkeley. His research interestsinclude network interface architectures, Inter-net economics and technology policy.

Nick McKeown (S'90-M'95) received theB.Eng degree in Electronic Engineering fromthe University of Leeds, England in 1986, andthe M.S. and Ph.D. degrees from the Univer-sity of California, Berkeley, in 1992 and 1995,respectively.

He is currently an Assistant Professor of Elec-trical Engineering and Computer Science atStanford University. His research interests in-clude high performance ATM switching and

economics of the Internet.

Pravin Varaiya (M'68{SM'78{F'80) receivedthe Ph.D. degree in electrical engineering fromthe University of California, Berkeley.

He is James Fife Professor of Electrical En-gineering and Computer Sciences at the Uni-versity of California, Berkeley, and Director ofCalifornia PATH, a multi-university programof research in Intelligent Transportation Sys-tems. From 1975 to 1992 he also was Professorof Economics at Berkeley. He has taught at

MIT and the Federal University of Rio de Janeiro. He was a memberof the technical sta� at Bell Laboratories during 1962-63. Dr. Varaiyahas held a Guggenheim Fellowship and a Miller Research Professor-ship. His areas of research are communication networks, transporta-tion systems, and electric power systems. He is writing a book withJean Walrand on high-speed, exible communication networks.