IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 13, NO. 7, SEPTEMBER 1995 1
Billing Users and Pricing for TCPRichard J. Edell, Nick McKeown and Pravin P. Varaiya
Abstract| This paper presents a system for billing users
for their TCP tra�c. This is achieved by postponing the
establishment of connections while the user is contacted,
verifying in a secure way that they are prepared to pay. By
presenting the user with cost and price information, the sys-
tem can be used for cost recovery and to encourage e�cient
use of network resources. The system requires no changes to
existing protocols or applications and can be used to recover
costs between cooperating sites. Statistics collected from a
four day trace of tra�c between the University of Califor-
nia, Berkeley and the rest of the Internet demonstrate that
such a billing system is practical and introduces acceptable
latency. An implementation based on the BayBridge pro-
totype router is described. Our study also indicates that
pricing schemes may be used to control network congestion
either by rescheduling time-insensitive tra�c to a less ex-
pensive time of the day, or by smoothing packet transfers to
reduce tra�c peaks.
Keywords| Internet economics, network tracing, usage-
accounting
I. Introduction
SINCE its creation in 1986, the NSFNET has sustainedan extremely rapid growth rate. With an estimated
20 million people reachable by electronic mail, the NSFfunded backbone is serving a multitude of academic andcommercial organizations. The NSF subsidy for NSFNETand various regional networks amounts to $20 million peryear, of which about $12 million is for NSFNET. The totalcost of the Internet is estimated to be about $200 millionper year. It is likely that NSFNET will soon be gone, sothat Internet will be run entirely by commercial and non-pro�t carriers, and all costs (and pro�ts) will be recoveredthrough charges and pricing schemes.
If all users within an organization generated similaramounts of tra�c, cost recovery could be equitably andsimply achieved by dividing the cost equally among users.However, as might be guessed, and as we shall demonstrate,di�erent Internet users generate widely di�erent amountsof tra�c. Pricing schemes or charges that are designed torecover costs fairly from this diverse population of usersand to allocate network resources in an e�cient mannerrequire the capability to meter individual user tra�c andto present users with prices and charges in a way that en-courages e�cient network use [1]. In this paper, we presenta billing system that can be used to meter users' wide area
Manuscript receivedNovember 1, 1994; revised April 15, 1995. Thisresearch was funded by grants from Paci�c Bell, the California StateMICRO Program 93-152 and NSF grant IRI-9120074.R. Edell and P. P. Varaiya are with the Department of Electri-
cal Engineering and Computer Sciences, University of California,Berkeley, Berkeley, California 94720 USA. Their email addresses arefedell,[email protected]. McKeown is with the Departments of Electrical Engineering and
Computer Science, StanfordUniversity, Stanford, CA 94305 USA. Hisemail address is [email protected].
c 1995 IEEE
TCP tra�c, involving the users in the decision to consumeresources and making them accountable for the tra�c thatthey generate.
A number of network billing systems have been previ-ously proposed [2],[3]; in addition, one billing system hasbeen implemented [4]. An excellent discussion of methods,costs and bene�ts of usage metering as well as a proposedbilling infrastructure is given in [2]. In [3], a ow-based ac-counting mechanism is de�ned based on IP data that owsbetween end hosts. Brownlee reports experiences with im-plementing a network billing system [4]. But none of thesesystems provide a mechanism for identifying and involvingindividual users; i.e. these systems identify hosts machinesbut not the users of those machines. In 1990, Estrin etal. suggest some research topics towards usage billing andfeedback [5]. In particular, they propose the instrumen-tation of current networks to determine the performanceimplications of usage metering, and highlight the problemof identifying and authenticating the end user.
Our billing system is designed to meter users (i.e. thepersons using hosts) of the TCP connection-oriented pro-tocol [6],[7], which represents almost all of the tra�c onthe Internet. It is because a connection is established priorto the transfer of data that we are able to identify reliablythe originating user and verify that they are prepared topay for their tra�c. An important feature of this billingsystem is that this can take place without modi�cation tothe TCP/IP protocols, or applications.A usage-based billing system in which the user is directly
involved will a�ect the way in which the Internet is used.Several schemes have been proposed for controlling conges-tion via pricing which require that tra�c is metered andinvolve the user in the feedback mechanism [8]. However,it has not been previously shown that real-time meteringof tra�c for high-speed networks is feasible, or that it ispractical to involve the user in the feedback.
A billing system that controls the access of individualusers will also a�ect the way in which the Internet grows.Users within an organization who need a higher bandwidthconnection to the commercial data carrier can pay for andget preferential use of the improved service.
Usage billing for network tra�c is a controversial subject[6],[9]. It is not the purpose of this paper to argue for thewidespread adoption of usage billing. Our aim is to demon-strate that it is technically feasible to implement a billingsystem that involves users and makes them accountable fortheir tra�c.
In Section II, we provide a detailed description of theproposed billing system. In Section III, we describe anextensive feasibility study of the system, based on a traceof the tra�c that leaves and enters the Berkeley campus.Section IV shows how the BayBridge, a prototype router,
2 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 13, NO. 7, SEPTEMBER 1995
could be to used to meter tra�c. Section V discusses someother relevant measurements. We draw some conclusionsfrom our study in Section VI.
II. Generic System Description
A number of features are essential to any billing system:� No changes to existing Internet protocols|Because ofthe huge number of installed end systems, bridges androuters, it is important that an Internet billing systemwork with the existing Internet protocols. It meansthat the billing system should not require the use ofany special option �elds by end systems (for example,IP options or TCP options).
� No changes to existing applications | Many applica-tions such as ftp, email, gopher and mosaic are inwidespread use today and collectively contribute tomost of the tra�c on the Internet [10]. A billing sys-tem should be able to meter tra�c for these and otherexisting applications without change.
� User Involvement | If a billing system charges indi-vidual users for their tra�c, it must �rst determine theidentity of the user. It should also obtain the explicitapproval from the user or an authorized agent thatthey will pay for the resources consumed. For securityand credibility, this approval should be authenticated,for example with Kerberos [11]. And �nally, the billingsystem should provide accurate and credible on-linefeedback to the user as they consume resources. Thisimplies that the metering of tra�c should be exact andnot based on tra�c sampling.
It would be desirable for a billing system to have thefollowing additional features:
� Provide on-line reporting of aggregate network usage
| This enables schemes that control network conges-tion based on global tra�c measures. The control maybe implemented using priorities or by a time varyingpricing policy [8].
� Allow continued sharing of information and resources
| The growth of the Internet can be attributed toapplications that encourage the sharing of informationand resources between remote sites. It would be ad-vantageous if billing systems could cooperate to iden-tify the user and bill them for their tra�c.
A. Detailed Description of Billing System
A block diagram of the billing system is shown in Fig-ure 1. This �gure shows a single administrative domainthat is connected to the outside world via a Billing Gate-way (BGW) and a billed link. The administrative domaincontains a collection of networks, users and hosts. Thebilling system controls users' access to the billed link byallowing or disallowing TCP connections and by meteringusers' TCP tra�c once a connection has been established.The basic operation is as follows: when a user attempts
to establish a TCP connection with the outside world, theBGW postpones the establishment of the connection whileit tries to identify the originating user. The system con-tacts the user, verifying that they want to establish the
Billed Link
PurchasingAgent
User-idDaemon
AccessController
BillingGateway
Host A
Host B
NameServer
12
Kerberos
3,4
5,6
2,11
7,10
8,9
Administrative Domain
Server
Router1
BillingRecords
Fig. 1. Components of the billing system showing the major com-munications required to establish a connection.
connection and that they are prepared to pay for it. If theuser accepts the connection, the normal connection estab-lishment is allowed to continue and the BGW will beginmetering this connection's tra�c.
We will describe the operation in detail by way of anexample and by referring to the communications markedon Figure 1 and the corresponding time diagram in Fig-ure 2b. The timeline in Figure 2b is an extension of Fig-ure 2a showing all the communications involved in settingup a connection between the originating local host and theremote host.
In the example, the user sits at Host A but is logged into Host B. The user's application on Host B attempts toestablish a TCP connection to a remote host. Consideringeach communication in turn:
1. Host B initiates the connection by sending a TCPSYN message. The BGW recognizes the TCP SYNmessage as an attempt to establish a new connectionto the outside world. The BGW holds onto the TCPSYN message while it determines whether or not theconnection should be allowed and access granted to thebilled link. This is achieved by communications (2-11)to identify and contact the user, determine whetherthe connection will be allowed and to set up the nec-essary state for metering and billing records. The con-nection is always referred to by its unique identi�er:[(source (address,port), destination (address,port)].
2. The BGW contacts the Access Controller which isresponsible for determining whether the connectionshould be allowed. It achieves this by communicat-ing with each of the components in turn.
3,4. The Access Controller asks the User Identi�cationDaemon (Useridd) to identify the user.
5,6. The Access Controller asks the User Name Serverto locate the user's Purchasing Agent.
7. The Access Controller asks the user's PurchasingAgent to verify that the user wishes to establish andpay for this connection.
8,9. The Purchasing Agent may be con�gured to respond
EDELL, MCKEOWN AND VARAIYA: BILLING USERS AND PRICING FOR TCP 3
Originating Host Destination Host
1st SYN
2nd SYNACK 1st SYN
ACK 2nd SYN
(a) Normal 3-Way handshake to establish connections
Originating Host Destination Host
1st SYN
2nd SYNACK 1st SYN
ACK 2nd SYN
(b) Billing System Communications Involved in Establishing a Connection
1st SYN (delayed)
BGW Access
Useridd
Name-server
Controller
UserPurchasingAgent
1 234567
10
8
911
12
S1S2
S3
Fig. 2. The communications required to establish a connection with-out and with the billing system.
automatically on behalf of the user, or may request theexplicit authorization of the user by means of a dialogbox on the user's screen.
10. The Purchasing Agent responds to the Access Con-troller authenticating its reply using Kerberos.
11. If the user con�rms that they wish to establish andpay for the connection, the Access Controller tells theBGW to allow access to the billed link and meter theconnection.
12. The BGW creates an entry for this connection in itsconnection tables and forwards the TCP SYN messagetowards the remote host.
Now that the BGW knows to allow this connection, all fu-ture messages for this connection in either direction will beforwarded without further delay. This means that when re-mote host responds with a TCP SYN+ACK, the BGW willforward the message, updating its connection table entry.
B. Components
The components in Figure 1 fall into three categories.The �rst category is the Access Controller which was de-scribed in detail above.The second category is user involvement. This contains
three components: User Identi�cation Daemon, User NameServer and Purchasing Agent. Together these componentsidentify the originating user, obtain veri�cation from the
user and provide feedback to the user as resources are used.� The User Identi�cation Daemon (Useridd) is a systemdaemon process that runs on all multi-user machineswithin the administrative domain. Request messagesare sent to the Useridd containing the unique con-nection identi�er. The Useridd examines kernel datastructures to determine and return the name of theuser who originated the connection.
� The User Name Server is a standard network namingservice which when supplied with a user name willreturn the location of the user's Purchasing Agent.
� The Purchasing Agent1 is a user-level process responsi-ble for verifying that the user is prepared to pay for theTCP connection. In its simplest form, the PurchasingAgent could just pop up a dialog box on the user'sscreen asking for a simple accept/reject response. Al-ternatively it may be con�gured to automatically re-spond to some or all of the requests that it receivesfrom the Access Controller. For example, it could becon�gured by the user to: (i) automatically accept allconnections below a certain price, but verify with theuser before accepting more expensive connections; (ii)automatically accept all connections to speci�c well-known ports, but verify with the user before acceptingothers; or (iii) automatically accept all connections tocertain destinations. The Purchasing Agent proves itsauthority to represent the user by including a Kerberosauthenticator in its response.
The third category is metering. The two components inthis category are the BGW and Billing Records.
� The BGW is a specialized router that maintains a tableof established TCP connections for metering tra�c, inaddition to performing the usual IP routing functions.The BGW must understand TCP connections so thatit can determine which connection record to update.New connections are TCP messages for any connec-tion that the BGW has not seen before. This includesconnections that establish via other BGWs.The BGW must determine when connections haveclosed so that the entry in the connection table canbe freed. Connections may close in a number of dif-ferent ways and the TCP FIN messages that close theconnection may travel via a di�erent BGW. So thatconnections can be removed from the tables in a timelymanner, the BGW times out connections that are in-active for a long period. If the connections becomeactive again, they are added back into the table.TheBGW maintains a table of all connections with a sepa-rate entry for each established connection keyed by itsunique connection identi�er. This means that the con-nection tables maintained by the BGW may be muchlarger than IP routing tables. It also means that thekey into the connection table is much larger: for an IProuter the key would normally just be the 32-bit desti-nation network address. For a BGW with TCP/IP thekey is 96-bits. In Section III.D we will consider how
1For security purposes, this process is usually run on the host atwhich the user sits.
4 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 13, NO. 7, SEPTEMBER 1995
large the connection tables need to be and compareseveral schemes for looking up entries in the connec-tion tables.
� The Billing Records are responsible for maintainingrecords of metered connections and for providing on-line feedback to the user as the connection progresses.
C. Simpler Cases
A number of possible simpli�cations could be used toreduce the number of communications described above.Each simpli�cation corresponds to bypassing communica-tions shown in Figure 2b, \S1" - \S3". The simpli�cationsreduce the number of communications involved in estab-lishing a connection, thus reducing the latency seen by theuser.1. No user interaction | Do not explicitly involve theuser in the connection acceptance. This was discussedabove and would be achieved by con�guring the Pur-chasing Agent to accept connections automatically.The advantages are that it would make the connec-tion setup faster and would not burden the user withinvolvement in every connection.
2. No Purchasing Agent interaction|Con�gure the Ac-cess Controller so that connections from some users orfor some applications are accepted automatically. Thiseliminates communications (5-10).
3. By machine billing|Con�gure the Access Controllerso that connections from some machines are acceptedautomatically. This would be used when a machine isused by a single user, or if it has been agreed that thetra�c will always be paid for regardless of the user. Infact, this would be necessary for machines that can-not support the Useridd process (this would includesingle-process personal computers). In this case, weeliminate communications (3-10).
It is important to note that both simpli�cations 2 and3 compromise the security of the billing system as neitherobtains authenticated feedback from the user.
D. Complications & Gotchas
We believe that the billing system described above is fea-sible and implementable. However, there are several thingsthat can make the implementation more di�cult, or makethe billing model less appropriate. We consider some ofthese below.
� IP Fragmentation| IP datagrams may be fragmentedby routers as they progress through the network. TheBGW must interpret TCP messages and so it must beable to reassemble fragmented IP datagrams. In ourfeasibility study, we found that less than 0.01% of ourwide-area datagrams are fragmented2.
� Malicious IP Fragmentation| IP datagrams may alsobe severely fragmented by the end station in an at-tempt to circumvent the billing system. This maybe overcome by refusing to forward fragmented data-grams with too few data bytes.
2All of the machines on the Berkeley campus that fragment theiroutgoing tra�c are Macintoshes.
� Hiding data in option �elds | To avoid being billed,the user may attempt to hide data in IP or TCP op-tions �elds. This may be overcome by metering pro-tocol header bytes as well as data bytes. This way,anomalies can be detected and billed for accordingly.
� Time varying route | The packet-switched Internetallows datagrams within a connection to be deliveredvia di�erent routes. If the change is due to load bal-ancing, we can expect the price for delivery to be un-changed. But if there is a change in topology, the pricemay change and should be communicated back to theuser.
� Connections on behalf of, but not originated by a user
| The main reason for involving the user is to makethem accountable for their tra�c. The basic systemachieves this by billing the originator of a TCP con-nection. However, it would be more appropriate tobill the bene�ciary of the connection. In practice, theoriginator of a connection is not necessarily the bene�-ciary. For example, if a process on one host originatesa connection so that it can transfer data to or from aremote host, who is the bene�ciary of the transfer?We discuss a solution to this problem below when wedescribe how cooperative billing can be achieved.
E. Enhancements to the Basic Billing System
� Cooperative Billing Between Administrative Domains
| If two Administrative Domains wish to bill for aconnection that ows between them, they must �rstagree which domain will pay for the connection. Thatdomain then needs to determine which of its users tobill. The user that is billed should be the bene�ciaryof the connection.The following method could be used to determine thebene�ciary. When the connection is established, theBGW asks the originating user if they will pay for theconnection. If the user is the bene�ciary, they shouldagree to pay. Alternatively, the user may request thatthe remote end pay. In this case the local BGW con-tacts the remote BGW which in turn contacts the re-mote user asking if they are prepared to pay. If neitheruser agrees to pay, the connection is refused. If eitherof the end points of the connection is a service, ratherthan a user, its local Access Controller could be con-�gured to accept connections on its behalf.
� User Speci�ed Limits | The Purchasing Agent couldrespond to the Access Controller with conditions andlimits to its acceptance. For example, the user maybe willing to pay up to a speci�ed amount for a con-nection, or for a speci�ed number of packets or bytes.The user may only want to pay for tra�c as long asthe price for the connection stays below a speci�edthreshold. This would allow the implementation of atime-varying threshold pricing policy agreed upon bythe user and policed by the BGW [8]. If during the lifeof the connection any of the limits or thresholds are ex-ceeded, the BGW contacts the user before forwardingany more messages.
EDELL,MCKEOWNANDVARAIYA:BILLINGUSERSANDPRICINGFORTCP
5
�Meterin
gUDP
Tra�c|
Ourbillin
gsystem
isde-
signed
tometer
TCPtra
�c,which
curren
tlycom
prises
alm
ost
allof
thetra�
contheIntern
et.How
ever,a
growingproportion
oftra
�cis
usin
gtheUDP
pro-
tocol,
e.g.MBONE.In
prin
ciple,
ourbillin
gsystem
couldalso
meter
tra�cfrom
these
applica
tions.There
aretwoprin
cipal
requirem
ents.
First,
thatprior
totheBGW
forward
ingdatagram
sitmust
bepossib
leto
identify
theuser
inanequivalen
tway
totheUser
Identi�
cation
Daem
on.Secon
d,that
itbepossib
lefor
theBGW
torelate
eachdatag
ramto
ametered
ow
.How
practical
thisiswill
dependontheapplication
.
III.FeasibilityStudy
Inthissectio
n,wewill
use
theresu
ltsfro
madetailed
studyto
dem
onstrate
that
ourbillin
gsystem
isfeasib
leeven
forlarge
campusnetw
orks.
Ourresu
ltsare
derived
from
atrace
ofnetw
ork
tra�con
theBerk
eleycam
pus
FDDIback
bonenetw
ork. 3
Most
ofourbillin
gsystem
'scom
plex
ityiscon
nection
re-lated
.Therefore,
ourfeasib
ilitystu
dymeasures
several
keystatistics
aboutTCP
connectio
ns.
These
statisticsare:
connectio
nsetu
ptim
e,con
nection
setupinten
sity,active
connectio
ncount,andcom
plex
ityof
search
ingthecon
nec-
tiontab
le.Under
ourbillin
gsystem
,con
nectio
nestab
-lish
mentisdelayed
while
locatin
gtheuser
andthen
ver-ify
ingthatthey
will
pay;therefore,
wemeasu
redcon
nec-
tionsetu
platencywith
outthebillin
gsystem
andestim
atetheaddition
alcon
nection
setuplaten
cyintro
duced
bythe
billin
gsystem
.Several
components
ofourbillin
gsystem
areactiva
tedonce
forevery
connectio
n;therefore,
wemea-
sured
connectio
nsetu
pinten
sityto
determ
inethereq
uired
throughputfor
these
components.
TheBGW
main
tainsa
tableof
active
connection
s;therefo
re,wemeasu
retheevo-
lution
ofcon
nectio
ncou
ntto
determ
inethereq
uired
size
ofthecon
nection
table.
Thecon
nection
tableisfreq
uently
searched;therefo
re,wemeasure
theperform
ance
ofdi�er-
entsea
rchmeth
odsto
determ
inethecomplexity
ofcon
nec-
tiontablelookup.
TheBGW
processes
every
data
gramfor
meterin
g;there-
fore,
ourfeasib
ilitystu
dymeasures
datag
raminten
sityto
determ
inethereq
uired
datagram
throughputfor
theBGW.
A.AbouttheTrace
TheBerkeley
campuscommunity
iscomposed
of40,000
students,
faculty
andsta
�mem
bers.
Thecampusnetw
orkintercon
nects
22,000hosts
ofwhich
6,312
hosts
partici-
pated
inWANTCPcon
nection
sdurin
gthestu
dyperio
d.
Thecam
pusnetw
ork
isconnected
totheIntern
etbytwo
1.5M
bps\T
-1"lin
ksandasin
gle
boundary
router.
We
consid
erevery
thingontheoth
ersid
eof
thisrouter
tobe
the\ou
tsideworld
."4
3Durin
gthemonth
ofSeptem
ber
1994,Berk
eleycontrib
uted
the
13th
largestamountofWANtra
�ctotheNSFNEToutofover22,000
registe
rednetw
orks[12].
4This
de�nitio
nof\outsid
eworld
"inclu
des
some
geographi-
cally
and
topologica
llynearby
sitessuch
asLawren
ceBerk
eleyLaboratorie
s.
0
10
0
20
0
30
0
40
0
50
0
60
0
70
0
80
0
90
0
10
00
11
00
Intensity (Packets/sec)
01
/17
/95
01
/18
/95
01
/19
/95
01
/20
/95
Avg
Ou
tbo
un
dA
vg In
bo
un
d
Fig.3.
Avera
geinten
sityofdatagramsenterin
gandlea
vingthe
Berk
eleycampus.
Ourtrace
was
obtain
edbyloggin
gall
TCP/IP
head
ersthat
appeared
ontheBerk
eleyback
bonenetw
ork.A50n
stim
estampisattach
edto
eachlog
entry.
Theanaly
sispre-
sented
inthispaper
consid
ersWAN
tra�cover
thefou
rdaysfrom
midnigh
tMonday
January
16throu
ghmidnigh
tFrid
ayJan
uary
20,1995.
Durin
gourstu
dyperio
d,we
measu
red89
gigabytes
ofWANtra�
cin
439million
data-
grams.
OfthisWANtra�
c,92%
ofthebytes
and92%
ofthedatagram
swere
forTCPtra�
c.OfthisWAN
TCP
tra�c,
86%of
thebytes
and84%
ofthedatagram
swere
frommulti-u
serhosts
(seeTableI).
Figu
re3show
show
the
averagerate
ofinboundandoutbounddatagram
schanged
overourstu
dyperio
d.
B.Additio
nalConnectionSetupLatency
Figu
re2a
illustrated
thecom
munication
involv
edin
es-tab
lishingaTCPcon
nection
with
outbillin
gandFigu
re2b
illustrated
theaddition
alcom
munication
required
byour
billin
gsystem
.Obviou
slythisaddition
alcom
munication
increases
connection
setuptim
e.Therelativ
esign
i�can
ceof
this
addition
aldelay
dependson
thedistrib
ution
ofcon
nection
setuptim
es.Figu
re4show
sthecumulative
distrib
ution
ofmeasu
redcon
nection
setuptim
esfor
out-
boundcon
nection
swith
outthebillin
gsystem
.Ouranaly
-sis
show
edthat
80%ofthecon
nection
stook
betw
een24m
sand734m
sto
establish
,with
amedian
timeof
116ms,and
anaverage
timeof
519ms.
6 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 13, NO. 7, SEPTEMBER 1995
TABLE I
Amount of TCP WAN Traffic by Host Type.
Number Per Host Averagesa Percentage in Category
Host Type of Hosts Connections Datagrams Bytes Datagrams Bytes
Multi-user Computers 2,119 1,306 153,226 32,428,047 83.8% 85.7%
Personal Computers 1,527 64 5,447 1,167,732 2.1% 2.2%
Unknown 2,666 237 20,413 3,617,429 14.0% 12.0%
aThese are four day averages.
0.001 0.01 0.1 1 100
10
20
30
40
50
60
70
80
90
100
Connection Setup Time (Seconds)
%
Fig. 4. Cumulative distribution of setup latencies for connectionsthat originate on the Berkeley campus and connect to a hostelsewhere on the Internet.
Our trace experiment included the identi�cation of userson selected machines using the \User Identi�cation Dae-mon" (Useridd) implemented as a user-level process. Ouranalysis5 showed that the average response time was 84ms.
We can use these results to estimate how long it wouldtake for the rest of the connection setup overhead. Let'sassume that each of the other three request/reply pairs willtake approximately the same amount of time. Then theaverage additional delay is 336ms. Therefore, the averageconnection setup time with the billing system would be855ms, an increase of approximately 65%.
5This data is from an earlier trace recorded from midnight Novem-ber 10 through midnight November 14, 1994.
0 10 20 30 40 50 60 70 800
1
2
3
4
5
6
7
8
9
Intensity (Connections/Second)
Pro
babi
lity
Den
sity
(%
)
50 percentile 99.9 percentile
Fig. 5. Rate at which connection are established to and from theBerkeley campus.
C. Connection Throughput Requirements
This section discusses the throughput requirements forthe functional blocks. Most of the billing system functionalblocks are activated once per connection setup. Therefore,the peak intensity of connection setup is an important pa-rameter. Figure 5 shows the distribution of connectionsetup intensities over one second intervals. The peak ratewas 79 connection setups per second, and 99.9% of sec-onds had 41 or fewer connection setups. Therefore, thethroughput requirements for the functional blocks can beeasily achieved.
D. Size of Connection Table
While a connection is active, an entry must be main-tained in the BGW connection table. For the billing system
EDELL,MCKEOWNANDVARAIYA:BILLINGUSERSANDPRICINGFORTCP
7
60
0
80
0
10
00
12
00
14
00
16
00
18
00
20
00
22
00
24
00
26
00
Active Connection Count
01
/17
/95
01
/18
/95
01
/19
/95
01
/20
/95
with
ou
t time
ou
t
Fig.6.
Acountofthenumberofcurren
tlyesta
blish
edconnectio
ns
toandfro
mtheBerk
eleycampus.
tobefea
sible,
these
tables
mustnotbetoolarge.
Wehave
used
ourtra
cedata
toreco
nstru
cttheconnectio
ncou
nt
evolutio
nin
twodi�eren
tways.
The�rst
reconstru
ction
meth
od
removes
connection
sfrom
thetab
leonly
when
thetrace
data
indica
testhat
aconnection
hasclosed
.Figure
6show
sthisrecon
struc-
tion.This�gure
clearlyexhibits
anupward
drift
inthe
connection
count.
Som
eofthisdrift
canbeattrib
uted
tolong-duratio
nconnectio
nsthatstart
butdonot
enddur-
ingthetrace
perio
d.Most
ofthedrift
isfro
mhosts
that
failto
prop
erlyclo
se-dow
ncon
nection
s.Fromthis,
wecon
-clu
dethat
aBGW
cannot
reliablydetect
theclo
singofall
connectio
ns.
Thesecon
drecon
structio
nmeth
odincorp
orates
aninac-
tivity
time-o
ut.
Figure
7show
stheevolution
ofcon
nec-
tioncou
ntifconnectio
nsare
timed-ou
tafter
60minutes.
With
thetim
e-outpolicy
themaximum
connectio
ncou
nt
isbelow
1,350entries
andthere
isnolongeranysign
i�can
tupward
drift.
E.Complexity
ofConnectionLookup
TheBGW
meters
every
TCPmessa
gethat
itforw
ards.
Itistherefore
importan
tthat
connection
tableentries
arelocated
quick
ly.These
entries
arekeyed
bya96-bitcon
nec-
tionidenti�
er.Usin
gthiskey
todirectly
index
a296entry
table
inRAM
orusin
ga96-b
itwideCAM
(conten
tsad-
dressab
lemem
ory,
sometim
escalled
asso
ciativemem
ory)
40
0
50
0
60
0
70
0
80
0
90
0
10
00
11
00
12
00
13
00
14
00
Active Connection Count
01
/17
/95
01
/18
/95
01
/19
/95
01
/20
/95
with
time
ou
t
Fig.7.Acountofthenumber
ofcurren
tlyesta
blish
edconnectio
ns.
Connectio
nsare
timed-outafter
aninactiv
ityperio
dof60min-
utes.
isim
practical.
Wehave
founde�ective
hash
ingfunction
sfor
hard
ware
andsoftw
areim
plem
entation
susin
geith
erCAM
orRAM
based
connection
tables.
Weem
ulate
thebehavior
ofeach
hash
ingfunction
with
ourtrace
data,
measu
ringtheaver-
agenumberoflookupsthat
arereq
uired
to�ndthecorrect
tableentry.
For
CAM
based
tables,
wecon
sidered
three
widthsthat
arecom
monlyavailab
leandused
incom
mercialIP
routers:
48,32
and16
bits
wide.
Ahash
ingfunction
isapplied
tothe96-b
itkey
tored
uce
itto
a48,
32or
16-bitindex
which
isused
to�ndthe�rst
match
intheCAM.Asso
ciatedwith
eachCAM
entry
isashadow
entry
atthesam
eo�set
inRAM.Theshadow
entry
contain
san
orthogon
alportion
ofthekey
that
when
combined
with
theindex
canbeused
torecover
thefull96-b
itkey.
Theorth
ogonalportion
sof
the
keyare
compared
toverify
that
theentries
match
.Ifthey
donot,
then
thesearch
isrep
eatedlookingfor
thenext
match
intheCAM.Figu
re8illu
stratesthisstru
cture.
Figu
re9show
show
CAM
hash
ingfunction
sfCAM;1 ,
fCAM;2 ,
fCAM;3
map
the96-b
itcon
nection
identi�
erto
48,32
or16
bits.
Figu
res10a-c
plot
theaverage
numberof
comparison
spertab
lelookupover
thetrace
perio
d.Wesee
that
function
fCAM;1ismuch
lesse�ective
than
theoth
-ers.
Thisisbecau
setheport
numbers
forftp
andftp
-data
di�er
byexactly
one.
Function
sfCAM;2andfCAM;3both
overcomethisprob
lem.Figu
re10c
show
sthat
theperfor-
8IEEEJOURNALONSELECTEDAREASIN
COMMUNICATIONS,VOL.13,NO.7,SEPTEMBER1995
Shadow
RA
MC
AM
IP A
ddressT
CP
portC
ounters Flags
Match?
48
4896
4848-bit key
HIT
fC
AM
3,
Source
Destination
Host A
ddressP
ortH
ost Address
PortC
onnection Table
Fig.8.
Thestru
cture
oftheCAM-based
connectio
ntables
inthe
BayBrid
geprototype.
CA
M H
ash Function
CAM Width (bits)
483216
A0
A1
⊕P
0P
1⊕
3216
A0
A1
⊕P
0P
1⊕
3216
A0
A1
⊕P
0P
1⊕16
32
A0
A1
P0
P1
⊕⊕
⊕
A0
A1
⊕1
6P
0P
1⊕
⊕
A0
A1
P0
P1
⊕⊕
⊕A
0A
1P
0P
1⊕
⊕⊕
A0
A1
⊕1
6P
0P
1⊕
⊕A
0A
1⊕
16
P0
P1
⊕⊕
fCA
M1,
fCA
M2,
fCA
M3,
RA
MA
0A
1P
0P
1⊕
⊕⊕
ind
ex
Siz
eC
RC
-32
RA
M H
ash Function
fRA
M1,
fRA
M2,
Key to sym
bols:A
0≡
P0
≡
A0
≡
P0
≡
IP address
TC
P port
16-bit swapped
byte swapped
AB
≡
AB
≡⊕
B least significant bits of A
AE
XO
R B
Fig.9.
Hashingfunctio
nsforCAM-basedandRAM-basedconnec-
tiontables.
mance
ofthese
hashingfunction
sissig
ni�can
tlydegrad
edforaCAM
width
ofjust16-bits.
Wenow
turn
ouratten
tionto
theRAM
based
imple-
mentation
.Aconnection
tableisim
plem
ented
inRAM
by
applyingahash
ingfunctio
nto
the96-bitkey
togen
erateamuch
smaller
index.Thisindex
isused
asthedirect
ad-
dress
ofahash
-bucket
inRAM.Ifthebucketcon
tainsmore
than
oneentry,
weassu
methatthebucket
issea
rched
asalin
kedlist.
Weconsid
eredindex
widthsof14throu
gh17
bits,
correspondingto
2kto
128khash
buckets.
Thehash
ingfunctio
nsfRAM;1
and
fRAM;2
are
show
nin
Figu
re9.
Functio
nfRAM;1
issim
ilarto
thefunction
sused
fortheCAM,whereas
fRAM;2uses
thesta
ndard
IEEE
802.X
32-b
itCRCfunctio
n.TheCRCfunctio
nisused
be-
cause
itisasim
pleway
togen
eratepseu
do-ran
dom
keys
inhard
ware.
Figures
11a-b
plottheavera
genumber
ofcomparison
sper
tablelookup.Wesee
thatboth
function
sfRAM;1andfRAM;2are
e�ectiv
eandconclu
dethat
thesim
-plebyte
manipulatio
nsof
fRAM;1are
su�cien
tfor
both
ahard
ware
orsoftw
are
implem
entation
.
0
10
00
20
00
30
00
40
00
50
00
60
00
Intensity (Packets/sec)
01
/17
/95
01
/18
/95
01
/19
/95
01
/20
/95
30
0s A
vera
ge
10
0m
s Pe
ak
Fig.12.
Rate
atwhich
datagramsenter
andlea
vetheBerk
eleycampus.
F.Datagram
ThroughputRequirements
TheBGW
processes
everydatagram
inaway
signi�-
cantly
di�eren
tfrom
norm
alboundary
routers.
Speci�
-cally,
theBGW
must
lookuptheTCPcon
nection
record,
update
thebyte
andpacket
counters,
andstore
theresu
ltsback
inthecon
nection
table.
Figu
re12
show
sthepeak
and
averagedatagram
inten
sities.Ouranaly
sisshow
edthat
the
BGW
mustbecap
ableofprocessin
gdatagram
sat
ratesup
to6,000
packets
persecon
d.Dependingon
thearch
itecture
oftheBGW,thisseem
sto
bequite
feasible.
IV.BayBridgePrototype
Inthissection
weshow
how
theBayBrid
ge,ahigh
per-
formance
prototy
perou
ter,cou
ldbeused
astheBGW
forourbillin
gsystem
.
A.OverviewofBayBrid
ge
TheBayBrid
ge[13],[14]
isatwo-p
ortbrid
geandrou
terdesign
edfor
high
perform
ance
and exibility.
With
both
ports
connected
toFDDIrin
gs,theBayBrid
gehas
been
dem
onstrated
tomake
routin
gdecision
sfor
over200,000
datagram
spersecon
d.Referrin
gtothesystem
architectu
reinFigu
re13,
datagram
sare
processed
attheport
onwhich
they
arrivebyaprogram
mableproto
colcon
verterwhich
consults
aCAM-based
address
table.
Thedatagram
may
beforw
arded
totheoth
erport
orsen
ttothelocalh
ost.The
address
tables
contain
1024entries,
eachentry
consists
of
EDELL,MCKEOWNANDVARAIYA:BILLINGUSERSANDPRICINGFORTCP
9
1
1.0
1
1.0
2
0.9
99
Avg Search Length
01
/19
/95
Ha
shin
g F
un
ction
1H
ash
ing
Fu
nctio
n 2
Ha
shin
g F
un
ction
3
(a)48-bitCAM
1
1.0
1
1.0
2
0.9
99
Avg Search Length
01
/19
/95
Ha
shin
g F
un
ction
1H
ash
ing
Fu
nctio
n 2
Ha
shin
g F
un
ction
3
(b)32-b
itCAM
1
1.0
1
1.0
2
1.0
3
1.0
4
Avg Search Length
01
/19
/95
Ha
shin
g F
un
ction
1H
ash
ing
Fu
nctio
n 2
Ha
shin
g F
un
ction
3
(c)16-b
itCAM
Fig.10.Averagenumberoflookupsforeach
connectio
nin
CAM-based
connectio
ntables.
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.7
3
Avg Search Length
01
/19
/95
2k B
ucke
ts4
k Bu
ckets
8k B
ucke
ts1
6k B
ucke
ts
32
k Bu
ckets
64
k Bu
ckets
12
8k B
ucke
ts
(a)fRAM;1
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.7
3
Avg Search Length
01
/19
/95
2k B
ucke
ts4
k Bu
ckets
8k B
ucke
ts1
6k B
ucke
ts
32
k Bu
ckets
64
k Bu
ckets
12
8k B
ucke
ts
(b)fRAM;2
Fig.11.Averagenumberoflookupsforeach
connectio
nin
RAM-based
connectio
ntables.
10 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 13, NO. 7, SEPTEMBER 1995
Bridge/Router Board
SBus
SBus Card
AddressTables
HostInterface
ProtocolConverter
MUX
ProtocolConverter
MUX
Host SPARCstation
PORTA
PORTB
NetworkInterface
NetworkInterface
Fig. 13. The architecture of the BayBridge prototype.
a 48-bit wide CAM entry, and a shadow entry in RAM.
B. Using the BayBridge as a BGW
The BayBridge may be used as a BGW in the followingway. The CAM-based address tables are implemented asdescribed in Section III.E. The connection tables are shownin Figure 8: hashing function fCAM;3 is applied to the 96-bit connection identi�er to generate a 48-bit index whichis used to �nd a match in the CAM. To verify the match,one of the IP addresses and one of the port numbers aremaintained in the shadow RAM entry. The RAM entry alsocontains counters for bytes and packets in each direction.The next hop routing information is also contained in theshadow RAM entry.When a TCP message arrives, the protocol converter
searches in the connection table for a matching connection.If the entry is not found then this message must be froma new connection. The message is forwarded to the Bay-Bridge's local host which forwards the connection identi�erto the Access Controller. When the connection has beenaccepted, the host accesses the connection tables directlyto insert the new entry.
If the entry is found in the connection table, the counterentries are read, incremented by the protocol converter andwritten back to the table.
C. Feasibility of BayBridge as a BGW
To operate as a BGW, the BayBridge must be able tohandle the intensity of connection setups and have su�-cient space in its connection table for all active connec-tions. It must also be fast enough to process all datagramsas they arrive, �nding the entries quickly in the connectiontables.We found in Section III.C that the intensity of new con-
nections does not exceed 79 per second. This is easilyachievable by the BayBridge: the protocol converter andhost can forward many hundreds of packets per second tothe Access Controller.Section III.D showed that if a 60 minute inactivity time-
out is used, the maximum number of active connections is1,350. The BayBridge has a table size of just 1,024 entries:
too small to hold all active connections. One way to over-come this limitation is to use the hardware CAM-basedaddress tables as a cache for a larger connection table inthe host's memory.In Section III.F we showed that the maximum arrival
rate for datagrams is less than 6,000 packets per second.Assuming that the �rst lookup is successful, the BayBridgecan perform the metering and routing functions in approx-imately 13�s6. This corresponds to a sustainable rate of75,000 packets per second simultaneously in each direction.We conclude that the BayBridge is easily capable of act-
ing as a BGW between the Berkeley campus and the restof the Internet.
V. Other Relevant Measurements
A. Variation in WAN Usage Levels
It is intuitive that di�erent Internet hosts and users gen-erate widely di�erent amounts of tra�c. This intuition isa crucial part of the motivation for usage based cost recov-ery. However, intuition alone should not justify complexcost recovery systems. We now show that there is a verylarge variation in WAN usage by hosts and by users on theBerkeley campus.Table II shows the usage per-host averaged over the du-
ration of the trace for the top-ten most active subdomainswithin Berkeley's network. These statistics indicate thatthe per host average usage varies widely by subdomain.The EECS Department subdomain has the largest num-ber of hosts. The amount of WAN TCP usage by thesehosts averaged 23 megabytes with a standard deviation of56 megabytes. Figures 14a-b show the large variation inusage for these hosts.Figures 15a-b show the distribution of usage per user.7
For this set of users, the average amount of usage was 1.0megabytes with a standard deviation of 3.7 megabytes.
B. Potential for Congestion Pricing
In addition to cost recovery, a billing system can also beused to control congestion by enforcing a pricing scheme.Using our trace data, we consider congestion control attwo di�erent time granularities and measure their e�ect onnetwork tra�c. These results are only intended to givean indication of the possible improvements in network e�-ciency. We do not propose speci�c techniques for achievingthese improvements.In the �rst scheme, we assume that the price to use
the network varies by time of day. For example, traf-�c sent during the day may be more expensive than atnight. Some tra�c is not sensitive to time of delivery andmay be postponed to a less expensive time of the day. Inparticular, we assume that under this scheme email andbulletin-board tra�c8 is \non-interactive" and therefore
6The publishedversion of this paper containeda typographical error(ms instead of �s). We have corrected it here.7This data is from an earlier trace recorded from midnight Novem-
ber 10 through midnight November 14, 1994.8These are the smtp and nntp protocols. They accounted for 9%
and 24%, respectively, of all wide area data bytes during our trace.
EDELL,MCKEOWNANDVARAIYA:BILLINGUSERSANDPRICINGFORTCP
11
TABLEII
AverageUsagebyHostsin
Top-TenSubdomains.
Number
PerHostAveragesa
PercentageinCategoryb
Subdomain
ofHosts
Connections
Datagrams
Bytes
Datagrams
Bytes
cs.berkeley.edu
469
1,393
130,737
23,422,981
26.6%
26.9%
eecs.berkeley.edu
482
594
56,633
9,703,412
11.8%
11.5%
lib.berkeley.edu
395
480
41,645
7,620,778
7.1%
7.4%
math.berkeley.edu
115
677
108,682
24,225,511
5.4%
6.8%
biochem.berkeley.edu
111
803
95,472
23,567,586
4.6%
6.4%
hip.berkeley.edu
1,058
146
13,364
1,935,854
6.1%
5.0%
ssl.berkeley.edu
79
314
77,873
19,349,504
2.7%
3.7%
me.berkeley.edu
235
249
35,406
6,427,425
3.6%
3.7%
cchem.berkeley.edu
295
195
27,295
5,143,642
3.5%
3.7%
geo.berkeley.edu
56
1,656
129,543
26,164,881
3.1%
3.6%
aThese
are
fourdayavera
ges.
bThistableandits
percen
tagecalcu
latio
nsexclu
dehosts
thatprovideserv
icesto
theentire
campus.
10
02
00
30
04
00
14
82
10
10
0
10
00
10
00
0
1e
+0
5
1e
+0
6
1e
+0
7
1e
+0
8
1e
+0
9
1e
+1
0
Bytes
me
an
=9
70
34
12
byte
ssd
ev=
56
35
19
97
byte
s
(a)Scatter
Plot
01
02
03
04
05
06
07
08
09
01
00
0 10
20
30
40
50
60
70
80
90
10
0
% M
achines
% Bytes8
5.3
%
94
.1%
(b)Cumulative
Distrib
ution
Fig.14.Varia
tionofnetw
ork
usagebymachinesin
theEECS.Berk
eley.EDUsubdoamin.
canbedelay
ed.Therem
ainingtra�
cis
assumed
tobe
\interactiv
e."9Wesprea
dthenon-in
teractiv
etra
�cover
timeby\w
ater-pourin
g,"
i.e.,wedistrib
ute
thistra�
cso
asto
level-outthetotal
(interactiv
e+
distrib
uted
non-
9Themostactiv
eofthese
\intera
ctive"
protocolswere
ftp,WWW,
shell,
telnet
and\X."
These
acco
unted
for33%,16%,4%,3%
and
3%,respectiv
ely,
ofallwideareadata
bytes
durin
gourtra
ce.
interactive)
inten
sityas
much
aspossib
le.
Figu
re16
show
stheim
provem
entthat
canbeach
ievedusin
g\w
ater-pourin
g."Thetop
lineshow
stheinten
sityof
allnetw
orktra�
cin
bytes
per
second.Thelow
estlin
eshow
stheinten
sityofjusttheinteractive
tra�c.
Thehori-
zontal
lineshow
stheinten
sityifthenon-in
teractivetra�
cis\poured
"over
theinteractive
tra�c.
Byspread
ingthe
12
IEEEJOURNALONSELECTEDAREASIN
COMMUNICATIONS,VOL.13,NO.7,SEPTEMBER1995
10
02
00
30
01
34
51
0
10
0
10
00
10
00
0
1e
+0
5
1e
+0
6
1e
+0
7
1e
+0
8
Bytes
me
an
=1
04
58
64
byte
ssd
ev=
36
82
39
4 b
ytes
(a)Scatter
Plot
01
02
03
04
05
06
07
08
09
01
00
0 10
20
30
40
50
60
70
80
90
10
0
% U
sers
% Bytes
80
.4%
93
.0%
(b)Cumulative
Distrib
ution
Fig.15.Varia
tionofnetw
ork
usageamongaselected
populatio
nofusers.
non-in
teractivetra
�cinthisway,
thenetw
orkutilization
isalm
ostcon
stantatan
inten
sityof150,00
0bytes
persecon
dwith
apeak
of170,0
00bytes
per
second.Thisrep
resents
a23%
reductio
nin
peak
bandwidth.
Intheseco
ndsch
eme,weassu
methatthenetw
orkiscon
-tro
lledmore
rapidly
inresp
onse
tocon
gestion.Thismay
correspondto
a ow
-control
mech
anism
,tra�
c-shapingus-
ingbu�erin
gor
apolicy
inwhich
theprice
variesquick
lyin
response
tocon
gestion.Toestim
atethepoten
tialgain
ofsuch
asch
eme,Figu
re17show
sthebusiest
0.1
second
and1secon
dinterva
lsdurin
gthetra
ce,measured
usin
gamoving-avera
gebu�er
(forcom
parison
,the�gure
alsoshow
stheavera
ge).
Thepeakinten
sityover
1secon
dis
22%
below
thepeakinten
sityover
0.1seco
nd,suggestin
gthatthepeak
netw
ork
load
canbered
uced
dramatically
by
such
asch
emeusin
gabu�er
ofjust250
,000
bytes.
VI.Conclusion
More
than90percen
tofanestim
ated
$200
million
an-
nual
Intern
etcost
istoday
recovered
fromusers
inthe
formofavariety
oforganiza
tionalcharg
esanduser
prices.
These
charges
andprices
are
design
edfor
administrative
convenien
cerather
thanto
prom
otee�
cientuse
ofnet-
work
resources.
Prices
design
edto
achieve
e�cien
tallo
ca-tio
nor
faircostrecov
ery,how
ever,
require
abillin
gsystem
that
canmeter
theindividual
user's
tra�candthat
canpresen
ttheuser
with
inform
ationthat
encou
ragese�
cient
use.
Thispaper
presen
tsabillin
gsch
emethat
caniden-
tifyandauthenticate
users,
monitor
individualuser
tra�c,
andpresen
ttheuser
with
realtim
epricin
ginform
ation.
Anim
plem
entation
ofthesch
emebased
ontheBayBrid
ge,aprototy
perou
ter,dem
onstrates
that
thebillin
gsch
emeis
practical,
andintro
duces
acceptab
lelaten
cy.
Itisan
open
question
wheth
erthetra�
cdem
andsand
correspondingcosts
ofoperatin
gtheIntern
etwillgrow
suf-
�cien
tlyto
make
pressin
gtheneed
foran
e�cien
tpricin
gandcost
recoverysch
eme.
That
will
dependon
theprolif-
erationofbandwidth
consumingapplication
son
theInter-
net.
Inturn,that
will
dependupon
theability
ofIntern
etproto
cols,lin
ksandrou
tersto
support
such
application
sin
competition
with
other
netw
orkserv
iceprov
iders,
inclu
d-
ingcab
leTVandtelep
honecom
panies.
Statistics
collectedfrom
afou
rday
traceofall
TCPtraf-
�cbetw
eentheUniversity
ofCaliforn
ia,Berkeley,
andthe
\outsid
eworld
"show
both
hom
ogeneities
anddiversities
with
inthecurren
tUniversity
population
.Thedata
reveal
severalfeatu
resthat
arerelevan
tto
thefeasib
ilityof
dif-
ferentpricin
gandcost
recoverysch
emes.
First,
ifpricin
gsch
emes
varybytim
eof
day,
then
upto
33%of
thetraf-
�cwhich
represen
tse-m
ailandbulletin
-board
tra�ccan
be
EDELL,MCKEOWNANDVARAIYA:BILLINGUSERSANDPRICINGFORTCP
13
0 20
40
60
80
10
0
12
0
14
0
16
0
18
0
20
0
22
0Intensity (Bytes/sec) (x 1000)
01
/19
/95
All T
raffic
Inte
ractive
Wa
ter L
eve
l
Fig.16.Reductio
nin
peaknetwork
bandwidth
by"pourin
g"e-m
ail
andbulletin
-board
tra�cover
theday.
shifted
overtim
ewith
littleloss
inbene�tto
theuser,
since
thistra
�cisnotsen
sitiveto
timeofdeliv
ery.Such
ashift
would
reduce
peakutiliza
tion.Seco
nd,on
amuch
smaller
timesca
le,we�ndthat
thepeakinten
sityover
1secon
dis22%
below
thepeakutilization
over0.1
second.Soif
pricin
gsch
emes
canshift
tra�cbyoneseco
nd,thepeak
router
andlin
kcapacity
canbesign
i�cantly
reduced
.Such
shifts
may
beautomatically
accommodated
byincreasin
gbu�er
sizesat
hosts
orin
routers.
Third
,itseem
sfeasib
leto
recovercost
forwidearea
tra�ceith
erfro
mtim
eofday
pricin
gorfrom
short
termpeakpricin
g.Recov
erythrou
ghtim
eofday
pricin
gwouldhave
abroaderbaseandwouldbe
much
simpler
toim
plem
ent.Recov
erythroughshort
termpeak
pricin
gwould
bemore
di�
cultto
implem
ent,butit
would
also
leadto
greater
reductio
nsin
thepeak
capacity
ofrou
tersandlin
ks.Acknowledgments
Theauthors
apprecia
tethegen
erousassista
nce
ofCli�
Frost,
Kevin
Mullally
andKevin
Zim
merm
anin
collectionof
netw
ork
tracedata.
References
[1]
J.Mackie-M
asonandH.Varia
n,\EconomicFAQsAboutthe
Internet",JournalofEconomic
Perspec
tives,vol.8,no.3,pp.
75-96,Summer1994.
[2]
C.Mills,
D.Hirsch
andG.Ruth,\Intern
etAcco
untin
g:Back-
ground",RFC1272,Intern
etEngineerin
gTask
Force
,1991.
0
10
0
20
0
30
0
40
0
50
0
60
0
70
0
80
0
90
0
10
00
11
00
12
00
Peak Bytes/sec (x 1000)
01
/17
/95
01
/18
/95
01
/19
/95
01
/20
/95
Pe
ak 1
00
ms
Pe
ak 1
sA
vera
ge
Fig.17.
Thepeakinten
sity,in
bytes
per
second,oftra
�clea
ving
theBerk
eleycampus,measured
over
0.1
secondand1seco
nd
interv
alscompared
with
theavera
geinten
sity.
[3]
H.W
.Braun,K.C.Cla�yandG.C.Polyzos,\AFramew
ork
for
Flow-Based
Acco
untin
gontheIntern
et",Proc.
oftheSingapore
IntlConferen
ceonNetw
orks,
1993.
[4]
N.Brownlee,
\New
ZealandExperien
ceswith
Netw
ork
Tra�c
Charging",ConneX
ions,
TheIntero
pCompany,vol.8,no.11,
Novem
ber1994.
[5]
D.Estrin
andL.Zhang,\Desig
nConsid
eratio
nsforUsageAc-
countin
gandFeed
back
inIntern
etworks",ACM
Computer
Com-
munica
tionReview
,vol.20,no5,pp.56-66.Octo
ber
1990.
[6]
R.Braden,\Requirem
entsforIntern
etHosts
{Communica
tion
Layers"
,RFC1122,Intern
etEngineerin
gTask
Force,
1989.
[7]
D.E.Comer,
Intern
etworkin
gwith
TCP/IP
|Volume
I,Pren
tice-Hall,2ndEd.,1991.
[8]
J.Mackie-M
asonandH.Varia
n,\Pricin
gtheIntern
et",In
Public
Access
totheIntern
et(K
ahin,B.andKeller,
J.eds.),
Pren
tice-Hall,1994.
[9]
TaxpayerAssets
Project
-Inform
atio
nPolicy
Note
May7,1994.
[10]K.Cla�y,
H.W
.BraunandG.Polyzos,\TrackingLong-Term
Growth
oftheNSFNET",Communica
tionsoftheACM,vol.
37,no.8,pp.34-45,Aug1994.
[11]J.Stein
er,C.NeumanandJ.Schiller,
\Kerb
eros:
AnAuthen-
ticatio
nServ
iceforOpen
Netw
ork
System
s",Proc.
ofUSENIX
Winter
Conferen
ce,pp.191-202,Feb
1988.
[12]Merit
Netw
ork,Inc.,
\NSFNET
BackboneServ
iceTra�cDis-
tributio
nby
Intern
etNetw
ork
Num-
ber,"
ftp://nic.m
erit.edu/nsfn
et/sta
tistics/1994/nsf-9
409.pnets.
Septem
ber1994.
[13]N.McK
eown,R.Edell
andM.T.Le,
\TheBayBrid
ge:
ahigh
speed
brid
ge/router"
.(ProtocolsforHigh-Speed
Netw
orksIII.
IFIP
WG6.1/WG6.4Third
Intern
atio
nalWorkshop,Stockholm
,Sweden,13-15May1992).IFIP
Transactio
nsC
(Communica
-tio
nSystem
s),vol.C
-9:203-18,1993.
[14]N.McK
eown,R.Edell
andM.T.Le,\TheBayBrid
ge:
AHigh
Speed
Brid
ge/Router
betw
eenFDDIandSMDSPart
I|
Sys-
temArch
itectureandPerfo
rmance"
,Submitted
.
14 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 13, NO. 7, SEPTEMBER 1995
Richard Edell (S'94) received the B.S. andM.S. degrees in electrical engineering from theUniversity of California, Berkeley, in 1991 and1994, respectively.
He is currently working toward the Ph.D. de-gree in electrical engineering at the Universityof California, Berkeley. His research interestsinclude network interface architectures, Inter-net economics and technology policy.
Nick McKeown (S'90-M'95) received theB.Eng degree in Electronic Engineering fromthe University of Leeds, England in 1986, andthe M.S. and Ph.D. degrees from the Univer-sity of California, Berkeley, in 1992 and 1995,respectively.
He is currently an Assistant Professor of Elec-trical Engineering and Computer Science atStanford University. His research interests in-clude high performance ATM switching and
economics of the Internet.
Pravin Varaiya (M'68{SM'78{F'80) receivedthe Ph.D. degree in electrical engineering fromthe University of California, Berkeley.
He is James Fife Professor of Electrical En-gineering and Computer Sciences at the Uni-versity of California, Berkeley, and Director ofCalifornia PATH, a multi-university programof research in Intelligent Transportation Sys-tems. From 1975 to 1992 he also was Professorof Economics at Berkeley. He has taught at
MIT and the Federal University of Rio de Janeiro. He was a memberof the technical sta� at Bell Laboratories during 1962-63. Dr. Varaiyahas held a Guggenheim Fellowship and a Miller Research Professor-ship. His areas of research are communication networks, transporta-tion systems, and electric power systems. He is writing a book withJean Walrand on high-speed, exible communication networks.