1Controlling the Cost of Reliability in Peer-to-Peer Overlays

Ratul Mahajan Miguel Castro Antony Rowstron

University of Washington Microsoft ResearchSeattle, WA Cambridge, UK

AbstractStructured peer-to-peer overlay networks pro-vide a useful substrate for building distributed applicationsbut there are general concerns over the cost of maintain-ing these overlays. The current approach is to configure theoverlays statically and conservatively to achieve the desiredreliability even under uncommon adverse conditions. Thisresults in high cost in the common case, or poor reliabilityin worse than expected conditions. We analyze the cost ofoverlay maintenance in realistic dynamic environments anddesign novel techniques to reduce this cost by adapting tothe operating conditions. With our techniques, the concernsover the overlay maintenance cost are no longer warranted.Simulations using real traces show that they enable high re-liability and performance even in very adverse conditionswith low maintenance cost.

I. INTRODUCTION

Structured peer-to-peer (p2p) overlay networks (e.g., [6,12, 7, 14]) are a useful substrate for building distributedapplications because they are scalable, self-organizing andreliable. They provide a hash table like primitive to routemessages using their keys. These messages are routedin a small number of hops using small per-node rout-ing state. The overlays update routing state automaticallywhen nodes join or leave, and can route messages correctlyeven when a large fraction of the nodes crash or the net-work partitions.

But scalability, self-organization, and reliability have acost; nodes must consume network bandwidth to main-tain routing state. There is a general concern over thiscost [10, 11] but there has been little work studying it. Thecurrent approach is to configure the overlays statically andconservatively to achieve the desired reliability and per-formance even under uncommon adverse conditions. Thisresults in high cost in the common case, or poor reliabilityin worse than expected conditions.

This paper studies the cost of overlay maintenance in re-alistic environments where nodes join and leave the systemcontinuously. We derive analytic models for routing relia-bility and maintenance cost in these dynamic conditions.

This work was done while Ratul Mahajan was visit-ing Microsoft Research. Emails: ratul@cs.washington.edu,{mcastro,antr}@microsoft.com

We also present novel techniques that reduce the mainte-nance cost by observing and adapting to the environment.First, we describe a self-tuning mechanism that minimizesthe overlay maintenance cost given a performance or relia-bility target. The current mechanism minimizes the proberate for fault detection given a target message loss rate. Itestimates both the failure rate and the size of the overlay,and uses the analytic models to compute the required proberate. Second, we present mechanisms to effectively and ef-ficiently deal with uncommon conditions such as networkpartitions and extremely high failure rates. These mecha-nisms enable the use of less conservative overlay configu-rations with lower maintenance cost. Though presented inthe context of Pastry [7, 2], our results and techniques canbe directly applied to other overlays.

Our results show that concerns over the maintenancecost in structured p2p overlays are not warranted anymore.It is possible to achieve high reliability and performanceeven in adverse conditions with low maintenance cost. Insimulations with a corporate network trace [1], over 99%of the messages were routed efficiently while control traf-fic was under 0.2 messages per second per node. Witha much more dynamic Gnutella trace [10], similar perfor-mance levels were achieved with a maintenance cost belowone message per second per node most of the time.

The remainder of the paper is organized as follows.We provide an overview of Pastry and our environmen-tal model in Section II, and present analytic reliability andcost models in Section III. Techniques to reduce mainte-nance cost appear in Section IV. In Section V we discusshow our techniques can be generalized for other structuredp2p overlays. Related work is in Section VI, and conclu-sions in Section VII.

II. BACKGROUND

This section starts with a brief overview of Pastry witha focus on aspects relevant to this paper. Then, it presentsour environment model.

PASTRY Nodes and objects are assigned random iden-tifiers from a large sparse 128-bit id space. These identi-fiers are called nodeIds and keys, respectively. Pastry pro-vides a primitive to send a message to a key that routes

2the message to the live node whose nodeId is numericallyclosest to the key in the id space.

The routing state maintained by each node consists ofthe leaf set and the routing table. Each entry in the rout-ing state contains the nodeId and IP address of a node. Theleaf set contains the l/2 neighboring nodeIds on either sideof the local nodes nodeId in the id space. In the routingtable, nodeIds and keys are interpreted as unsigned inte-gers in base 2b (where b is a parameter with typical value4). The routing table is a matrix with 128/b rows and 2bcolumns. The entry in row r and column c of the routingtable contains a nodeId that shares the first r digits with thelocal nodes nodeId, and has the (r + 1)th digit equal to c.If there is no such nodeId or the local nodeId satisfies thisconstraint, the entry is left empty. On average only log2bNrows have non-empty entries.

Pastry routes a message to a key using no more thanlog2bN hops on average. At each hop, the local node nor-mally forwards the message to a node whose nodeId shareswith the key a prefix that is at least one digit longer thanthe prefix that the key shares with the local nodes nodeId.If no such node is known, the message is forwarded to anode whose nodeId is numerically closer to the key andshares a prefix with the key at least as long. If there is nosuch node, the message is delivered to the local node.

Pastry updates routing state when nodes join and leavethe overlay. Joins are handled as described in [2] and fail-ures are handled as follows. Pastry uses periodic probingfor failure detection. Every node sends a keep-alive to themembers of its leaf set every Tls seconds. Since the leafset membership is symmetric, each node should receive akeep-alive message from each of its leaf set members. If itdoes not, the node sends an explicit probe and assumes thata member is dead if it does not receive a response withinTout seconds. Additionally, every node sends a livenessprobe to each entry in its routing table every Trt seconds.Since routing tables are not symmetric, nodes respond tothese probes. If no response is received within Tout, an-other probe is sent. The node is assumed faulty if no re-sponse is received to the second probe within Tout.

Faulty entries are removed from the routing state but itis necessary to replace them with other nodes. It is suffi-cient to replace leaf set entries to achieve correctness but itis important to replace routing table entries to achieve log-arithmic routing cost. Leaf set replacements are obtainedby piggybacking information about leaf set membershipin keep-alive messages. Routing table maintenance is per-formed by periodically asking a node in each row of therouting table for the corresponding row in its routing ta-ble, and when a routing table slot is found empty duringrouting, the next hop node is asked to return any entry it

may have for that slot. These mechanisms are described inmore detail in [2].

ENVIRONMENT MODEL Our analysis and some of ourcost reduction techniques assume that nodes join accord-ing to a Poisson process with rate and leave accordingto an exponential distribution with rate parameter (asin [4]). But we also evaluate our techniques using realisticnode arrival and departure patterns and simulated massivecorrelated failures such as network partitions. We assumea fail-stop model and conservatively assume that all nodesleave ungracefully without informing other nodes and thatnodes never return with the same nodeId.

III. RELIABILITY AND COST MODELS

Pastry forwards messages using UDP with no acknowl-edgments by default. This is efficient and simple, but mes-sages forwarded to a faulty node are lost. The probabilityof forwarding a message to a faulty node at each hop isPf (T, ) = 1

1T(1 e

T), where T is the maximumtime it takes to detect the fault. There are no more thanlog2bN overlay hops in a Pastry route on average. Typi-cally, the last hop uses the leaf set and the others use therouting table. If we ignore messages lost by the underlyingnetwork, the message loss rate, L, is:

L = 1(1Pf (Tls+Tout, )).(1Pf (Trt+2Tout, ))log

2bN1

Reliability can be improved by applications if re-quired [9]. Applications can retransmit messages and seta flag indicating that they should be acknowledged at eachhop. This provides very strong reliability guarantees [2]because nodes can choose an alternate next hop if the pre-viously chosen one is detected to be faulty. But waitingfor timeouts to detect that the next hop is faulty can leadto very bad routing performance. Therefore, we use themessage loss rate, L, in this paper because it models bothperformance and reliability the probability of being ableto route efficiently without waiting for timeouts.

We can also derive a model to compute the cost of main-taining the overlay. Each node generates control trafficfor five operations: leaf set keep-alives, routing table entryprobes, node joins, background routing table maintenance,and locality probes. The control traffic in our setting isdominated by the first two operations. So for simplicity,we only consider the control traffic per second per node,C, due to leaf set keep-alives and routing table probes:

C =l

Tls+

2 128

b

r=0((2b 1) (1 b(0; N, 1

(2b)(r+1))))

Trt

30 50 100session time in minutes

0

5

10

15

20lo

ss r

ate

(%)

(a) loss rate

T_rt = 10sT_rt = 30sT_rt = 60s

0 50 100session time in minutes

0

5

10

15

20

msg

s/sec

per

nod

e(b) control traffic

T_rt = 10sT_rt = 30sT_rt = 60s

Fig. 1. Verifying loss rate and control traffic models.

The first term is the cost of leaf set keep-alives: l keep-alives every Tls seconds. The second is the cost of rout-ing table probing: two messages (probe and response) foreach routing table entry every Trt. The summation com-putes the expected number of routing table entries, whereb(k; n, p) is the binomial distribution.

We verified these equations using simulation. Westarted by creating a Pastry overlay with 10,000 nodes.Then we let new nodes arrive and depart according to aPoisson processes with the same rate to keep the num-ber of nodes in the system roughly constant. After tensimulated minutes, 500,000 messages were sent over thenext ten minutes from randomly selected nodes to ran-domly selected keys. Figure 1a shows the message lossrate for three different values of Trt (10, 30 and 60 sec-onds) with Tls fixed at 30 seconds. The x-axis shows themean session lifetime of a node ( = 1/lifetime). Thelines correspond to the values predicted with the loss rateequation and the dots correspond to the simulation results(three simulation runs for each parameter setting). Fig-ure 1b shows a similar graph for control traffic. The resultsshow that both equations are quite accurate. As expected,the loss rate decreases when Trt (or Tls) decrease but thecontrol traffic increases.

IV. REDUCING MAINTENANCE COST

This section describes our techniques to reduce theamount of control traffic required to maintain the overlay.We start by motivating the importance of observing andadapting to the environment by discussing the characteris-tics of realistic environments. Then, we explain the self-tuning mechanism and the techniques to deal with massivefailures.

A. Node arrivals and departures in realistic environments

We obtained traces of node arrivals and failures fromtwo recent measurement studies of p2p environments. Thefirst study [10] monitored 17,000 unique nodes in theGnutella overlay over a period of 60 hours. It probed eachnode every seven minutes to check if it was still part of

the overlay. The average session time over the trace wasapproximately 2.3 hours and the number of active nodesin the overlay varied between 1300 and 2700. Figure 2ashows the failure rate over the period of the trace averagedover 10 minute intervals. The arrival rate is similar. Thereare large daily variations in the failure rate of more than afactor of 3.

The Gnutella trace is representative of conditions in anopen Internet environment. The second trace [1] is repre-sentative of a more benign corporate environment. It mon-itored 65,000 nodes in the Microsoft corporate network,probing each node every hour for a month. The averagesession time over the trace was 37.7 hours. This traceshowed large daily as well as weekly variations in the fail-ure rate, presumably because machines are switched offduring nights and weekends.

These traces show that failure rates vary significantlywith both daily and weekly patterns, and the failure ratein the Gnutella overlay is more than an order of magni-tude higher than in the corporate environment. Therefore,the current static configuration approach would require notonly different settings for the two environments, but alsoexpensive configurations if good performance is desired atall times. The next sections show how to achieve high re-liability with lower cost in all scenarios.

B. Self-tuning

The goal of a self-tuning mechanism is to enable anoverlay to operate at the desired trade-off point betweencost and reliability. In this paper we show how to operateat one such point achieve a target routing reliability whileminimizing control traffic. The methods we use to do thiscan be easily generalized to make arbitrary trade-offs.

In the loss rate and control traffic equations, there arefour parameters that we can set: Trt, Tls, Tout, and l. Cur-rently, we choose fixed values for Tls and l that achievethe desired resilience to massive failures (Section IV-C).Tout is fixed at a value higher than the maximum expectedround trip delay between two nodes; it is set to 3 secondsin our experiments (same as the TCP SYN timeout). Wetune Trt to achieve the specified target loss rate by period-ically recomputing it using the loss rate equation with thecurrent estimates of N and . Below we describe mech-anisms that, without any additional communication, esti-mate N and .

We use the density of nodeIds in the leaf set to esti-mate N . Since nodeIds are picked randomly with uniformprobability from the 128-bit id space, the average distancebetween nodeIds in the leaf set is 2128/N . It can be shownthat this estimate is within a small factor of N with very

40 20 40 60hours from start of trace

0.000

0.005

0.010

0.015

0.020

failu

re r

ate

(1/mi

n)

(a) Gnutella failure rate0 20 40 60

hours from start of trace0

1

2

loss

rat

e (%

)

(b) loss rate / hand-tuned0 20 40 60

hours from start of trace0

1

2

loss

rat

e (%

)

(c) loss rate / self-tuned0 20 40 60

hours from start of trace0.0

0.5

1.0

1.5

2.0

2.5

msg

s/s p

er n

ode

(d) control traffic

hand-tunedself-tuned

Fig. 2. Self-tuning: (a) shows the measured failure rate in the Gnutella network; (b) and (c) show the loss rate in the hand-tunedand self-tuned versions of Pastry; and (d) shows the control traffic in the two systems.

high probability, which is sufficient for our purposes sincethe loss rate depends only on log2bN .

The value of is estimated by using node failures inthe routing table and leaf set. If nodes fail with rate ,a node with M unique nodes in its routing state shouldobserve K failures in time KM . Every node remembers thetime of the last K failures. A node inserts its current timein the history when it joins the overlay. If there are onlyk < K failures in the history, we compute the estimate asif there was a failure at the current time. The estimate of is kMTkf , where Tkf is the time span between the firstand the last failure in the history.

The accuracy of s estimate depends on K; increas-ing K increases accuracy but decreases responsiveness tochanges in the failure rate. We improve responsivenesswhen the failure rate decreases by using the current esti-mate of to discard old entries from the failure history thatare unlikely to reflect the current behavior of the overlay.When the probability of observing a failure given the cur-rent estimate of reaches a threshold (e.g., 0.90) withoutany new failure being observed, we drop the oldest failuretime from the history and compute a new estimate for .

We evaluated our self-tuning mechanism using simula-tions driven by the Gnutella trace. We simulated two ver-sions of Pastry: self-tuned uses the self-tuning mechanismto adjust Trt to achieve a loss rate of 1%; and hand-tunedsets Trt to a fixed value that was determined by trial anderror to achieve the same average loss rate. Hand-tuningis not possible in real settings because it requires perfectknowledge about the future. Therefore, a comparison be-tween these two versions of Pastry provides a conservativeevaluation of the benefits of self-tuning.

Figures 2b and 2c show the loss rates achieved by theself-tuned and the best hand-tuned versions, respectively.The loss rate is averaged over 10 minute windows, andis measured by sending 1,000 messages per minute fromrandom nodes to random keys. Tls was fixed at 30 secondsin both versions and Trt at 120 seconds for hand-tuned.The results show that self-tuning works well, achieving thetarget loss rate independent of the failure rate.

Figure 2d shows the control traffic generated by both thehand-tuned and the self-tuned versions. The control trafficgenerated by hand-tuned is roughly constant whereas theone generated by self-tuned varies according to the failurerate to meet the loss rate target. It is interesting to note thatthe loss rate of hand-tuned increases significantly above1% between 52 and 58 hours due to an increased failurerate. The control traffic generated by the self-tuned versionclearly increases during this period to achieve the targetloss rate with the increased failure rate. If the hand-tunedversion was instead configured to always keep loss ratebelow 1%, it would have generated over 2 messages persecond per node all the time.

We simulated the Microsoft corporate network tracealso and obtained similar results. The self-tuned versionachieved the desired loss rate with under 0.2 messages persecond per node. The hand-tuned version required a dif-ferent setting for Trt as the old value would have resultedin an unnecessarily high overhead.

C. Dealing with massive failuresNext we describe mechanisms to deal with massive but

rare failures such as network partitions.

BROKEN LEAF SETS Pastry relies on the invariant thateach node has at least one live leaf set member on eachside. This is necessary for the current leaf set repair mech-anism to work. Chord relies on a similar assumption [12].Currently, Pastry uses large leaf sets (l = 32) to ensure thatthe invariant holds with high probability even when thereare massive failures and the overlay is large [2].

We describe a new leaf set repair algorithm that usesthe entries in the routing table. It can repair leaf setseven when the invariant is broken. So it allows the useof smaller leaf sets, which require less maintenance traffic.

The algorithm works as follows. When a node n detectsthat all members in one side of its leaf set are faulty, it se-lects the nodeId that is numerically closest to ns nodeIdon that side from among all the entries in its routing state.Then it asks this seed node to return the entry in its routing

5state with the nodeId closest to ns nodeId that lies be-tween the seeds nodeId and ns nodeId in the id space.This process is repeated until no more live nodes withcloser nodeIds can be found. The node with the closestnodeId is then inserted in the leaf set and its leaf set is usedto complete the repair. To improve the reliability of the re-pair process, we explore several paths in parallel startingwith different seeds.

The expected number of rounds to complete the repairis log2bN . We improve convergence by adding a shadowleaf set to the routing state. The shadow leaf set of noden contains the l/2 nodes in the right leaf set of its furthestleaf on the right, and the l/2 nodes in the left leaf set of itsfurthest leaf on the left. This state is acquired at no addi-tional cost as leaf set changes are already piggybacked onkeep-alive messages, and is inexpensive to maintain sincenodes in it are not directly probed. Most leaf set repairscomplete in one round using the nodes in the shadow leafset. We also use the shadow leaf set to increase the accu-racy of the estimate of N .

The experiments in this paper use a small leaf set withl = 8. This is sufficient to ensure reliable operation withhigh probability even when half the nodes fail simultane-ously. At the same time it ensures that the leaf set repairalgorithm does not need to be invoked very often. For ex-ample, the probability of all nodes in half of a leaf set fail-ing within Tls = 30s is less than 1010 with the averagesession time in the Gnutella trace.

MASSIVE FAILURE DETECTION The failure of a largenumber of nodes in a small time interval results in a largenumber of faulty entries in nodes routing tables, whichincreases loss rate. This is resolved when nodes detect thatthe entries are faulty and repair the routing tables. Butit can take very long in environments with low averagefailure rate because the self-tuning mechanism sets Trt toa large value. We reduce the time to repair routing tablesusing a mechanism to detect massive failures.

The members of a leaf set are randomly distributedthroughout the underlying network. So a massive failuresuch as a network partition is likely to manifest itself asfailure of several nodes in the leaf set within the sameprobing period. When a node detects a number of faultsin the same probing period that exceeds a specified frac-tion of l, it signals the occurrence of a massive failure andprobes all entries in its routing table (failures discoveredduring this probing are not used for estimating as thebackground failure rate has not changed). This reducesthe time to repair the routing tables after the failure toTls + Tout seconds. We set an upper bound on the valueof Tls that achieves the desired repair time. In the experi-ments described in this paper, the threshold on the numberof faults for failure detection is 30% and Tls = 30s.

0 20 40 60 80 100 120simulation time in minutes

010203040506070

loss

rat

e (%

) loss rate

0

1

2

3

4

5 avg. number of hops

hops

Fig. 3. Impact of half the nodes in the overlay failing together.

We evaluated our mechanisms to deal with massive fail-ures using a simulation with 10,000 nodes where we failedhalf the nodes 40 minutes into the simulation. In additionto this failure, nodes were arriving and departing the over-lay according to a Poisson process with a mean lifetimeof 2 hours. 10,000 messages per minute were sent fromrandom sources to random keys. Figure 3 shows the aver-age loss rate and number of overlay hops in each minute.There is a large peak in the loss rate when the massive fail-ure occurs but Pastry is able to recover in about a minute.Recovery involved not only detecting the partition and re-pairing routing tables but also repairing several broken leafsets.

V. APPLICABILITY BEYOND PASTRY

The work presented in this paper is relevant to otherstructured p2p overlay networks as well. In this section,we briefly outline how it applies to other networks. Due tospace constraints, we only describe how it can be appliedto CAN [6] and Chord [12], and we assume that the readerhas a working knowledge of CAN and Chord.

The average number of hops in CAN is d4N1/d (where d

is the number of dimensions) and is 12 log2N in Chord. Wecan use these equations to compute the average loss ratefor a given failure rate () and number of nodes (N ) as wedid for Pastry. The average size of the routing state is 2d inCAN and log2N + l (where l is the successor set size) inChord. We can use these equations and the probing ratesused in CAN and Chord to compute the amount of controltraffic.

Self-tuning requires an estimate of N and . Our ap-proach for estimating N using density can be generalizedeasily the size of local and neighboring zones, and thedensity of the successor set can be used respectively forCAN and Chord. We can estimate in CAN and Chordusing the failures observed in the routing state exactly aswe did in Pastry.

The leaf set repair algorithm applies directly to Chordssuccessor set. A similar iterative search mechanism can

6be used when a CAN node loses all its neighbors alongone dimension (the current version uses flooding). Finally,partition detection can be done using the successor set inChord and neighbors in CAN.

VI. RELATED WORK

Most previous work has studied overlay maintenanceunder static conditions but the following studied dynamicenvironments where nodes continuously join and leave theoverlay. Saia et al use a butterfly network to build an over-lay that routes efficiently even with large adversarial fail-ures provided that the network keeps growing [8]. Pan-durangan et al present a centralized algorithm to ensureconnectivity in the face of node failures [5]. Liben-Nowellet al provide an asymptotic analysis of the cost of main-taining Chord [12]. Ledlie et al [3] present some simula-tion results of Chord in an idealized model with Poissonarrivals and departures. We too study the overlay main-tenance cost in dynamic environments but we provide anexact analysis in an idealized model together with simu-lations using real traces. Additionally, we describe newtechniques to reduce this cost while providing high relia-bility and performance.

Weatherspoon and Kubiatowicz have looked at efficientnode failure discovery [13]; they propose that nodes fur-ther away be probed less frequently to reduce wide areatraffic. In contrast, we reduce the cost of failure discoverythrough adapting to the environment. The two approachescan potentially be combined though their approach makesthe later hops in Tapestry (and Pastry) less reliable, withmessages more likely to be lost after having been routedfor a few initial hops.

VII. CONCLUSIONS AND FUTURE WORK

There are general concerns over the cost of maintain-ing structured p2p overlay networks. We examined thiscost in realistic dynamic conditions, and presented noveltechniques to reduce this cost by observing and adaptingto the environment. These techniques adjust control trafficbased on observed failure rate and they detect and recoverfrom massive failures efficiently. We evaluated these tech-niques using mathematical analysis and simulation withreal traces. The results show that concerns over the overlaymaintenance cost are no longer warranted. Our techniquesenable high reliability and performance even in adverseconditions with low maintenance cost. Though done in thecontext of Pastry, this work is relevant to other structuredp2p networks such as CAN, Chord and Tapestry.

As part of ongoing work, we are exploring differentself-tuning goals and methods. These include i) oper-ating at arbitrary points in the reliability vs. cost curve

and using different performance or reliability targets; ii)choosing a self-tuning target that takes into account theapplications retransmission behavior, such that total traf-fic is minimized; and iii) varying Tls (has implications fordetecting leaf set failures since keep-alives are unidirec-tional) along with Trt under the constraint that Tls has anupper bound determined by the desired resilience to mas-sive failures. We are also studying the impact of failureson other performance criteria such as locality.

ACKNOWLEDGEMENTS

We thank Ayalvadi Ganesh for help with mathematicalanalysis, and John Douceur and Stefan Saroiu for the tracedata used in this paper.

REFERENCES[1] W. J. Bolosky, J. R. Douceur, D. Ely, and M. Theimer. Feasibility

of a serverless distributed file system deployed on an existing setof desktop PCs. In ACM SIGMETRICS, June 2000.

[2] M. Castro, P. Druschel, Y. C. Hu, and A. Rowstron. Exploitingnetwork proximity in peer-to-peer overlay networks. TechnicalReport MSR-TR-2002-82, Microsoft Research, May 2002.

[3] J. Ledlie, J. Taylor, L. Serban, and M. Seltzer. Self-organizationin peer-to-peer systems. In ACM SIGOPS European Workshop,Sept. 2002.

[4] D. Liben-Nowell, H. Balakrishnan, and D. Karger. Analysis ofthe evolution of peer-to-peer systems. In ACM Principles of Dis-tributed Computing (PODC), July 2002.

[5] G. Pandurangan, P. Raghavan, and E. Upfal. Building low-diameter peer-to-peer networks. In IEEE FOCS, Oct. 2001.

[6] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker. Ascalable content-addressable network. In SIGCOMM, Aug. 2001.

[7] A. Rowstron and P. Druschel. Pastry: Scalable, distributed ob-ject location and routing for large-scale peer-to-peer systems. InIFIP/ACM Middleware, Nov. 2001.

[8] J. Saia, A. Fiat, S. Gribble, A. Karlin, and S. Saroiu. Dynamicallyfault-tolerant content addressable networks. In IPTPS, Mar. 2002.

[9] J. Saltzer, D. Reed, and D. Clarke. End-to-end arguments in sys-tem design. ACM TOCS, 2(4), Nov. 1984.

[10] S. Saroiu, K. Gummadi, and S. Gribble. A measurement study ofpeer-to-peer file sharing systems. In MMCN, Jan. 2002.

[11] S. Sen and J. Wang. Analyzing Peer-to-Peer Traffic AcrossLarge Networks. In Proc. ACM SIGCOMM Internet Measure-ment Workshop, Marseille, France, Nov. 2002.

[12] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrish-nan. Chord: A scalable peer-to-peer lookup service for Internetapplications. In ACM SIGCOMM, Aug. 2001.

[13] H. Weatherspoon and J. Kubiatowicz. Efficient heartbeats and re-pair of softstate in decentralized object location and routing sys-tems. In ACM SIGOPS European Workshop, Sept. 2002.

[14] B. Y. Zhao, J. D. Kubiatowicz, and A. D. Joseph. Tapestry: Aninfrastructure for fault-resilient wide-area location and routing.Technical Report UCB-CSD-01-1141, U. C. Berkeley, Apr. 2001.