the distributed object consistency protocol version 1 · the distributed object consistency...

The Distributed Object ConsistencyProtocol Version 1.0

John Dilley, Martin Arlitt, Stephane Perret, Tai JinInternet Systems and Applications LaboratoryHP Laboratories Palo AltoHPL-1999-109September, 1999

World Wide Web,cache consistency,cacheinvalidation,publish/subscribeprotocol, proxycache, HTTPcache control,strongconsistency, deltaconsistency

This report describes a protocol for improving content consistency in webproxy cache servers, which leads to reduced response time and server load.The Distributed Object Consistency Protocol (DOCP) is an extension toHTTP, replacing some of HTTP’s current cache control mechanism. Itsupports incremental evolution: proxy servers that implement the protocolcan interoperate with non-DOCP proxy servers and gradually learn aboutpeers as they are deployed in the network.

The DOCP uses a publish/subscribe mechanism with server invalidationwhen objects change instead of client validation to test for changes.Stronger semantic consistency assures content providers that pages servedfrom cache are the same as are on the master origin server, and assures endusers that the information they are getting is fresh, limiting the need to“Reload”.

Strong consistency is difficult to achieve in wide area networks, so theDOCP provides “delta consistency”, where an object is consistent for all buta short, bounded time after a modification. This allows the DOCP to meetweb user expectations for content availability and responsiveness.

Protocol simulation shows that DOCP consumes fewer network and serverresources than the current HTTP cache consistency protocol does, whileproviding greatly improved consistency. DOCP eliminates the need for usersand content providers to guess at correct values for HTTP cache controlheaders by providing a clean framework for wide area web object replication.

Copyright Hewlett-Packard Company 1999

hek-thehstsoer,

li--b.rvendaleor-

thef

enty

nd-

c-ndn.ebofesonec-a-n

me

ce,er-

nce

rmsal,t is

The Distributed Object Consistency ProtocolVersion 1.0

John Dilley, Martin Arlitt, Stéphane Perret, Tai Jin

Hewlett-Packard LaboratoriesPalo Alto, CA

Abstract

This report describes a protocol for improving contentconsistency in web proxy cache servers, which leads toreduced response time and server load. The DistributedObject Consistency Protocol (DOCP) is an extension toHTTP, replacing some of HTTP’s current cache controlmechanism. It supports incremental evolution: proxyservers that implement the protocol can interoperatewith non-DOCP proxy servers and gradually learnabout peers as they are deployed in the network.

The DOCP uses a publish/subscribe mechanism withserver invalidation when objects change instead ofclient validation to test for changes. Stronger semanticconsistency assures content providers that pages servedfrom cache are the same as are on the master originserver, and assures end users that the information theyare getting is fresh, limiting the need to “Reload”.

Strong consistency is difficult to achieve in wide areanetworks, so the DOCP provides “delta consistency”,where an object is consistent for all but a short, boundedtime after a modification. This allows the DOCP to meetweb user expectations for content availability andresponsiveness.

Protocol simulation shows that DOCP consumes fewernetwork and server resources than the current HTTPcache consistency protocol does, while providinggreatly improved consistency. DOCP eliminates theneed for users and content providers to guess at correctvalues for HTTP cache control headers by providing aclean framework for wide area web object replication.

1 IntroductionTo improve service to web clients, web proxy cacheservers have been deployed throughout the network.These cache servers store copies of objects requested byweb users and subsequently serve that object to thoseusers if requested again. By serving cached objects, theproxy reduces network and origin server demand. How-ever, the objects they serve are not necessarily currentwith the origin server. Thisweak consistencyleads con-tent providers to disable caching for some objects andforces users to reload pages when they suspect inconsis-tency. This increases server load, especially duringflashcrowds when many users visit a site at the same time.

Cache servers can not know whether an object in cacis consistent without querying the origin server. Checing consistency every time assures consistency atcost of additional connections to origin servers, whicadds considerable delay to the servicing of user reque[8]. Use of polling to achieve strong consistency alsincreases demand on the network and origin servwhich reduces the benefit of caching.

Weak consistency is currently accepted for most appcations of the web. Client-driven polling is the only current alternative to provide strong consistency in the weWhen consistency is needed, origin servers must seevery request, which requires sufficient hardware anetwork resources. However this scheme does not scwell; implementing strong consistency by polling is toexpensive for all but a few content providers. Furthemore these solutions can not improve response time.

Stronger consistency is important for business use ofweb such as for interaction with customers, viewing osupplies from vendors, and access to time-dependinformation. Greater commercial use of web technologwill lead to increased expectations of performance apredictability. The Distributed Object Consistency Protocol is designed to address these needs.

The remainder of the paper is organized as follows. Setion 2 presents an overview of cache consistency asome issues with today’s web cache implementatioSection 3 discusses requirements for a wide area wconsistency protocol. Section 4 presents an overviewthe DOCP consistency model and Section 5 describthe protocol in detail. Section 6 discusses the interactibetween DOCP and HTTP cache control headers. Stion 7 analyzes the protocol’s performance and summrizes the results of a simulation of the protocol. Sectio8 discusses related work. Section 9 concludes with soobservations and suggestions for future work.

2 Web Cache ConsistencyCaching enhances system scalability and performanparticularly where remote communication has highcost (in latency or resource utilization) than local communication with the cache. With caching or replicatiocomes the issue of maintaining consistency or coherenamong the copies. Caches can be characterized in teof the coherence they provide to their users. In generthe weaker the consistency requirement, the easier ito build a scalable distributed system.

DOCP.fm September 28, 1999 2:50 pm

The Distributed Object Consistency Protocol Web Cache Consistency

heyndkes-sed

ak.

na

er-es

e5]

oftd)

li-

t

seofi-Preof

stn-the

rA)LLver-deisghe

2.1 Strong Consistency ModelsUnder a strong consistency model a cache guaranteesnot to serve data that is different from a request for theoriginal data to the origin server. This requirement isfrequently seen in operating systems (such as in a mem-ory or disk cache) and transaction-processing systems.Mosberger [22] provides an overview of the followingtypes of strong consistency.

• Atomic. Any read on a shared item returns the mostrecent write of that item. In a distributed system eachparticipant has an independent notion of event order;there is no global “most recent”. Atomic action isonly defined locally.

• Sequential. The result of any set of operations is thesame as if the operations were executed in somesequential order [16]. In other words, there is sometotal ordering for all operations taken together. Anyexecution must obey this total ordering to be sequen-tial.

• Causal. Writes that are potentially causally relatedmust be seen by all processes in the same order, butconcurrent writes that are not related may be seen ina different order by different processes [15].

• Pipelined RAM(PRAM). Writes by a single processare received by all other processes in the order theywere issued, but writes from different processes maybe seen in a different order by different processes[18].

Strong consistency in a distributed system can be client-driven or server-driven: either clients query the server tovalidate the object on each request, or servers invalidatecached copies when an object changes.

In the client-driven case the load on the origin server canbe nearly as high as it would be without caching sincethe server has to handle each request (perhaps not withfull object data). If communication latency between theclient and server is great, the cache can not reduceresponse time significantly.

In the server-driven case the server must maintain stateabout all cached replicas, and notify each replica beforea change to the data is committed. In general, readaccess is not permitted while the data is changing (afterinvalidations begin to be delivered but before they areacknowledged by all replicas). A write can not completeuntil all replicas have been notified to flush the previouscopy of the object. This will prevent a replica servinginconsistent data relative to its peers, and supportssequential consistency. In wide area distributed systems,communication delay between peers can be quite high,so this protocol may require a long time to complete.

In any strong consistency implementation, network orreplica delay or failure will either delay or prevent readand write requests from completing.

2.2 Weak Consistency ModelsRelaxing the consistency requirement can improve tavailability, fault resilience, and performance of wideldistributed systems, such as the Grapevine [5] aLOCUS [25] distributed systems; [28] discusses weaconsistency and suggests a mechanism to provide ssion guarantees to reduce the potential confusion cauby weak consistency on users and applications.

To describe web cache consistency we classify weconsistency models for information access as follows

• Delta consistency. The result of any read operation isconsistent with a read on the original copy of aobject except for a (short) bounded interval aftermodification.

• Eventual consistency. The result of any read opera-tion is consistent with the last known update, but thpropagation time of updates is not prescribed. Infomation about updates is distributed to replica sitand if updates cease replicas will eventually becomconsistent. This is the model used in Grapevine [by the Clearinghouse directory service.

• Loose. There is no guarantee about consistencyany read with the original copy of an object, buthere are systematic (possibly manually invokemechanisms to validate and restore consistency.

• Periodic. Replicas are periodically synchronizedwith their original copy, after which they may againdiverge. There is no systematic mechanism to vadate object consistency on demand.

• Offline. The state of the original copy of an objeccan not be determined on demand.

Most current web cache implementations provide looconsistency. They serve object data for some periodtime without contacting (or being contacted by) the orgin server. Loose consistency is inherent in the HTTprotocol: there is no protocol mechanism to assustronger consistency other than to prevent cachingobjects. HTTP does provide for validation and mocaches implement occasional validation of object cosistency. One widely used mechanism is described innext section.

2.3 The Alex ProtocolAvoiding contact with the origin server improves useresponse time [8] and reduces network utilization.cache does this by determining a time to live (TTLvalue for each object in the cache. During the TTperiod the cache will serve the object locally. The TTvalue could be chosen as some constant value, howeweb objects have widely varying lifetimes and modification rates. The Alex file system protocol [6] developean adaptive TTLmechanism to address this need. ThHarvest cache [7] employed this mechanism, whichfairly widely used by Harvest-derived caches, includinthe Squid open-source proxy cache [27]. We use tSquid implementation to illustrate the protocol.

2

The Distributed Object Consistency Protocol DOCP Requirements

n-ndmetly

dyn-otndre-henuldir

ri-a

ndsnayrs

nleolng

ste,fe

omi-s

aner-il-orehests

r-y

ct-

byrsey

In the Squid adaptive TTL implementation, the TTL isdetermined to be the time indicated in theExpiresheader, if one is present; if not it uses the time in theLast-Modified header, computes a percentage of theage of the object in the cache at each new request anduses that value as the TTL ([8] has an illustration anddescription of this protocol). After the TTL periodexpires the cache will make an HTTPGETrequest withan If-Modified-Since (IMS) header field indicatingthe last modification time of the object as previouslyreported by the origin server. If the object has changedthe origin server will respond with a fresh copy of theobject and the HTTP200 OK response status; otherwiseit will return the response status304 Not Modified ,indicating that the cached copy is still current.

In most cases theseGET IMSrequests are unnecessary,since web objects change much less frequently than theyare accessed. In a study of five months of web accessfrom a group of users with cable modems we observed15% of total requests resulted in304 Not Modifiedresponses [2]. In another study of the World Cup website we observed as high as 37% of responses from theserver were these “positive validations” [4]. These per-centages are of total requests, including first timerequests and requests for objects evicted from caches.We can not determine how many requests wereGETIMS requests from the logs.

Eliminating theseGET IMS requests would seriouslycompromise object consistency. It is not possible toassess how many stale objects were served by caches;that requires global state, which is not available. Therate and distribution of object modification and objectaccesses both affect the stale hit ratio.

The TTL mechanism creates a trade-off between objectconsistency on the one hand, and network utilizationand response time on the other. By choosing a large TTLvalue, a cache will make fewer external requests andservice data more quickly, but it will serve more incon-sistent copies of information to end users. By choosing asmall value, the cache will make more unnecessaryGETIMS requests but achieve higher consistency. Cacheadministrators are sometimes able to make this trade-offthrough cache configuration based upon local systemand user requirements. Note that this decision may notreflect content provider or user preferences.

3 DOCP RequirementsThe broad goals of the DOCP are to improve the userresponse time, predictability, accountability, scalability,and fault tolerance of HTTP access to web objects. Thissection presents some key requirements for a protocoldesigned to meet these goals.

3.1 Incremental EvolutionServers in the Internet are not owned by any centralorganization nor do they obey any common set of rules.Proxy cache servers can come and go at any time; originservers may be here one day but gone the next. Any sys-

tem to improve web consistency and performance canot alter the fundamental independent ownership aoperation model inherent in the Internet. The systemust work with, not against, Internet philosophy. Thprotocol must be able to be run between independenowned and operated caches and origin servers.

The web is a large-scale, widely deployed distributesystem. Such systems are impossible to update in a schronized manner due to their size and complexity, nto mention that servers are independently owned amanaged. The consistency protocol must be an incmental addition to the system. Implementations of cacservers running the new protocol must work with origiservers and caches that do not support it. They shoautomatically discover the protocol capability in thepeers if possible, and use it where possible.

3.2 Internet ScaleA user may access dozens to hundreds of individual ogin servers and hundreds to thousands of objects inday. Web servers may receive requests from thousato millions of users in a day. Proxy servers are ibetween, often serving thousands of users who maccess millions of objects on thousands of origin servethroughout the Internet.

The protocol should reduce network traffic and load oorigin servers; implementations must be able to handcurrent web workloads better than the current protoccan; and should demonstrate the potential of handlifuture workloads better than the current protocol.

3.3 Fault ToleranceAny large-scale, wide-area distributed system muaccommodate component or infrastructure failurbecause failureswill happen. Since any component othe system may fail or disappear from the network, thparties sharing system state must be able to recover frsuch failure. When failures occur there should be minmal disruption to the operation of the non-failing partof the network.

3.4 Web Consistency and PerformanceThe protocol should support stronger consistency thcurrent cache servers can provide and improve user pformance. Objects in cache should be more highly avaable than in today’s caches and should be served mquickly to end users. Communication between a cacand origin server that is synchronous to user requeshould be eliminated where possible.

The protocol should support high service quality, in paticular low delay since the object is close, and low delavariance since the end network is usually more prediable than the wide area Internet.

3.5 Accounting and Content DistributionMost cache solutions mask user requests servedcaches from content providers. When content providewant control for consistency or accounting reasons th

3

The Distributed Object Consistency Protocol DOCP Overview

o-

er-

ro-rsthea-

ted

ed

e.

onird

h

esnt

nt

r’seer

ionb-entnibed

have few options. They can attach HTTP cache controlheaders, such as theExpires header, so user requestswill be sent to the origin server each time. Attachingsuch headers is generally referred to ascache-busting.

To limit cache-busting, caches should assure contentproviders that their content is consistent with what iscurrently published, and should deliver access account-ing information to the provider.

Proxies that can serve objects authoritatively at the edgeof the network must provide accounting informationabout user requests to the content provider. They alsoallow more flexible content distribution and hostingacross wide area networks.

4 DOCP OverviewThe DOCP improves consistency and accountabilitythrough a publish/subscribe mechanism in conjunctionwith soft state and other optimizations. This helps toaccommodate the scale of the Internet and to limit thedamage caused by unanticipated failures or changes innetwork configuration. DOCP is similar to the HTTPcache control mechanism, and allows incremental evo-lution of that mechanism. The DOCP relies on informa-tion push between cooperating proxies to achieve betterperformance when objects change.

The protocol operates between a publisher (master) andsubscriber (slave). Since most web objects are requestedonly once [2], the slave will only attempt to subscribe toan object that has been requested before. If the slave isgranted a subscription it does not have to validate theobject’s freshness with the master, but can serve thatobject directly from cache. The master will send a notifi-cation to the slave if the object changes. If the slave getsanother request for that object it will request a freshcopy of the object (and maybe a renewed subscription)from the master. This provides improved consistency ofcontent; in a simulation of over 40 M user requests not asingle object was served inconsistently.

Objects can be subscribed to only after they have beenre-referenced without an intervening modification. Thislimits subscriptions to uninteresting or very rapidlychanging objects.

Serving objects directly from cache speeds the deliveryof web objects to end users and reduces the demand onorigin servers and networks. In addition to the strongerconsistency, content is served faster and cheaper.

Strong consistency is difficult to achieve in large, widelydistributed systems, due to latency, coordination andobject availability requirements. Instead of strong con-sistency the protocol allowsdelta consistency, whereeach object returned by the cache is consistent to withinsome bounded time, delta, since its last modification.The period of potential inconsistency is roughly thepropagation time of a notification from the server to thecache after an object is modified. Note that this period ison the same order as the propagation time of aGET IMSresponse from a server to a cache, currently used to

check object consistency. Broadly speaking this protcol replaces severalGET IMSrequests with zero or oneinvalidation message (one if there was a change; othwise no communication).

The DOCP defines a new set of HTTP headers to pvide object consistency between content origin serveand edge proxy cache servers. The DOCP distributesability to serve objects authoritatively on behalf ofcontent provider throughout the network. The key components and roles in the DOCP architecture are depicin Figure 1 and described in the sections below.

4.1 Web ObjectsWeb objects are identified using a Uniform ResourcLocator (URL, defined in RFC 1738) and accesseusing HTTP (RFC 2616 [9]), usually with a webbrowser application, which may use a web proxy cachThe DOCP does not change any of these definitions.

4.2 Content ProviderA content provider develops objects to be publishedthe web. A content provider should be able to use thechoice of web publishing and preparation tools, anpublish to any supported web server platform.

A content provider must provide the DOCP service witobject modification notification. This is done via amechanism external to the DOCP. The DOCP makaccess accounting information available to the conteprovider as described in Section 5.9.

4.3 DOCP MasterA DOCP master serves content for one or more conteproviders. It behaves like areverse proxy server: itresponds to HTTP requests for the content provideobjects. If the content is not currently in its cache thDOCP master accesses it from an origin web servaffiliated with the content provider.

The DOCP master serves objects as well as subscriptrequests for the objects. When a content provider pulishes new content a consistency manager compondetects object modifications and delivers invalidatiomessages to the subscribed DOCP slaves, as descrin Section 5.7.

FIGURE 1. DOCP Architecture

ContentProvider

DOCPMaster

DOCPSlave

EndUser

WebServer

HTTP GETUpdate:FTP, HTTP

Accounting:HTTP GET

Access:HTTP GET

Consistency:DOCP Inv

4

The Distributed Object Consistency Protocol DOCP Protocol Operation

tbye

es asP

tent

ct

d-

r antne.

noed

inge

ngrsta-xtvee

asb-d

ldely

i-n-ls

4.4 Web ServerThe DOCP master communicates with content on a webserver. This allows a content provider to select the pub-lishing tools they wish, while improving the content ser-vice. The DOCP master accesses content on the webserver via HTTP, caching a copy of the response. Allnew information published to the web server must passthrough the DOCP master so that it can generate changenotifications to subscribed slaves.

4.5 DOCP SlaveA DOCP slave serves requests for one or more clients. Itbehaves like a regular web proxy cache: it acceptsHTTP GETrequests, checks its cache for a current copyof a requested object. If it has a current copy that isreturned to the client; otherwise the DOCP slave makesan HTTPGETor GET IMSrequest to retrieve or validatethe object. If the object is served by a DOCP master, theslave may also request to subscribe to the object.

DOCP proxies can be arranged in a hierarchy, and maybe a master for some slaves and a slave to others. Themaster and slave terms describe roles in the protocol.

4.6 ClientClients issue HTTP requests to DOCP proxies as usual.The fact that they are using a DOCP enabled cache istransparent to them except for improved consistency andperformance. No modification is required to browsers.

5 DOCP Protocol OperationThis section presents the operation of the DistributedObject Consistency Protocol.

5.1 Protocol OverviewThe DOCP uses a publish/subscribe replication modelfor popular web objects. When an object becomes suffi-ciently popular in a DOCP slave cache the slave willrequest to subscribe to that object with the originserver’s DOCP subscription manager. If the objectchanges during the subscription interval assigned by thesubscription manager, the DOCP master will send anotification of the change to all current subscribers.Until then a slave can serve subscribed objects authori-tatively without contacting the origin server.

Figure 2illustrates the operation of the protocol. Clienrequests for an object arrive from the top to be serveda DOCP slave. On the first request for the object thslave makes aGET request to load the object into itscache. On subsequent client requests the slave makGET IMS Sub request to determine if the object habeen modified and request to subscribe to it. The DOCslave guarantees that every object it serves is consiswith what the content provider has published. TheGETIMS Sub request can return a fresh copy of the obje(and HTTP status200 OK), or a positive validation(HTTP status304 Not Modified ) with no object data.

The slave requests to subscribe to the object by incluing a new HTTPDOCP-Subscribe header with therequest on the second and subsequent requests foobject. It will not request to subscribe to an objecretrieved only once. In earlier workload characterizatio[2] we found 60% of objects were requested only oncThere is no value to subscribing to these objects.

If the object was modified, as in the firstGET Subrequest, a fresh version of the object is returned butsubscription lease. If the object has not been modifisince the prior access, as in the secondGET Subrequest, the master grants a subscription lease. Durthe lease interval (shaded intervals in Figure 2). thslave serves client requests directly without contactithe master. The lease can either expire, as in the filease in the figure, or be terminated through invalidtion, as in the second lease in the figure. On the nerequest after a lease expires or is invalidated, the slamust contact the master. The slave will refresh thobject and request a fresh lease for the object. If it wmodified only once, as depicted in the figure, the suscription request will be granted. If it had been modifieagain before the finalGET Subdepicted in the figure thelease would not have been renewed. A client must hoan unmodified object to receive a lease. This limits thneed for masters to send invalidations for frequentchanging objects.

The following sections describe the subscription decsion and some high level details of the protocol. Appedix A contains further detail about the protocoincluding the syntax of the HTTP protocol extensionand the specific responses from master to slave.

FIGURE 2. DOCP Consistency Protocol Operation

Client requests

Slave

Content updates

Master

GE

T

20

0 O

K

GE

T S

ub

20

0 O

K

GE

T S

ub

30

4/le

ase

GE

T S

ub

30

4/le

ase

DO

CP

In

v

GE

T S

ub

20

0/le

ase

5


wner

terib-d

so-se

sese’ster

hees-

veeru-

eesAta-ewrk

the

thet

ip--esshe

genet

lns-g

,l

5.2 Subscription Decision - SlaveThe request for a subscription is a local decision madeby a DOCP slave. The slave does not need to coordinatewith the DOCP master or any other client to make a sub-scription decision.

The slave may use the object’s local popularity (refer-ence count), size, last modification time and last accesstime to make a decision whether to request to subscribeto the object. To request a subscription, the slave makesan HTTPGETrequest with an extra header field,DOCP-Subscribe .

The slave must include anIf-Modified-Sinceheader indicating the slave’s last known modificationtime of the object. A slave should not request validationwith an IMS request without requesting a subscription,since a subscription is more efficient than polling for theslave and the network.

5.3 Subscription Decision - MasterA subscription request must be acknowledged by theDOCP master in its HTTP reply. The master decideswhether to grant a subscription based upon local policy,which may include an estimate of the object’s globalpopularity, its size, modification history, and the numberof existing subscriptions to that object.

The master’s HTTP reply indicates if the subscriptionwas granted, and if so for how long, using aDOCP-Lease response header. The choices of the lease valueare described in Appendix A, in particular in Table 2,“DOCP-Lease Header Values,” on page 19.

Upon granting a subscription, the DOCP master mustrecord the subscribing slave’s return (notification)address in case the object changes.

5.4 Subscription Decision - Lease IntervalEach subscription is bounded by a lease interval calcu-lated by the master. Each object may have zero or onelease associated with it; all subscriptions to a singleobject share the same lease value (expiration time) at theDOCP master. This prevents the need for the master tomaintain different expiration times for different slaves.

If an object is not modified when its lease expires, themaster simply clears the subscriber list. No communica-tion takes place between master and slaves.

The lease provides a simple, robust method for limitingthe amount of state that must be kept by the master. Itprovides a network-efficient mechanism for subscriberlist clean-up, since no communication is required torelease a lease (no unsubscribe exchange is requiredbetween master and slave). The lease also provides abound on the amount of time a master will attempt todeliver an invalidation to a slave that is unreachable, andtherefore the maximum period of object inconsistency.A lease interval will typically be days to weeks long.

The master should set leases to expire at quiet periods inthe day if possible. This helps to avoid bursts of requests

when object leases expire. At most sites the pre-daperiod between 2-5 AM receives the fewest requests phour. Frequently changing content may require shorlease intervals. Lease expiration times should be distruted (for example randomly) during an interval to avoimay object leases expiring together.

5.5 Leases and Clock SkewClock synchronization in wide area distributed systemhas long been studied [15]. Protocols exist to synchrnize time across the Internet [20][26]. However, clockmay still not be synchronized for many reasons. ThDOCP must be resilient to unsynchronized clocks.

A DOCP master accommodates this by returning a leaas an absolute time in the slave’s time domain as cloas possible to the expiration time in the masterdomain. To support this the slave must send the masits notion of the current time. The master assigns tlease expiration time for every object. The master usthe expiration time and the slave’s current time to compute the lease expiration time at the slave. The slaexpiration time is set to expire at or before the mastlease expiration time. This prevents slaves serving docments that have expired at the master.

5.6 Modification of Subscribed ObjectsIf an object is modified during the lease interval thDOCP master must attempt to notify any DOCP slavcurrently subscribed to the object that it has changed.that time the lease is canceled and no further invalidtion messages will be sent for that object unless a nsubscription is begun. Note that the cost to the netwoand origin server of deliveringn invalidation messagesfor a changed object is approximately the same ascost of serving oneIMS request fromn proxies checkingthe freshness of that object, orn IMS requests from oneproxy.

5.7 Change NotificationWhen a change to a subscribed object is detectedDOCP master will transfer the current subscription listo a notification agent. This transfer resets the subscrtion list for that object so no further invalidation messages will be sent upon subsequent modifications, unla DOCP slave re-subscribes to (the new version of) tobject.

The notification agent attempts to deliver the channotification to each slave indicated on the subscriptiolist. Each notification must be acknowledged by thslave. If delivery fails (is not acknowledged) the agenwill attempt to re-deliver the notification after an initiatimeout interval. The interval between retransmissioshould employ a backoff mechanism if delivery continues to fail. The notification agent must stop attemptinto deliver the notification when the lease expires.

Delivery is accomplished using one of two protocolsunder the DOCP notification agent’s control (it wilattempt both if the first choice fails):

6


eedm-

sti-

r. Ifngesteis-ion

n.d-illldnat

e-lute

to-alisterly

Lwldfored

rsPonerns-etndterrt-

erer

seeA

aeraren

• DOCP/UDP. The DOCP master sends a notificationmessage to the slave’s UDP notification port. Thereis an issue with such UDP traffic traversing firewalls,so other alternatives must also be supported.

• HTTP/TCP. The DOCP master makes an HTTPPOSTrequest to the DOCP slave’s HTTP notificationport. The HTTP channel should be persistent whilethe slave and master are actively communicating; thesame channel may be used for notifications frommaster to slave, acknowledgments from slave tomaster, and subscribe requests from slave to master.

• Remote Procedure Call (RPC). The DOCP mastermakes a direct RPC call to the slave’s notificationport (address). An interface has not been defined forany specific RPC mechanism, but the DOCP shouldsupport direct client/server communication. RPCcan also have issues with traversing firewalls, but hasother benefits such as efficient message encoding,well defined security mechanism, development toolsupport, and procedure call/return semantics.

The slave communicates its notification port (or bindinghandle) to the master prior to the first subscriptionrequest. Each notification message body identifies allmodified objects at that master to which the slave isbelieved to be subscribed. This follows from the obser-vation that many objects can change at or near the sametime, as in the case of a tree update operation by a con-tent publisher. The invalidation indicates the former andcurrent modification time of the changed objects and anobject digital signature (for example a checksum). Eachinvalidation message carries a sequence number to allowthe slave to detect a missing notification.

The slave must acknowledge each notification; it maynegatively acknowledge a notification if it detects a gapin sequence numbers. The slave may send one acknowl-edgment message for multiple notifications.

When the slave receives an invalidation notification itmust annotate the object’s metadata such that it will notserve the object again from its cache. After an invalida-tion the object must not be served to clients, even ifcommunication with the master for a fresh copy fails,since the object may have been removed by the contentprovider. The slave may remove the object data fromdisk although removal does not have to be synchronouswith the request; it can be done at a quiet time or whendisk space is next needed.

The slave could keep object data on disk and use a deltaencoding to update the data when it is next requested, asproposed by Mogul et al [21].

5.8 Optimistic DiscoveryThe web is a large-scale, widely deployed distributedsystem. Such systems are impossible to update in a syn-chronized manner due to their size and complexity, notto mention that the origin and proxy servers in the webare independently owned and managed. ThereforeDOCP technology must be incrementally added to the

system. To accommodate this, DOCP slave servers nto be able to discover DOCP masters. This is accoplished throughoptimistic discovery.

To enable optimistic discovery, a DOCP master muinclude aDOCP-Lease header in each response in addtion to standard HTTP headers (e.g.,Expires ) for theobject. Non-DOCP proxies will ignore the extra headefield and use standard HTTP cache consistency rulesdesired, this can enforce strong consistency by makiproxy cache servers validate consistency every requusing GET IMS requests. DOCP slaves will recognizthat the object should be served with the DOCP constency guarantee, and that they can make subscriptrequests to that server and receive a valid lease.

This approach provides another potential optimizatioMany objects on the web are written only once, but reamany times, for example all the little colored dots, company logos, and so on. Since a read-only object wnever change (by definition), a DOCP master coualways include an “infinite” lease for these objects upoevery access. The “infinite” lease can be interpretedthe slave as “a very long time from now”, for examplthe end of time of the local system’s clock. It is not necessary that all slaves treat this value as the same absotime; any time in the distant future will do. A DOCPslave receiving such an unsolicited lease should aumatically mark the object as subscribed for that intervand never request to validate the object (unless itevicted from the cache and re-referenced). The masdoes not need to maintain a subscriber list for read-onobjects since an invalidation will never be sent.

If such an object needed to change, a new object URmust be created and links updated to point to the neobject. Using this technique, many more objects coube made to be read-only. There is no mechanismdeleting these read-only objects, so they must be uswith care.

5.9 Access LoggingWhen content is served from caches content providedo not see as many hits at their site. The DOCaddresses this by transmitting access log informatifrom DOCP slaves to the DOCP master for each servfor which it served content. These access logs are tramitted periodically and should be transmitted at quitimes (e.g., pre-dawn), as negotiated by the master athe slave. Logs should be made available by the masin one of the accepted standard formats for log repoing, for example the Common Log Format (CLF). ADOCP slave may log all requests by DOCP mastinstead of using a single log file for all requests and latextracting them.

Content providers can use the access information togreater detail of user access patterns on their site.server log today may contain only a single hit fromproxy for the first request of an object. Subsequent usrequests to the proxy result in a cached response andtherefore not seen by the origin server until a validatio

7


a

x-a

t-isa-al-vealto

ofs-

es

or-

es.

nheheme

eb-

an

atToayousttoi-

or

ar-

ing

hil-ndrd-ta

rip-

request (If-Modified-Since ) is made. Furthermore,access by all users of a proxy appears to the originserver as coming from the same source (the IP addressof the proxy server). Therefore log analysis on serversprovides only an approximation of user activity.

By having full proxy logs at the origin server the contentprovider is able to see all user requests that arrived at theproxy and use that information to better understand tra-versal patterns and optimize their site’s layout. A webhosting provider can supply detailed analysis of theaccess patterns at all DOCP slaves as a value added ser-vice to a content provider.

5.10 Predictive Subscription RenewalIf an object is deemed to be extremely popular, a DOCPslave may request a subscription prior to the next userrequest. In general it is hard to predict both the time ofthe next request and the time of the next modification.We have observed that modifications to our own webpages tend to occur in clusters where a single page willreceive several updates in a relatively short time, some-times without intervening reads.

For this reasons, the DOCP uses a passive model thatachieves the best use of the network while providingbetter average retrieval latency than the TTL-basedmodel. However, renewal of a subscription after expira-tion is allowed by the protocol. This is particularly use-ful to support high availability of content at the slave orassurance of fast response time.

Pushing copies of modified objects is also being consid-ered, for example to support dynamic mirror mainte-nance. This is not currently defined by the protocolpending definition of an economic model for push cach-ing in the DOCP.

5.11 Content Provider UpdatesThe mechanism for content provider updates is externalto the DOCP protocol. Various methods may be used tocommunicate changes to the DOCP master.

• A content publishing system can transmit informa-tion about published objects as they are changed.

• An application can scan the content for changes.

• Manual notification can be sent to a DOCP master.

The most effective and efficient mechanism is to inte-grate with the content publishing system. In any case aconsistency manager must check each published object.It computes the object’s modification date and check-sum, compares these with the previous state of theobject, and notifies the subscription manager if theobject differs. The subscription manager determines ifany of the changed objects had current subscriptions andgenerates change notifications for them.

5.12 Hierarchy for ScalabilityIn order to scale to very large networks DOCP proxiescan be configured into a hierarchy. The hierarchy should

reflect network topology with parent proxies servingset of child proxies logically farther from the originservers they are attempting to contact; for example proies within an ISP should route their requests throughregional parent proxy for that ISP.

Many organizations implement security firewalls, creaing few places in the network where the organizationconnected to the external network. Caching at this loction is natural, and satisfies the goal of increasing scability through hierarchy. Each organization should haone or a few DOCP proxies connected to the externnetwork, and possibly internal proxies connectedthose “parent” proxies.

A DOCP parent will serve as a master for some setDOCP child proxies, and as a slave to other DOCP maters. The intermediate parent proxy’s role is

• to aggregate HTTPGETand DOCP subscribe mes-sages for the child slave proxies it serves

• to manage subscription lists on behalf of these slavand the DOCP masters to which they subscribe

• to aggregate access information from slaves and fward logs to parent DOCP masters

• to redistribute DOCP invalidation messages fromexternal masters to the slaves when content chang

If there are many DOCP child proxies interested in aobject, only one subscription needs to be made to torigin’s DOCP master; the others can be served by tDOCP parent. All subscribed slaves share the salease value granted by the master.

Not all DOCP proxies need to maintain a copy of thobject data. A DOCP parent may maintain only a suscription list for its child proxies. The children wouldmanage object data and receive notification whenobject changes, purging it from their caches.

A data-less parent will have knowledge of the slaves thare subscribed to popular objects but not object data.serve a request for a subscribed object the parent mrequest a copy of the data from one of its children; if nchild has a copy of the subscribed object the parent mobtain a copy from the origin. A slave can subscribean object with a DOCP parent without further communcation with the origin. When anIMS request arrives for asubscribed object (whether the parent has the datanot), the parent may respond with an HTTP304 NotModified (if it has not been invalidated) and includeDOCP-Lease header with the lease expiration time fothe object. The request is handled locally, while providing the consistency guarantee.

A DOCP parent also aggregates access accountinformation from its children. It periodically receivesupdates on subscribed objects it has served to its cdren, aggregates them with data from direct access aother children, separates the access information accoing to DOCP master, and forwards the appropriate dato each of the DOCP masters that have granted subsctions to the DOCP parent.

8

The Distributed Object Consistency Protocol HTTP Cache Control

til

n-

rtes.ute.asr-ntthe

for

omg

cyarengstofseurey.

is-b-ntinot

Aa-ontoat-,s-rnse

in-ornrm

al-o, as

6 HTTP Cache ControlThe DOCP makes several HTTP cache control headersunnecessary. In a DOCP slave the HTTP cache controlheaders are subordinate to DOCP headers. Note that aDOCP slave may communicate both with DOCP mas-ters and ordinary web servers and other non-DOCPproxies. A DOCP slave must still implement HTTPcache control headers.

6.1 Pragma: No-CacheThis HTTP/1.0 header can be attached to a clientGETrequest or a server response. It indicates that the proxyshould not serve a copy from the cache but rather a cur-rent copy from the origin server. In practice, some cacheimplementations modify this into aGET IMSrequest sothey do not have to retrieve the whole object if it has notchanged since the last modification time. This is themechanism by which users can check object consistencyif they believe their cache to be out of date.

With DOCP, subscribed objects are served authorita-tively from the DOCP slave cache. Therefore the slavemay respond with a304 Not Modified responsewithout communicating with the DOCP master. If anobject is modified the DOCP slave will be informed andwill serve a consistent object on the next request.

6.2 ExpiresThe Expires header on a returned object includes atimestamp that identifies how long an object may remainin cache before it is validated. Prior to the object’s expi-ration a cache may serve the object without checking itsvalidity. After expiration the cache must request a newversion from the origin server. Some cache servers inter-pret this as allowing a validation with the origin server.

It is difficult for content providers to use this featureeffectively because it is hard to predict when an objectwill be modified. Even for objects with regular publica-tion schedules, there are cases where a retraction is nec-essary to correct factual or other errors. If the contentprovider uses theExpires header to prevent unneces-sary IMS requests, the incorrect object will remain incaches until it expires. An HTTPPURGEmethod hasbeen proposed to address this (as an IETF work inprogress report; no published version exists yet).

This header is also used to prevent a cache from holdingan object too long before checking its consistency. Thisis the de-facto mechanism to assure consistency in theweb today. However, this polling mechanism leads toincreased request traffic at origin servers and responsetime for end users, with little benefit since most objectsrarely change.

With DOCP, this header becomes unnecessary. Insteadof expiring, objects are either modified or deleted, inwhich case a DOCP Inv message is sent to all DOCPslave servers holding a copy of the object. Upon subse-quent requests for the object the slave will retrieve a

fresh copy or get an error if the object was deleted. Unthat time the cache may serve the object directly.

6.3 Cache-Control: HeadersThe HTTP/1.1 specification defines explicit cache cotrol headers, which are handled as follows.

6.3.1 max-stale, min-fresh, min-stale,...

It is not clear whether or how web browsers will suppothese headers, nor how end users will choose the valuThere is little reason for end users to set anything b“always give me the most current version” since thlatency trade-off is difficult to quantify to end usersPractical experience also indicates end users will dolittle as possible, and that they have minimal undestanding of the operation of the web service. Conteproviders and server operators also improperly useexisting headers.

DOCP will ignore these headers the same as it doesPragma: No-Cache : it will serve objects authorita-tively, and await notification of change from the DOCPmaster.

6.3.2 Do-Not-Cache, Private,...

These headers are intended to prevent a cache frholding a copy of an object. The reason for preventincaching of objects is often for accounting or consistenreasons, which DOCP addresses. However thereother reasons to prevent caching of an object includifor intellectual property protection. DOCP caches muhonor these HTTP/1.1 headers and not hold a copyobjects so marked. Content providers should not uthese headers just to obtain hit accounting or to assconsistency since DOCP provides that function alread

7 Protocol Analysis and SimulationDuring the design of DOCP we explored several constency models from per-server to per-object (and per-sutree in between). A per-server model limits the amouof state that must be maintained by proxy and origservers, however it is imprecise: the server does nknow specifically which of its objects are subscribed.per-object model carries the most precise state informtion about subscriptions, but requires the most statethe origin and proxy servers. In the end we chosemaintain consistency on a per-object basis after evaluing the object popularity distribution within a serverand therefore the number of imprecise invalidation mesages that would have been sent. To address conceabout servers being able to maintain all that state, wobserved that currently web proxy cache servers matain per-object state; in fact they require more state fthe TTL value and to perform positive validations thaDOCP requires to manage subscriptions and perfoinvalidations.

Next we wanted to validate the assumptions about scability and efficiency (that this protocol was at least nworse than the current weak consistency mechanism)

9

The Distributed Object Consistency Protocol Protocol Analysis and Simulation

lts.

ponofanassthe

.

ofrs

ingoris-d

,s-a

28e

tes

aks.the

ndbe

ce-stm

nde

well as explore various trade-offs in protocol design,such as the computation of the lease interval and thedetermination of when an object was sufficiently popu-lar. In order to accomplish this we extended a simulatorpreviously used to measure the performance of variousreplacement policies [3] such that it could also simulatethe consistency protocol.

The method we used to analyze the characteristics andperformance of the new protocol is trace-driven simula-tion. This method consists of replaying logs (traces)from real users through a simulator that is as similar tothe protocol and underlying network as is feasible. Wechose trace-driven simulation because Internet traffichas been shown to be difficult to model accurately. Byusing a trace of actual activity we are assured of havinga valid model of the activity, although only at a singlelocation and for a limited time. Fortunately, we hadaccess to a large data set from a busy proxy server usedearlier to perform a workload characterization [2]. Thedata were gathered by logging every request made by apopulation of thousands of home users connected to theweb via cable modem technology over a five-monthperiod (a total of 117 million requests). Using this log,we were able to simulate protocol behavior for a set ofweb objects accessed through a cache by a large groupof users representative of other high-speed home users.

We also built a statistical model of object updates, sincethat information is not present in the logs. We did this bystudying busy web server logs and the existing litera-ture. Based upon this we used a power-law function todistribute updates among objects within a site, and anexponential distribution (Poisson process) to distributethe updates for an object within the simulation period.We simulated two modification levels:regular andheavy. The regular workload corresponds to our bestestimate of object modification profiles for web objects.We are aware that the modification rate varies greatlyamong sites, so we also simulated a heavy modificationworkload, in which objects change much more fre-quently. This provides a worst-case scenario for theDOCP. We used the same access workload (and popu-larity distribution) for both modification rates.

The simulation examined the following characteristicsof the protocols under study.

• Network demand. We measured the number ofrequests by type: misses (cold and capacity), slowhits (where the data was in cache but validation withthe origin server was required), fast hits (the datawas in cache but no communication with the originserver was required), and invalidations.

• Server demand. We measured the number ofinbound object requests and outbound DOCP invali-dation requests between origin servers and proxiesthat were all either DOCP or non-DOCP.

• Protocol performance. We examined performance ofthe DOCP under varying modification levels andlease intervals to optimize the protocol.

The subsections below present a summary of our resuDetails about the simulation can be found in [24].

We have also explored cache response time based ucache consistency [8]. Proxy logs indicate an ordermagnitude improvement in response time whenobject is served directly from cache (as a fast hit)compared with a positive validation. Positive validationare again much faster than misses. This establishesbenefit of cache object consistency on response time

7.1 Network and Server DemandExternal network demand includes the total numberexternal requests made by a proxy to all origin serveand the total number of bytes transferred.

Figure 3 presents the request demand a proxy runnthe DOCP, Alex, or strong consistency protocol, and fa range of cache replacement policies. Strong constency is achieved through polling. This simulation usethenormalmodification workload, as explained in [24]and includes DOCP invalidations and subscription mesages, and Alex validations. The simulation examinedrange of cache sizes (along the x axis) from 1 GB to 1GB. A 128 GB cache was sufficient to hold the entirworking set for the simulation interval (104 GB ofunique content was requested), and therefore simulaan “infinite cache”.

The simulation of the Alex consistency protocol used20% age factor and a max stay in cache of three weeThese are the default consistency parameters used bypopular Squid proxy cache [27].

FIGURE 3. Request Demand

As cache size grows the external request demadecreases because objects in cache do not need totransferred again. For smaller cache sizes the replament policy plays a significant role in the total requedemand. The cache replacement policy is the algoriththat determines which objects remain in cache awhich are evicted to make room for new objects. Wexplored the following replacement policies [1]:

• The classical Least Recently Used (LRU) policy.

24

26

28

30

32

34

36

38

40

42

44

1 2 4 8 16 32 64 128

Serv

er r

eque

sts

(mill

ion)

Cache size (GB)

Alex/STRONGAlex/LRU

Alex/GDSFAlex/LFUDA

DOCP/LRUDOCP/GDSF

DOCP/LFUDA

10


ofnd

nwo

nentenersth

be

seetheoe.:teore

ed,

rkdoIt

ewsthel-

cenotize

o-est

• The Greedy Dual-Size with Frequency (GDSF) pol-icy, which is designed to maximize object hit rate bykeeping more of the popular objects in cache.

• The Least Frequently Used with Dynamic Aging(LFUDA) policy, which is designed to maximizebyte hit rate by keeping more popular bytes in cache.

LFUDA ignores object size in makin its replacement deci-sion. GDSF keeps more objects in cache by evictinglarger objects before smaller ones (assuming they havethe same popularity or reference count).

As cache size grows to the size of the working set(effectively an “infinite” cache), the replacement policydoes not affect the request demand. In an infinite cacheall objects that are cachable will be in cache after thefirst request. At this point only the consistency policyplays a role in determining request demand. The DOCPreduces the number of external requests by approxi-mately 19% as compared with the Alex protocol; and by42% when compared with strong consistency. It doesthis while delivering object consistency equivalent to thestrong policy (polling every time).

At smaller cache sizes, the consistency policy reducesexternal demand for each of the replacement policies byapproximately the same ratio as at larger cache sizes.

Figure 4 presents the total bandwidth demand under thesame configuration of proxies.

FIGURE 4. Bandwidth Demand

The consistency protocol does not play a significant rolein the reduction of bandwidth demand between proxiesand origin servers. Bandwidth demand is dominated bythe cost of transferring object data across the network.The consistency protocol reduces the number of consis-tency validation messages required between proxy andorigin servers, but these are small messages, as are theinvalidation messages that take their place.

In Figure 4 the lines for the Alex protocol with theGDSFand LFUDA policies have been left off for clarity. Theyfall exactly on theDOCP/GDSF and DOCP/LFUDA lines,just as Alex/LRU andDOCP/LRU are overlaid.

7.2 Protocol PerformanceDuring the simulation we explored the performancethe DOCP with various cache replacement policies achoices of lease interval.

To quantify the performance of the DOCP subscriptiomechanism under varying lease durations we added tsensors to our simulator. The sensors measure theinval-idation miss rateandsubscription miss rate. The invali-dation miss rate is the ratio of object invalidatiorequests for an object that the slave’s cache replacempolicy had already removed to total invalidations. Thsubscription miss rate is the ratio of subscriptiorequests at a master for an object the master considthe slave already subscribed to total subscriptions. Boof these types of misses are inefficient and shouldminimized.

During protocol development we explored various leaintervals including static intervals of one day, thredays, and one week; and a lease interval based onlast modification time of an object, which attempted texpire at or near the next predicted modification timPredicting next modification time proved to be difficultthe decision was often wrong, resulting in inapproprialease durations. In the end we chose not to use the mcomplex dynamic lease computation mechanism.

Figure 5 shows the invalidation miss rate for the threlease intervals we explored using the normal workloaa 4 GB cache, and theGDSF replacement policy.

FIGURE 5. Invalidation Miss Rate by Lease

Extending the lease interval does not affect netwobandwidth demand because renewals and validationsnot carry much content relative to object data transfer.only alters the length of time a popular object will bserved during a subscription. Increasing the lease alloa slave to deliver more fast hits, and also increaseschance of eviction of a subscribed object. Sending invaidations increases DOCP master workload, but sinthey occur asynchronously to user requests they doaffect response time. A long lease also increases the sof the master’s subscriber list. We did not study the prtocol’s effect on master sate in this simulation. Thimpact of the lease interval on fast hit rate, reque

180

200

220

240

260

280

300

320

340

1 2 4 8 16 32 64 128

Ban

dwid

th d

eman

d (G

B)

Cache size (GB)

Alex/LRUDOCP/LRU

DOCP/GDSFDOCP/LFUDA

0

5

10

15

20

25

30

35

1 2 4 8 16 32 64 128

Inva

lidat

ion

mis

s (%

of

inva

lidat

ions

)

Cache size (GB)

DOCP/1 dayDOCP/3 days

DOCP/1 week

11


ctren-ger

nt

ltics.d

P

ven

tts.

lso

demand, and bandwidth demand is summarized inTable 1.

7.3 Hit and Miss RatesThe next two figures present the fast hit rate and total hitrate of the two consistency protocols, three replacementpolicies, and six cache sizes we studied. These are allbased upon the normal workload, default Alex consis-tency parameters, and a static three day DOCP lease.

FIGURE 6. Fast Hit Rate

Figure 6 shows an improvement in fast hit rate whenusing the DOCP protocol as compared with the Alexprotocol regardless of cache size. In large caches thereplacement policy is not a factor; the consistency proto-col alone determines the fast hit rate.

FIGURE 7. Hit Rate

The consistency protocol does not significantly affethe overall cache hit rate nor the byte hit rate. These adetermined by the cache replacement policy. The cosistency protocol improves the fast hit rate by allowinthe cache to serve objects directly from cache raththan wait for a consistency validation.

We also explored the impact of the cache replacemepolicy on the subscription and invalidation miss rate.

The following two figures illustrate that for a smalcache the choice of replacement policy has a dramaeffect on the subscription and invalidation miss rateWhen the cache is a large enough to hold all modifieobjects the miss rate drops to zero.

FIGURE 8. Subscription Miss Rate

FIGURE 9. Invalidation Miss Rate

Note the synergy between theGDSF replacement policyand the DOCP consistency protocol. TheGDSF policytries to keep more popular objects in cache. The DOCsubscribes based upon object popularity.

By keeping more popular objects in cache, a 1 GB slausing theGDSFpolicy generates 95% fewer subscriptiomisses masters than when using theLRU policy.

The LFUDA policy also keeps popular objects, but iignores size and therefore keeps fewer, larger objecThis increased the overall miss rate and therefore athe subscription and invalidation miss rate.

Table 1 Lease Impact - 4 GB cache,DOCP/GDSF

Lease 1 day 3 days 1 week

Fast hits 36.78% 41.78% 44.79%

Bandwidth (GB) 283.337 283.354 283.719

Requests (million) 26.696 24.582 23.312

Invalidation miss(% invalidations)

0.38% 1.92% 6.05%

15

20

25

30

35

40

45

1 2 4 8 16 32 64 128

Fast

hit

rate

(pe

rcen

t)

Cache size (GB)

Alex/LRUAlex/GDSF

Alex/LFUDADOCP/LRU

DOCP/GDSFDOCP/LFUDA

30

35

40

45

50

55

60

65

1 2 4 8 16 32 64 128

Hit

rate

(pe

rcen

t)

Cache size (GB)

Alex/LRUDOCP/LRU

DOCP/GDSFDOCP/LFUDA

0

20

40

60

80

100

120

140

1 2 4 8 16 32 64 128

Subs

crip

tion

mis

s (%

of

subs

crip

tions

)

Cache size (GB)

DOCP/LRUDOCP/GDSF

DOCP/LFUDA

0

10

20

30

40

50

60

70

80

90

1 2 4 8 16 32 64 128

Inva

lidat

ion

mis

s (%

of

inva

lidat

ions

)

Cache size (GB)

DOCP/LRUDOCP/GDSF

DOCP/LFUDA

12

The Distributed Object Consistency Protocol Related Work

ll-Inedig-eto

eb

ris-de

ch-es-

rthe].ingdsthchngntsiftost

edent).

asen-

ede.res

is-rstrs

thet.

is-err--

n:

myotxy

7.4 Simulation SummaryIn summary, the simulation of the DOCP in a wide areanetwork showed an overall reduction in connectiondemand while serving zero stale hits. The eliminatedrequests were positive validations for cached objects:instead of a slow hit, where a cache must make a widearea round trip, the DOCP slave achieves a fast hit byserving subscribed objects immediately from its cache.This reduces client response time and server demand.

We simulated the three replacement policies defined in[1] using the DOCP and Alex adaptive TTL consistencyprotocols. The replacement policy was the dominantfactor in bandwidth reduction; the consistency protocolhad little effect on bandwidth demand since the connec-tions eliminated are validations, which only exchangeheader information. The new replacement policies showa significant improvement in bandwidth demand andtotal in hit rate, as reported in the earlier work.

The choice of replacement policy affects the perfor-mance of the DOCP for smaller cache sizes. Frequencybased policies show better synergy with the frequencybased subscription mechanism, and cause less wastedeffort on the part of DOCP masters and slaves.

8 Related WorkThe DOCP is based upon a significant body of work,most notably in the area of scalability, caching and repli-cation in wide area distributed systems. The focus of thework presented here is to bring together a number ofprevious results in order to construct an efficient proto-col for object consistency in the world-wide web.

Neuman [23] presents an overview of scale in distrib-uted systems, motivating the need for scaling on thebasis of improved reliability and performance, andobserving the complexity of multiple administrativedomains in large-scale widely distributed systems. Neu-man discusses distributed naming systems, which areoften used to locate system components. Neuman alsodiscusses replication, caching, and consistency, providesa set of guidelines for designing scalable systems, andexamples of several such systems. Another useful set ofguidelines can be found in Lampson [17].

Kermarrec et al [11] have defined a framework for con-sistent, replicated web objects. Their framework is partof the Globe wide-area distributed systems researchproject. In their paper they define a hierarchy of coher-ence models and a framework to allow clients and storesto negotiate the coherence they wish according to a setof implementation parameters. Their framework is moregeneral than what we propose. No HTTP implementa-tion was proposed as part of this work.

Liu and Cao [19] demonstrated that strong consistencycould be achieved at about the same cost as the currentweak consistency approach implemented by proxycaches. In their paper, every object that changes causedan invalidation message to be sent to the caches thataccessed that object. They demonstrated that with this

model they could support consistency equivalent to poing every time without increased network demand.the proxy workloads we have studied we have observthat most objects are never re-referenced even over snificant durations [2], so sending notifications for thesobjects should be avoided. Their report encouraged usexplore ways to provide stronger consistency in the wmore efficiently and with better scalability.

Krishnamurthy and Wills [13] demonstrated that cleveuse of existing HTTP requests can improve the constency of web objects. However, no guarantees are maof the degree of object consistency; the proposed meanism also may cause a significant number of unnecsary invalidation messages to be exchanged.

Yin et al [29] propose using volume leases to suppoconsistency in large-scale systems. This builds on tearlier notion of leases from Gray and Cheriton [10Volume leases allow a server to make progress updatobjects even if clients (proxies) fail. The server neeonly wait until a relatively short volume lease expirebefore modifying an object. Each proxy must have boa short volume lease and a longer object lease for eaconsistent object. This approach supports true stroconsistency since no object will be served without aactive lease, and no object will be modified withourevoking or waiting for termination of a lease. There istill the possibility of a delayed read or write responsea server can not be contacted. This is fundamentalstrong consistency. Their algorithm also supports a “beeffort” consistency level similar to our notion of deltaconsistency, which allows some content to be servwith a strong consistency guarantee and other contwith best effort (with corresponding update semantics

Version 1.0 of the DOCP uses only object leases and ha relaxed consistency requirement, with a delta betwemodification and eventual consistency. This delta is similar to the normal lag between when an object is creatand when it is made available through a hosting servicWe believe there is an opportunity to integrate ouapproach with theirs to take advantage of volume leasto provide improved consistency.

Sandpiper and Akamai have developed services that dtribute content over the Internet from content provideinto the geographic locality of clients, to reduce clienresponse time and server load. Their work and oushare the same goals; our approach is to improvecore HTTP protocol while they layer a service above i

WebSpective and others also have products to constently replicate content. They focus on a data centunder common administrative control. We allow owneship of proxies and content providers to differ and propose a standard protocol for wide area consistency.

There is some other related work worth a brief mentio

• The IETF hit metering RFC proposed a mechanisfor proxy caches to inform origin servers how manhits they received for cached objects. This has nbeen widely adopted, possibly because the pro

13

The Distributed Object Consistency Protocol Conclusions

er

e-

.

e.

lrk-r-

theo-hen-ess

itor-sohto

etostsheicre,key-ndnt

a)thisvy-

co.

eorere

m-

-ct

caches cannot be audited and therefore the informa-tion is suspect. Our approach is auditable andaccountable, and furthermore log aggregation pro-vides the opportunity to see specific user browsingpatterns in proxy caches.

• The SkyCache service populates its subscribingcaches with data that has been found to be popularelsewhere. The service uses an out-of-band satellitepath. Cache misses from subscribing caches arerelayed to SkyCache Central where a set of rulesdetermine if an object is sufficiently popular to besent “up to the bird.” If so the object is sent up onceand broadcast to all participating caches.

• Multicast and other push technologies have beensuggested, although the asynchronous web modelhas been shown not to match well with the synchro-nous multicast model. Multicast and push may beviable for simultaneous popular media or objects,but the infrastructure has yet to mature.

There are several proposals and much work in progressaimed at improving web performance. Some, like ours,require modification to core infrastructure. This type ofmodification is an expensive, long-term effort. Webelieve that when such modifications are being made,they should yield the broadest possible improvement tothe overall system.

9 ConclusionsWe have designed and simulated a distributed consis-tency protocol for web objects in the current Internetenvironment. We have confirmed earlier research find-ings that stronger consistency can be achieved in theweb at lower cost than weak consistency. Through pro-tocol simulation we have demonstrated a reasonablereduction in origin server load and improvement in pageload times as a result of the elimination of unnecessaryconsistency validation requests in the current system.

Providing consistency along with access accounting caneliminate the need for cache-busting techniques on thepart of content providers. Since objects are served con-sistently with what the provider intended, and sinceaccess information is returned to the provider by theDOCP service, there should be no need for cache-bust-ing. Furthermore, inappropriate use of headers such asExpires and Last-Modified (accidentally, throughignorance, or intentionally) will not cause the DOCPsystem to behave improperly, since only actual modifi-cation will cause object invalidation and subsequent net-work demand.

We believe this service can enable accelerated growth ofcaching and provide for new service opportunities. Forexample, improved consistency can increase confidencein web information and lead to greater business use ofthe web.

As bandwidth continues to grow and bandwidth costfalls we believe the primary benefits proxy cache servers

will provide are to reduce response time and servdemand. This protocol is focused on those two goals.

In summary, we believe the benefits of the protocol prsented here are as follows.

• Better consistency for content providers and users

• Faster response times for popular objects.

• Cheaper for the origin server to serve fewer hits.

• Simpler for everyone by replacing confusing cachcontrol headers with a cleaner, simpler mechanism

9.1 Future WorkWork is currently in progress to integrate this protocodefinition with the volume lease and consistency wodone by Dahlin, Alvisi, and their colleagues at UT Austin. We believe our approaches are complimentary. Vesion 1.1 of the DOCP will incorporate volume leases.

Further research work needs to be done to definesecurity trust model and a combination of security prtocols and services identified and incorporated into tDOCP architecture. We believe this work will focus oprotecting the integrity of content provider objects, protecting the DOCP infrastructure from intentional abusof cached objects, and on providing auditable acceaccounting information back to content providers fromthe edge of the network. In order to accomplish thismay be necessary to define a DOCP certification authity in order for DOCP masters to trust DOCP slavewithout expensive manual configuration. We must alsformally define and describe the mechanism by whicaccess accounting information will be conveyed backthe DOCP master from DOCP slaves.

In addition to providing access to hot static objects ware looking at how to enable consistent accessdynamic and personalized data. Many dynamic requeare data-driven, however the underlying data and tapplications that access it to create the dynamresponse do not change with every request. Therefosome of these requests (such as searches on popularwords, or price or availability quotes) can be cached aeven subscribed, provided an invalidation can be sewhen the data (or the relevant portion of the datchanges. Distributed databases can also addressissue, however some of those mechanisms are heaweight for a web workload.

Multicast may help to reduce the burstiness of traffifrom a DOCP master. Multicast may be well suited tsynchronous distribution of invalidations to slavesFuture work will explore using (reliable) multicast todistribute update notifications. Creating a multicast treis an expensive task relative to retrieval of a web pagesubscription, so the protocol must minimize the numbof multicast trees. We envision a multicast tree could bset up by a DOCP master for the busiest slaves it is comunicating with.

Other future work may include optimization of the subscription and lease mechanism to account for the fa

14

The Distributed Object Consistency Protocol Acknowledgments

lop-s-edeoli-on,

hch-eys,d

yddahture

that objects are distributed together. For example a webpage and all of its inline images can be thought of as asingle “package”. Additional web pages may be logi-cally contained within a single package, such as for ahome page, its images, and the most popular destina-tions off the home page (many of which share a set ofimages). By distributing packages instead of individualobjects, efficiency could be improved. This also wouldhelp reduce the burstiness of change notifications.

In addition to researching multicast, packaging, andother techniques, we are working to characterize serverdemand under a standard, unicast model.

Finally, we would like to pursue an open standards-based approach to the formal definition and adoption ofthis technology. Only through broad adoption will thismechanism deliver its full value in improving webresponse time and reducing network and server load.

10 AcknowledgmentsThis work was motivated by the desire to improve thecurrently inefficient object replication in the web. It is

based upon a substantial body of research and devement in wide area distributed systems, in particular ditributed file systems like xfs and afs. Some of the relatwork is referenced, but other is part of our “collectivconsciousness.” We would like to be the first tacknowledge that we did not invent consistency, invadations, leases, or the Internet. To paraphrase Newtwe are standing on the shoulders of giants...

The authors are specifically grateful to Rich Friedricwho has provided guidance and feedback on the tenology and the business aspects of this work, to GodfrTan who assisted in early protocol design discussionand to John Barton, Nina Bhatti, Mark Nottingham, anCraig Wills, for their helpful review comments.

This version of the protocol was also significantlimproved through discussions with Michael Dahlin anhis research group at UT Austin. Their team contributethe method to handle clock skew in particular, andwide variety of other useful comments - some of whichave been addressed, others will be addressed in a fuversion of this technical report.

15

The Distributed Object Consistency Protocol References

d

y

h

-

.

s

,

f

11 References[1] M. Arlitt, L. Cherkasova, J. Dilley, R. Friedrich, T. Jin,

“Evaluating Content Management Techniques for WebProxy Caches”, in Proceedings of the 2nd Workshop onInternet Server Performance, Atlanta GA, May 1999.

[2] M. Arlitt, R. Friedrich, T. Jin, “WorkloadCharacterization of a Web Proxy in a Cable ModemEnvironment”, in ACM SIGMETRICS PerformanceEvaluation Review, vol 27 no 2, pp25-36, August 1998.

[3] M. Arlitt, R. Friedrich, T. Jin, “Performance Evaluationof Web Proxy Cache Replacement Policies”, to appear inPerformance Evaluation, 1999.

[4] M. Arlitt, T. Jin, “Workload Characterization of the 1998World Cup Web Site”, Technical Report HPL-1999-35,February 1999.

[5] A. Birrell, R. Levin, R. Needham, M. Schroeder.“Grapevine: An exercise in distributed computing.”Communications of the ACM, April 1982.

[6] V. Cate. “Alex -- A Global Filesystem”. In Proceedings ofthe USENIX File System Workshop, pages 1--12, May1992. USENIX Association.

[7] A. Chankhunthod, P. Danzig, C. Neerdaels, M. Schwartz,K. Worrell. “A Hierarchical Internet Object Cache”. InProceedings of the 1996 USENIX Annual TechnicalConference, January 1996.

[8] J. Dilley, “The Effect of Consistency on Cache ResponseLatency”, Technical Report HPL-1999-107, HPLaboratories, September 1999.

[9] R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter,P. Leach, T. Berners-Lee, “Hypertext Transfer Protocol -- HTTP/1.1”, IETF RFC 2616, June 1999. The InternetSociety.

[10] C. Gray, D. Cheriton, “Leases: An efficient fault-tolerantmechanism for distributed file cache consistency”. InProceedings of the 12th ACM Symposium on OperatingSystems Principles, pp 202-210, Dec 1989.

[11] A.M. Kermarrec, I. Kuz, M. van Steen, A. S. Tanenbaum,“A Framework for Consistent, Replicated Web Objects”.In Proceedings of the 18th International Conference onDistributed Computing Systems, May 1998.

[12] M. Korupolu, M. Dahlin. “Coordinated Placement andReplacement for Large-Scale Distributed Caches”, IEEEWorkshop on Internet Applications, pp 62-71, July 1999.

[13] B. Krishnamurthy, C. Wills. “Study of piggyback cachevalidation for proxy caches in the world wide web”,USENIX Symposium on Internet Technology andSystems, pp 1-12, Monterey CA, December 1997.

[14] B. Krishnamurthy, C. Wills. “Proxy Cache Coherencyand Replacement - Towards a More Complete Picture”.In Proceedings of the 19th International Conference on

Distributed Systems, Austin, TX, June 1999.

[15] L. Lamport, “Time, Clocks, and the ordering of events ina distributed system.” Communications of the ACM, v 21no 7, pp 558-565, July 1978.

[16] L. Lamport. “How to make a multiprocessor computerthat correctly executes multiprocess programs”. IEEETransactions on Computers, September 1979.

[17] B. Lampson, “Hints for Computer System Design”, InOperating Systems Review, v 15 nr 5, pp 33-48, Oct1983.

[18] R. J. Lipton, J. S. Sandberg. “PRAM: A Scalable ShareMemory.” Technical Report CS-TR-180-88, PrincetonUniversity, September 1988.

[19] C. Liu, P. Cao, “Maintaining Strong Cache Consistencin the World-Wide Web”, Proceedings of the 17th IEEEInternational Conference on Distributed ComputingSystems, May 1997.

[20] D. Mills, “Simple Network Time Protocol (SNTP)Version 4 for IPv4, IPv6 and OSI”, Internet RFC 2030,October 1996.

[21] J. Mogul, F. Douglis, A. Feldmann, B. Krishnamurthy.“Potential benefits of delta encoding and datacompression for HTTP”. In Proc. ACM SIGCOMM,pages 181-194, September 1997. AT&T Labs ResearcTR 97.22.1.

[22] D. Mosberger. “Memory Consistency Models”.Operating Systems Reviews, 27(1):18--26, Jan. 1993.

[23] B. C. Neuman, “Scale in Distributed Systems”, InReadings in Distributed Computing Systems, IEEEComputer Society Press, 1994.

[24] S. Perret, J. Dilley, M. Arlitt, “Performance Evaluation ofthe Distributed Object Consistency Protocol In a WebProxy Cache”. Technical Report HPL-99-108, HewlettPackard Laboratories, September 1999.

[25] G. Popek, B. Walker, J. Chow, D. Edwards, C. Kline, GRudisin, G. Thiel. “LOCUS: A Network Transparent,High Reliability Distributed System”. In Proceedings ofthe Eights Symposium on Operating Systems Principle(SOSP), pp 169-177, December 1981.

[26] J. Postel, K. Harrenstien, “Time Protocol”, Internet RFC868, May 1983, Internet Society.

[27] Squid Internet Object Cache, http://squid.nlanr.net/

[28] D. B. Terry, A. J. Demers, K. Petersen, M. J. SpreitzerM. M. Theimer, B. B. Welsh. “Session Guarantees forWeakly Consistent Replicated Data”. In Proceedings othe 1994 Symposium on Parallel and DistributedInformation Systems, pp. 140-149, Austin, TX,September 1994.

[29] J. Yin, L. Alvisi, M. Dahlin, C. Lin. “Using Leases toSupport Server-Driven Consistency in Large-ScaleSystems”. IEEE Transactions on Knowledge and Data

16


tberenthe

stheta

tas

r)s-

allyve

a

t itThe

t,

.

emm-he

toe.

Appendix A: Protocol DetailsThis section presents the protocol interactions between aDOCP slave and master using DOCP atop HTTP. Thesyntax of HTTP requests is described in RFC 1945,“Hypertext Transfer Protocol -- HTTP/1.0” and updatedin RFC 2616, “Hypertext Transfer Protocol -- HTTP/1.1”. We do not describe the core HTTP protocol herebut rather its use by and interaction with DOCP

HTTP GETAn HTTPGETrequest is used to retrieve an object whenit is not in cache or when itsLast-Modified date isunknown. It always retrieves object data and should alsoretrieve the modification date for later object validation.

The response to an HTTPGET request is either HTTPstatus200 OK and an object body, a redirection, or anHTTP error. A redirection is a response that refers therequestor to another URL. Most browsers will then fetchthe new object.

DOCP compliant slaves and masters must implementthese requests according to the specification. A DOCPmaster will include an additional header in its response:

DOCP-Lease: Granted 0

This supports optimistic discovery. See section 5.8,Optimistic Discovery on page 7 for a discussion of thismechanism and Table 2, “DOCP-Lease Header Values,”on page 19 for a complete discussion of lease values.

HTTP GET IMSAn HTTP GET If-Modified-Since request is usedto validate an object and retrieve a fresh copy if it hasbeen modified since itsLast-Modified date. A slavecan only use this request type if the modification date isknown.

The response to an HTTPGET IMSrequest is either anHTTP 304 Not Modified indication with no objectdata or an HTTP200 OK response with a new copy ofthe object, which had been modified since theLast-Modified date supplied by the requestor. The200 OKresponse should also include aLast-Modified timefor future validation.

A DOCP master should include theDOCP-Lease:Granted 0 response the same as with aGET response.

DOCP-InvA DOCP Invalidation message is sent from a master toinvalidate one or more subscribed objects on a slave.TheDOCP-Inv message carried over HTTP contains thefollowing headers.

DOCP-Master: Master-IdentDOCP-Host: Master-Host TxnIdDOCP-Inv: URI Last-mod Mod-time

Master-Ident is the identity of the DOCP master asdescribed below.Master-Hostis the host whose objecthas changed (the host part of the URL).URI is the Uni-form Resource Identifier for the changed object.

Last-modis the previous modification time of the objecat the master. When received by the slave this shouldthe last modified time of the object. If this time is earliethan the slave’s notion of the modification time thDOCP-Inv message is ignored; this is a late invalidatiomessage arrival, after the client has already refreshedobject. If theLast-modtime is greater than the slave’snotion of the object modification time the object iinvalidated as usual and the new value marked asobject’s more recent known modification time. Any geor subscribe response from the master must includeMod-timeat or after this time or it will not be honored.This is the case if aDOCP-Inv arrives early.

Mod-timeis the new modification time of the object athe DOCP master. It is used in the lease calculationdescribed in the DOCP-Lease section below.

TxnId is a transaction identifier (sequence numbeassigned by the DOCP master. Each notification mesage sent from a master to a slave has a monotonicincreasing sequence number. This allows a DOCP slato detect that it has missed aDOCP-Inv message andrequest a retransmission.

There may be multipleDOCP-Inv lines for eachDOCP-Host line. There may be multipleDOCP-Host lines foreach DOCP-Master . There may be multipleDOCP-Master header lines sent on a single connection.

A master notification agent must keep invalidations forslave until theTxnId (or a laterTxnId) is acknowledgedby the slave. A slave may request anyTxnId after thelast acknowledgedTxnId

DOCP-Inv-AckThe slave must respond with an acknowledgment thahas received and processed each DOCP-Inv request.slave should not acknowledge aTxnId until it hasreceived and acknowledged all previousTxnIds. Theslave’s response may acknowledge multipleTxnIds. Itmust apply each invalidation immediately upon receipeven if waiting for a missing invalidation.

DOCP-Inv-Ack: TxnId mInv nInv

TxnId is the identifier from the previousDOCP-Inv .

mInv is the number of invalidations made by the slave

nInv is the number of invalidations requested in thInvalidation message. This allows the master to confirthat all invalidations were processed, and also to copute the number of invalidation messages for which tcache no longer had data.

A cache may replace a subscribed object at any timemake room for other objects as per local policy. Thinvalidation miss rate helps to tune the lease duration

17


edds

-vere

ary.

-rs

as

itIf

t it

s-dhe

d

rel toe

ed,

ndanve

l-

DOCP-Inv-NackThe slave may indicate negative acknowledgment for arange of missingTxnId sequence numbers.

DOCP-Inv-Nack: TxnMin TxnMax

TxnMin and TxnMaxare the minimum and maximumsequence numbers that have been seen by the slave. Thesequence numbers between were missed (and notacknowledged) by the slave.

Upon receipt of aDOCP-Inv-Nack the master mustretransmit the notifications betweenTxnMin and Txn-Max to the slave in aDOCP-Inv message. The masternotification agent should cache these messages untilthey have been acknowledged.

DOCP-Subscribe RequestIf a slave learns that a remote master supports the DOCPprotocol through optimistic discovery (DOCP-Lease:Granted 0 in a response), it may request to subscribeto objects from that server. To subscribe a slave includesthe following additional headers in a subsequent HTTPGET IMS Sub request.

GET URI HTTP/1.1If-Modified-Since: IMS-timeDOCP-Subscribe: Slave-Ident Slave-time [Mod-time]

URI is the Uniform Resource Identifier for the request.

IMS-time is the value of the most recentLast-Modi-fied header for that object, as usually used with anIf-Modified-Since request.

Slave-Ident is the identity of the slave.

Slave-timeis the current clock time at the slave in sec-onds and microseconds, expressed assec.usec . Thisallows the master to estimate the slave’s current timeand to assign a lease expiration time in the client’s timeframe.

Mod-time is the value of the most recent modify timereturned by the master in aDOCP-Inv message. It isoptional and can only be included if the object has beenpreviously subscribed and invalidated.Mod-time mustbe less than or equal to IMS-time.

DOCP-Lease ResponseA lease header has the following syntax.

DOCP-Lease: Resp-code Slave-time Lease-time

Resp-codeindicates whether the response was grantor if not why. The response code and lease time fielcan take on the values proposed in Table 2.

Slave-timeis the slave’s time reference from the subscribe request. If desired this may be used by the slato identify the request. If so used the slave must assueach request occurs on a unique microsecond bound

Lease-timeis the expiration time of the lease in the client’s local time frame. It is an absolute time and occuon a second boundary.

The lease expiration time is computed by the masterdescribed below and depicted in Figure 10.

• When the slave receives a request for an objectchecks whether that object can be served locally.not it records the current time,Slave-time(ts), makesa subscribe (GET Sub) request to the master.

• When the master receives the subscribe requesrecordsSlave-time(ts) and records its value of thecurrent time,Master-time (tm).

• If no lease is currently active for the object, the mater calculates the local lease expiration time ansaves that value in object metadata. This is called tMaster-expiration-time, (tx).

• The lease expiration at the slave,Lease-time(tl), iscomputed according to the following equations anreturned to the slave.

∆t = t x - tmtl = ts + ∆t

Note that the lease at the slave will expire at or befothe lease expires at the master, by an amount equapropagation time of the request from the slave to thmaster. Note that the propagation time,tm- ts, can not beaccurately determined unless clocks are synchronizbut this does not matter.

The skew in expiration times between the master aslave may cause a master to send a notification forobject the slave believes is already expired. The slamust acknowledge this invalidation request withmInvequal to zero for that object (i.e., record this as an invaidation miss by the master).

FIGURE 10. Clock Skew and Lease Interval

Master

Slave

ts

tm tx

tl

∆t = tx - tm

tl = ts + ∆t

18


ndatw

heata

ct

at

ndat

at

dn.s

ter

as-

Theesot

lly

not

g asth-

The skew may also cause a slave to renew a lease priorto the master considering the lease to be expired. Toaccommodate this the master must start a second sub-scriber list prior to the expiration of the object. Requestsfor subscription to an object by slaves when there is lessthan a minimum lease interval remaining should be puton the new subscriber list.

Note that the lease response is orthogonal to the HTTPresponse code. ADOCP-Lease header will not accom-pany an HTTP error response unless that error responseshould be cacheable. For example, HTTP error code404 Not Found may be cached and even be subscribedto if it is a sufficiently popular response from a server.

HTTP/1.1 cache control headers may also be included ina DOCP response. These cache control headers aresuperseded by theDOCP-Lease, but may be passedalong to subordinate (non-DOCP) proxies.

Response Calculation and Modifications

The master responds to the request based upon theIMS-time andMod-timecommunicated by the slave and theobject’s true modification time,True-modtimeat themaster using the following rules. Note thatIMS-timemust be less than or equal toMod-timeat the slave ifMod-time is supplied in the request headers.

IMS = Mod-time AND IMS = True-modtime

This object was not modified at the master since the lasttime it was retrieved by the slave. The object lease hasexpired or there was no lease. The subscription requestshould be granted.

The master will respond with the following response.

HTTP: 304 Not ModifiedDOCP-Lease: Granted timestamp

<no object body>

IMS < Mod-time AND Mod-time = True-modtime

This object was previously subscribed at the slave aan invalidation was sent when the object was modifiedMod-time. The slave does not have a copy of the neobject (IMS-time < Mod-time). The object has notchanged at the master since its first modification. Tsubscription request should be granted, but object dneeds to be sent to the slave.

The master will respond with a new copy of the objeand a granted subscription.

HTTP: 200 OKDOCP-Lease: Granted timestamp

<object body included in response>

Mod-time < True-modtime

The relationship betweenIMS-time and Mod-time isirrelevant (although it should always be the case thIMS-time < Mod-time).

This object was previously subscribed at the slave aan invalidation was sent when the object was modifiedMod-time. The object was modified again between thfirst modification and the current request, atTrue-mod-time. The slave does not have a copy of the modifieobject nor does it know the most recent modificatioThis is a “rapidly changing object” relative to the slave’requests; the subscription should not be granted.

Table 2 DOCP-Lease Header Values

Resp-code Lease-time Explanation of Use

Granted 0 No subscription was requested and none is granted. This (unsolicited) DOCP masresponse supports optimistic discovery. When a slave sees a lease value of zero itknows it may request subscriptions from that master. Until it requests a subscriptionthe slave must use polling to validate its freshness (as if an Expires: 0 header wasused).

Granted Lease-time A lease was granted for this object and expires at this clock time at the slave. The mter will deliver a notification if the object changes within that time.

Granted (unsigned 32)-1

or MAXINT

Whether a subscription was requested or not it has been granted and never expires.slave may serve it forever without validating freshness but should still report accessto this object to the master as with any other subscribed object. The master does nneed to maintain a subscription list for such read-only objects.

Was-Modified Mod-time The object was modified since the last-modified time supplied by the slave, specificaatMod-time. The master returns the new object and HTTP code200 OK . This simi-lar to the behavior of a traditional proxy to anIf-Modified-Since request.

Was-Modified 0 The object is dynamically generated and is considered modified each request. Dorequest a subscription for this object.

Use-Parent Parent-addr Used by a master to indicate that a slave should subscribe through a parent, servina DOCP master in the geography of the requestor. This supports peer discovery wiout requiring an explicit location service.

19


tlybe

as-

of

toea

The master will respond with a new copy of the objectand a denied subscription.

HTTP: 200 OKDOCP-Lease: Was-Modified Mod-time

<object body included in response>

The master returns the newMod-time value for theobject. On a subsequent request the slave may begranted a subscription if the object has not been modi-fied again (Mod-time= True-Modtime).

In the case of an object that is modified more frequenthan it is requested, the slave will make a subscrirequest each time it validates the object (an HTTPGETIMS Sub request) and the master will respond withfresh copy of the object and a lease denied with a Wamodified indication. This polling assures consistencyrapidly changing objects.

A Mod-timevalue of zero may be used by the masterindicate that an object is dynamically generated. Thslave may use this information to avoid requestingsubscription on future requests.

20

the distributed object consistency protocol version 1 · the distributed object consistency...

Documents