Na Kika: Secure Service Execution and Compositionin an Open Edge-Side Computing NetworkRobert Grimm, Guy Lichtman, Nikolaos Michalakis,

Amos Elliston, Adam Kravetz, Jonathan Miller, and Sajid RazaNew York University

AbstractMaking the internet’s edge easily extensible fosters col-laboration and innovation on web-based applications, butalso raises the problem of how to secure the executionplatform. This paper presents Na Kika, an edge-sidecomputing network, that addresses this tension betweenextensibility and security; it safely opens the internet’sedge to all content producers and consumers. First, NaKika expresses services as scripts, which are selectedthrough predicates on HTTP messages and composedwith each other into a pipeline of content processingsteps. Second, Na Kika isolates individual scripts fromeach other and, instead of enforcing inflexible a-prioriquotas, limits resource consumption based on overallsystem congestion. Third, Na Kika expresses securitypolicies through the same predicates as regular applica-tion functionality, with the result that policies are as eas-ily extensible as hosted code and that enforcement is anintegral aspect of content processing. Additionally, NaKika leverages a structured overlay network to supportcooperative caching and incremental deployment withlow administrative overhead.

1 IntroductionWeb-based applications increasingly rely on the dynamiccreation and transformation of content [5]. Scaling suchapplications to large and often global audiences requiresplacing them close to clients, at the edge of the inter-net. Edge-side content management provides the CPUpower and network bandwidth necessary to meet theneeds of local clients. As a result, it reduces load on ori-gin servers, bandwidth consumption across the internet,and latency for clients. It also absorbs load spikes, e.g.,the Slashdot effect, for underprovisioned servers. Basedon similar observations, commercial content distributionnetworks (CDNs) already offer edge-side hosting ser-vices. For example, Akamai hosts customer-suppliedJ2EE components on edge-side application servers [1].Furthermore, many ISPs provide value-added services,such as “web accelerators”, by dynamically transformingweb content on the edge. However, commercial CDNsand ISPs have limited reach. To manage the trust nec-essary for exposing their hosting infrastructure to otherpeople’s code, they rely on traditional, contract-basedbusiness relationships. As a result, commercial CDNs

and ISPs are ill-suited to collaborative and community-based development efforts; they best serve as amplifiersof (large) organizations’ web servers.

At the same time, many community-based efforts areexploring the use of web-based collaboration to addresslarge-scale societal and educational problems. For in-stance, researchers at several medical schools, includingNew York University’s, are moving towards web-basededucation [10, 43, 45] to address nationally recognizedproblems in medical education [28, 49]. The basic ideais to organize content along narrative lines to re-establishcontext missing in clinical practice, complement textualpresentation with movies and animations to better illus-trate medical conditions and procedures, and leverageelectronic annotations (post-it notes) and discussions forbuilding a community of students and practitioners. Fur-thermore, such web-based educational environments dy-namically adapt content to meet students’ learning needsand transcode it to enable ubiquitous access, independentof devices and networks. A crucial challenge for theseefforts is how to combine the content and services cre-ated by several groups and organizations into a seamlesslearning environment and then scale that environment tonot only the 67,000 medical students in the U.S., but alsothe 850,000 physicians in the field as well as to medicalpersonnel in other countries facing similar problems.

Taking a cue from peer-to-peer CDNs for static con-tent, such as CoDeeN [47, 48] and Coral [13], Na Kika1

targets cooperative efforts that do not (necessarily) havethe organizational structure or financial resources to con-tract with a commercial CDN or cluster operator andseeks to provide an edge-side computing network that isfully open: Anyone can contribute nodes and bandwidthto Na Kika, host their applications on it, and access con-tent through it. In other words, by opening up the in-ternet’s edge, Na Kika seeks to provide the technologicalbasis for improved collaboration and innovation on large-scale web-based applications. In this paper, we explorehow Na Kika addresses the central challenge raised bysuch an open architecture: how to secure our executionplatform while also making it easily extensible.

Na Kika, similar to other CDNs, mediates all HTTP in-

1Our system is named after the octopus god of the Gilbert Islands,who put his many arms to good use during the great earth constructionproject.

teractions between clients and servers through edge-sideproxies. Also similar to other CDNs, individual edge-side nodes coordinate with each other to cache content,through a structured overlay in our case. Na Kika’s keytechnical difference—and our primary contribution—isthat both hosted applications and security policies areexpressed as scripted event handlers, which are selectedthrough predicates on HTTP messages and composedinto a pipeline of content processing stages. Our archi-tecture builds on the fact that HTTP messages containconsiderable information about clients, servers, and con-tent to expose the same high-level language for express-ing functionality and policies alike—with the result thatpolicies are as easily extensible as hosted code and thatenforcement is an integral aspect of content processing.A second difference and contribution is that Na Kika’sresource controls do not rely on a-priori quotas, whichare too inflexible for an open system hosting arbitraryservices with varying resource requirements. InsteadNa Kika limits resource consumption based on conges-tion: If a node’s resources are overutilized, our archi-tecture first throttles requests proportionally to their re-source consumption and eventually terminates the largestresource consumers.

Our use of scripting and overlay networks providesseveral important benefits. First, scripting provides a uni-form and flexible mechanism for expressing applicationlogic and security policies alike. Second, scripting sim-plifies the task of securing our edge-side computing net-work, as we can more easily control a small executionengine and a small number of carefully selected plat-form libraries than restricting a general-purpose comput-ing platform [20, 41]. Third, scripting facilitates an APIwith low cognitive complexity: Na Kika’s event-basedAPI is not only easy to use but, more importantly, al-ready familiar to programmers versed in web develop-ment. Fourth, the overlay ensures that Na Kika is in-crementally scalable and deployable. In particular, theoverlay supports the addition of nodes with minimal ad-ministrative overhead. It also helps with absorbing loadspikes for individual sites, since one cached copy (of ei-ther static content or service code) is sufficient for avoid-ing origin server accesses.

At the same time, Na Kika does have limitations. No-tably, it is unsuitable for applications that need to processlarge databases, as the databases need to be moved tothe internet’s edge as well. Furthermore, since Na Kikaexposes all functionality as scripts, applications whosecode needs to be secret cannot utilize it (though obfus-cation can help). Next, by utilizing Na Kika, contentproducers gain capacity but also give up control overtheir sites’ performance. We expect that any deploymentof our edge-side computing network is regularly mon-itored to identify persistent overload conditions and to

rectify them by adding more nodes. Finally, while NaKika protects against untrusted application code, it doestrust edge-side nodes to correctly cache data and executescripts. As a result, it is currently limited to deploymentsacross organizations that can be trusted to properly ad-minister local Na Kika nodes. We return to this issue inSection 6.

2 Related WorkDue to its palpable benefits, several projects have beenexploring edge-side content management. A major-ity of these efforts, such as ACDN [33], ColTrES [8],Tuxedo [38], vMatrix [2], and IBM’s WebSphere EdgeServer [17] (which is used by Akamai), explore how tostructure the edge-side hosting environment. Since theyare targeted at closed and trusted deployments, they donot provide an extension model, nor do they include thesecurity and resource controls necessary for hosting un-trusted code. In contrast, the OPES architecture for edge-side services recognizes the need for extensibility andservice composition [4, 23]. While it does not specifyhow composition should be achieved, OPES does definepotential security threats [3]. Their scope and magnitudeis illustrated by experiences with the CoDeeN open con-tent distribution network [48].

Next, Active Cache [9] and SDT [19] enable contentprocessing in proxy caches. While they do not providean extension mechanism, they do provide precise con-trol over edge-side processing through server-specifiedHTTP headers. Furthermore, while SDT enforces onlycoarse-grained resource controls for Perl and none forJava, Active Cache executes Java code with resource lim-its proportional to the size of the content being processed.Unlike these systems, Pai et al.’s proxy API [31] pro-vides fine-grained extensibility for web proxies throughan event-based API akin to ours. At the same time, theirwork focuses on enabling high-performance extensionsin trusted deployments, while our work focuses on con-taining arbitrary extensions in untrusted deployments.Finally, Active Names [46] are explicitly designed forextensibility and service composition, chaining process-ing steps in a manner comparable to Na Kika’s scriptingpipeline. In fact, by introducing a new naming interface,Active Names offer more flexibility for content process-ing than our work. However, they also require a newservice infrastructure, while Na Kika integrates with theexisting web.

While cooperative caching has its limitations [50],coordination between edge-side nodes is still impor-tant for scaling a system, in particular to balance loadand absorb load spikes. To this end, CoDeeN [47],ColTrES [8], and Tuxedo [38] are exploring the use ofdomain-specific topologies and algorithms. In contrast,Na Kika leverages previous work on structured overlay

networks [13, 16, 35, 42, 52] for coordinating betweenlocal caches. We believe that structured overlays providea robust and scalable alternative to domain-specific co-ordination strategies. Additionally, structured overlayshave already been used successfully for caching staticcontent [13, 18].

In most edge-side systems, nodes cannot be entrustedwith the sole copies of application data, and hard staterequiring stronger consistency than the web’s expiration-based guarantees (or lack thereof) must remain on ori-gin servers. In contrast, ACDN [33] reduces access la-tency for such data by replicating it across edge nodesand by providing full serializability through a primaryreplica. Gao et al. [14] explore alternative replicationstrategies by exposing a set of distributed objects thatmake different trade-offs between consistency, perfor-mance, and availability. Alternatively, the continuousconsistency model provides a framework for express-ing such trade-offs through a uniform interface to hardstate [51]. Na Kika’s support for application state buildson Gao et al.’s approach, with the primary difference thatreplicated state is subject to Na Kika’s security and re-source controls.

Web content processing is (obviously) not limited toedge nodes and can be performed on servers and clientsas well. For example, Na Kika has several similaritieswith the cluster-based TACC architecture [12]. Both NaKika and TACC rely on a pipeline of programs that pro-cess web content, and both build on the expiration-basedconsistency model of the web to cache both original andprocessed content. Na Kika differs in that it targets prox-ies distributed across the wide area and and thus needs tocarefully contain hosted code. Comparable to Na Kika,DotSlash [53] helps absorb load spikes by moving scriptexecution to other servers in a “mutual-aid community”.Unlike Na Kika, it has no extension model and does notprovide security and resource controls. At the other end,client side includes [34] (CSI) move the assembly of dy-namic content to the client, which can improve latencyfor clients relying on low bandwidth links. However, dueto their focus on assembling content fragments, CSI arenot suitable for content processing in general. The un-derlying edge side includes [29, 44] (ESI) can easily besupported within Na Kika.

Finally, based on the realization that system securitycan clearly benefit from a dedicated and concise spec-ification of policies, a considerable number of effortshave explored policy specification languages. For ex-ample, domain and type enforcement [7], XACML [22],and trust management systems such as PolicyMaker,KeyNote, and SPKI [6, 11] include languages for ex-pressing and enforcing policies. All these systems re-quire explicitly programmed calls to the respective ref-erence monitor. In contrast, previous work on security

Internet

Origin Server

Client

Client

Client

Na Kika

Na KikaNa Kika

ScriptsOverlayNetwork

HTTPProxyRuntime

Figure 1: Illustration Na Kika’s architecture. Edge-sideproxies mediate all HTTP interactions between clientsand servers by executing scripts; proxies also coordinatewith each other through an overlay network.

for extensible systems advocates the separation of poli-cies, enforcement, and functionality and relies on binaryinterposition to inject access control operations into ex-ecuting code [15, 39]. The WebGuard policy languagerelies on a similar approach for securing web-based ap-plications [40]. Since Na Kika’s programming modelis already based on interposition, we leverage the samepredicate selection mechanism for application logic andpolicies, thus eliminating the need for a separate policyspecification language.

3 Architecture

Like other extensions to the basic web infrastructureand as illustrated in Figure 1, Na Kika relies on prox-ies that mediate HTTP interactions between clients andservers. To utilize these proxies, content producers andconsumers need to change existing web practices alongtwo lines. First, content producers need to publish thenecessary edge-side processing scripts on their web sites.Content producers need not provide scripts for an en-tire site at once. Rather, they can transition to Na Kikapiecemeal, starting with content whose creation or trans-formation exerts the highest resource demands on theirservers. Second, links need to be changed by append-ing “.nakika.net” to a URL’s hostname, so that NaKika’s name servers can redirect clients to (nearby) edgenodes. As described in [13], URLs can be modified bycontent publishers, third parties linking to other sites, aswell as by users. Furthermore, URLs can be rewrittenthrough a service in our architecture. While Na Kika alsosupports static proxy configuration in browsers, we pre-fer URL rewriting as it allows for more fine-grained loadbalancing between edge nodes and presents a uniform,location-independent interface for using our architecture.

3.1 Programming ModelThe functionality of hosted services and applications isspecified through two event handlers, which are writ-ten in JavaScript. Our architecture does not depend onthe choice of scripting language and could support sev-eral languages. We chose JavaScript because it alreadyis widely used by web developers. Additionally, wefound its C-like syntax and prototype-based object modelhelpful in writing scripts quickly and with little code;though we had to add support for byte arrays to avoid un-necessarily copying data. The onRequest event han-dler accepts an HTTP request and returns either a re-quest for continued processing or a response represent-ing the corresponding content or error condition. TheonResponse event handler accepts an HTTP responseand always returns a response. A pair of onRequestand onResponse event handlers mimics the high-levelorganization of any HTTP proxy and represents the unitof composition in Na Kika: a scripting pipeline stage.

In providing two interposition points for HTTP pro-cessing, Na Kika differs from other systems, such asActive Cache [9], SDT [19], and TACC [12], whichonly interpose on HTTP responses. Interposition on re-quests is necessary for HTTP redirection and, more im-portantly, as a first-line defense for enforcing access con-trols. It also is more efficient if responses are createdfrom scratch, as it avoids accessing a resource beforeedge-side processing. To facilitate the secure composi-tion of untrusted services, Na Kika relies on fewer eventhandlers than Pai et al.’s proxy API [31]; though it doesprovide similar expressivity, notably, to control the proxycache, through its platform libraries.

Similar to ASP.NET and JSP, requests and responsesare not passed as explicit arguments and return values,but are represented as global JavaScript objects. Us-ing global objects provides a uniform model for access-ing functionality and data, since native-code libraries,which we call vocabularies, also expose their functional-ity through global JavaScript objects. Na Kika providesvocabularies for managing HTTP messages and state andfor performing common content processing steps. Inparticular, it provides support for accessing URL com-ponents, cookies, and the proxy cache, fetching otherweb resources, managing hard state, processing regu-lar expressions, parsing and transforming XML docu-ments, and transcoding images. We expect to add vo-cabularies for performing cryptographic operations andtranscoding movies as well. Figure 2 illustrates an ex-ample onResponse event handler.

For HTTP responses, the body always represents theentire instance [25] of the HTTP resource, so that the re-source can be correctly transcoded [19]. If the responserepresents an unmodified or partial resource, it is instan-tiated, for example, by retrieving it from the cache, when

onResponse = function() {var buff = null, body = new ByteArray();while (buff = Response.read()) {body.append(buff);

}

var type = ImageTransformer.type(Response.contentType);

var dim = ImageTransformer.dimensions(body, type);

if (dim.x > 176 || dim.y > 208) {var img;if (dim.x/176 > dim.y/208) {img = ImageTransformer.transform(body,

type, "jpeg", 176, dim.y/dim.x*208);} else {img = ImageTransformer.transform(body,

type, "jpeg", dim.x/dim.y*176, 208);}Response.setHeader("Content-Type","image/jpeg");

Response.setHeader("Content-Length",img.length);

Response.write(img);}

}

Figure 2: An example onResponse event handler,which transcodes images to fit onto the 176 by 208 pixelscreen of a Nokia cell phone. It relies on the image trans-former vocabulary to do the actual transcoding. The re-sponse body is accessed in chunks to enable cut-throughrouting; though the transformer vocabulary does not yetsupport it, with the script buffering the entire body.

a script accesses the body.

Event Handler SelectionTo provide script modularity and make individualpipeline stages easily modifiable, stages do not consist ofa fixed pair of event handlers; rather, the particular eventhandlers to be executed for each stage are selected froma collection of event handlers. To facilitate this selec-tion process, pairs of onRequest and onResponseevent handlers are associated with predicates on HTTPrequests, including, for example, the client’s IP addressor the resource’s URL. Conceptually, Na Kika first eval-uates all of a stage’s predicates and then selects the pairwith the closest valid match for execution.

The association between event handlers and predi-cates is expressed in JavaScript by instantiating pol-icy objects. As illustrated in Figure 3, each policyobject has several properties that contain a list of al-lowable values for the corresponding HTTP messagefields. Each policy object also has two properties forthe onRequest and onResponse event handlers andan optional nextStages property for scheduling ad-ditional stages as discussed below. Lists of allowablevalues support prefixes for URLs, CIDR notation for IP

p = new Policy();p.url = [ "med.nyu.edu",

"medschool.pitt.edu" ];p.client = [ "nyu.edu",

"pitt.edu" ];p.onResponse = function() { ... }p.register();

Figure 3: An example policy object. The policy appliesthe onResponse event handler to all content on serversat NYU’s or University of Pittsburgh’s medical schoolsaccessed from within the two universities. The call toregister() activates the policy.

addresses, and regular expressions for arbitrary HTTPheaders. When determining the closest valid match, dif-ferent values in a property’s list are treated as a disjunc-tion, different properties in a policy object are treated asa conjunction, and null properties are treated as truth val-ues. Furthermore, precedence is given to resource URLs,followed by client addresses, then HTTP methods, and fi-nally arbitrary headers. Null event handlers are treated asno-ops for event handler execution, thus making it possi-ble to process only requests or responses or to use a stagesolely for scheduling other stages.

Selecting event handlers by declaring predicates onHTTP messages avoids long sequences of if-else state-ments in a single, top-level event handler, thus resultingin more modular event processing code. When comparedto the additional HTTP headers used by Active Cacheand SDT for selecting edge-side code, predicate-basedscript selection also enables the interposition of codenot specified by the origin server, an essential require-ment for both composing services and enforcing security.While designing Na Kika, we did consider a domain-specific language (DSL) for associating predicates withevent handlers instead of using JavaScript-based policyobjects. While a DSL can be more expressive (for exam-ple, by allowing disjunction between properties), we re-jected this option because it adds too much complexity—both for web developers targeting Na Kika and for imple-mentors of our architecture—while providing little addi-tional benefits. We also considered performing predicateselection on HTTP responses, but believe that pairingevent handlers results in a simpler programming model,with little loss of expressivity. Also matching responsesrequires a very simple change to our implementation.

Scripting Pipeline CompositionBy default, each scripting pipeline has three stages. Thefirst stage provides administrative control over clients’access to our edge-side computing network. It can, forexample, perform rate limiting, redirect requests, or re-ject them altogether. The second stage performs site-specific processing, which typically serves as a surrogate

procedure EXECUTE-PIPELINE(request)forward ← EMPTYbackward ← EMPTY. Start with administrative control and site-specific stagesPUSH(forward , “http://nakika.net/serverwall.js”)PUSH(forward , SITE(request .url) + “/nakika.js”)PUSH(forward , “http://nakika.net/clientwall.js”)repeat . Schedule stages and execute onRequest

script ← FETCH-AND-EXECUTE(POP(forward))policy ← FIND-CLOSEST-MATCH(script , request)PUSH(backward , policy)if policy.onRequest 6= NIL then

response ← RUN(policy.onRequest , request). If handler creates response, reverse directionif response 6= NIL then exit repeat end if

end ifif policy.nextStages 6= NIL then . Add new stages

PREPEND(forward , policy.nextStages)end if

until forward = EMPTYif response = NIL then . Fetch original resource

response ← FETCH(request)end ifrepeat . Execute onResponse

policy ← POP(backward)if policy.onResponse 6= NIL then

RUN(policy.onResponse, response)end if

until backward = EMPTYreturn response

end procedure

Figure 4: Algorithm for executing a pipeline. The algo-rithm interleaves computing a pipeline’s schedule withonRequest event handler execution, so that matchingcan take into an account when an event handler modifiesthe request, notably to redirect it.

for the origin server and actually creates dynamic con-tent. For example, this stage adapts medical content in aweb-based educational environment to a students’ learn-ing needs. The third stage provides administrative con-trol over hosted scripts’ access to web resources. Similarto the first stage, it can redirect or reject requests.

To perform additional processing, each pipeline stagecan dynamically schedule further stages by listing thecorresponding scripts in a policy object’s nextStagesproperty. As shown in Figure 4, the dynamically sched-uled stages are placed directly after the scheduling stagebut before other, already scheduled stages. A site-specific script can thus delay content creation until alater, dynamically scheduled stage, while also schedul-ing additional processing before that stage. Examplesfor such intermediate services include providing anno-tations (electronic post-it notes) for textual content andtranscoding movies for access from mobile devices. Toput it differently, each site can configure its own pipelineand thus has full control over how its content is cre-ated and transformed—within the bounds of Na Kika’sadministrative control. At the same time, new services,

such as visualization of the spread of diseases, can eas-ily be layered on top of existing services, such as geo-graphical mapping, even when the services are providedby different sites: the new service simply adjusts the re-quest, including the URL, and then schedules the originalservice after itself. Both services are executed within asingle pipeline on the same Na Kika node.

The scripts for each stage are named through regu-lar URLs, accessed through regular HTTP, and subjectto regular HTTP caching. As shown in Figure 4, theadministrative control scripts are accessed from well-known locations; though administrators of Na Kikanodes may override these defaults to enforce their own,location-specific security controls. Site-specific scriptsare accessed relative to the server’s domain, in a filenamed nakika.js, which is comparable to the use ofrobots.txt and favicon.ico for controlling webspiders and browser icons, respectively. All other ser-vices, that is, dynamically scheduled pipeline stages, canbe hosted at any web location and are accessed throughtheir respective URLs.

In combining content creation with content transfor-mation, our architecture’s scripting pipeline is reminis-cent of the Apache web server and Java servlets. Atthe same time, both Apache and Java servlets have amore complicated structure. They first process a re-quest through a chain of input filters, then create a re-sponse in a dedicated module (the content handler forApache and the actual servlet for Java servlets), and fi-nally process the response through a chain of output fil-ters. In mirroring an HTTP proxy’s high-level organi-zation, Na Kika’s scripting pipeline stages have a sim-pler interface—requiring only two event handlers—andare also more flexible, as any onRequest event han-dler can generate a response. Furthermore, the contentprocessing pipelines for Apache and Java servlets canonly be configured by code outside the pipelines, whileeach stage in Na Kika’s scripting pipelines can locallyschedule additional stages—with the overall result thatNa Kika is more flexible and more easily extensible, evenin the presence of untrusted code.

Na Kika Pages

While our architecture’s event-based programmingmodel is simple and flexible, a large portion of dynamiccontent on the web is created by markup-based contentmanagement systems, such as PHP, JSP, and ASP.NET.To support web developers versed in these technologies,Na Kika includes an alternative programming model forsite-specific content. Under this model, HTTP resourceswith the nkp extension or text/nkp MIME type aresubject to edge-side processing: all text between the<?nkp start and ?> end tags is treated as JavaScriptand replaced by the output of running that code. These

bmj = "bmj.bmjjournals.com/cgi/reprint";nejm = "content.nejm.org/cgi/reprint";p = new Policy();p.url = [ bmj, nejm ];

p.onRequest = function() {if (! System.isLocal(Request.clientIP)) {Request.terminate(401);

}}p.register();

Figure 5: An example policy that prevents access to thedigital libraries of the BMJ (British Medical Journal) andthe New England Journal of Medicine from clients out-side a Na Kika node’s hosting organization. The 401HTTP error code indicates an unauthorized access.

so-called Na Kika Pages are implemented on top of NaKika’s event-based programming model through a sim-ple, 60 line script. We expect to utilize a similar tech-nique to also support edge side includes [29, 44] (ESI)within the Na Kika architecture.

3.2 Security and Resource ControlsNa Kika’s security and resource controls need to pro-tect (1) the proxies in our edge-side computing networkagainst client-initiated exploits, such as those encoun-tered by CoDeeN [48], (2) the proxies against exploitslaunched by hosted code, and (3) other web serversagainst exploits carried through our architecture. Weaddress these three classes of threats through admissioncontrol by the client-side administrative control stage, re-source controls for hosted code, and emission control bythe server-side administrative control stage, respectively.Of course, it is desirable to drop requests early, beforeresources have been expended [26], and, consequently,requests that are known to cause violations of Na Kika’ssecurity and resource controls should always be rejectedat the client-side administrative control stage.

Because the two administrative control stages mediateall HTTP requests and responses entering and leaving thesystem, they can perform access control based on clientand server names as well as rate limiting based on requestrates and response sizes. The corresponding policies arespecified as regular scripts and can thus leverage the fullexpressivity of Na Kika’s predicate matching. For in-stance, Figure 5 shows a policy object rejecting unau-thorized accesses to digital libraries, which is one typeof exploit encountered by CoDeeN. For more flexibility,security policies can also leverage dynamically sched-uled stages. For example, the two administrative controlstages can delegate content blocking to separate stageswhose code, in turn, is dynamically created by a scriptbased on a blacklist.

To enforce resource controls, a resource manager

procedure CONTROL(resource)priorityq ← EMPTYif IS-CONGESTED(resource) then . Track usage and throttle

for site in ACTIVE-SITES() doUPDATE(site.usage, resource)ENQUEUE(priorityq, site)THROTTLE(site, resource)

end forelse if ¬ IS-RENEWABLE(resource) then . Track usage

for site in ACTIVE-SITES() doUPDATE(site.usage, resource)

end forend ifWAIT(TIMEOUT) . Let throttling take effectif IS-CONGESTED(resource) then

TERMINATE(DEQUEUE(priorityq)) . Kill top offenderelse

UNTHROTTLE(resource) . Restore normal operationend if

end procedure

Figure 6: Algorithm for congestion control. The CON-TROL procedure is periodically executed for each trackedresource. Note that our implementation does not blockbut rather polls to detect timeouts.

tracks CPU, memory, and bandwidth consumption aswell as running time and total bytes transferred for eachsite’s pipelines. It also tracks overall consumption forthe entire node. As shown in Figure 6, if any of these re-sources is overutilized, the resource manager starts throt-tling requests proportionally to a site’s contribution tocongestion and, if congestion persists, terminates thepipelines of the largest contributors. A site’s contribu-tion to congestion captures the portion of resources con-sumed by its pipelines. For renewable resources, i.e.,CPU, memory, and bandwidth, only consumption underoverutilization is included. For nonrenewable resources,i.e., running time and total bytes transferred, all con-sumption is included. In either case, the actual value isthe weighted average of past and present consumptionand is exposed to scripts—thus allowing scripts to adaptto system congestion and recover from past penalization.

To complete resource controls, all pipelines are fullysandboxed. They are isolated from each other, running,for example, with their own heaps, and can only accessselect platform functionality. In particular, all regular op-erating system services, such as processes, files, or sock-ets, are inaccessible. The only resources besides comput-ing power and memory accessible by scripts are the ser-vices provided by Na Kika’s vocabularies (that is, native-code libraries).

We believe that Na Kika’s congestion-based resourcemanagement model is more appropriate for open sys-tems than more conventional quota-based resource con-trols for two reasons. First, open systems such as NaKika have a different usage model than more conven-

tional hosting platforms: they are open to all contentproducers and consumers, with hosting organizations ef-fectively donating their resources to the public. In otherwords, services and applications should be able to con-sume as many resources as they require—as long as theydo not interfere with other services, i.e., cause conges-tion. Second, quota-based resource controls require anadministrative decision as to what resource utilization islegitimate. However, even when quotas are set relative tocontent size [9], it is hard to determine appropriate con-stants, as the resource requirements may vary widely. Wedid consider setting fine-grained quotas through predi-cates on HTTP messages, comparable to how our archi-tecture selects event handlers. However, while predicate-based policy selection is flexible, it also amplifies the ad-ministrative problem of which constants to choose forwhich code.

Our architecture’s utilization of scripting has two ad-vantages for security and resource control when com-pared to other edge-side systems. First, administrativecontrol scripts simplify the development and deploymentof security policy updates. Once a fix to a newly discov-ered exploit or abuse has been implemented, the updatedscripts are simply published on the Na Kika web siteand automatically installed across all nodes when cachedcopies of the old scripts expire. In contrast, CoDeeN andother edge-side systems that hard code security policiesrequire redistribution of the system binaries across allnodes. Though Na Kika still requires binary redistribu-tion to fix security holes in native code. Second, provid-ing assurance that hosted services and applications areeffectively secured is simpler for scripts than for Java ornative code. Our starting point is a bare scripting en-gine to which we selectively add functionality, throughvocabularies, rather than trying to restrict a general pur-pose platform after the fact.

3.3 Hard StateThe web’s expiration-based consistency model forcached state is sufficient to support a range of edge-sideapplications, including content assembly (through, forexample, edge-side includes [29, 44]) or the transcod-ing of multi-media content. However, a complete plat-form for edge-side content management also requiressupport for managing hard state such as edge-side accesslogs and replicated application state. Edge-side loggingprovides accurate usage statistics to content producers,while edge-side replication avoids accessing the originserver for every data item.

Na Kika performs access logging on a per-site basis.Logging is triggered through a site’s script, which speci-fies the URL for posting log updates. Periodically, eachNa Kika node scans its log, collects all entries for eachspecific site, and posts those portions of the log to the

specified URLs.Na Kika’s support for edge-side replication builds on

Gao et al.’s use of distributed objects, which, internally,rely on domain-specific replication strategies to synchro-nize state updates and support both pessimistic and op-timistic replication [14]. Like Gao et al., Na Kika’shard state replication relies on a database for local stor-age and a reliable messaging service for propagating up-dates, which are exposed through vocabularies. UnlikeGao et al., Na Kika’s hard state replication is imple-mented by regular scripts. Updates are accepted by ascript, written to local storage, and then propagated toother nodes through the messaging layer. Upon receiptof a message on another node, a regular script processesthe message and applies the update to that node’s localstorage. As a result, Na Kika provides content produc-ers with considerable flexibility in implementing theirdomain-specific replication strategies. For example, thescript accepting updates can propagate them only to theorigin server to ensure serializability or to all nodes tomaximize availability. Furthermore, the script acceptingmessages can easily implement domain-specific conflictresolution strategies. To secure replicated state, Na Kikapartitions hard state amongst sites and enforces resourceconstraints on persistent storage. Since update process-ing is performed by regular scripts, it already is subjectto Na Kika’s security and resource controls.

3.4 Overlay NetworkThe Na Kika architecture relies on a structured over-lay network for coordinating local caches and for en-abling incremental deployment with low administrativeoverhead. From an architectural viewpoint, the overlayis treated largely as a black box, to be provided by anexisting DHT [13, 16, 35, 42, 52]. This reflects a con-scious design decision on our end and provides us witha test case for whether DHTs can, in fact, serve as ro-bust and scalable building blocks for a global-scale dis-tributed system. Our prototype implementation builds onCoral [13], which is well-suited to the needs of our archi-tecture, as Coral explicitly targets soft state and includesoptional support for DNS redirection to local nodes. Aswe deploy Na Kika, we expect to revisit the functional-ity provided by the DHT. Notably, load balancing, whichis currently provided at the DNS level, can likely benefitfrom application-specific knowledge, such as the num-ber of concurrent HTTP exchanges being processed by anode’s scripting pipelines.

3.5 SummaryThe Na Kika architecture leverages scripting and overlaynetworks to provide an open edge-side computing net-work. First, Na Kika exposes a programming model al-ready familiar to web developers by organizing hosted

services and applications into a pipeline of scripted eventhandlers that process HTTP requests and responses. Sec-ond, it provides a secure execution platform by medi-ating all HTTP processing under administrative control,by isolating scripts from each other, and by limiting re-source utilization based on overall system congestion.Third, it provides extensibility by dynamically schedul-ing event handlers within a pipeline stage as well as ad-ditional pipeline stages through predicate matching. Fi-nally, it provides scalability by organizing all nodes intoan automatically configured overlay network, which sup-ports the redirection of clients to (nearby) edge nodes andthe addition of new nodes with low administrative over-head.

At the same time, web integration is not entirely com-plete, as URLs need to be rewritten for Na Kika access.As already discussed, URLs can be automatically rewrit-ten by web browsers, hosted code, as well as serversand, consequently, the need for manual rewriting willdecrease over time. Furthermore, while our architec-ture protects against client- and script-initiated exploits,it does not currently protect against misbehaving edge-side nodes. In particular, nodes can arbitrarily modifycached content, which is especially problematic for ad-ministrative control scripts. We return to the issue of con-tent integrity in Section 6.

4 ImplementationOur prototype implementation of Na Kika builds on threeopen source packages: the Apache 2.0 web server, theMozilla project’s SpiderMonkey JavaScript engine [27],and the Coral distributed hashtable [13]. We choseApache for HTTP processing because it represents a ma-ture and cross-platform web server. Similarly, Spider-Monkey is a mature and cross-platform implementationof JavaScript and used across the Mozilla project’s webbrowsers. Additionally, our prototype includes a prelim-inary implementation of hard state replication, which re-lies on the Java-based JORAM messaging service [30]and exposes a vocabulary for managing user registra-tions, as required by the SPECweb99 benchmark. Ourimplementation adds approximately 23,000 lines of Ccode to the 263,000 lines of code in Apache, the 123,000lines in SpiderMonkey, and the 60,000 lines in Coral.The majority of changes is to Apache and mostly con-tained in Apache modules. Our modified Apache binary,including dynamically loaded libraries, is 10.6 MBytelarge and the Coral DHT server is 13 MByte.

As already mentioned in Section 3.1, Apache struc-tures HTTP processing into a chain of input filters thatoperate on requests, followed by a content handler thatgenerates responses, followed by a chain of output fil-ters that operate on responses. Our prototype imple-ments the scripting pipeline by breaking each stage

into a pair of input and output filters, which executethe onRequest and onResponse event handlers, re-spectively, and by dynamically inserting the pair intoApache’s filter chain. The content handler is a modi-fied version of Apache’s mod proxy, which implementsthe proxy cache and, in our version, also interfaces withthe DHT. If an onRequest event handler generates anHTTP response, our implementation sets a flag that pre-vents the execution of scripts in later pipeline stages andof the proxy caching code, while still conforming withApache’s sequencing of input filters, content handler,and output filters.

To provide isolation, our implementation executeseach pipeline in its own process and each script, in turn,in its own user-level thread and with its own scriptingcontext, including heap. Scripting contexts are reused toamortize the overhead of context creation across severalevent handler executions; this is safe because JavaScriptprograms cannot forge pointers and the heap is automat-ically garbage collected. A separate monitoring processtracks each pipeline’s CPU, memory, and network con-sumption and periodically executes the congestion con-trol algorithm in Figure 6. To throttle a site’s pipelines,the monitoring process sets a flag in shared memory,which causes the regular Apache processes to reject re-quests for that site’s content with a server busy error. Toterminate a site’s pipelines, the monitoring process killsthe corresponding Apache processes, thus putting an im-mediate stop to processing even if a pipeline is executinga vocabulary’s native code.

Employing per-script user-level threads also helps in-tegrate script execution with Apache, while still exposinga simple programming model. In particular, Apache’s se-quence of input filters, content handler, and output filtersis not necessarily invoked on complete HTTP requestsand responses. Rather, each filter is invoked on chunks ofdata, the so-called bucket brigades, as that data becomesavailable. As a result, Apache may interleave the exe-cution of several onRequest and onResponse eventhandlers. Per-script user-level threads hide this piece-meal HTTP processing from script developers, provid-ing the illusion of scripts running to completion beforeinvoking the next stage. To avoid copying data betweenApache and the scripting engine, our implementationadds byte arrays as a new core data type to SpiderMon-key. Whenever possible, these byte arrays directly refer-ence the corresponding bucket brigade buffers.

The policy matching code trades off space for dynamicpredicate evaluation performance. While loading a scriptand registering policy objects, the matcher builds a deci-sion tree for that pipeline stage, with nodes in the treerepresenting choices. Starting from the root of the tree,the nodes represent the components of a resource URL’sserver name, the port, the components of the path, the

Name DescriptionProxy A regular Apache proxy.DHT The proxy with an integrated DHT.Admin A Na Kika node evaluating one matching pred-

icate and executing empty event handlers foreach of the two administrative control stages.

Pred-n The Admin configuration plus another stageevaluating predicates for n policy objects, withno matches.

Match-1 The Admin configuration plus another stageevaluating one matching predicate and execut-ing the corresponding, empty event handlers.

Table 1: The different micro-benchmark configurations.

components of the client address, the HTTP methods,and, finally, individual headers. If a property of a pol-icy object does not contain any values, the correspondingnodes are skipped. Furthermore, if a property containsmultiple values, nodes are added along multiple paths.When all properties have been added to the decision tree,the event handlers are added to the current nodes, oncefor each path. With the decision tree in place, dynamicpredicate evaluation simply is a depth-first search acrossthe tree for the node closest to the leaves that also ref-erences an appropriate event handler. Decision trees arecached in a dedicated in-memory cache. The implemen-tation also caches the fact that a site does not publisha policy script, thus avoiding repeated checks for thenakika.js resource.

5 EvaluationTo evaluate Na Kika, we performed a set of local micro-benchmarks and a set of end-to-end experiments, whichinclude wide area experiments on the PlanetLab dis-tributed testbed [32]. The micro-benchmarks charac-terize (1) the overhead introduced by Na Kika’s DHTand scripting pipeline and (2) the effectiveness of ourcongestion-based resource controls. The end-to-end ex-periments characterize the performance and scalability ofa real-world application and of a modified SPECweb99benchmark. We also implemented three new servicesto characterize the extensibility of our edge-side com-puting network. In summary, our experimental resultsshow that, even though the scripting pipeline introducesnoticeable overheads, Na Kika is an effective substrateboth for scaling web-based applications and for extend-ing them with new functionality.

5.1 Micro-BenchmarksTo characterize the overheads introduced by Na Kika’sDHT and scripting pipeline, we compare the perfor-mance of a Na Kika node with a regular Apache proxycache for accessing a single, static 2,096 byte doc-ument representing Google’s home page (without in-

Configuration Cold Cache Warm CacheProxy 3 1DHT 5 1Admin 16 2Pred-0 19 2Pred-1 20 2Match-1 21 2Pred-10 22 2Pred-50 30 2Pred-100 41 2

Table 2: Latency in milliseconds for accessing a staticpage under the different configurations.

line images). Since static resources are already well-served by existing proxy caches and CDNs, these micro-benchmarks represent a worst-case usage scenario for NaKika. After all, any time spent in the DHT or in thescripting pipeline adds unnecessary overhead. For allexperiments, we measured the total time of a client ac-cessing the static web page through a proxy—with client,proxy, and server being located on the same, switched100 Mbit ethernet. The proxy runs on a Linux PC with a2.8 GHz Intel Pentium 4 and 1 GB of RAM.

We performed 18 experiments, representing 9 differ-ent configurations under both a cold and a warm proxycache. The different configurations are summarized inTable 1 and determine the overhead of DHT integration,baseline administrative control, predicate matching, andevent handler invocation, respectively. For the cold cachecase of the Admin, Pred-n, and Match-1 configurations,the administrative control and site-specific scripts arefetched from the local server and evaluated to producethe corresponding decision tree. For the warm cachecase, the cached decision tree is used. Resource controlis disabled for these experiments.

Table 2 shows the latency in milliseconds for the 18different experiments. Each number is the average of 10individual measurements. Overall, the results clearly il-lustrate the basic cost of utilizing Na Kika: its script-ing pipeline. For the Pred-n and Match-1 configura-tions under a cold cache, loading the actual page takes2.9 ms and loading the script takes between 2.5 ms and5.6 ms, depending on size. Additionally, the creationof a scripting context takes 1.5 ms. Finally, parsingand executing the script file takes between 0.08 ms and17.8 ms, again, depending on size. However, the resultsalso illustrate that our implementation’s use of caching—for resources, scripting contexts, and decision trees—iseffective. Retrieving a resource from Apache’s cachetakes 1.1 ms and retrieving a decision tree from the in-memory cache takes 4 µs. Re-using a scripting con-text takes 3 µs. Finally, predicate evaluation takes lessthan 38 µs for all configurations. However, these op-erations also result in a higher CPU load: the Na Kika

node reaches capacity with 30 load-generating clientsat 294 requests/second (rps) under Match-1, while theplain Apache proxy reaches capacity with 90 clients at603 rps on the same hardware. Since both resources andscripts only need to be accessed when reaching their ex-piration times, we expect that real world performance iscloser to warm cache than cold cache results. Further-more, most web resources are considerably larger thanGoogle’s home page, so that network transfer times willdominate scripting pipeline latency.

Resource ControlsTo characterize the effectiveness of Na Kika’scongestion-based resource management, we com-pare the performance of a Na Kika node with andwithout resource controls under high load, such as thatcaused by a flash crowd. For these experiments, the NaKika proxy runs on the same Linux PC as before. Load isgenerated by accessing the same 2,096 byte page underthe Match-1 configuration in a tight loop. With 30 loadgenerators (i.e., at the proxy’s capacity), we measure294 rps without and 396 rps with resource controls. With90 load generators (i.e., under overload), we measure229 rps without and 356 rps with resource controls. Ifwe also add one instance of a misbehaving script, whichconsumes all available memory by repeatedly doublinga string, the throughput with 30 load generators drops to47 rps without but only 382 rps with resource controls.For all experiments, the runs with resource controlsreject less than 0.55% of all offered requests due tothrottling and drop less than 0.08% due to termination,including the one triggering the misbehaving script.These results illustrate the benefits of Na Kika’s resourcecontrols. Even though resource management is reactive,throttling is effective at ensuring that admitted requestshave sufficient resources to run to completion, andtermination is effective at isolating the regular load fromthe misbehaving one.

5.2 Web-based Medical EducationTo evaluate a real-world application running on Na Kika,we compare the Surgical Interactive Multimedia Mod-ules [43] (SIMMs) in their original single-server con-figuration with an initial port to our edge-side comput-ing network. The SIMMs are a web-based educationalenvironment that is being developed by NYU’s medi-cal school. Each SIMM focuses on a particular medi-cal condition and covers the complete workup of a pa-tient from presentation to treatment to follow-up. It con-sists of rich-media enhanced lectures, annotated imagingstudies, pathology data, and animated and real-life sur-gical footage—comprising around 1 GB of multimediacontent per module. The five existing SIMMs alreadyare an integral part of the curriculum at NYU’s medicalschool and are also used at four other medical schools in

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 5 10 15 20 25 30 35 40 45 50 55 60Total time (sec)

120 Clients Warm180 Clients Warm240 Clients Warm120 Clients Cold180 Clients Cold240 Clients Cold120 Clients180 Clients240 Clients

Figure 7: Cumulative distribution function (CDF) for la-tency to access HTML content in the SIMMs’ single-server and Na Kika cold and warm cache configurations.

the U.S. and Australia, with more institutions to follow.The SIMMs rely heavily on personalized and multimediacontent but do not contain any real patient data (with itscorrespondent privacy requirements), thus making themgood candidates for deployment on Na Kika.

The SIMMs are implemented on top of Apache Tom-cat 5.0 and MySQL 4.1. They utilize JSP and Javaservlets to customize content for each student as wellas to track her progress through the material and the re-sults of sectional assessments. To facilitate future in-terface changes as well as different user interfaces, cus-tomized content is represented as XML and, before be-ing returned to the client, rendered as HTML by an XSLstylesheet (which is the same for all students)2. The ini-tial Na Kika port off-loads the distribution of multimediacontent, since it is large, and the (generic) rendering ofXML to HTML, since it is processor intensive, to ouredge-side computing network. Content personalizationis still performed by the central server; we expect that fu-ture versions will also move personalization to the edge.

The port was performed by one of the main developersof the SIMMs and took two days. The developer spentfour hours on the actual port—which entailed changingURLs to utilize Na Kika, making XML and XSL con-tent accessible over the web, and switching from cookiesto URL-based session identifiers as well as from HTTPPOSTs to GETs—and the rest of the two days debug-ging the port. In fact, the main impediment to a fasterport was the relative lack of debugging tools for our pro-totype implementation. The port adds 65 lines of code tothe existing code base of 1,900 lines, changes 25 lines,and removes 40 lines. The new nakika.js policy con-sists of 100 lines of JavaScript code.

To evaluate end-to-end performance, we compare the

2An earlier version relied on a custom-built Macromedia Directorclient for rendering XML. It was abandoned in favor of regular webbrowsers due to the extra effort of maintaining a dedicated client.

single-server version with the Na Kika port accessedthrough a single, local proxy—which lets us comparebaseline performance—and with the Na Kika port run-ning on proxies distributed across the wide area—whichlets us compare scalability. For all experiments, we mea-sure the total time to access HTML content—which rep-resents client-perceived latency—and the average band-width when accessing multimedia files—which deter-mines whether playback is uninterrupted. Load is gen-erated by replaying access logs for the SIMMs collectedby NYU’s medical school; log replay is accelerated 4×to produce noticeable activity. For the local experiments,we rely on four load-generating nodes. For the wide-area experiments, 12 load-generating PlanetLab nodesare distributed across the U.S. East Coast, West Coast,and Asia—thus simulating a geographically diverse stu-dent population—and, for Na Kika, matched with nearbyproxy nodes. For Na Kika, we direct clients to randomlychosen, but close-by proxies from a preconfigured listof node locations. For the local experiments, the originserver is the same PC as used in Section 5.1; for the wide-area experiments, it is a PlanetLab node in New York.

The local experiments show that, under a cold cacheand heavy load, the performance of the single Na Kikaproxy trails that of the single server. Notably, for 160clients (i.e., 40 instances of the log replay program run-ning on each of 4 machines), the 90th percentile la-tency for accessing HTML content is 904 ms for thesingle server and 964 ms for the Na Kika proxy. Thefraction of accesses to multimedia content consistentlyseeing a bandwidth of at least 140 Kbps—the SIMMs’video bitrate—is 100% for both configurations. How-ever, when adding an artificial network delay of 80 msand bandwidth cap of 8 Mbps between the server onone side and the proxy and clients on the other side(to simulate a wide-area network), the single Na Kikaproxy already outperforms the single server, illustratingthe advantages of placing proxies close to clients. For160 clients, the 90th percentile latency for HTML con-tent is 8.88 s for the single server and 1.21 s for theNa Kika proxy. Furthermore, only 26.2% of clients seesufficient bandwidth for accessing video content for thesingle server, while 99.9% do for the Na Kika proxy.As illustrated in Figure 7, the advantages of our edge-side computing network become more pronounced forthe wide-area experiments. For 240 clients (i.e., 20 pro-grams running on each of 12 machines), the 90th per-centile latency for accessing HTML content is 60.1 s forthe single server, 31.6 s for Na Kika with a cold cache,and 9.7 s with a warm cache. For the single server, thefraction of clients seeing sufficient video bandwidth is0% and the video failure rate is 60.0%. For Na Kika witha cold cache, the fraction is 11.5% and the failure rateis 5.6%. With a warm cache, the fraction is 80.3% and

the failure rate is 1.9%. For Na Kika, accesses to mul-timedia content benefit to a greater extent from a warmcache than accesses to HTML, since PlanetLab limits thebandwidth available to each hosted project.

5.3 Hard State ReplicationTo further evaluate end-to-end performance in the widearea, we compare the performance of a single ApachePHP server and the same server supported by NaKika running a modified version of the SPECweb99benchmark. For this experiment, we re-implementedSPECweb99’s server-side scripts in PHP and Na KikaPages. The single-server version relies on PHP be-cause it is the most popular add-on for creating dynamiccontent to the most popular web server [36, 37]. TheNa Kika version relies on replicated hard state to man-age SPECweb99’s user registrations and profiles. Withclients and five Na Kika nodes on the U.S. West Coastand the server located on the East Coast, 80% dynamicrequests, 160 simultaneous connections, and a runtimeof 20 minutes, the PHP server has a mean response timeof 13.7 s and a throughput of 10.8 rps. With a cold cache,the Na Kika version has a response time of 4.3 s anda throughput of 34.3 rps. Additional experiments showthat the results are very sensitive to PlanetLab CPU load,thus indicating that Na Kika’s main benefit for these ex-periments is the additional CPU capacity under heavyload (and, conversely, that Na Kika requires ample CPUresources to be effective). Our SPECweb99 compliancescore is 0 due to the limited bandwidth available betweenPlanetLab nodes. Nonetheless, this benchmark showsthat Na Kika can effectively scale a complex workloadthat includes static content, dynamic content, and dis-tributed updates.

5.4 ExtensionsTo evaluate Na Kika’s extensibility, we implementedthree extensions in addition to the Na Kika Pages ex-tension discussed in Section 3.1: electronic annotationsfor the SIMMs, image transcoding for small devices, andcontent blocking based on blacklists. As described be-low, our experiences with these extensions confirm thatNa Kika is, in fact, easily extensible. In particular, theyillustrate the utility of predicate-based event handler se-lection and dynamically scheduled pipeline stages. Fur-thermore, they illustrate that developers can build usefulextensions quickly, even if they are not familiar with NaKika or JavaScript programming.

Our first extension adds electronic annotations, i.e.,post-it notes, to the SIMMs, thus providing another layerof personalization to this web-based educational environ-ment. The extended SIMMs are hosted by a site outsideNYU’s medical school and utilize dynamically sched-uled pipeline stages to incorporate the Na Kika version

of the SIMMs. The new functionality supports elec-tronic annotations by injecting the corresponding dy-namic HTML into the SIMMs’ HTML content. It alsorewrites request URLs to refer to the original contentand URLs embedded in HTML to refer to itself, thus in-terposing itself onto the SIMMs. The resulting pipelinehas three non-administrative stages, one each for URLrewriting, annotations, and the SIMMs. The annotationsthemselves are stored on the site hosting the extendedversion. This extension took one developer 5 hours towrite and debug and comprises 50 lines of code; it lever-ages a previously developed implementation of elec-tronic annotations, which comprises 180 lines of code.

In contrast to the extension for electronic annotations,which represents one site building on another site’s ser-vice, our second extension represents a service to be pub-lished on the web for use by the larger community. Thisextension scales images to fit on the screen of a Nokiacell phone and generalizes the onResponse event han-dler shown in Figure 2 to cache transformed content. Theextension can easily be modified to support other typesand brands of small devices by (1) parameterizing theevent handler’s screen size and (2) adding new policyobjects that match other devices’ User-Agent HTTPheaders. This extension took a novice JavaScript devel-oper less than two hours to write and debug and com-prises 80 lines of code.

Our third extension does not provide new functional-ity, but rather extends Na Kika’s security policy with theability to block sites based on blacklists. Its intendeduse is to deny access to illegal content through Na Kika.The extension is implemented through two dynamicallyscheduled pipeline stages. The first new stage relies on astatic script to dynamically generate the JavaScript codefor the second new stage, which, in turn, blocks accessto the URLs appearing on the blacklist. The static scriptreads the blacklist from a preconfigured URL and thengenerates a policy object for each URL appearing on thatblacklist. The onRequest event handler for all pol-icy objects is the same handler, denying access as illus-trated in Figure 5. This extension took 4.5 hours to writeand debug, with an additional 1.5 hours for setting up atestbed. Since this extension represents the developer’sfirst Na Kika as well as JavaScript code, the 4.5 hoursinclude one hour mostly spent familiarizing himself withJavaScript. The extension comprises 70 lines of code.

6 Discussion and Future WorkAs presented in this paper, Na Kika assumes that edge-side nodes are trusted, which effectively limits the or-ganizations participating in a deployment. To allow NaKika to scale to a larger number of edge networks andnodes, we are currently working towards eliminatingthis requirement by automatically ensuring the integrity

of content served by our edge-side computing network.Content integrity is important for producers and con-sumers so that, for example, the results of medical stud-ies cannot be falsified. It also is important for the op-eration of the network itself, as scripts, including thoseused for administrative control, are accessed through andcached within Na Kika.

For original content, protecting against inadver-tent or malicious modification reduces to detectingsuch changes and then retrieving the authoritative ver-sion from the origin server. However, using cryp-tographic hashes, for example, through self-certifyingpathnames [24] as suggested in [13], is insufficient, asthey cannot ensure freshness. To provide both integrityand freshness, we have already implemented an alterna-tive solution that integrates with HTTP’s cache controlby adding two new headers to HTTP responses. TheX-Content-SHA256 header specifies a cryptographichash of the content for integrity and, to reduce load, canbe precomputed. The X-Signature header specifiesa signature over the content hash and the cache controlheaders for freshness. Our solution requires the use ofabsolute cache expiration times instead of the relativetimes introduced in HTTP/1.1 [21] as nodes cannot betrusted to correctly decrement relative times.

For processed or generated content, content integritycannot be established through hashes and signaturesalone, as content processing is performed by potentiallyuntrusted nodes. Instead, we are exploring a probabilisticverification model. Under this model, a trusted registrymaintains Na Kika membership. To detect misbehav-ing nodes, clients forward a fraction of content receivedfrom Na Kika proxies to other proxies, which then re-peat any processing themselves. If the two versions donot match, the original proxy is reported to the registry,which uses this information to evict misbehaving nodesfrom the edge-side computing network.

7 ConclusionsEdge-side content management reduces load on originservers, bandwidth consumption across the internet, andlatency for clients. It also absorbs load spikes for under-provisioned servers. To make these benefits available toall content producers and consumers and thus to fostercollaboration and innovation on web-based applications,Na Kika provides an open architecture for edge-side con-tent creation, transformation, and caching.

Services and applications hosted by Na Kika are ex-pressed through scripted event handlers. Event handlersare selected through predicates on HTTP messages andare composed into a pipeline that combines administra-tive control and site-specific processing. The resultingprogramming model is not only familiar to web devel-opers versed in client-side scripting and the content pro-

cessing pipelines of Apache and Java servlets, but it alsois more secure and more easily extensible. To providesecurity, Na Kika’s scripting pipeline mediates all re-quests and responses passing through the system. Fur-thermore, all hosted services and applications are iso-lated from each other and the underlying operating sys-tem and subject to congestion-based resource manage-ment: hosted code can consume resources without re-striction as long as it does not cause overutilization. Toprovide incremental scalability, all Na Kika nodes are or-ganized into a structured overlay network, which enablesDNS redirection of clients to nearby nodes and coop-erative caching of both original and processed content.The experimental evaluation demonstrates that Na Kika’sprototype implementation is effective at reducing load onorigin servers and latency for clients, supporting signifi-cantly larger user populations than a single dynamic webserver. It also demonstrates that Na Kika is, in fact, easilyprogrammable and extensible.

AcknowledgmentsBill Holloway ported the SIMMs to Na Kika and JakeAviles helped with their evaluation. Robert Soule imple-mented static content integrity and the security policy ex-tension. Our shepherd, Emin Gun Sirer, and the anony-mous reviewers provided valuable feedback on earlierversions of this paper. This material is based in part uponwork supported by the National Science Foundation un-der Grant No. 0537252 and by the New York SoftwareIndustry Association.

References[1] Akamai Technologies, Inc. A developer’s guide to on-demand

distributed computing, Mar. 2004.[2] A. Awadallah and M. Rosenblum. The vMatrix: A network of

virtual machine monitors for dynamic content distribution. InProc. 7th IWCW, Aug. 2002.

[3] A. Barbir, O. Batuner, B. Srinivas, M. Hofmann, and H. Or-man. Security threats and risks for open pluggable edge services(OPES). RFC 3837, IETF, Aug. 2004.

[4] A. Barbir, R. Penno, R. Chen, M. Hofmann, and H. Orman. Anarchitecture for open pluggable edge services (OPES). RFC 3835,IETF, Aug. 2004.

[5] L. Bent, M. Rabinovich, G. M. Voelker, and Z. Xiao. Characteri-zation of a large web site population with implications for contentdelivery. In Proc. 13th WWW, pp. 522–533, May 2004.

[6] M. Blaze, J. Feigenbaum, J. Ioannidis, and A. D. Keromytis.The role of trust management in distributed systems security. InJ. Vitek and C. D. Jensen, eds., Secure Internet Programming,vol. 1603 of LNCS, pp. 185–210. Springer, 1999.

[7] W. E. Boebert and R. Y. Kain. A practical alternative to hierar-chical integrity policies. In Proc. 17th NCSC, pp. 18–27, 1985.

[8] C. Canali, V. Cardellini, M. Colajanni, R. Lancellotti, and P. S.Yu. Cooperative archictectures and algorithms for discovery andtranscoding of multi-version content. In Proc. 8th IWCW, Sept.2003.

[9] P. Cao, J. Zhang, and K. Beach. Active Cache: Caching dynamiccontents on the web. In Proc. Middleware ’98, pp. 373–388, Sept.1998.

[10] D. M. D’Alessandro, T. E. Lewis, and M. P. D’Alessandro. A pe-diatric digital storytelling system for third year medical students:The virtual pediatric patients. BMC Medical Education, 4(10),July 2004.

[11] C. M. Ellison, B. Frantz, B. Lampson, R. Rivest, B. Thomas, andT. Ylonen. SPKI certificate theory. RFC 2693, IETF, Sept. 1999.

[12] A. Fox, S. D. Gribble, Y. Chawathe, E. A. Brewer, and P. Gau-thier. Cluster-based scalable network services. In Proc. 16thSOSP, pp. 78–91, Oct. 1997.

[13] M. J. Freedman, E. Freudenthal, and D. Mazieres. Democratizingcontent publication with Coral. In Proc. 1st NSDI, Mar. 2004.

[14] L. Gao, M. Dahlin, A. Nayate, J. Zheng, and A. Iyengar. Appli-cation specific data replication for edge services. In Proc. 12thWWW, pp. 449–460, May 2003.

[15] R. Grimm and B. N. Bershad. Separating access control policy,enforcement and functionality in extensible systems. ACM TOCS,19(1):36–70, Feb. 2001.

[16] I. Gupta, K. Birman, P. Linga, A. Demers, and R. van Renesse.Kelips: Building an efficient and stable P2P DHT through in-creased memory and background overhead. In Proc. 2nd IPTPS,vol. 2735 of LNCS, pp. 160–169. Springer, Feb. 2003.

[17] IBM Corporation. WebSphere Edge Server Administration Guide.3rd edition, Dec. 2001.

[18] S. Iyer, A. Rowstron, and P. Druschel. Squirrel: A decentralizedpeer-to-peer web cache. In Proc. 21st PODC, pp. 213–222, July2002.

[19] B. Knutsson, H. Lu, J. Mogul, and B. Hopkins. Architectureand performance of server-directed transcoding. ACM TOIT,3(4):392–424, Nov. 2003.

[20] D. Kotz, R. Gray, S. Nog, D. Rus, S. Chawla, and G. Cybenko.AGENT TCL: Targeting the needs of mobile computers. IEEEInternet Computing, 1(4):58–67, Jul./Aug. 1997.

[21] B. Krishnamurthy, J. C. Mogul, and D. M. Kristol. Key differ-ences between HTTP/1.0 and HTTP/1.1. In Proc. 8th WWW,May 1999.

[22] M. Lorch, S. Proctor, R. Lepro, D. Kafura, and S. Shah. Firstexperiences using XACML for access control in distributed sys-tems. In Proc. 2003 XMLSEC, pp. 25–37, 2003.

[23] W.-Y. Ma, B. Shen, and J. Brassil. Content services network: Thearchitecture and protocols. In Proc. 6th IWCW, June 2001.

[24] D. Mazieres and M. F. Kaashoek. Escaping the evils of central-ized control with self-certifying pathnames. In Proc. 8th SIGOPSEurop. Workshop, pp. 118–125, Sept. 1998.

[25] J. C. Mogul. Clarifying the fundamentals of HTTP. In Proc. 11thWWW, pp. 25–36, May 2002.

[26] J. C. Mogul and K. K. Ramakrishnan. Eliminating receive live-lock in an interrupt-driven kernel. ACM TOCS, 15(3):217–252,Aug. 1997.

[27] Mozilla Foundation. SpiderMonkey (JavaScript-C) engine.http://www.mozilla.org/js/spidermonkey/.

[28] National Center for Postsecondary Improvement. Beyond deadreckoning: Research priorities for redirecting American highereducation. http://www.stanford.edu/group/ncpi/documents/pdfs/beyond dead reckoning.pdf, Oct.2002.

[29] M. Nottingham and X. Liu. Edge architecture specifi-cation, 2001. http://www.esi.org/architecturespec 1-0.html.

[30] ObjectWeb. JORAM. http://joram.objectweb.org/.[31] V. S. Pai, A. L. Cox, V. S. Pai, and W. Zwaenepoel. A flexible

and efficient application programming interface for a customiz-able proxy cache. In Proc. 4th USITS, Mar. 2003.

[32] L. Peterson, T. Anderson, D. Culler, and T. Roscoe. A blueprintfor introducing disruptive technology into the internet. In Proc.1st HotNets, Oct. 2002.

[33] M. Rabinovich, Z. Xiao, and A. Aggarwal. Computing on theedge: A platform for replicating internet applications. In Proc.8th IWCW, Sept. 2003.

[34] M. Rabinovich, Z. Xiao, F. Douglis, and C. Kalmanek. Movingedge-side includes to the real edge—the clients. In Proc. 4thUSITS, Mar. 2003.

[35] A. Rowstron and P. Druschel. Pastry: Scalable, distributed objectlocation and routing for large-scale peer-to-peer systems. In Proc.Middleware ’01, pp. 329–350, Nov. 2001.

[36] Security Space. Apache module report. http://www.securityspace.com/s survey/data/man.200501/apachemods.html, Feb. 2005.

[37] Security Space. Web server survey. http://www.securityspace.com/s survey/data/200501/index.html, Feb. 2005.

[38] W. Shi, K. Shah, Y. Mao, and V. Chaudhary. Tuxedo: A peer-to-peer caching system. In Proc. 2003 PDPTA, pp. 981–987, June2003.

[39] E. G. Sirer, R. Grimm, A. J. Gregory, and B. N. Bershad. De-sign and implementation of a distributed virtual machine for net-worked computers. In Proc. 17th SOSP, pp. 202–216, Dec. 1999.

[40] E. G. Sirer and K. Wang. An access control language for webservices. In Proc. 7th SACMAT, pp. 23–30, June 2002.

[41] N. Spring, D. Wetherall, and T. Anderson. Scriptroute: A publicinternet measurement facility. In Proc. 4th USITS, pp. 225–238,Mar. 2003.

[42] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakr-ishnan. Chord: A scalable peer-to-peer lookup service for Inter-net applications. In Proc. 2001 SIGCOMM, pp. 149–160, Aug.2001.

[43] M. M. Triola, M. A. Hopkins, M. J. Weiner, W. Holloway, R. I.Levin, M. S. Nachbar, and T. S. Riles. Surgical interactive mul-timedia modules: A novel, non-browser based architecture formedical education. In Proc. 17th CBMS, pp. 423–427, June 2004.

[44] M. Tsimelzon, B. Weihl, and L. Jacobs. ESI language specifica-tion 1.0, 2001. http://www.esi.org/language spec1-0.html.

[45] S. H. J. Uijtdehaage, S. E. Dennis, and C. Candler. A web-baseddatabase for sharing educational multimedia within and amongmedical schools. Academic Medicine, 76:543–544, 2001.

[46] A. Vahdat, M. Dahlin, T. Anderson, and A. Aggarwal. ActiveNames: Flexible location and transport of wide-area resources.In Proc. 2nd USITS, pp. 151–164, Oct. 1999.

[47] L. Wang, V. Pai, and L. Peterson. The effectiveness of requestredirection on CDN robustness. In Proc. 5th OSDI, pp. 345–360,Dec. 2002.

[48] L. Wang, K. Park, R. Pang, V. Pai, and L. Peterson. Reliabilityand security in the CoDeeN content distribution network. In Proc.2004 USENIX, pp. 171–184, June 2004.

[49] S. A. Wartman. Research in medical education: The challengefor the next decade. Academic Medicine, 69(8):608–614, 1994.

[50] A. Wolman, G. M. Voelker, N. Sharma, N. Cardwell, A. Karlin,and H. M. Levy. On the scale and performance of cooperativeweb proxy caching. In Proc. 17th SOSP, pp. 16–31, Dec. 1999.

[51] H. Yu and A. Vahdat. Design and evaluation of a continuousconsistency model for replicated services. In Proc. 4th OSDI, pp.305–318, Oct. 2000.

[52] B. Y. Zhao, L. Huang, J. Stribling, S. C. Rhea, A. D. Joseph, andJ. D. Kubiatowicz. Tapestry: A resilient global-scale overlay forservice deployment. IEEE J-SAC, 22(1):41–53, Jan. 2004.

[53] W. Zhao and H. Schulzrinne. DotSlash: Providing dynamic scal-ability to web applications with on-demand distributed query re-sult caching. Tech. Report CUCS-035-05, Columbia University,Sept. 2005.