adaptsize: orchestrating the hot object memory …...adaptsize: orchestrating the hot object memory...

This paper is included in the Proceedings of the 14th USENIX Symposium on Networked Systems

Design and Implementation (NSDI rsquo17)March 27ndash29 2017 bull Boston MA USA

ISBN 978-1-931971-37-9

Open access to the Proceedings of the 14th USENIX Symposium on Networked

Systems Design and Implementation is sponsored by USENIX

AdaptSize Orchestrating the Hot Object Memory Cache in a Content Delivery Network

Daniel S Berger University of Kaiserslautern Ramesh K Sitaraman University of Massachusetts Amherst and Akamai Technologies

Mor Harchol-Balter Carnegie Mellon University

httpswwwusenixorgconferencensdi17technical-sessionspresentationberger

AdaptSize Orchestrating the Hot Object Memory Cachein a Content Delivery Network

Daniel S Berger1 Ramesh K Sitaraman2 and Mor Harchol-Balter31University of Kaiserslautern 2UMass Amherst amp Akamai Technologies 3Carnegie Mellon University

AbstractMost major content providers use content delivery net-works (CDNs) to serve web and video content to theirusers A CDN is a large distributed system of serversthat caches and delivers content to users The first-levelcache in a CDN server is the memory-resident Hot ObjectCache (HOC) A major goal of a CDN is to maximize theobject hit ratio (OHR) of its HOCs But the small sizeof the HOC the huge variance in the requested objectsizes and the diversity of request patterns make this goalchallenging

We propose AdaptSize the first adaptive size-awarecache admission policy for HOCs that achieves a highOHR even when object size distributions and requestcharacteristics vary significantly over time At the core ofAdaptSize is a novel Markov cache model that seamlesslyadapts the caching parameters to the changing requestpatterns Using request traces from one of the largestCDNs in the world we show that our implementation ofAdaptSize achieves significantly higher OHR than widely-used production systems 30-48 and 47-91 higherOHR than Nginx and Varnish respectively AdaptSizealso achieves 33-46 higher OHR than state-of-the-artresearch systems Further AdaptSize is more robust tochanging request patterns than the traditional tuning ap-proach of hill climbing and shadow queues studied inother contexts

1 IntroductionContent delivery networks (CDNs) [18] enhance perfor-mance by caching objects in servers close to users andrapidly delivering those objects to users A large CDNsuch as that operated by Akamai [57] serves trillionsof user requests a day from 170000+ servers locatedin 1500+ networks in 100+ countries around the worldCDNs carry the majority of todayrsquos Internet traffic and areexpected to carry almost two thirds within five years [22]

A CDN server employs two levels of caching a smallbut fast in-memory cache called the Hot Object Cache(HOC) and a large second-level Disk Cache (DC) Eachrequested object is first looked up in the HOC If absentit is looked up in the DC If also absent there the object isfetched over the WAN from the content providerrsquos origin

Serving requests from the HOC is much faster andmore efficient than serving from the DC Thus the goalof a CDN is to maximize the object hit ratio (OHR)which is the fraction of requests served from the HOC

0

25

50

75

100

1B 1KiB 1MiB 1GiBRequest size

CD

F

HKUS

Figure 1 The cumulative distribution for object sizes intwo Akamai production traces from Hong Kong and theUS Sizes vary by more than nine orders of magnitude

In this paper we show how to attain this goal in a robustefficient and scalable manner (see Section 2)

11 Why HOC cache management is hardHOC cache management entails two types of decisionsFirst the cache can decide whether or not to admit anobject (cache admission) Second the cache can decidewhich object to evict from the cache (cache eviction) ifthere is no space for a newly admitted object While cachemanagement is well studied in other contexts HOC cachemanagement poses the following new challenges

1) The HOC is subject to extreme variability in requestpatterns and object sizes CDNs serve multiple trafficclasses using a shared server infrastructure Such classesinclude web sites videos and interactive applicationsfrom thousands of content providers each class with itsown distinctive object size distributions and request pat-terns [57] Figure 1 shows the object size distribution ofrequests served by two Akamai production servers (one inthe US the other in Hong Kong) We find that object sizesspan more than nine orders of magnitude and that thelargest objects are often of the same order of magnitude asthe HOC size itself This extreme variability underscoresthe need for cache admission as placing one large objectin the HOC can result in the eviction of many small oneswhich can severely degrade the OHR (Section 3)

2) Prior academic research is largely inapplicable asit focuses on caching objects of similar sizes While theacademic literature on caching policies is extensive itfocuses on situations where objects are of the same sizeFurther prior work almost exclusively focuses on evictionpolicies ie all requested objects are admitted to thecache and space is only managed by eviction (Section 7)Thus there is little emphasis on cache admission in theprior literature even as we show that cache admission iskey in our context (Section 3)

3) Production systems implement a cache admis-sion scheme with a static threshold that performs sub-

USENIX Association 14th USENIX Symposium on Networked Systems Design and Implementation 483

00

02

04

06

08

1B 1KiB 1MiB 1GiB

Size threshold

OH

R

OPT

too high

too low

(a) OHR versus threshold

0

25

50

75

100

Request Size

CD

F

1B 1KiB 1MiB 1GiBOPT

80 getadmitted

20dont getadmitted

(b) Request size distribution

Figure 2 Experimental results with different size thresh-olds (a) A OHR-vs-threshold curve shows that the ObjectHit Ratio (OHR) is highly sensitive to the size thresholdand that the optimal threshold (red arrow) can significantlyimprove the OHR (b) The optimal threshold admits therequested object for only 80 of the requests

optimally for CDN workloads In contrast to much ofthe academic research production systems recognize thefact that not all objects can be admitted into the HOCA common approach is to define a static size thresholdand to only admit objects with size below this thresholdFigure 2a shows how OHR is affected by the size thresh-old for a production CDN workload While the optimalthreshold (OPT) almost doubles the OHR compared toadmitting all objects conservative thresholds that are toohigh lead to marginal gains and the OHR quickly dropsto zero for aggressive thresholds that are too low

Unfortunately the ldquobestrdquo threshold changes signifi-cantly over time Figures 3a and 3b show the OHR as afunction of the size threshold at two different times of theday Note that the optimal thresholds can vary by as muchas two orders of magnitude during a day Since no priormethod exists for dynamically tuning such a thresholdcompanies have resorted to either setting the size admis-sion threshold conservatively high or (more commonly)not using size-aware admission at all [67 54 21]

4) Simple strategies for dynamically tuning cache ad-mission parameters do not work well While it may seemthat simple tuning approaches can be used to adapt thesize threshold parameter over time this turns out to bea non-trivial problem This is probably why size-awareadmission is not used effectively in practice In Sec-tion 3 we consider common tuning approaches such ashill climbing with shadow caches or using a thresholdthat is a fixed function of a request size percentile (egthe 80-th percentile as in Figure 2b) We also considerusing probabilistic size-aware admission where smallsizes are ldquomore likelyrdquo to be admitted and large files areldquoless likelyrdquo to be admitted We find that none of theseapproaches is sufficiently robust to traffic mix changesthat occur in daily operation due in part to the CDNrsquosglobal load balancer

12 Our contributionsWe propose AdaptSize a lightweight and near-optimaltuning method for size-aware cache admission Adapt-

00

02

04

06

08

1B 1KiB 1MiB 1GiB

Size threshold

OH

R

OPT

(a) Morning web traffic

00

02

04

06

08

1B 1KiB 1MiB 1GiB

Size threshold

OH

R

OPT

(b) Evening webvideo mix

Figure 3 The optimal size threshold changes significantlyover time (a) In the morning hours small objects (egnews items) are more popular which requires a small sizethreshold of a few tens of KiBs (b) In the evening hoursweb traffic gets mixed with video traffic which requires asize threshold of a few MiBs

Size is based on a novel statistical representation of thecache using a Markov model This model is unique inthat it incorporates all correlations between object sizesand current request rates (prior caching models assumedunit-sized objects) This model is analytically tractableallowing us to quickly find the optimal size-aware admis-sion policy and repeat this process at short intervals

We have implemented AdaptSize within the Varnishproduction caching system1 Varnish is known as a high-throughput system that makes extensive use of concur-rency [76 43 42] and is used by several prominent con-tent providers and CDNs including Wikipedia Facebookand Twitter Through evaluations of AdaptSize on produc-tion request traces from one of the worldrsquos largest CDNsAkamai we observe the following key features

1 AdaptSize improves the OHR by 47-91 over anunmodified Varnish system and by 30-48 over anoffline-tuned version of the Nginx caching system(Figure 4 and Section 61) Varnish and Nginx areused by almost 80 of the top 5000 websites whichwe determined by crawling Alexarsquos top sites list [74]

2 In addition to improving upon production systemsAdaptSize also improves the OHR by 33-46 overstate-of-the-art research caching systems (Figure 5and Section 62)

3 Compared to other tuning methods such as the clas-sical hill climbing technique using shadow queuesAdaptSize improves the OHR on average by 15-20and in some cases by more than 100 (Figure 6 andSection 63) In particular we found classical tuningmethods can get ldquostuckrdquo at poor local optima that areavoided by AdaptSizersquos model-based optimization

4 We compare AdaptSize with SIZE-OPT which tunesthe size threshold parameter using a priori knowl-edge of the next one million requests AdaptSizestays within 90 of the OHR of SIZE-OPT in themedian across all experiments and is never worsethan 80 of the OHR of SIZE-OPT (Sections 61

1The source code of AdaptSize and installation instructions areavailable at httpsgithubcomdasebeAdaptSize

484 14th USENIX Symposium on Networked Systems Design and Implementation USENIX Association

SIZE-OPT

AdaptSize

Nginx

Varnish

00 02 04 06Object Hit Ratio

+91+48

Figure 4 Comparison of AdaptSizersquos implementation tothe Varnish and Nginx production systems We also showthe SIZE-OPT policy which has future knowledge anduses this to set the optimal size threshold at every momentin time AdaptSize improves the OHR by 48-91 overthe production systems and also achieves 95 of the OHRof SIZE-OPT These results are for the US trace and atypical HOC size (details in Section 51)

and 63) ndash even when subjected to extreme changesin the request traffic

5 In addition to improving the OHR AdaptSize alsoreduces request latencies by 43 in the median andby more than 30 at the 90-th and 999-th percentileAdaptSize is able to maintain the high throughput ofVarnish without adding (concurrent) synchronizationoverheads and reduces the disk utilization of thesecond-level cache by 20 (Section 64)

Roadmap The rest of this paper is structured as fol-lows Sections 2 and 3 discuss our goals and our rationalein designing AdaptSize Section 4 details AdaptSizersquosdesign and implementation Sections 5 and 6 describeour setup and experimental results respectively Section 7reviews related work We conclude in Section 8

2 HOC Design GoalsIn designing AdaptSize we seek to maximize the OHRwhile maintaining a robust and scalable HOC and avoid-ing adverse side-effects on second-level caches

Maximizing the OHR The HOCrsquos primary designobjective is user performance which it optimizes by pro-viding fast responses for as many requests as possible Anatural way to measure this objective is the object hit ratio(OHR) which gives equal weight to all user requests

While our evaluations are based on production CDNservers without SSDs HOCs are also key to hybrid CDNservers that typically have both hard disks and SSDs Thisis because HOCs are more CPU efficient and can servetraffic at higher rates Further HOCs offload requestsfrom the SSDs that are often io bound HOCs are usedin SSD-based servers at Akamai and also at Fastly [54]and Wikipedia [67] These production deployments seekto maximize the OHR of the HOC which is the mainperformance metric used throughout this paper

There are other cache performance metrics that are lessrelevant to the HOC For example the much larger DCfocuses on the byte hit rate (BHR) that is the fraction ofbytes that are served from the cache [71] The HOC has

Figure 5 Comparison of AdaptSize to state-of-the-artresearch caching systems Most of these use sophisticatedadmission and eviction policies that combine recencyand frequency (striped blue bars) AdaptSize improvesthe OHR by 46 over the next best system Policiesannotated by ldquo++rdquo are actually optimistic because weoffline-tuned their parameters to the trace These resultsare for the US trace and a HOC size 12 GiB

little impact on the BHR as it is typically three orders ofmagnitude smaller than the DC

Robustness against changing request patterns AHOC is subjected to a variety of traffic changes each dayFor example web content popularity changes during theday (eg news in the morning vs video at night) whichincludes rapid changes due to flash crowds Anothersource of traffic changes is the sharing of the server in-frastructure between traffic classes Such classes includeweb sites videos software downloads and interactiveapplications from thousands of content providers [57]As a shared infrastructure is more cost effective a CDNserver typically serves a mix of traffic classes Due toload balancing decisions this mix can change abruptlyThis poses a particular challenge as each traffic class hasits own distinctive request and object size distributionstatistics large objects can be unpopular during one hourand popular during the next A HOC admission policymust be able to rapidly adapt to all these changing requestpatterns in order to achieve consistently high OHRs

Low overhead and high concurrency As the firstcaching level in a CDN the HOC needs to both respondquickly to requests and deliver high throughput Thisrequires that the admission and eviction policies havea small processing overhead ie a constant time com-plexity per request (see the O(1) policies in Table 2 inSection 7) and that they have concurrent implementations(see the corresponding column in Table 2)

No negative side-effects While the HOC achieves ahigh OHR and fast responses it must not impede the per-formance of the overall CDN server Specifically changesto the HOC must not negatively affect the BHR and diskutilization of the DC

3 Rationale for AdaptSizeThe goal of this section is to answer why the HOC needssize-aware admission why such an admission policy


04

06

08

6pm 7pm 8pm 9pm 10pm

OH

Rwebvideoweb

Time

AdaptSize

Static

HillClimb

Figure 6 Comparison of AdaptSize threshold tuning viahill climbing and shadow caches (HillClimb) and a staticsize threshold (Static) under a traffic mix change fromonly web to mixed webvideo traffic While AdaptSizequickly adapts to the new traffic mix HillClimb gets stuckin a suboptimal configuration and Static (by definition)does not adapt AdaptSize improves the OHR by 20over HillClimb and by 25 over Static on this trace

needs to be adaptively tuned and why a new approach toparameter tuning is needed

31 Why HOCs need size-aware admissionWe start with a toy example Imagine that there are onlytwo types of objects 9999 small objects of size 100 KiB(say web pages) and 1 large object of size 500 MiB (saya software download) Further assume that all objectsare equally popular and requested forever in round-robinorder Suppose that our HOC has a capacity of 1 GiB

A HOC that does not use admission control cannotachieve an OHR above 05 Every time the large objectis requested it pushes out asymp5000 small objects It doesnot matter which objects are evicted when the evictedobjects are requested they cannot contribute to the OHR

An obvious solution for this toy example is to controladmissions via a size threshold If the HOC admits onlyobjects with a size at most 100 KiB then it can achievean OHR of 09999 as all small objects stay in the cache

This toy example is illustrative of what happens underreal production traffic We observe from Figure 1 thatapproximately 5 of objects have a size bigger than 1MiB Every time a cache admits a 1 MiB object it needsto evict space equivalent to one thousand 1 KiB objectswhich make up about 15 of requests Again thoseevicted objects will not be able to contribute to the OHRA well-designed cache admission algorithm could helpavoid such evictions that have a large impact on OHR

32 Why we need a new tuning methodThe key question when implementing size-aware admis-sion is picking its parameters Figure 2 shows that astatic size threshold is inadequate Next we explore threecanonical approaches for tuning a size threshold Theseapproaches are well-known in prior literature and havebeen applied in other contexts (unrelated to the tuningof size thresholds) However we show that these knownapproaches are deficient in our context motivating theneed for AdaptSizersquos new tuning mechanism

00

02

04

06

1B 1KiB 1MiB 1GiB

Size Threshold

OH

R Trace 1Trace 2Trace 3

same80 tileof size

OPT3OPT2OPT1

Figure 7 Experimental results showing that setting thesize threshold to a fixed function does not work All threetraces shown here have the same 80-th size percentile buttheir optimal thresholds differ by two orders of magnitude

Tuning based on request size percentiles A com-mon approach used in many contexts (eg capacity pro-visioning) is to derive the required parameter as somefunction of the request size distribution and arrival rateA simple way of using this approach in our context is toset the size threshold for cache admission to be a fixedpercentile of the object size distribution However forproduction CDN traces there is no fixed relationshipbetween the percentiles of the object size distributionand optimal size threshold that maximizes the OHR InFigure 2 the optimal size threshold lands on the 80-thpercentile request size However in Figure 7 note thatall three traces have the same 80-th percentile but verydifferent optimal thresholds In fact we found many ex-amples of multiple traces that agree on all size percentilesand yet have different optimal size thresholds The rea-son is that for maximizing OHR it matters whether thenumber of requests seen for a specific object size comefrom one (very popular) object or from many (unpopular)objects This information is not captured by the requestsize distribution

Tuning via hill climbing and shadow caches A com-mon tool for the tuning of caching parameters is the useof shadow caches For example in the seminal paperon ARC [53] the authors tune their eviction policy tohave the optimal balance between recency and frequencyby using a shadow cache (we discuss other related workusing shadow caches in Section 72) A shadow cacheis a simulation which is run in real time simultaneouslywith the main (implemented) cache but using a differentparameter value than the main cache Hill climbing thenadapts the parameter by comparing the hit ratio achievedby the shadow cache to that of the main cache (or anothershadow cache) In theory we could exploit the same ideato set our size-aware admission threshold Unfortunatelywhen we tried this we found that the OHR-vs-thresholdcurves are not concave and that they can have severallocal optima in which the hill climbing gets frequentlystuck Figure 3b shows such an example in which thelocal optima result from mixed traffic (web and video)As a consequence we will demonstrate experimentally inSection 63 that hill climbing is suboptimal AdaptSizeachieves an OHR that is 29 higher than hill climbing on


average and 75 higher in some cases We tried addingmore shadow caches and also randomizing the evalu-ated parameters but could not find a robust variant thatconsistently optimized the OHR across multiple traces2

In conclusion our extensive experiments show thattuning methods like shadow caches with hill climbing aresimply not robust enough for the problem of size-awareadmission with CDN traffic

Avoiding tuning by using probabilistic admissionOne might imagine that the difficulty in tuning the sizethreshold lies in the fact that we are limited to a singlestrict threshold The vast literature on randomized algo-rithm suggests that probabilistic parameters are more ro-bust than deterministic ones [55] We attempted to applythis idea to size-aware tuning by considering probabilisticadmission policies which ldquofavor the smallsrdquo by admit-ting them with high probability whereas large objects areadmitted with low probability We chose a probabilisticfunction that is exponentially decreasing in the object size(eminussizec) Unfortunately the parameterization of the ex-ponential curve (the c) matters a lot ndash and itrsquos just as hardto find this c parameter as it is to find the optimal sizethreshold Furthermore the best exponential curve (thebest c) changes over time In addition to exponentiallydecreasing probabilities we also tried inversely propor-tional linear and log-linear variants Unfortunately noneof these variants resolves the problem that there is at leastone parameter without an obvious way how to choose it

In conclusion even randomized admission control re-quires the tuning of some parameter

4 The AdaptSize Caching SystemAdaptSize admits objects with probability eminussizec andevicts objects using a concurrent variant of LRU [43]Observe that the function eminussizec is biased in favor ofadmitting small sizes with higher probability

Why a probabilistic admission function The sim-plest size-based admission policy is a deterministic thresh-old c where only objects with a size lt c are admitted Aprobabilistic admission function like eminussizec is moreflexible objects greater than c retain a low but non-zeroadmission probability which results in eventual admissionfor popular objects (but not for unpopular ones) In ourexperiments eminussizec consistently achieves a 10 higherOHR than the best deterministic threshold

What parameter c does AdaptSize use in the eminussizec

function AdaptSizersquos tuning policy recomputes the op-timal c every ∆ requests A natural approach is to usehill-climbing with shadow caches to determine the opti-mal c parameter Unfortunately that leads to a myopicview in that only a local neighborhood of the current c can

2While there are many complicated variants of shadow-cache searchalgorithms they all rely on a fundamental assumption of stationaritywhich does not need to apply to web traffic

Figure 8 AdaptSizersquos Markov chain model for object irepresents irsquos position in the LRU list and the possibil-ity that the object is out of the cache Each object isrepresented by a separate Markov chain but all Markovchains are connected by the common ldquopushdownrdquo rate microcSolving these models yields the OHR as a function of c

be searched This leads to sub-optimal results given thenon-convexities present in the OHR-vs-c curve (Figure 3)By contrast we derive a full Markov chain model of thecache This model allows AdaptSize to view the entireOHR-vs-c curve and perform a global search for the opti-mal c The challenge of the Markov model approach is indevising an algorithm for finding the solution quickly andin incorporating that algorithm into a production system

In the following we describe the derivation of Adapt-Sizersquos Markov model (Section 41) and how we incorpo-rate AdaptSize into a production system (Section 42)

41 AdaptSizersquos Markov chain tuning modelTo find the optimal c AdaptSize uses a novel Markovchain model which is very different from that typi-cally used for cache modeling Traditionally peoplehave modeled the entire state of the cache trackingall objects in the cache and their ordering in the LRUlist [44 30 15 52 10 33 25 24 19 68] While thisis 100 accurate it also becomes completely infeasiblewhen the number of objects is high because of a combi-natorical state space explosion

AdaptSize instead creates a separate Markov chain foreach object (cf Figure 8) Each objectrsquos chain tracksits position in the LRU list (if the object is in the cache)as well as a state for the possibility that the object is outof the cache Using an individual Markov chain greatlyreduces the model complexity which now scales linearlywith the number of objects rather than exponentially inthe number of objects

AdaptSizersquos Markov chain Figure 8 shows theMarkov chain for the ith object The chain has two im-portant parameters The first is the rate at which object iis moved up to the head of the LRU list due to accessesto the object We get the ldquomove uprdquo rate ri by collect-ing aggregate statistics for object i during the previous∆ time interval The second parameter is the averagerate at which object i is pushed down the LRU list Theldquopushdownrdquo rate microc depends on the rate with which anyobject is moved to the top of the LRU list (due to a hit


or after cache admission) As it does not matter whichobject is moved to the top microc is approximately the samefor all objects So we consider a single ldquopushdownrdquo ratefor all objects We calculate microc by solving an equationthat takes all objects into account and thus captures theinteractions between all the objects3 Specifically we findmicroc by solving an equation that says that the expected sizeof all cached objects canrsquot exceed the capacity K that isactually available to the cache

N

sumi=1

P[object i in cache] si = K (1)

Here N is the number of all objects observed over theprevious ∆ interval and si is the size of object i Note thatP[object i in cache] is a monotonic function in terms ofmicroc which leads to a unique solution

Our new model enables us to find P[object i in cache]as a function of c by solving for the limiting probabilitiesof all ldquoinrdquo states in Figure 8 Appendix A shows how thisis done We obtain a function of c in closed formTheorem 1 (proof in Appendix A)

P[object i in cache] =(erimicroc minus1) middot eminuscmiddotsi

1+(erimicroc minus1) middot eminuscmiddotsi

Note that the size admission parameter c affects both theadmission probability (eminussic) and the pushdown rate (microc)For example a lower c results in fewer admissions whichresults in fewer evictions and in a smaller pushdown rate

The OHR as a function of c Theorem 1 and Equa-tion (1) yield the OHR by observing that the expectednumber of hits of object i equals ri (irsquos average requestrate) times the long-term probability that i is in the cacheThe OHR predicted for the threshold parameter c is thensimply the ratio of expected hits to requests

OHR(c) =sum

Ni=1 riP[object i in cache]

sumNi=1 ri

If we consider a discretized range of c values we can nowcompute the OHR for each c in the range which gives usa ldquocurverdquo of OHR-vs-c (similar to the curves in Figure 9)

Global search for the optimal c Every ∆ steps wederive the OHR-vs-c curve using our Markov model Wesearch this curve for the c that maximizes the OHR usinga standard global search method for non-concave func-tions [64] This c is then used for the next ∆ steps

Accuracy of AdaptSizersquos model Our Markov chainrelies on several simplifying assumptions that can poten-tially impact the accuracy of the OHR predictions Fig-ure 9 shows that AdaptSizersquos OHR equation matches ex-perimental results across the whole range of the thresholdparameter c on two typical traces of length ∆ In additionwe continuously compared AdaptSizersquos model to mea-surements during our experiments (Section 6) AdaptSizeis very accurate with an average error of about 1

3Mean-field theory [45] provides analytical justification for why itis reasonable to assume a single average pushdown rate when there arethousands of objects (as in our case)

00

02

04

06

08

1B 1KiB 1MiB 1GiB

Size admission parameter c

OH

R

(a) HK trace

00

02

04

06

08

1B 1KiB 1MiB 1GiB


OH

R

(b) US trace

Figure 9 AdaptSizersquos Markov model predicts the OHRsensitivity curve (red solid line) This is very accuratewhen compared to the actual OHR (black dots) that resultswhen that threshold is chosen Each experiment involvesa portion of the production trace of length ∆ = 250K

42 Integration with a production systemWe implemented AdaptSize on top of Varnish [76 32] aproduction caching system by modifying the miss requestpath On a cache miss Varnish accesses the second-levelcache to retrieve the object and places it in its HOCWith AdaptSize the probabilistic admission decision isexecuted which is evaluated independently for all cachethreads and adds a constant number of instructions to therequest path If the object is not admitted it is servedfrom Varnishrsquos transient memory

Our implementation uses a parameter ∆ which is thesize of the window of requests over which our Markovmodel for tuning is computed In addition to statisticsfrom the current window we also incorporate the statisti-cal history from prior windows via exponential smoothingwhich makes AdaptSize more robust and largely insen-sitive to ∆ on both of our production traces In our ex-periments we choose ∆=250K requests (about 5-10 minson average) which allows AdaptSize to react quickly tochanges in the request traffic

Lock-free statistics collection A key problem in im-plementing AdaptSize lies in efficient statistics collectionfor the tuning model Gathering request statistics can addsignificant overhead to concurrent caching designs [69]Varnish and AdaptSize use thousands of threads in ourexperiments so centralized request counters would causehigh lock contention In fact we find that Varnishrsquosthroughput bottleneck is lock contention for the few re-maining synchronization points (eg [43])

Instead of a central request counter AdaptSize hooksinto the internal data structure of the cache threads Eachcache thread keeps debugging information in a concurrentring buffer to which all events are simply appended (over-writing old events after some time) AdaptSizersquos statisticscollection frequently scans this ring buffer (read only) anddoes not require any synchronization

Robust and efficient model evaluation The OHRprediction in our statistical model involves two more im-plementation challenges The first challenge lies in effi-ciently solving equation (1) We achieve a constant time


HK trace US traceTotal Requests 450 million 440 million

Total Bytes 1575 TiB 1523 TiBUnique Objects 25 million 55 millionUnique Bytes 147 TiB 89 TiB

Start Date Jan 29 2015 Jul 15 2015End Date Feb 06 2015 Jul 20 2015

Table 1 Basic information about our web traces

overhead by using a fixed-point solver [26] The secondchallenge is due to the exponential function in the The-orem 1 The value of the exponential function outgrowseven 128-bit float number representations We solve thisproblem by using an accurate and efficient approxima-tion for the exponential function using a Pade approxi-mant [63] that only uses simple float operations which arecompatible with SSEAVX vectorization speeding up themodel evaluation by about 10-50times in our experiments

5 Evaluation MethodologyWe evaluate AdaptSize using both trace-based simulations(Section 52) and a Varnish-based implementation (Sec-tion 53) running on our experimental testbed For boththese approaches the request load is derived from tracesfrom Akamairsquos production CDN servers (Section 51)

51 Production CDN request tracesWe collected request traces from two production CDNservers in Akamairsquos global network Table 1 summarizesthe main characteristics of the two traces Our first traceis from urban Hong Kong (HK trace) Our second traceis from rural Tennessee in the US (US trace) Both spanmultiple consecutive days with over 440 million requestsper trace during the months of February and July 2015Both production servers use a HOC of size 12 GiB andseveral hard disks as second-level caches They serve atraffic mix of several thousand popular web sites whichrepresents a typical cross section of the web (news so-cial networks downloads ecommerce etc) with highlyvariable object sizes Some content providers split verylarge objects (eg videos) into smaller (eg 2 MiB)chunks The chunking approach is accurately representedin our request traces For example the cumulative dis-tribution function shown in Figure 1 shows a noticeablejump around the popular 2 MiB chunk size

52 Trace-based simulatorWe implemented a cache simulator in C++ that incor-porates AdaptSize and several state-of-the-art researchcaching policies The simulator is a single-threaded im-plementation of the admission and eviction policies andperforms the appropriate cache actions when it is fed theCDN request traces Objects are only stored via theirids and the HOC size is enforced by a simple check onthe sum of bytes currently stored While actual cachingsystems (such as Varnish [43 42]) use multi-threaded con-

current implementations our simulator provides a goodapproximation of the OHR when compared with our pro-totype implementations that we describe next

53 Prototype Evaluation TestbedOur implementation testbed is a dedicated (university)data center consisting of a client server an origin serverand a CDN server that incorporates the HOC We useFUJITSU CX250 HPC servers which run RHEL 65kernel 2632 and gcc 447 on two Intel E5-2670 CPUswith 32 GiB RAM and an IB QDR networking interface

In our evaluation the HOC on our CDN server is ei-ther running Nginx Varnish or AdaptSize Recall thatwe implemented AdaptSize by adding it to Varnish4 asdescribed in Section 42 We use Nginx 1912 (February2016) with its build-in frequency-based admission policyThis policy relies on one parameter how many requestsneed to be seen for an object before being admitted to thecache We use an optimized version of Nginx since wehave tuned its parameter offline for both traces We useVarnish 412 (March 2016) with its default configurationthat does not use an admission policy

The experiments in Section 61 62 and 63 focus onthe HOC and do not use a DC The DC in Section 61 usesVarnish in a configuration similar to that of the WikimediaFoundationrsquos CDN [67] We use four equal dedicated1 TB WD-RE3 7200 RPM 32 MiB-Cache hard disksattached via a Dell 6 Gbs SAS Host Bus Adapter Card inraw mode (RAID disabled)

The client fetches content specified in the request tracefrom the CDN server using libcurl The request trace iscontinuously read into a global queue which is distributedto worker threads (client threads) Each client threadcontinually requests objects in a closed-loop fashion Weuse up to 200 such threads and verified that the numberof client threads has a negligible impact on the OHR

If the CDN server does not have the requested contentit is fetched from the origin server Our origin server is im-plemented in FastCGI As it is infeasible to store all traceobjects (23 TB total) on the origin server our implementa-tion creates objects with the correct size on the fly beforesending them over the network In order to stress testour caching implementation the origin server is highlymulti-threaded and intentionally never the bottleneck

6 Empirical EvaluationThis section presents our empirical evaluation of Adapt-Size We divide our evaluation into three parts In Sec-tion 61 we compare AdaptSize with production cachingsystems as well as with an offline caching system calledSIZE-OPT that continuously optimizes OHR with knowl-edge of future requests While SIZE-OPT is not imple-mentable in practice it provides an upper bound on the

4We refer to AdaptSize incorporated into Varnish as ldquoAdaptSizerdquoand Varnish without modifications as ldquoVarnishrdquo


SIZE-OPT

AdaptSize

Nginx

Varnish

00 02 04 06 08Object Hit Ratio

+47+30

Figure 10 Comparison of AdaptSizersquos implementationto the Varnish and Nginx production systems and SIZE-OPT AdaptSize improves the OHR by 30-47 over theproduction systems and also achieves 99 of the OHRof SIZE-OPT These results are for the HK trace corre-sponding results for the US trace are shown in Figure 4

achievable OHR to which AdaptSize can be compared InSection 62 we compare AdaptSize with research cachingsystems that use more elaborate eviction and admissionpolicies In Section 63 we evaluate the robustness ofAdaptSize by emulating both randomized and adversar-ial traffic mix changes In Section 64 we evaluate theside-effects of AdaptSize on the overall CDN server

61 Comparison with production systemsWe use our experimental testbed outlined in Section 53and answer four basic questions about AdaptSize

What is AdaptSizersquos OHR improvement over pro-duction systems Quick answer AdaptSize improvesthe OHR by 47-91 over Varnish and by 30-48 overNginx We compare the OHR of AdaptSize to Nginx andVarnish using the 12 GiB HOC configuration from thecorresponding Akamai production servers (Section 51)For the HK trace (Figure 10) we find that AdaptSizeimproves over Nginx by 30 and over Varnish by 47For the US trace (Figure 4) the improvement increases to48 over Nginx and 91 over Varnish

The difference in the improvement over the two tracesstems from the fact that the US trace contains 55 millionunique objects as compared to only 25 million uniqueobjects in the HK trace We further find that AdaptSizeimproves the OHR variability (the coefficient of variation)by 19times on the HK trace and by 38times on the US trace(compared to Nginx and Varnish)

How does AdaptSize compare with SIZE-OPTQuick answer for the typical HOC size AdaptSizeachieves an OHR within 95 of SIZE-OPT We bench-mark AdaptSize against the SIZE-OPT policy whichtunes the threshold parameter c using a priori knowledgeof the next one million requests Figures 4 and 10 showthat AdaptSize is within 95 of SIZE-OPT on the UStrace and within 99 of SIZE-OPT on the HK tracerespectively

How much is AdaptSizersquos performance affected bythe HOC size Quick answer AdaptSizersquos improvementover production caching systems becomes greater forsmaller HOC sizes and decreases for larger HOC sizesWe consider the OHR when scaling the HOC size between512 MiB and 32 GiB under the production server traffic

00

02

04

06

08

512MiB 2GiB 8GiB 32GiB

HOC size

SIZE-OPTAdaptSizeNginxVarnish

OH

R

(a) HK trace

00

02

04

06


HOC size

OH

R


(b) US trace

Figure 11 Comparison of AdaptSize to SIZE-OPT Var-nish and Nginx when scaling the HOC size under theproduction server traffic of two 12 GiB HOCs Adapt-Size always stays close to SIZE-OPT and significantlyimproves the OHR for all HOC sizes

of a 12 GiB HOC Figures 11a and 11b shows that theperformance of AdaptSize is close to SIZE-OPT for allHOC sizes The improvement of AdaptSize over Nginxand Varnish is most pronounced for HOC sizes close tothe original configuration As the HOC size increasesthe OHR of all caching systems improves since the HOCcan store more objects This leads to a smaller relativeimprovement of AdaptSize for a HOC size of 32 GiB10-12 over Nginx and 13-14 over Varnish

How much is AdaptSizersquos performance affectedwhen jointly scaling up HOC size and traffic rateQuick answer AdaptSizersquos improvement over produc-tion caching systems remains constant for larger HOCsizes We consider the OHR when jointly scaling theHOC size and the traffic rate by up 128x (153 GiB HOCsize) This is done by splitting a production trace into128 non-overlapping segments and replaying all 128 seg-ments concurrently We find that the OHR remains ap-proximately constant as we scale up the system and thatAdaptSize achieves similar OHR improvements as underthe original 12 GiB HOC configuration

What about AdaptSizersquos overhead Quick answerAdaptSizersquos throughput is comparable to existing produc-tion systems and AdaptSizersquos memory overhead is reason-ably small AdaptSize is build on top of Varnish whichfocuses on high concurrency and simplicity In Figure 12we compare the throughput (bytes per second of satisfiedrequests) of AdaptSize to an unmodified Varnish systemWe use two micro experiments The first benchmarks thehit request path (100 OHR scenario) to verify that thereis indeed no overhead for cache hits (see section 42)The second benchmarks the miss request path (0 OHRscenario) to assess the worst-case overhead due to theadmission decision

We replay one million requests and configure differentconcurrency levels via the number of client threads Notethat a client thread does not represent an individual user(Section 53) The results are based on 50 repetitions


0

10

20

0 100 200 300Concurrency ( client threads)

Thro

ughp

ut (G

bps)

AdaptSizeVarnish

(a) 100 OHR scenario

0

5

10


Thro

ughp

ut (G

bps)

AdaptSizeVarnish

(b) 0 OHR scenario

Figure 12 Comparison of the throughput of AdaptSizeand Varnish in micro experiments with (a) 100 OHR and(b) 0 OHR Scenario (a) stress tests the hit request pathand shows that there is no difference between AdaptSizeand Varnish Scenario (b) stress tests the miss requestpath (every request requires an admission decision) andshows that the throughput of AdaptSize and Varnish isvery close (within confidence intervals)

Figure 12a shows that the application throughput ofAdaptSize and Varnish are indistinguishable in the 100OHR scenario Both systems achieve a peak through-put of 175 Gbs for 50 clients threads Due to lockcontention the throughput of both systems decreases toaround 15 Gbs for 100-300 clients threads Figure 12bshows that the application throughput of both systems inthe 0 OHR scenario is very close and always withinthe 95 confidence interval

The memory overhead of AdaptSize is small The mem-ory overhead comes from the request statistics needed forAdaptSizersquos tuning model Each entry in this list describesone object (size request count hash) which requires lessthan 40 bytes The maximum length of this list acrossall experiments is 15 million objects (58 MiB) whichalso agrees with the memory high water mark (VmHWM)reported by the Kernel for AdaptSizersquos tuning process

62 Comparison with research systemsWe have seen that AdaptSize performs very well againstproduction systems We now ask the following

How does AdaptSize compare with researchcaching systems which involve more sophisticated ad-mission and eviction policies Quick answer Adapt-Size improves by 33-46 over state-of-the-art researchcaching system We use the simulation evaluation setupexplained in Section 52 with eight systems from Table 2which are selected with the criteria of having an efficientconstant-time implementation Four of the eight systemsuse a recency and frequency trade-off with fixed weightsbetween recency and frequency Another three systems(ending with ldquo++rdquo) use sophisticated recency and fre-quency trade-offs with variable weights which we hand-tuned to our traces to create optimistic variants5 The

5There are self-tuning variants of recency-frequency trade-offs suchas ARC [53] Unfortunately we could not test ARC itself because itslearning rule relies on the assumption of unit-sized object sizes

Figure 13 Comparison of AdaptSize to state-of-the-artresearch caching systems Most of these are sophisticatedadmission and eviction policies that combine recency andfrequency (striped blue bars) LRU-S is the only systemndash besides AdaptSize ndash that incorporates size AdaptSizeimproves the OHR by 33 over the next best system Poli-cies annotated by ldquo++rdquo are actually optimistic becausewe offline-tuned their parameters to the trace These re-sults are for the HK trace correspondings results for theUS trace are shown in Figure 5

remaining system is LRU-S [72] which uses size-awareeviction and admission with static parameters

Figure 13 shows the simulation results for a HOC ofsize 12 GiB on the HK trace We find that AdaptSizeachieves a 33 higher OHR than the second best systemwhich is SLRU++ Figure 5 shows the simulation resultsfor the US trace AdaptSize achieves a 46 higher OHRthan the second best system which is again SLRU++Note that SLRUrsquos performance heavily relies on offlineparameters as can be seen by the much smaller OHRof S4LRU which is a static-parameter variant of SLRUIn contrast AdaptSize achieves its superior performancewithout needing offline parameter optimization In con-clusion we find that AdaptSizersquos policies outperformsophisticated eviction and admission policies which donot depend on the object size

63 Robustness of alternative tuning methodsfor size-aware admission

So far we have seen that AdaptSize significantly improvesthe OHR over caching systems without size-aware admis-sion including production caching systems (Section 61)and research caching systems (Section 62) We nowfocus on different cache tuning methods for the size-aware admission parameter c (see the beginning of Sec-tion 4) Specifically we compare AdaptSize with hillclimbing (HillClimb) based on shadow caches (cf Sec-tion 3) HillClimb uses two shadow caches and we hand-optimized its parameters (interval of climbing steps stepsize) on our production traces We also compare to astatic size threshold (Static) where the value of this staticthreshold is offline optimized on our production tracesWe also compare to SIZE-OPT which tunes c based onoffline knowledge of the next one million requests Allfour policies are implemented on Varnish using the setupexplained in Section 53


00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(a) Randomized

00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(b) Adversarial

Figure 14 Comparison of cache tuning methods undertraffic mix changes We performed 50 randomized trafficmix changes (a) and 25 adversarial traffic mix changes(b) The boxes show the range of OHR from the 25-th tothe 75-th percentile among the 25-50 experiments Thewhiskers show the 5-th to the 95-th percentile

We consider two scenarios 1) randomized traffic mixchanges and 2) adversarial traffic mix changes A ran-domized traffic mix change involves a random selectionof objects which abruptly become very popular (similar toa flash crowd event) An adversarial traffic mix change in-volves frequently changing the traffic mix between classesthat require vastly different size-aware admission parame-ters (eg web video or download traffic) An exampleof an adversarial change is the case where objects largerthan the previously-optimal threshold suddenly becomevery popular

Is AdaptSize robust against randomized traffic mixchanges Quick answer AdaptSize performs within 95of SIZE-OPTrsquos OHR even for the worst 5 of experimentswhereas HillClimb and Static achieve only 47-54 ofSIZE-OPTrsquos OHR We create 50 different randomizedtraffic mix changes Each experiment consists of two partsThe first part is five million requests long and allows eachtuning method to converge to a stable configuration Thesecond part is ten million requests long and consists of50 production-trace requests and 50 of very popularobjects The very popular objects consist of a randomnumber of objects (between 200 and 1000) which arerandomly sampled from the trace

Figure 14a shows a boxplot of the OHR for eachcaching tuning method across the 50 experiments Theboxes indicate the 25-th and 75-th percentile the whiskersindicate the 5-th and 95-th percentile AdaptSize im-proves the OHR over HillClimb across every percentileby 9 on average and by more than 75 in five of the 50experiments AdaptSize improves the OHR over Staticacross every percentile by 30 on average and by morethan 100 in five of the 50 experiments Compared toSIZE-OPT AdaptSize achieves 95 of the OHR for allpercentiles

Is AdaptSize robust against adversarial traffic mixchanges Quick answer AdaptSize performs within 81of SIZE-OPTrsquos OHR even for the worst 5 of experimentswhereas HillClimb and Static achieve only 5-15 of SIZE-

OPTrsquos OHR Our experiment consists of 25 traffic mixchanges Each traffic mix is three million requests longand the optimal c parameter changes from 32-256 KiB to1-2 MiB then to 16-32 MiB and back again

Figure 14b shows a boxplot of the OHR for eachcaching tuning method across all 50 experiments Theboxes indicate the 25-th and 75-th percentile the whiskersindicate the 5-th and 95-th percentile AdaptSize im-proves the OHR over HillClimb across every percentileby 29 on average and by more than 75 in seven ofthe 25 experiments AdaptSize improves the OHR overStatic across every percentile by almost 3x on averageand by more than 10x in eleven of the 25 experimentsCompared to SIZE-OPT AdaptSize achieves 81 of theOHR for all percentiles

64 Side effects of Size-Aware AdmissionSo far our evaluation has focused on AdaptSizersquos improve-ment with regard to the OHR We evaluate AdaptSizersquosside-effects on the DC and on the clientrsquos request latency(cf Section 2) Specifically we compare AdaptSize toan unmodified Varnish system using the setup explainedin Section 53 Network latencies are emulated using theLinux kernel (tc-netem) We set a 30ms round-trip latencybetween client and CDN server and 100ms round-triplatency between CDN server and origin server We an-swer the following three questions on the CDN serverrsquosperformance

How much does AdaptSize affect the BHR of theDC Quick answer AdaptSize has a neutral effect onthe BHR of the DC The DCrsquos goal is to maximize theBHR which is achieved by a very large DC capacity [49]In fact compared to the DC the HOC has less than onthousandth the capacity Therefore changes to the HOChave little effect on the DCrsquos BHR

In our experiment we measure the DCrsquos byte hit ratio(BHR) from the origin server Figure 15a shows thatthere is no noticeable difference between the BHR underAdaptSize and under an unmodified Varnish

Does AdaptSize increase the load of the DCrsquos hard-disks Quick answer No In fact AdaptSize reducesthe average disk utilization by 20 With AdaptSize theHOC admits fewer large objects but caches many moresmall objects The DCrsquos request traffic therefore consistsof more requests to large objects and significantly fewerrequests to small objects

We measure the request size distribution at the DC andreport the corresponding histogram in Figure 16 Weobserve that AdaptSize decreases the number of cachemisses significantly for all object sizes below 256 KiBFor object sizes above 256 KiB we observe a slight in-crease in the number of cache misses Overall we findthat the DC has to serve 60 fewer requests with Adapt-Size but that the disks have to transfer a 30 higher


000

025

050

075

100

AdaptS

ize

Varnis

h

Byt

e H

it R

atio

(a) BHR

0

50

100

150

200

AdaptS

ize

Varnis

h

Dis

k IO

PS

(b) IOPS

0

20

40

60

AdaptS

ize

Varnis

h

Dis

k U

til [

]

(c) Utilization

0

200

400

600

AdaptS

ize

Varnis

h

Late

ncy

[ms]

(d) Latency

Figure 15 Evaluation of AdaptSizersquos side effects acrossten different sections of the US trace AdaptSize has aneutral impact on the byte hit ratio and leads to a 10reduction in the median number of IO operations goingto the disk and a 20 reduction in disk utilization

byte volume The average request size is also 4x largerwith AdaptSize which improves the sequentiality of diskaccess and thus makes the DCrsquos disks more efficient

To quantify the performance impact on the DCrsquos hard-disks we use iostat [31] Figure 15b shows that the aver-age rate of IO operations per second decreases by about10 Moreover Figure 15c shows that AdaptSize reducesthe diskrsquos utilization (the fraction of time with busy peri-ods) by more than 20 We conclude that the increasein byte volume is more than offset by the fact that Adapt-Size shields the DC from many small requests and alsoimproves the sequentiality of requests served by the DC

How much does AdaptSize reduce the request la-tency Quick answer AdaptSize reduces the requestlatency across all percentiles by at least 30

We measure the end-to-end request latency (time untilcompletion of a request) from the client server Figure 15dshows that AdaptSize reduces the median request latencyby 43 which is mostly achieved by the fast HOC an-swering a higher fraction of requests The figure alsoshows significant reduction of tail latency eg the 90-thand 99-th latency percentiles are reduced by more than30 This reduction in the tail latency is due to the DCrsquosimproved utilization factor which leads to a much smallernumber of outstanding requests which makes it easier toabsorb traffic bursts

7 Related WorkThe extensive related work in caching can be dividedinto two major lines of work research caching systems(Section 71) and cache tuning methods (Section 72)

71 Research caching systemsTable 2 surveys 33 caching systems proposed in the re-search literature between 1993 and 2016 We classifythese systems in terms of the per-request time complexitythe eviction and admission policies used the support fora concurrent implementation and the evaluation method

Not all of the 33 caching systems fulfill the low over-head design goal of Section 2 Specifically the complex-ity column in Table 2 shows that some proposals before2002 have a computational overhead that scales logarith-

0

05m

1m

lt128B 256KiB 1KiB 4KiB 16KiB 64KiB 256KiB 1MiB 4MiB gt4MiBObject size

requ

ests

to D

C

AdaptSizeVarnish

Figure 16 Comparison of the distribution of request sizesto the disk cache under a HOC running AdaptSize versusunmodified Varnish All object sizes below 256 KiBare significantly less frequent under AdaptSize whereaslarger objects are slightly more frequent

mically in the number of objects in the cache which isimpractical AdaptSize differs from these systems be-cause it has a constant complexity and a low synchroniza-tion overhead which we demonstrated by incorporatingAdaptSize into the Varnish production system

Of those caching systems that have a low overheadalmost none (except LRU-S and Threshold) incorporateobject sizes In particular these systems admit and evictobjects based only on recency frequency or a combi-nation thereof AdaptSize differs from these systemsbecause it is size aware which improves the OHR by33-46 (as shown in Section 62)

There are only three low-overhead caching systems thatare size aware Threshold [2] uses a static size thresholdwhich has to be determined in advance The correspond-ing Static policy in Section 63 performs poorly in our ex-periments LRU-S [72] uses size-aware admission whereit admits objects with probability 1size Unfortunatelythis static probability is too low6 AdaptSize achieves a61-78 OHR improvement over LRU-S (Figures 5 and13) The third system [56] also uses a static parameter andwas developed in parallel to AdaptSize AdaptSize differsfrom these caching systems by automatically adapting thesize-aware admission parameter over time

While most of these caching systems share our goal ofimproving the OHR an orthogonal line of research seeksto achieve superior throughput using concurrent cacheimplementations (compare the concurrent implementationcolumn in Table 2) AdaptSize also uses a concurrentimplementation and achieves throughput comparable toproduction systems (Section 61) AdaptSize differs fromthese systems by improving the OHR ndash without sacrificingcache throughput

The last column in Table 2 shows that most recentcaching systems are evaluated using prototype implemen-tations Likewise this work evaluates an actual imple-mentation of AdaptSize (Sections 5 and 6) through ex-periments in a dedicated data center We additionallyuse trace-driven simulations to compare to some of thosesystems that have only been used in simulations

6We also tested several variants of LRU-S We were either confrontedwith a cache tuning problem with no obvious solution (Section 32) or(by removing the admission component) with an OHR similar to LRU


Name Year Complexity Admission Policy Eviction Policy Concurrent Imp EvaluationAdaptSize 2016 O(1) size recency yes implementationCliffhanger [14] 2016 O(1) none recency no implementationBillion [47] 2015 O(1) none recency yes implementationBloomFilter [49] 2015 O(1) frequency recency no implementationSLRU [29] 2015 O(1) none recency+frequency no analysisLama [34] 2015 O(1) none recency no implementationDynaCache [13] 2015 O(1) none recency no implementationMICA [48] 2014 O(1) none recency yes implementationTLRU [20] 2014 O(1) frequency recency no simulationMemC3 [23] 2013 O(1) none recency yes implementationS4LRU [35] 2013 O(1) none recency+frequency no simulationCFLRU [62] 2006 O(1) none recency+cost no simulationClock-Pro [38] 2005 O(1) none recency+frequency yes simulationCAR [7] 2004 O(1) none recency+frequency yes simulationARC [53] 2003 O(1) none recency+frequency no simulationLIRS [39] 2002 O(1) none recency+frequency no simulationLUV [6] 2002 O(logn) none recency+size no simulationMQ [81] 2001 O(1) none recency+frequency no simulationPGDS [12] 2001 O(logn) none recency+frequency+size no simulationGD [40] 2001 O(logn) none recency+frequency+size no simulationLRU-S [72] 2001 O(1) size recency+size no simulationLRV [66] 2000 O(logn) none frequency+recency+size no simulationLFU-DA [5 70] 2000 O(1) none frequency no simulationLRFU [46] 1999 O(logn) none recency+frequency no simulationPSS [3] 1999 O(logn) frequency frequency+size no simulationGDS [11] 1997 O(logn) none recency+size no simulationHybrid [79] 1997 O(logn) none recency+frequency+size no simulationSIZE [1] 1996 O(logn) none size no simulationHyper [1] 1996 O(logn) none frequency+recency no simulationLog2(SIZE) [2] 1995 O(logn) none recency+size no simulationLRU-MIN [2] 1995 O(n) none recency+size no simulationThreshold [2] 1995 O(1) size recency no simulation2Q [41] 1994 O(1) frequency recency+frequency no simulationLRU-K [58] 1993 O(logn) none recency+frequency no implementation

Table 2 Historical overview of web caching systems

72 Cache tuning methodsWhile tuning for size-based admission is entirely newtuning has been used in other caching contexts such astuning for the optimal balance between recency and fre-quency [41 53 46 39 81 7 9] and for the allocation ofcapacity to cache partitions [14 34 13 69]

In these other contexts the most common tuning ap-proach is hill climbing with shadow caches [41 53 46 3981 7 14] Section 32 discusses why this approach oftenperforms poorly when tuning size-aware admission andSection 6 provides corresponding experimental evidence

Another method involves a prediction model togetherwith a global search algorithm The most widely usedprediction model is the calculation of stack distances [514 78 77] which has been recently used as an alternativeto shadow caches [69 13 69] Unfortunately the stackdistance model is not suited to optimizing the parametersof an admission policy since each admission parameterleads to a different request sequence and thus a differentstack distance distribution that needs to be recalculated

Many cache models have been studied in the CS the-ory community [44 30 15 52 10 33 17 75 25 2436 19 68 16 37 61 65 28 80 59 27 50 8 29 9]Unfortunately all these models assume unit-sized objects

AdaptSizersquos Markov model allows to model size-awareadmission and variable-sized objects

8 ConclusionAdaptSize is a new caching system for the hot objectcache in CDN servers The power of AdaptSize stemsfrom a size-aware admission policy that is continuouslyoptimized using a new Markov model of the HOC Inexperiments with Akamai production traces we showthat AdaptSize vastly improves the OHR over both state-of-the-art production systems and research systems Wealso show that our implementation of AdaptSize is robustand scalable and improves the DCrsquos disk utilization

As more diverse applications with richer content mi-grate onto the Internet future CDNs will experience evengreater variability in request patterns and object sizes Webelieve that AdaptSize and its underlying mathematicalmodel will be valuable in addressing this challenge

9 AcknowledgmentsWe thank Jens Schmitt Sebastian Michel our shepherdMahesh Balakrishnan and our anonymous reviewers fortheir valuable feedback

This research is supported in part by a Google FacultyAward and NSF grants CNS-1413998 CMMI-1538204CMMI-1334194 and XPS-1629444


AppendixA Proof of Theorem 1The result in Theorem 1 is achieved by solving the Markovchain shown in Figure 8

The key challenge when solving this chain is that thelength of the LRU list changes over time We solve thisby using a mathematical convergence result

We consider a fixed object i and a fixed size-awareadmission parameter c Let ` denote the length of theLRU list Now the Markov chain has `+1 states one foreach position in the list and one to represent the object isout of the cache as shown below

Over time ` changes as either larger or small objectspopulate the cache However what remains constant isthe expected time for an object to get evicted (if it is notrequested again) as this time only depends on the overalladmission rate (ie the size-aware admission parameterc) which is independent of ` Using this insight wemodify the Markov chain to increase the push-down ratemicro by a factor of ` now the expected time to traverse fromposition 1 to `+1 (without new requests) is constant at1micro

We now solve the Markov chain for a fixed ` andobtain the limiting probability πi of each position i isin0 ` `+ 1 Using the πi we can now derive thelimiting probability of being ldquoinrdquo the cache πin = sum

`i=0 πi

which can be algebraically simplified to

πin = 1minus

(`

`+rimicro

)`

eminussic +(

``+rimicro

)`minus eminussic

(`

`+rimicro

)`

We observe that the πin quickly converges in ` nu-merically convergence happens around ` gt 100 In oursimulations the cache typically holds many more objectsthan 100 simultaneously Therefore it is reasonable toalways use the converged result `rarrinfin We formally solvethis limit for πin and obtain the closed-form solution ofthe long-term probability that object i is present in thecache as stated in Theorem 1

We remark that our convergence result uses a similarintuition as recent studies on equal-sized objects [60 27]which is that the time it takes an object to get from po-sition 1 to `+ 1 (if there are no further requests to it)converges to a constant in a LRU cache What makes

AdaptSizersquos model different from these models is that weconsider size-aware admission and variable object sizes

References[1] ABRAMS M STANDRIDGE C R ABDULLA

G FOX E A AND WILLIAMS S Removalpolicies in network caches for World-Wide Webdocuments In ACM SIGCOMM (1996) pp 293ndash305

[2] ABRAMS M STANDRIDGE C R ABDULLAG WILLIAMS S AND FOX E A CachingProxies Limitations and Potentials Tech repVirginia Polytechnic Institute amp State UniversityBlacksburgh VA 1995

[3] AGGARWAL C WOLF J L AND YU P SCaching on the world wide web IEEE Transac-tions on Knowledge and Data Engineering 11 1(1999) 94ndash107

[4] ALMASI G CASCAVAL C AND PADUA D ACalculating stack distances efficiently In ACM SIG-PLAN Notices (2002) vol 38 pp 37ndash43

[5] ARLITT M CHERKASOVA L DILLEY JFRIEDRICH R AND JIN T Evaluating contentmanagement techniques for web proxy caches Per-formance Evaluation Review 27 4 (2000) 3ndash11

[6] BAHN H KOH K NOH S H AND LYUL SEfficient replacement of nonuniform objects in webcaches IEEE Computer 35 6 (2002) 65ndash73

[7] BANSAL S AND MODHA D S CAR Clockwith adaptive replacement In USENIX FAST (2004)vol 4 pp 187ndash200

[8] BERGER D S GLAND P SINGLA S ANDCIUCU F Exact analysis of TTL cache networksPerform Eval 79 (2014) 2 ndash 23 Special IssuePerformance 2014

[9] BERGER D S HENNINGSEN S CIUCU FAND SCHMITT J B Maximizing cache hit ratiosby variance reduction ACM SIGMETRICS Perfor-mance Evaluation Review 43 2 (2015) 57ndash59

[10] BURVILLE P AND KINGMAN J On a model forstorage and search Journal of Applied Probability(1973) 697ndash701

[11] CAO P AND IRANI S Cost-aware WWW proxycaching algorithms In USENIX symposium on In-ternet technologies and systems (1997) vol 12pp 193ndash206


[12] CHERKASOVA L AND CIARDO G Role of ag-ing frequency and size in web cache replacementpolicies In High-Performance Computing and Net-working (2001) pp 114ndash123

[13] CIDON A EISENMAN A ALIZADEH M ANDKATTI S Dynacache dynamic cloud caching InUSENIX HotCloud (2015)

[14] CIDON A EISENMAN A ALIZADEH M ANDKATTI S Cliffhanger Scaling performance cliffsin web memory caches In USENIX NSDI (2016)pp 379ndash392

[15] COFFMAN E G AND DENNING P J Operatingsystems theory Prentice-Hall 1973

[16] COFFMAN E G AND JELENKOVIC P Perfor-mance of the move-to-front algorithm with Markov-modulated request sequences Operations ResearchLetters 25 (1999) 109ndash118

[17] DAN A AND TOWSLEY D An approximateanalysis of the LRU and FIFO buffer replacementschemes In ACM SIGMETRICS (1990) pp 143ndash152

[18] DILLEY J MAGGS B M PARIKH J PROKOPH SITARAMAN R K AND WEIHL W E Glob-ally distributed content delivery IEEE Internet Com-puting 6 5 (2002) 50ndash58

[19] DOBROW R P AND FILL J A The move-to-front rule for self-organizing lists with Markov de-pendent requests In Discrete Probability and Algo-rithms Springer 1995 pp 57ndash80

[20] EINZIGER G AND FRIEDMAN R Tinylfu Ahighly efficient cache admission policy In IEE Eu-romicro PDP (2014) pp 146ndash153

[21] ELAARAG H ROMANO S AND COBB JWeb Proxy Cache Replacement Strategies Simu-lation Implementation and Performance Evalua-tion Springer Briefs in Computer Science SpringerLondon 2013

[22] ERAmdashTRENDS C V G I T F T Z ANDANALYSIS CISCO VNI global IP traffic fore-cast The zettabyte eramdashtrends and analysis May2015 Available at httpgooglwxuvVk ac-cessed 091216

[23] FAN B ANDERSEN D G AND KAMINSKY MMemC3 Compact and concurrent memcache withdumber caching and smarter hashing In USENIXNSDI (2013) pp 371ndash384

[24] FILL J A AND HOLST L On the distributionof search cost for the move-to-front rule RandomStructures amp Algorithms 8 (1996) 179ndash186

[25] FLAJOLET P GARDY D AND THIMONIER LBirthday paradox coupon collectors caching algo-rithms and self-organizing search Discrete AppliedMathematics 39 (1992) 207ndash229

[26] FOFACK N C DEHGHAN M TOWSLEY DBADOV M AND GOECKEL D L On the perfor-mance of general cache networks In VALUETOOLS(2014) pp 106ndash113

[27] FRICKER C ROBERT P AND ROBERTS J Aversatile and accurate approximation for LRU cacheperformance In ITC (2012) p 8

[28] GALLO M KAUFFMANN B MUSCARIELLOL SIMONIAN A AND TANGUY C Performanceevaluation of the random replacement policy fornetworks of caches In ACM SIGMETRICS PER-FORMANCE (2012) pp 395ndash396

[29] GAST N AND VAN HOUDT B Transient andsteady-state regime of a family of list-based cachereplacement algorithms In ACM SIGMETRICS(2015) pp 123ndash136

[30] GELENBE E A unified approach to the evalua-tion of a class of replacement algorithms IEEETransactions on Computers 100 (1973) 611ndash618

[31] GODARD S Iostat 2015 Available at httpgooglJZmbUp accessed 091216

[32] GRAZIANO P Speed up your web site with VarnishLinux Journal 2013 227 (2013) 4

[33] HENDRICKS W The stationary distribution ofan interesting Markov chain Journal of AppliedProbability (1972) 231ndash233

[34] HU X WANG X LI Y ZHOU L LUO YDING C JIANG S AND WANG Z LAMA Op-timized locality-aware memory allocation for key-value cache In USENIX ATC (2015) pp 57ndash69

[35] HUANG Q BIRMAN K VAN RENESSE RLLOYD W KUMAR S AND LI H C An anal-ysis of Facebook photo caching In ACM SOSP(2013) pp 167ndash181

[36] JELENKOVIC P R Asymptotic approximation ofthe move-to-front search cost distribution and least-recently used caching fault probabilities The Annalsof Applied Probability 9 (1999) 430ndash464


[37] JELENKOVIC P R AND RADOVANOVIC ALeast-recently-used caching with dependent re-quests Theoretical computer science 326 (2004)293ndash327

[38] JIANG S CHEN F AND ZHANG X CLOCK-Pro An effective improvement of the clock replace-ment In USENIX ATC (2005) pp 323ndash336

[39] JIANG S AND ZHANG X LIRS an efficientlow inter-reference recency set replacement policyto improve buffer cache performance ACM SIG-METRICS 30 1 (2002) 31ndash42

[40] JIN S AND BESTAVROS A GreedyDual webcaching algorithm exploiting the two sources oftemporal locality in web request streams ComputerCommunications 24 (2001) 174ndash183

[41] JOHNSON T AND SHASHA D 2Q A low over-head high performance buffer management replace-ment algorithm In VLDB (1994) pp 439ndash450

[42] KAMP P-H Varnish notes from the architect 2006Available at httpswwwvarnish-cache

orgdocstrunkphknoteshtml accessed091216

[43] KAMP P-H Varnish LRU architecture June 2007Available at httpswwwvarnish-cache

orgtracwikiArchitectureLRU accessed091216

[44] KING W F Analysis of demand paging algorithmsIn IFIP Congress (1) (1971) pp 485ndash490

[45] LE BOUDEC J-Y MCDONALD D ANDMUNDINGER J A generic mean field convergenceresult for systems of interacting objects In Quantita-tive Evaluation of Systems (2007) IEEE pp 3ndash18

[46] LEE D CHOI J KIM J-H NOH S H MINS L CHO Y AND KIM C S On the existenceof a spectrum of policies that subsumes the leastrecently used (LRU) and least frequently used (LFU)policies In ACM SIGMETRICS (1999) vol 27pp 134ndash143

[47] LI S LIM H LEE V W AHN J H KALIAA KAMINSKY M ANDERSEN D G SEONGILO LEE S AND DUBEY P Architecting toachieve a billion requests per second throughputon a single key-value store server platform In ACMISCA (2015) pp 476ndash488

[48] LIM H HAN D ANDERSEN D G ANDKAMINSKY M MICA A holistic approach tofast in-memory key-value storage In USENIX NSDI(2014) pp 429ndash444

[49] MAGGS B M AND SITARAMAN R K Algorith-mic nuggets in content delivery ACM SIGCOMMCCR 45 (2015) 52ndash66

[50] MARTINA V GARETTO M AND LEONARDI EA unified approach to the performance analysis ofcaching systems In IEEE INFOCOM (2014)

[51] MATTSON R L GECSEI J SLUTZ D R ANDTRAIGER I L Evaluation techniques for storagehierarchies IBM Systems journal 9 2 (1970) 78ndash117

[52] MCCABE J On serial files with relocatable recordsOperations Research 13 (1965) 609ndash618

[53] MEGIDDO N AND MODHA D S ARC A self-tuning low overhead replacement cache In USENIXFAST (2003) vol 3 pp 115ndash130

[54] Modern network design November 2016 Avail-able at httpswwwfastlycomproducts

modern-network-design accessed 021717

[55] MOTWANI R AND RAGHAVAN P Randomizedalgorithms Chapman amp HallCRC 2010

[56] NEGLIA G CARRA D FENG M JANARD-HAN V MICHIARDI P AND TSIGKARI DAccess-time aware cache algorithms In IEEE ITC(2016) vol 1 pp 148ndash156

[57] NYGREN E SITARAMAN R K AND SUNJ The Akamai Network A platform for high-performance Internet applications ACM SIGOPSOperating Systems Review 44 3 (2010) 2ndash19

[58] OrsquoNEIL E J OrsquoNEIL P E AND WEIKUMG The LRU-K page replacement algorithm fordatabase disk buffering ACM SIGMOD 22 2(1993) 297ndash306

[59] OrsquoNEIL E J OrsquoNEIL P E AND WEIKUM GAn optimality proof of the LRU-K page replacementalgorithm JACM 46 (1999) 92ndash112

[60] OSOGAMI T A fluid limit for a cache algorithmwith general request processes Advances in AppliedProbability 42 3 (2010) 816ndash833

[61] PANAGAKIS A VAIOS A AND STAVRAKAKISI Approximate analysis of LRU in the case of shortterm correlations Computer Networks 52 (2008)1142ndash1152

[62] PARK S-Y JUNG D KANG J-U KIM J-S AND LEE J CFLRU a replacement algorithmfor flash memory In ACMIEEE CASES (2006)pp 234ndash241


[63] PETRUSHEV P P AND POPOV V A Rationalapproximation of real functions vol 28 CambridgeUniversity Press 2011

[64] POSIK P HUYER W AND PAL L A compari-son of global search algorithms for continuous blackbox optimization Evolutionary computation 20 4(2012) 509ndash541

[65] PSOUNIS K ZHU A PRABHAKAR B ANDMOTWANI R Modeling correlations in web tracesand implications for designing replacement policiesComputer Networks 45 (2004) 379ndash398

[66] RIZZO L AND VICISANO L Replacement poli-cies for a proxy cache IEEEACM TON 8 (2000)158ndash170

[67] ROCCA E Running Wikipediaorg June 2016Available at httpswwwmediawikiorg

wikiFileWMF_Traffic_Varnishcon_2016

pdf accessed 091216

[68] RODRIGUES E R The performance of the move-to-front scheme under some particular forms ofMarkov requests Journal of applied probability(1995) 1089ndash1102

[69] SAEMUNDSSON T BJORNSSON H CHOCKLERG AND VIGFUSSON Y Dynamic performanceprofiling of cloud caches In ACM SoCC (2014)pp 1ndash14

[70] SHAH K MITRA A AND MATANI D An O(1)algorithm for implementing the LFU cache evictionscheme Tech rep Stony Brook University 2010

[71] SITARAMAN R K KASBEKAR M LICHTEN-STEIN W AND JAIN M Overlay networks AnAkamai perspective In Advanced Content DeliveryStreaming and Cloud Services John Wiley amp Sons2014

[72] STAROBINSKI D AND TSE D Probabilisticmethods for web caching Perform Eval 46 (2001)125ndash137

[73] TANGE O Gnu parallel - the command-line powertool login The USENIX Magazine 36 1 (Feb2011) 42ndash47

[74] Alexa top sites on the web March 2016 httpwwwalexacomtopsites accessed 031616

[75] TSUKADA N HIRADE R AND MIYOSHI NFluid limit analysis of FIFO and RR caching forindependent reference model Perform Eval 69(Sept 2012) 403ndash412

[76] VELAZQUEZ F LYNGSTOslashL K FOG HEEN TAND RENARD J The Varnish Book for Varnish 40Varnish Software AS March 2016

[77] WALDSPURGER C A PARK N GARTHWAITEA AND AHMAD I Efficient MRC constructionwith SHARDS In USENIX FAST (2015) pp 95ndash110

[78] WIRES J INGRAM S DRUDI Z HARVEYN J AND WARFIELD A Characterizing storageworkloads with counter stacks In USENIX OSDI(2014) pp 335ndash349

[79] WOOSTER R P AND ABRAMS M Proxycaching that estimates page load delays ComputerNetworks and ISDN Systems 29 8 (1997) 977ndash986

[80] YOUNG N E Online paging against adversariallybiased random inputs Journal of Algorithms 37(2000) 218ndash235

[81] ZHOU Y PHILBIN J AND LI K The multi-queue replacement algorithm for second level buffercaches In USENIX ATC (2001) pp 91ndash104


Introduction

Why HOC cache management is hard

Our contributions

HOC Design Goals

Rationale for AdaptSize

Why HOCs need size-aware admission

Why we need a new tuning method

The AdaptSize Caching System

AdaptSizes Markov chain tuning model

Integration with a production system

Evaluation Methodology

Production CDN request traces

Trace-based simulator

Prototype Evaluation Testbed

Empirical Evaluation

Comparison with production systems

Comparison with research systems

Robustness of alternative tuning methods for size-aware admission

Side effects of Size-Aware Admission

Related Work

Research caching systems

Cache tuning methods

Conclusion

Acknowledgments

Proof of Theorem 1

AdaptSize Orchestrating the Hot Object Memory Cachein a Content Delivery Network

Daniel S Berger1 Ramesh K Sitaraman2 and Mor Harchol-Balter31University of Kaiserslautern 2UMass Amherst amp Akamai Technologies 3Carnegie Mellon University

AbstractMost major content providers use content delivery net-works (CDNs) to serve web and video content to theirusers A CDN is a large distributed system of serversthat caches and delivers content to users The first-levelcache in a CDN server is the memory-resident Hot ObjectCache (HOC) A major goal of a CDN is to maximize theobject hit ratio (OHR) of its HOCs But the small sizeof the HOC the huge variance in the requested objectsizes and the diversity of request patterns make this goalchallenging

We propose AdaptSize the first adaptive size-awarecache admission policy for HOCs that achieves a highOHR even when object size distributions and requestcharacteristics vary significantly over time At the core ofAdaptSize is a novel Markov cache model that seamlesslyadapts the caching parameters to the changing requestpatterns Using request traces from one of the largestCDNs in the world we show that our implementation ofAdaptSize achieves significantly higher OHR than widely-used production systems 30-48 and 47-91 higherOHR than Nginx and Varnish respectively AdaptSizealso achieves 33-46 higher OHR than state-of-the-artresearch systems Further AdaptSize is more robust tochanging request patterns than the traditional tuning ap-proach of hill climbing and shadow queues studied inother contexts

1 IntroductionContent delivery networks (CDNs) [18] enhance perfor-mance by caching objects in servers close to users andrapidly delivering those objects to users A large CDNsuch as that operated by Akamai [57] serves trillionsof user requests a day from 170000+ servers locatedin 1500+ networks in 100+ countries around the worldCDNs carry the majority of todayrsquos Internet traffic and areexpected to carry almost two thirds within five years [22]

A CDN server employs two levels of caching a smallbut fast in-memory cache called the Hot Object Cache(HOC) and a large second-level Disk Cache (DC) Eachrequested object is first looked up in the HOC If absentit is looked up in the DC If also absent there the object isfetched over the WAN from the content providerrsquos origin

Serving requests from the HOC is much faster andmore efficient than serving from the DC Thus the goalof a CDN is to maximize the object hit ratio (OHR)which is the fraction of requests served from the HOC

0

25

50

75

100

1B 1KiB 1MiB 1GiBRequest size

CD

F

HKUS

Figure 1 The cumulative distribution for object sizes intwo Akamai production traces from Hong Kong and theUS Sizes vary by more than nine orders of magnitude

In this paper we show how to attain this goal in a robustefficient and scalable manner (see Section 2)

11 Why HOC cache management is hardHOC cache management entails two types of decisionsFirst the cache can decide whether or not to admit anobject (cache admission) Second the cache can decidewhich object to evict from the cache (cache eviction) ifthere is no space for a newly admitted object While cachemanagement is well studied in other contexts HOC cachemanagement poses the following new challenges

1) The HOC is subject to extreme variability in requestpatterns and object sizes CDNs serve multiple trafficclasses using a shared server infrastructure Such classesinclude web sites videos and interactive applicationsfrom thousands of content providers each class with itsown distinctive object size distributions and request pat-terns [57] Figure 1 shows the object size distribution ofrequests served by two Akamai production servers (one inthe US the other in Hong Kong) We find that object sizesspan more than nine orders of magnitude and that thelargest objects are often of the same order of magnitude asthe HOC size itself This extreme variability underscoresthe need for cache admission as placing one large objectin the HOC can result in the eviction of many small oneswhich can severely degrade the OHR (Section 3)

2) Prior academic research is largely inapplicable asit focuses on caching objects of similar sizes While theacademic literature on caching policies is extensive itfocuses on situations where objects are of the same sizeFurther prior work almost exclusively focuses on evictionpolicies ie all requested objects are admitted to thecache and space is only managed by eviction (Section 7)Thus there is little emphasis on cache admission in theprior literature even as we show that cache admission iskey in our context (Section 3)

3) Production systems implement a cache admis-sion scheme with a static threshold that performs sub-


00

02

04

06

08

1B 1KiB 1MiB 1GiB

Size threshold

OH

R

OPT

too high

too low


0

25

50

75

100

Request Size

CD

F


80 getadmitted

20dont getadmitted







00

02

04

06

08

1B 1KiB 1MiB 1GiB

Size threshold

OH

R

OPT


00

02

04

06

08

1B 1KiB 1MiB 1GiB

Size threshold

OH

R

OPT











SIZE-OPT

AdaptSize

Nginx

Varnish


+91+48
















04

06

08


OH

Rwebvideoweb

Time

AdaptSize

Static

HillClimb








00

02

04

06

1B 1KiB 1MiB 1GiB

Size Threshold

OH


same80 tileof size

OPT3OPT2OPT1






















N

sumi=1








OHR(c) =sum


sumNi=1 ri





00

02

04

06

08

1B 1KiB 1MiB 1GiB


OH

R

(a) HK trace

00

02

04

06

08

1B 1KiB 1MiB 1GiB


OH

R

(b) US trace

























SIZE-OPT

AdaptSize

Nginx

Varnish


+47+30








00

02

04

06

08


HOC size


OH

R

(a) HK trace

00

02

04

06


HOC size

OH

R


(b) US trace







0

10

20


Thro

ughp

ut (G

bps)

AdaptSizeVarnish


0

5

10


Thro

ughp

ut (G

bps)

AdaptSizeVarnish

(b) 0 OHR scenario













00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(a) Randomized

00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(b) Adversarial














000

025

050

075

100

AdaptS

ize

Varnis

h

Byt

e H

it R

atio

(a) BHR

0

50

100

150

200

AdaptS

ize

Varnis

h

Dis

k IO

PS

(b) IOPS

0

20

40

60

AdaptS

ize

Varnis

h

Dis

k U

til [

]

(c) Utilization

0

200

400

600

AdaptS

ize

Varnis

h

Late

ncy

[ms]

(d) Latency









0

05m

1m


requ

ests

to D

C

AdaptSizeVarnish


























`i=0 πi


πin = 1minus

(`

`+rimicro

)`

eminussic +(

``+rimicro

)`minus eminussic

(`

`+rimicro

)`















































































pdf accessed 091216
















Introduction


Our contributions

HOC Design Goals
















Related Work



Conclusion

Acknowledgments

Proof of Theorem 1

00

02

04

06

08

1B 1KiB 1MiB 1GiB

Size threshold

OH

R

OPT

too high

too low


0

25

50

75

100

Request Size

CD

F


80 getadmitted

20dont getadmitted







00

02

04

06

08

1B 1KiB 1MiB 1GiB

Size threshold

OH

R

OPT


00

02

04

06

08

1B 1KiB 1MiB 1GiB

Size threshold

OH

R

OPT











SIZE-OPT

AdaptSize

Nginx

Varnish


+91+48
















04

06

08


OH

Rwebvideoweb

Time

AdaptSize

Static

HillClimb








00

02

04

06

1B 1KiB 1MiB 1GiB

Size Threshold

OH


same80 tileof size

OPT3OPT2OPT1






















N

sumi=1








OHR(c) =sum


sumNi=1 ri





00

02

04

06

08

1B 1KiB 1MiB 1GiB


OH

R

(a) HK trace

00

02

04

06

08

1B 1KiB 1MiB 1GiB


OH

R

(b) US trace

























SIZE-OPT

AdaptSize

Nginx

Varnish


+47+30








00

02

04

06

08


HOC size


OH

R

(a) HK trace

00

02

04

06


HOC size

OH

R


(b) US trace







0

10

20


Thro

ughp

ut (G

bps)

AdaptSizeVarnish


0

5

10


Thro

ughp

ut (G

bps)

AdaptSizeVarnish

(b) 0 OHR scenario













00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(a) Randomized

00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(b) Adversarial














000

025

050

075

100

AdaptS

ize

Varnis

h

Byt

e H

it R

atio

(a) BHR

0

50

100

150

200

AdaptS

ize

Varnis

h

Dis

k IO

PS

(b) IOPS

0

20

40

60

AdaptS

ize

Varnis

h

Dis

k U

til [

]

(c) Utilization

0

200

400

600

AdaptS

ize

Varnis

h

Late

ncy

[ms]

(d) Latency









0

05m

1m


requ

ests

to D

C

AdaptSizeVarnish


























`i=0 πi


πin = 1minus

(`

`+rimicro

)`

eminussic +(

``+rimicro

)`minus eminussic

(`

`+rimicro

)`















































































pdf accessed 091216
















Introduction


Our contributions

HOC Design Goals
















Related Work



Conclusion

Acknowledgments

Proof of Theorem 1

SIZE-OPT

AdaptSize

Nginx

Varnish


+91+48
















04

06

08


OH

Rwebvideoweb

Time

AdaptSize

Static

HillClimb








00

02

04

06

1B 1KiB 1MiB 1GiB

Size Threshold

OH


same80 tileof size

OPT3OPT2OPT1






















N

sumi=1








OHR(c) =sum


sumNi=1 ri





00

02

04

06

08

1B 1KiB 1MiB 1GiB


OH

R

(a) HK trace

00

02

04

06

08

1B 1KiB 1MiB 1GiB


OH

R

(b) US trace

























SIZE-OPT

AdaptSize

Nginx

Varnish


+47+30








00

02

04

06

08


HOC size


OH

R

(a) HK trace

00

02

04

06


HOC size

OH

R


(b) US trace







0

10

20


Thro

ughp

ut (G

bps)

AdaptSizeVarnish


0

5

10


Thro

ughp

ut (G

bps)

AdaptSizeVarnish

(b) 0 OHR scenario













00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(a) Randomized

00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(b) Adversarial














000

025

050

075

100

AdaptS

ize

Varnis

h

Byt

e H

it R

atio

(a) BHR

0

50

100

150

200

AdaptS

ize

Varnis

h

Dis

k IO

PS

(b) IOPS

0

20

40

60

AdaptS

ize

Varnis

h

Dis

k U

til [

]

(c) Utilization

0

200

400

600

AdaptS

ize

Varnis

h

Late

ncy

[ms]

(d) Latency









0

05m

1m


requ

ests

to D

C

AdaptSizeVarnish


























`i=0 πi


πin = 1minus

(`

`+rimicro

)`

eminussic +(

``+rimicro

)`minus eminussic

(`

`+rimicro

)`















































































pdf accessed 091216
















Introduction


Our contributions

HOC Design Goals
















Related Work



Conclusion

Acknowledgments

Proof of Theorem 1

04

06

08


OH

Rwebvideoweb

Time

AdaptSize

Static

HillClimb








00

02

04

06

1B 1KiB 1MiB 1GiB

Size Threshold

OH


same80 tileof size

OPT3OPT2OPT1






















N

sumi=1








OHR(c) =sum


sumNi=1 ri





00

02

04

06

08

1B 1KiB 1MiB 1GiB


OH

R

(a) HK trace

00

02

04

06

08

1B 1KiB 1MiB 1GiB


OH

R

(b) US trace

























SIZE-OPT

AdaptSize

Nginx

Varnish


+47+30








00

02

04

06

08


HOC size


OH

R

(a) HK trace

00

02

04

06


HOC size

OH

R


(b) US trace







0

10

20


Thro

ughp

ut (G

bps)

AdaptSizeVarnish


0

5

10


Thro

ughp

ut (G

bps)

AdaptSizeVarnish

(b) 0 OHR scenario













00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(a) Randomized

00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(b) Adversarial














000

025

050

075

100

AdaptS

ize

Varnis

h

Byt

e H

it R

atio

(a) BHR

0

50

100

150

200

AdaptS

ize

Varnis

h

Dis

k IO

PS

(b) IOPS

0

20

40

60

AdaptS

ize

Varnis

h

Dis

k U

til [

]

(c) Utilization

0

200

400

600

AdaptS

ize

Varnis

h

Late

ncy

[ms]

(d) Latency









0

05m

1m


requ

ests

to D

C

AdaptSizeVarnish


























`i=0 πi


πin = 1minus

(`

`+rimicro

)`

eminussic +(

``+rimicro

)`minus eminussic

(`

`+rimicro

)`















































































pdf accessed 091216
















Introduction


Our contributions

HOC Design Goals
















Related Work



Conclusion

Acknowledgments

Proof of Theorem 1


















N

sumi=1








OHR(c) =sum


sumNi=1 ri





00

02

04

06

08

1B 1KiB 1MiB 1GiB


OH

R

(a) HK trace

00

02

04

06

08

1B 1KiB 1MiB 1GiB


OH

R

(b) US trace

























SIZE-OPT

AdaptSize

Nginx

Varnish


+47+30








00

02

04

06

08


HOC size


OH

R

(a) HK trace

00

02

04

06


HOC size

OH

R


(b) US trace







0

10

20


Thro

ughp

ut (G

bps)

AdaptSizeVarnish


0

5

10


Thro

ughp

ut (G

bps)

AdaptSizeVarnish

(b) 0 OHR scenario













00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(a) Randomized

00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(b) Adversarial














000

025

050

075

100

AdaptS

ize

Varnis

h

Byt

e H

it R

atio

(a) BHR

0

50

100

150

200

AdaptS

ize

Varnis

h

Dis

k IO

PS

(b) IOPS

0

20

40

60

AdaptS

ize

Varnis

h

Dis

k U

til [

]

(c) Utilization

0

200

400

600

AdaptS

ize

Varnis

h

Late

ncy

[ms]

(d) Latency









0

05m

1m


requ

ests

to D

C

AdaptSizeVarnish


























`i=0 πi


πin = 1minus

(`

`+rimicro

)`

eminussic +(

``+rimicro

)`minus eminussic

(`

`+rimicro

)`















































































pdf accessed 091216
















Introduction


Our contributions

HOC Design Goals
















Related Work



Conclusion

Acknowledgments

Proof of Theorem 1


















SIZE-OPT

AdaptSize

Nginx

Varnish


+47+30








00

02

04

06

08


HOC size


OH

R

(a) HK trace

00

02

04

06


HOC size

OH

R


(b) US trace







0

10

20


Thro

ughp

ut (G

bps)

AdaptSizeVarnish


0

5

10


Thro

ughp

ut (G

bps)

AdaptSizeVarnish

(b) 0 OHR scenario













00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(a) Randomized

00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(b) Adversarial














000

025

050

075

100

AdaptS

ize

Varnis

h

Byt

e H

it R

atio

(a) BHR

0

50

100

150

200

AdaptS

ize

Varnis

h

Dis

k IO

PS

(b) IOPS

0

20

40

60

AdaptS

ize

Varnis

h

Dis

k U

til [

]

(c) Utilization

0

200

400

600

AdaptS

ize

Varnis

h

Late

ncy

[ms]

(d) Latency









0

05m

1m


requ

ests

to D

C

AdaptSizeVarnish


























`i=0 πi


πin = 1minus

(`

`+rimicro

)`

eminussic +(

``+rimicro

)`minus eminussic

(`

`+rimicro

)`















































































pdf accessed 091216
















Introduction


Our contributions

HOC Design Goals
















Related Work



Conclusion

Acknowledgments

Proof of Theorem 1

0

10

20


Thro

ughp

ut (G

bps)

AdaptSizeVarnish


0

5

10


Thro

ughp

ut (G

bps)

AdaptSizeVarnish

(b) 0 OHR scenario













00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(a) Randomized

00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(b) Adversarial














000

025

050

075

100

AdaptS

ize

Varnis

h

Byt

e H

it R

atio

(a) BHR

0

50

100

150

200

AdaptS

ize

Varnis

h

Dis

k IO

PS

(b) IOPS

0

20

40

60

AdaptS

ize

Varnis

h

Dis

k U

til [

]

(c) Utilization

0

200

400

600

AdaptS

ize

Varnis

h

Late

ncy

[ms]

(d) Latency









0

05m

1m


requ

ests

to D

C

AdaptSizeVarnish


























`i=0 πi


πin = 1minus

(`

`+rimicro

)`

eminussic +(

``+rimicro

)`minus eminussic

(`

`+rimicro

)`















































































pdf accessed 091216
















Introduction


Our contributions

HOC Design Goals
















Related Work



Conclusion

Acknowledgments

Proof of Theorem 1

00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(a) Randomized

00

02

04

06

08

OPT

AdaptS

ize

HillClim

bStat

ic

Obj

ect H

it R

atio

(b) Adversarial














000

025

050

075

100

AdaptS

ize

Varnis

h

Byt

e H

it R

atio

(a) BHR

0

50

100

150

200

AdaptS

ize

Varnis

h

Dis

k IO

PS

(b) IOPS

0

20

40

60

AdaptS

ize

Varnis

h

Dis

k U

til [

]

(c) Utilization

0

200

400

600

AdaptS

ize

Varnis

h

Late

ncy

[ms]

(d) Latency









0

05m

1m


requ

ests

to D

C

AdaptSizeVarnish


























`i=0 πi


πin = 1minus

(`

`+rimicro

)`

eminussic +(

``+rimicro

)`minus eminussic

(`

`+rimicro

)`















































































pdf accessed 091216
















Introduction


Our contributions

HOC Design Goals
















Related Work



Conclusion

Acknowledgments

Proof of Theorem 1

000

025

050

075

100

AdaptS

ize

Varnis

h

Byt

e H

it R

atio

(a) BHR

0

50

100

150

200

AdaptS

ize

Varnis

h

Dis

k IO

PS

(b) IOPS

0

20

40

60

AdaptS

ize

Varnis

h

Dis

k U

til [

]

(c) Utilization

0

200

400

600

AdaptS

ize

Varnis

h

Late

ncy

[ms]

(d) Latency









0

05m

1m


requ

ests

to D

C

AdaptSizeVarnish


























`i=0 πi


πin = 1minus

(`

`+rimicro

)`

eminussic +(

``+rimicro

)`minus eminussic

(`

`+rimicro

)`















































































pdf accessed 091216
















Introduction


Our contributions

HOC Design Goals
















Related Work



Conclusion

Acknowledgments

Proof of Theorem 1


















`i=0 πi


πin = 1minus

(`

`+rimicro

)`

eminussic +(

``+rimicro

)`minus eminussic

(`

`+rimicro

)`















































































pdf accessed 091216
















Introduction


Our contributions

HOC Design Goals
















Related Work



Conclusion

Acknowledgments

Proof of Theorem 1































































pdf accessed 091216
















Introduction


Our contributions

HOC Design Goals
















Related Work



Conclusion

Acknowledgments

Proof of Theorem 1

adaptsize: orchestrating the hot object memory …...adaptsize: orchestrating the hot object memory...

Documents