Energy Proportionality and WorkloadConsolidation for Latency-critical Applications

George Prekas1 Mia Primorac1 Adam Belay2 Christos Kozyrakis2 Edouard Bugnion1

1EPFL 2Stanford

AbstractEnergy proportionality and workload consolidation are im-portant objectives towards increasing efficiency in large-scale datacenters Our work focuses on achieving these goalsin the presence of applications with micros-scale tail latency re-quirements Such applications represent a growing subset ofdatacenter workloads and are typically deployed on dedi-cated servers which is the simplest way to ensure low taillatency across all loads Unfortunately it also leads to lowenergy efficiency and low resource utilization during the fre-quent periods of medium or low load

We present the OS mechanisms and dynamic controlneeded to adjust core allocation and voltagefrequency set-tings based on the measured delays for latency-critical work-loads This allows for energy proportionality and frees themaximum amount of resources per server for other back-ground applications while respecting service-level objec-tives Monitoring hardware queue depths allows us to detectincreases in queuing latencies Carefully coordinated adjust-ments to the NICrsquos packet redirection table enable us to re-assign flow groups between the threads of a latency-criticalapplication in milliseconds without dropping or reorderingpackets We compare the efficiency of our solution to thePareto-optimal frontier of 224 distinct static configurationsDynamic resource control saves 44ndash54 of processor en-ergy which corresponds to 85ndash93 of the Pareto-optimalupper bound Dynamic resource control also allows back-ground jobs to run at 32ndash46 of their standalone through-put which corresponds to 82ndash92 of the Pareto bound

Categories and Subject Descriptors D4 [OS] Scheduling

General Terms Design Performance

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page Copyrights for components of this work owned by others than ACMmust be honored Abstracting with credit is permitted To copy otherwise or republishto post on servers or to redistribute to lists requires prior specific permission andor afee Request permissions from permissionsacmorgSoCC rsquo15 August 27-29 2015 Kohala Coast HI USAccopy 2015 ACM ISBN 978-1-4503-3651-21508 $1500

DOI httpdxdoiorg10114528067772806848

Keywords Latency-critical applications Microsecond-scalecomputing Energy proportionality Workload consolidation

1 IntroductionWith the waste in the power delivery and cooling systemslargely eliminated [4] researchers are now focusing on theefficient use of the tens of thousands of servers in large-scaledatacenters The first goal is energy proportionality whichminimizes the energy consumed to deliver a particular work-load [3 29] Hardware enhancements primarily in dynamicvoltagefrequency scaling (DVFS) and idle modes in modernprocessors [22 45] provide a foundation for energy propor-tionality The second goal is workload consolidation whichraises server utilization and minimizes the number of serversneeded for a particular set of workloads [14 46 50] Ad-vances in cluster management [11 32] and server consoli-dation using virtual machines or container systems have alsoplayed an important role in datacenter efficiency by enablingmultiple workloads to be consolidated on each machine

The two goals map to distinct economic objectives en-ergy proportionality reduces operational expenses (opex)whereas workload consolidation reduces capital expenses(capex) Since capital costs often dominate the datacenterrsquostotal cost of ownership (TCO) consolidation is highly desir-able Nevertheless it is not always possible eg when oneapplication consumes the entirety of a given resource egmemory In such cases energy proportionality is a necessity

Datacenter applications have evolved with the advent ofweb-scale applications User-facing large-scale applicationsnow rely extensively on high fan-out patterns between low-latency services [9] Such services exhibit low per-requestlatencies (a handful of micros for a key-value store) have strictservice-level objectives (SLO eg lt 500micros at the 99th per-centile) and must sustain massive request rates and highclient fan-in connections Although functionally simplethese services are deployed on thousands of servers of large-scale datacenters Because of their importance there areseveral proposals to improve their throughput or latency us-ing user-level networking stacks [20 31] networking toolk-its [16 18 44] or dataplane operating systems [6 43]

Such latency-critical services are challenging to run in ashared infrastructure environment They are particularly sen-sitive to resource allocation and frequency settings and theysuffer frequent tail latency violations when common powermanagement or consolidation approaches are used [26 27]As a result operators typically deploy them on dedicatedservers running in polling mode forgoing opportunities forworkload consolidation and for energy-proportional com-puting at below-peak utilization levels

To gain a principled understanding of the challenges forresource management in the presence of latency-critical ser-vices we performed an exhaustive analysis of static con-figurations for a latency-critical service (memcached [35])running on a modern server We explored up to 224 possi-ble settings for core allocation use of hyperthreads DVFSfrequencies and Turbo Boost While our experiments usea single application the implications have broad applica-bility because memcached has aggressive latency require-ments short service times and a large number of indepen-dent clients that are common among many latency-criticalapplications Our experiments reveal that there is an inher-ent tradeoff for any given static configuration between themaximum throughput and the overall efficiency when oper-ating below peak load Furthermore the experiments reveala Pareto-optimal frontier [41] in the efficiency of static con-figurations at any given load level which allows for close tolinear improvements in energy-proportionality and workloadconsolidation factors

Based on these insights we introduce a set of novel dy-namic resource management mechanisms and policies thatimprove energy proportionality and workload consolidationin the presence of latency-sensitive applications We inte-grate these techniques in IX a state-of-the-art dataplane op-erating system that optimizes both throughput and latencyfor latency-critical workloads [6]

Fig 1 illustrates our approach the dynamic controller(ixcp) adjusts the number of cores allocated to a latency-sensitive application running on top of IX and the DVFS set-tings for these cores The remaining cores can be placed inidle modes to reduce power consumption or can be safelyused to run background tasks The controller builds upontwo key mechanisms The first mechanism detects back-log and increases in queuing delays that exceed the allow-able upper bound for the specific latency-critical applica-tion It monitors CPU utilization and signals required adjust-ments in resource allocation The second mechanism quicklymigrates both network and application processing betweencores transparently and without dropping or reordering pack-ets

We evaluated our system with two control policies thatoptimize for energy proportionality and workload consolida-tion respectively A policy determines how resources (coreshyperthreads and DVFS settings) are adjusted to reduce un-derutilization or to restore violated SLO The two policies

IXCP

C C

ho

st

ap

plicati

on

C

Dune Linux

IX

latency-critical

workload

C

background

task

C

+ -

kern

el

user

C

dvfs

Figure 1 Dynamic resource controls with IX for a workloadconsolidation scenario with a latency-sensitive application(eg memcached) and a background batch task (eg an-alytics) The controller ixcp partitions cores among theapplications and adjusts the processorrsquos DVFS settings

are derived from the exhaustive analysis of the 224 staticconfigurations For the platform studied (a Xeon E5-2665)we conclude that for best energy proportionality (i) we startwith the lowest clock rate and allocate additional cores to thelatency-critical task as its load grows using at first only onehyperthread per core (ii) we enable the second hyperthreadonly when all cores are in use and finally (iii) we increasethe clock rate for the cores running the latency-critical taskFor best consolidation (i) we start at the nominal clock rateand add cores with both hyperthreads enabled as load in-creases and (ii) finally enable Turbo Boost as a last resort

This paper makes the following contributions We develop techniques for fine-grain resource manage-ment for latency-critical workloads This includes mecha-nisms for detection of load changes in sub-second timescalesand for rebalancing flow-groups between cores withoutcausing packet drops or reordered deliveries In our experi-ments this mechanism completes in less than 2 ms 95 ofthe time and in at most 35 ms We provide a methodology that uses the Pareto frontier of aset of static configurations to derive resource allocation poli-cies We derive two policies for memcached that respectthe SLO constraints of a latency-critical in-memory key-value store These policies lead to 44ndash54 energy sav-ings for a variety of load patterns and enable workload con-solidation with background jobs executing at 32ndash46 oftheir peak throughput on a standalone machine These gainsare close to the Pareto-optimal bounds for the server usedwithin 88 and 82ndash92 respectively We demonstrate that precise and fine-grain control ofcores hyperthreads and DVFS has a huge impact on bothenergy proportionality and workload consolidation espe-

cially when load is highly variable DVFS is a necessary butinsufficient mechanism to control latency-critical applica-tions that rely on polling

The rest of this paper is organized as follows sect2 describesthe experimental setup sect3 characterizes resource usage us-ing static configurations and derives insights for dynamic al-location policies sect4 describes the design and implementa-tion of the mechanisms and policies for dynamic resourcemanagement in IX sect5 evaluates the impact of the dynamiccontroller for energy proportionality and workload consol-idation We discuss insights and open issues in sect6 relatedwork in sect7 and conclude in sect8

2 Workloads and Experimental SetupOur experimental setup consists of a cluster of 11 clientsand one server connected by a low-latency 10 GbE switch10 clients generate load and the 11th measures latency ofrequests to the latency-sensitive application The client ma-chines are a mix of Xeon E5-2637 35 Ghz and XeonE5-2650 26 Ghz The server is a Xeon E5-2665 24 Ghz with 256 GB of DRAM typical in datacenters Eachclient and server socket has 8 cores and 16 hyperthreads Allmachines are configured with one Intel x520 10 GbE NIC(82599EB chipset) Our baseline configuration in each ma-chine is an Ubuntu LTS 1404 distribution updated to the3161 Linux kernel Although the server has two socketsthe foreground and background applications run on a singleprocessor to avoid any NUMA effects We run the controlplane on the otherwise empty second socket but do not ac-count for its energy draw

Our primary foreground workload is memcached awidely deployed in-memory key-value store built on topof the libevent framework [35] It is frequently usedas a high-throughput low-latency caching tier in front ofpersistent database servers memcached is a network-bound application with threads spending over 75 of ex-ecution time in kernel mode for network processing whenusing Linux [26] It is a difficult application to scale be-cause the common deployments involve high connectioncounts for memcached servers and small-sized requestsand replies [2 39] Combined with latency SLOs in the fewhundreds of microseconds these characteristics make mem-cached a challenging workload for our study Any short-comings in the mechanisms and policies for energy propor-tionality and workload consolidation will quickly translateto throughput and latency problems In contrast workloadswith SLOs in the millisecond range are more forgivingFurthermore memcached has well-known scalability lim-itations [28] To improve its scalability we configure mem-cached with a larger hash table size (-o hashpower=20)and use a random replacement policy instead of the built-inLRU which requires a global lock This last change pro-vides memcached with similar multicore scalability as state-

of-the-art key-value stores such as MICA [28] We configurememcached similarly for Linux and IX

Our primary background application is a simple syntheticproxy for computationally intense batch processing applica-tions it encrypts a memory stream using SHA-1 in parallelthreads that each run in an infinite loop and report the aggre-gate throughput The workload streams through a 40 MB ar-ray which exceeds the capacity of the processorrsquos L3-cacheand therefore interferes (by design) with the memory sub-system performance of the latency-sensitive application

We use the mutilate load-generator to place a se-lected load on the foreground application in terms of re-quests per second (RPS) and measure response latency [25]Mutilate coordinates a large number of client threadsacross 10 machines to generate the desired RPS load andone additional unloaded client to measure latency for atotal of 2752 connections to the memcached server Weconfigure mutilate to generate load representative of theFacebook USR workload [2] this is a large-scale deploy-ment dominated by GET requests (99) with short keys(lt20B) and 2B values Each request is issued separately (nomultiget operations) However clients are permitted topipeline up to four requests per connection if needed to keepup with their target request rate We enhance mutilateto support dynamically changing load levels used in theseexperiments As the memcached server is multi-threadedwe also enhance the latency-measuring client to randomlychoose among 32 open connections for each request as toensure statistically significant measurements

We compute metrics as follows the unloaded client mea-sures latency by issuing one request at the time chosen ran-domly among its 32 open connections The SLO is spec-ified so that the 99th percentile of requests issued by thatunloaded client return in le 500micros Latency distributionachieved requests and power metrics for all graphs are re-ported at one-second intervals Energy consumption is mea-sured using the Intel RAPL counters [8] for the processor inquestion these provide an accurate model of the power drawof the CPU but ignore platforms effectsA note on Turbo Boost For any given throughput levelwe observe that the reported power utilization is stable forall CPU frequencies except for Turbo Boost When runningin Turbo Boost the temperature of the CPU gradually risesover a few minutes from 58C to 78C and with it the dissi-pated energy rises by 4 W for the same level of performanceThe experiments in sect31 run for a long time in Turbo Boostmode with a hot processor we therefore report those resultsas an energy band of 4 W

3 Understanding Dynamic Resource UsageThe purpose of dynamic resource allocation is to adapt tochanges of load of a latency-critical workload by adding orremoving resources as needed The remaining resources canbe placed in idle modes to save energy or used by other con-

0

20

40

60

80

100

0 2

Pow

er

(W)

Memcached RPS x 106 at SLO

(a) Energy Proportionality (Linux)

0

20

40

60

80

100

0 2 4 6 8

Memcached RPS x 106 at SLO

cfg - all others

cfg at Turbo mode

cfg w 8 cores + HT

cfg at 12 GHz

Pareto frontier

(b) Energy Proportionality (IX)

0

20

40

60

80

100

0 2

o

f P

eak f

or

Best-

eff

ort

Task

Memcached RPS x 106 at SLO

(c) Consolidation (Linux)

0

20

40

60

80

100

0 2 4 6 8

Memcached RPS x 106 at SLO

(d) Server Consolidation (IX)

Figure 2 Pareto efficiency for energy proportionality and workload consolidation for Linux 3161 and IX The Paretoefficiency is in red while the various static configurations are color-coded according to their distinctive characteristics

solidated applications In general dynamic control can beapplied to any resource including processor cores mem-ory and IO We focus on managing CPU cores becausethey are the largest contributor to power consumption anddraw significant amounts of energy even when lightly uti-lized [4 29] Moreover the ability to dynamically reassigncores between latency-critical and background applicationsis key to meet latency SLOs for the former and to make rea-sonable progress using underutilized resources for the latter

Dynamic resource control is a multi-dimensional prob-lem on current CPUs multi-core hyperthreading and fre-quency each separately impact the workloadrsquos performanceenergy consumption and consolidation opportunities sect31uses a Pareto methodology to evaluate this three-dimensionalspace and sect32 discusses the derived allocation policies

31 Pareto-Optimal Static ConfigurationsStatic resource configurations allow for controlled experi-ments to quantify the tradeoff between an applicationrsquos per-formance and the resources consumed Our approach limitsbias by considering many possible static configurations inthe three-dimensional space of core hyperthread and fre-quency For each static configuration we characterize themaximum load that meets the SLO (le 500micros99th per-

centile) we then measure the energy draw and throughputof the background job for all load levels up to the maximumload supported From this large data set we derive the setof meaningful static configurations and build the Pareto effi-ciency frontier [41] The frontier specifies for any possibleload level the optimal static configuration and the resultingminimal energy draw or maximum background throughputdepending on the scenario

Fig 2 presents the frontier for four experiments theenergy proportionality and workload consolidation scenar-ios for both Linux and IX The four graphs each plot theobjectivemdashwhich is either to minimize energy or maximizebackground throughputmdashas a function of the foregroundthroughput provided that the SLO is met Except for the redlines each line corresponds to a distinct static configurationof the system the green curves correspond to configurationat the minimal clock rate of 12 Ghz the blue curves use allavailable cores and hyperthreads other configurations are inblack In Turbo Boost mode the energy drawn is reported asa band since it depends on operating temperature (see sect2)

Finally the red line is the Pareto frontier which corre-sponds for any load level to the optimal result using any

0

20

40

60

80

100

0 2 4 6 8

Pow

er (

W)

Memcached RPS x 106 at SLO

Pareto IX

DVFS-only IX

Pareto Linux

DVFS-only Linux

Figure 3 Energy-proportionality comparison between thePareto-optimal frontier considering only DVFS adjustmentsand the full Pareto frontier considering core allocation hy-perthread allocations and frequency

of the static configurations available Each graph only showsthe static configurations that participate in the frontierEnergy proportionality We evaluate 224 distinct combi-nations from one to eight cores using consistently eitherone or two threads per core for 14 different DVFS levelsfrom 12 Ghz to 24 Ghz as well as Turbo Boost Fig 2a andFig 2b show the 43 and 32 static configurations (out of 224)that build the Pareto frontier for energy proportionality Thefigures confirm the intuition that (i) various static configura-tions have very different dynamic ranges beyond which theyare no longer able to meet the SLO (ii) each static config-uration draws substantially different levels of energy for thesame amount of work (iii) at the low-end of the curve manydistinct configurations operate at the minimal frequency of12 Ghz obviously with a different number of cores andthreads and contribute to the frontier these are shown ingreen in the Figure (iv) at the high-end of the range manyconfigurations operate with the maximum of 8 cores withdifferent frequencies including Turbo BoostConsolidation The methodology here is a little differentWe first characterize the background job and observe that itdelivers energy-proportional throughput up to 24 Ghz butthat Turbo Boost came at an energythroughput premiumConsequently we restrict the Pareto configuration space at24 Ghz the objective function is the throughput of the back-ground job expressed as a fraction of the throughput of thatsame job without any foreground application Backgroundjobs run on all cores that are not used by the foreground ap-plication Fig 2c and Fig 2d show the background through-put expressed as a fraction of the standalone throughput asa function of the foreground throughput provided that theforeground application meets the SLO as the foregroundapplication requires additional cores to meet the SLO thebackground throughput decreases proportionally

Linux vs IX Fig 2a and Fig 2b show that Linux and IXbehave differently (i) IX improves the throughput of Linuxby 48times (ii) IX can clearly take advantage of hyperthread-ing and has its best performance with configurations with16 threads on 8 cores In contrast Linux cannot take ad-vantage of hyperthreading and has its best performance with8 threads of control running on 8 cores (iii) the efficiencyof individual static configurations of Linux which relieson interrupts is much more subject to load than individ-ual static configurations of IX which relies on a pollingmodel As a consequence an improperly configured over-sized IX dataplane will be extremely energy-inefficient atlow-to-moderate loads (iv) the comparison of the Paretofrontiers shows that IXrsquos frontier always dominates the fron-tier of Linux ie provides better energy efficiency for thesame load level This suggests that a properly configureddataplane design is always more energy-efficient than anyLinux-based configuration The difference is substantial in-cluding at low-to-moderate load levels IXrsquos frontier is lessthan half of Linuxrsquos when the throughput load ge 780 KRPS(sim54 of Linuxrsquos capacity)DVFS-only alternative Fig 3 further analyzes the data andcompares the Pareto frontiers with an alternate frontier thatonly considers changes in DVFS frequency The comparisonis limited to the energy proportionality scenario for consoli-dation available cores are necessary for the background ap-plication to run and a DVFS-only solution would not suffice

We observe that the impact of DVFS-only controls differsnoticeably between Linux and IX with Linux the DVFS-only alternate frontier is very close to the Pareto frontiermeaning that a DVFS-only approach such as Pegasus [29] orAdrenaline [15] would be adequate This is due to Linuxrsquosidling behavior which saves resources In the case of IXhowevermdashand likely for any polling-based dataplanemdashaDVFS-only scheduler would provide worse energy propor-tionality at low-moderate loads than a corresponding Linux-based solution As many datacenter servers operate in the10-30 range [4] we conclude that a dynamic resourceallocation scheme involving both DVFS and core allocationis necessary for dataplane architectures

32 Derived PolicyWe use the results from sect31 to derive a resource config-uration policy framework whose purpose is to determinethe sequence of configurations to be applied as a functionof the load on the foreground application to both the fore-ground (latency-sensitive) and background (batch) applica-tions Specifically given an ever-increasing (or -decreasing)load on the foreground applications the goal is to determinethe sequence of resource configurations minimizing energyconsumption or maximizing background throughput respec-tively

We observe that (i) the latency-sensitive application(memcached) can scale nearly linearly up to the 8 cores of

the processor (ii) it benefits from running a second thread oneach core with a consistent speedup of 13times (iii) it is mostenergy-efficient to first utilize the various cores and onlythen to enable the second hyperthread on each core ratherthan the other way around and (iv) it is least energy-efficientto increase the frequency

We observe that the background application (i) also scaleslinearly but (ii) does not benefit from the 2nd hyperthread(iii) is nearly energy-proportional across the frequency spec-trum with the exception of Turbo Boost From a total cost ofownership perspective the most efficient operating point forthe workload consolidation of the background task is there-fore to run the system at the processorrsquos nominal 24 Ghzfrequency whenever possible We combine these observa-tions with the data from the Pareto analysis and derive thefollowing policiesEnergy Proportional Policy As a base state run with onlyone core and hyperthread with the socket set at the mini-mal clock rate (12Ghz) To add resources first enable ad-ditional cores then enable hyperthreads on all cores (as asingle step) and only after that gradually increase the clockrate until reaching the nominal rate (24Ghz) finally enableTurbo Boost To remove resources do the opposite This pol-icy leads to a sequence of 22 different configurationsWorkload Consolidation Policy As a base state run thebackground jobs on all available cores with the processor atthe nominal clock rate To add resources to the foregroundapplication first shift cores from the background thread tothe foreground application one at a time This is done by firstsuspending the background threads use both hyperthreadsof the newly freed core for the foreground application Nextstop the background job entirely and allocate all cores to theforeground applications As a final step enable Turbo BoostThis policy leads to a sequence of 9 different configurations

These policies closely track the corresponding Paretofrontier For energy proportionality (i) the 32 different staticconfigurations of the frontier are a superset of the configura-tions enabled by the policy and (ii) the difference in overallimpact in terms of energy spent is marginal For consolida-tion Pareto and policy nearly identically overlap

Obviously these conclusions may vary with platformsand applications but the use of underlying Pareto method-ology remains valid In fact it should provide a robust guideto determine allocation policies for more complex workloads(eg with natural scalability limitations) and more flexibleplatforms (eg the newer Haswell processors can set theclock frequency separately for individual cores)

4 Dynamic Resource Controls of DataplanesWe now present the design of the mechanisms and poli-cies that are necessary to implement dynamic resource con-trols in dataplane operating systems Although the imple-mentation is specific to the IX dataplane operating systemthe design principles generally apply to all high-throughput

dataplane operating systems such as Arrakis [43] and user-level networking stacks running latency-sensitive applica-tions [18 20 31] Traditionally such environments have notfocused on dynamic resource controls instead they focusedsolely on throughput with statically allocated resourceswithout considerations for the energy consumed Addingresource controls in such environments poses the followingchallenges

1 How to rebalance flows across cores without impactingnormal performance This is particularly challenging asdataplane environments achieve high performance be-cause of their coherence-free execution model By thatwe mean a programming where all common case oper-ations execute without requiring communication of syn-chronization between threads

2 How to rebalance flows across processing cores withoutimpacting network performance given that dataplane op-erating systems implement their own TCPIP network-ing stack and that TCPIP performance is negatively im-pacted whenever packets are either lost or processed outof order [24] This is particularly challenging wheneverthe rebalancing operation also involves the reconfigura-tion of the NIC which generates inherent race conditions

3 How to determinemdashboth robustly and efficientlymdashwhetherresources need to be added or can be removed from thedataplane given the complexity of modern applicationsand the difficulty to estimate the impact of resource con-trols on application throughput

Our design addresses these three challenges We firstprovide in sect41 the necessary background on the IX data-plane and in sect42 the background on the few assumptionsmade on the hardware sect43 describes the implementationof the controller which relies exclusively on application-independent queuing delay metrics to adjust resources sect44describes how a dataplane operating system can have both acoherence-free execution model and at the same time allowfor rebalancing of flows Finally sect45 describes the mecha-nism that migrates flows without reordering packets

41 Background on IX

The IX system architecture separates the control plane re-sponsible for the allocation of resources from the dataplanewhich operates on them using an approach similar to Ar-rakis [43] IX relies on hardware virtualization to protect thekernel from applications it uses Dune [5] to load and exe-cute the IX dataplane instances and uses the Linux host asits control plane environment

The IX dynamic controller is a Python daemon that con-figures and launches the IX dataplane then monitors its ex-ecution makes policy decisions and communicates them tothe IX dataplane and the host operating system

Unlike conventional operating systems which decouplenetwork processing from application execution the IX data-

app

libix

tcpip tcpip

timer

adaptive batch

1

2

4

5

Event

ConditionsBatched

Syscalls

3

6

kernel

user

Figure 4 Adaptive batching in the IX data plane

plane relies on a run to completion with adaptive batchingscheme to deliver both low latency and high throughput toapplications

Fig 4 illustrates the key run-to-completion mechanismbuilt into the original IX dataplane [6] each elastic threadhas exclusive access to NIC queues In step (1) the data-plane polls the receive descriptor ring and moves all receivedpackets into an in-memory queue and potentially posts freshbuffer descriptors to the NIC for use with future incomingpackets The thread then executes steps (2) ndash (6) for an adap-tive batch of packets the TCPIP network stack processesthese packets (2) and the application then consumes the re-sulting events (3) Upon return from user-space the threadprocesses batched systems calls and in particular the onesthat direct outgoing TCPIP traffic (4) runs necessary timers(5) and places outgoing Ethernet frames in the NICrsquos trans-mit descriptor ring for transmission (6)

Adaptive batching is specifically defined as follows (i)we never wait to batch requests and batching only occurs inthe presence of congestion (ii) we set an upper bound on thenumber of batched packets Using batching only during con-gestion allows us to minimize the impact on latency whilebounding the batch size prevents the live set from exceed-ing cache capacities and avoids transmit queue starvationBatching improves packet rate because it amortizes systemcall transition overheads and improves instruction cache lo-cality prefetching effectiveness and branch prediction ac-curacy When applied adaptively batching also decreases la-tency because these same efficiencies reduce head-of-lineblocking

42 Dynamic NIC ConfigurationModern NICs provide hardware mechanisms such as VMDqFlow Director [17] and Receive Side Scaling (RSS) [36] todirect traffic to one of multiple queues as an essential mech-anism for multicore scalability For example the Intel x520

relies on an on-chip table (called the RSS Redirection Ta-ble or RETA [17]) to partition flows into multiple queueson the data path the NIC first computes the Toeplitz hashfunction of the packet 5-tuple header (as specified by RSS)and uses the lower 7 bits of the result as an index into theRETA table A flow group is the set of network flows with anidentical RETA index The host updates the table using PCIewrites on a distinct control path which can lead to packetreordering [52]

Such anomalies have severe consequences in generalon the performance of the TCPIP networking stack [24]IXrsquos coherence-free design exposes a further constraint be-cause there are no locks within the TCPIP stack two elasticthreads cannot process packets from the same flow simulta-neously For correctness packets received by the thread thatdoes not own the flow group must be dropped This furthernegatively impacts network performance

The IX dataplane implements the actual flow group mi-gration and relies on the RETA to change the mappings Thechallenge is to design a migration mechanism that updatesthe steering of packets to different queues (and by extensionto different threads) without (i) impacting the steady stateperformance of the dataplane and its coherence-free designor (ii) creating network anomalies during migration such asdropping packets or processing them out of order in the net-working stack

43 Control LoopWe first describe the control loop at the heart of IXrsquos dy-namic controller implemented within the control plane Thecontroller adjusts processor resources by suspending andresuming IX elastic threads and specifying the mappingbetween flow groups and threads It migrates flow groupsfrom multiple existing threads to a newly resumed threadand conversely from a thread to be suspended to the re-maining threads For the server consolidation scenario thecontrol loop also allocates resources dedicated to the back-ground applications In our experimental setup it simply is-sues SIGSTOP and SIGCONT signals to transparently con-trol the background application

The control loop does not depend on any detailed charac-terization of the Pareto frontier of sect31 Instead it merely im-plements the derived high-level policy of sect32 using queuingdelay estimates Specifically the user sets the upper boundon the acceptable queuing delay in our experiments withmemcached characterized by high packet rates and low ser-vice times we set the acceptable bound at 300micros to minimizelatency violations given our SLO of 500micros at the 99th per-centile as measured end-to-end by the client machine

For this we rely on a key side effect of IXrsquos use of adap-tive batching unprocessed packets that form the backlogare queued in a central location namely in step (1) in thepipeline of Fig 4 Packets are then processed in order inbounded batches to completion through both the networkingstack and the application logic In other words each IX core

Prepare Wait RPC Defer processIXCPpy

migrate(fgs)

CPU A

update HW

hold

finalize

done

NIC

first

packet

release

CPU B

prepare

timeout

(a) Thread-centric view

NICHWQ-A

HWQ-B

RETA

FGPoll

Poll

defaultQ-A

remoteQ-B

defaultQ-B

localQ-B

Thread-A

Thread-B

(b) Packet-centric view

Figure 5 Flow-group Migration Algorithm

operates like a simple FIFO queuing server onto which clas-sic queuing and control theory principles can be easily ap-plied In contrast conventional operating systems distributebuffers throughout the system in the NIC (because of coa-lesced interrupts) in the driver before networking process-ing and in the socket layer before being consumed by theapplication Furthermore these conventional systems pro-vide no ordering guarantees across flows which makes itdifficult to pinpoint congestion

To estimate queuing delays the controller monitors theiteration time τ and the queue depth Q With B the maximalbatch size the tail latency issimmax(delay) = dQBelowastτ Thedataplane computes each instantaneous metric every 10msfor the previous 10ms interval As these metrics are subjectto jitter the dataplane computes the exponential weightedmoving averages using multiple smoothing factors (α) inparallel For example we track the queue depth as Q(tα) =α lowastQnow +(1minusα) lowastQ(tminus 1α) The control loop executesat a frequency of 10 Hz which is sufficient to adapt to loadchanges

Deciding when to remove resources is trickier than de-ciding when to add them as shallow and near-empty queuesdo not provide reliable metrics Instead we measure idletime and observe that each change in the configuration addsor removes a predictable level of throughput (i) increas-ing cores from n to n+ 1 increases throughput by a factorof 1n (ii) enabling hyperthreads adds 30 (for our fore-ground workload at least) and (iii) the corersquos performanceis largely proportional to its clock rate We can thereforedefine the function threshold[ f rom to] as the ratio of maxthroughput between the f rom and to configurations Whenoperating at configuration f rom and the idle time exceedsthreshold[ f rom to] we can safely switch to configuration towithout loss of performance

These settings are not universal as a number of assump-tions such as load changes requests bursts and variationsin request service times will noticeably affect the metricsThese are policies not mechanisms In particular there is an

inherent tradeoff between aggressive energy saving policieswhich may lead to SLO violations and conservative poli-cies which will consume an excessive amount of energyFortunately the policy can be easily modified the dataplaneexposes naturally all relevant metrics and the control planedaemon issim500 lines of Python includingsim80 lines for thecontrol loop

44 Data Structures for Coherence-free ExecutionWe designed our memory management and data structurelayout to minimize the performance impact of flow groupmigration and to preserve the coherence-free property of ourdataplane in the steady-state Our memory allocator drawsinspiration from magazine-based SLAB allocation [7] Eachthread maintains a thread-local cache of every data typeIf the size of the thread-local cache exceeds a thresholdmemory items are returned using traditional locking to ashared memory pool Such rebalancing events are commonduring flow group migration but infrequent during normaloperation

We further organize our data structures so the pointerreferences building linked lists hash tables etc are onlybetween objects either both permanently associated with thesame thread (per-cpu) or both permanently associated withthe same flow group (per-fg) or both global in scope Asa result we can efficiently transfer ownership of an entireflow-grouprsquos data structures during migration while avoidinglocking or coherence traffic during regular operation

For example the event condition arrays batch syscallarrays timer wheels incoming and outgoing descriptorrings posted and pending mbufs are all per-cpu Theprotocol control blocks 5-tuple hash table listening sock-ets TW WAIT queues out-of-order fragments and unac-knowledged sequences are all per-fg The ARP table isglobal Some objects change scope during their lifetimebut only at well-defined junctures For example an mbuf ispart of the per-cpu queue while waiting to be processed(step 1 in Fig 4) but then becomes part of per-fg struc-tures during network processing

A global data structure the software flow group tablemanages the entry points into the per-fg structures IntelNICs conveniently store the RSS hash result as meta-dataalong with each received packet the bottom bits are used toindex into that table

We encountered one challenge in the management of TCPtimers We design our implementation to maximize normalexecution performance each thread independently managesits timers as a hierarchical timing wheel [49] ie as a per-cpu structure As a consequence a flow group migrationrequires a scan of the hierarchical wheels to identify andremove pending timers during the hold phase of Fig 5athose will be restored later in the destination thread at therelease phase of the flow group migration

45 Flow Group MigrationWhen adding or removing a thread the dynamic controllergenerates a set of migration requests Each individual re-quest is for a set of flow groups (fgs) currently handledby one elastic thread A to be handled by elastic thread B Tosimplify the implementation the controller serializes the mi-gration requests and the dataplane assumes that at most onesuch request is in progress at any point in time

Each thread has three queues that can hold incoming net-work packets the defaultQ contains packets received dur-ing steady state operation when no flow group migration ispending the remoteQ contains packets belonging to outgo-ing flow groups the localQ contains packets belonging toincoming flow groups The use of these two extra queuesis critical to ensure the processing of packets in the properorder by the networking stack

Fig 5 illustrates the migration steps in a thread-centricview (Fig 5a) and in a packet-centric view (Fig 5b) Thecontroller and the dataplane threads communicate via lock-free structures in shared memory First the controller signalsA to migrate fgs to B A first marks each flow group of theset fgs with a special tag to hold off normal processing onall threads moves packets which belong to the flow groupset fgs from defaultQ-A to remoteQ-B and stops alltimers belonging to the flow group set A then reprogramsthe RETA for all entries in fgs Packets still received byA will be appended to remoteQ-B packets received by Bwill go to localQ-B

Upon reception of the first packet whose flow group be-longs to fgs by B B signals A to initiate the final stageof migration Then B finalizes the migration by re-enablingfgsrsquos timers removing all migration tags and pre-pendingto its defaultQ-B the packets from remoteQ-B andthe packets from localQ-B Finally B notifies the con-trol plane that the operation is complete There are obviouslycorner-cases and subtleties in the algorithm For example Ainstalls a migration timer to finalize the migration even if Bhas not yet received any packets We set the timeout to 1 ms

5 EvaluationWe use three synthetic time-accelerated load patterns toevaluate the effectiveness of the control loop under stressfulconditions All three vary between nearly idle and maximumthroughput within a four minute period the slope patterngradually raises the target load from 0 and 62M RPS andthen reduces its load the step pattern increases load by 500KRPS every 10 seconds and finally the sine+noise patternis a basic sinusoidal pattern modified by randomly addingsharp noise that is uniformly distributed over [-250+250]KRPS and re-computed every 5 seconds The slope patternprovides a baseline to study smooth changes the step patternmodels abrupt and massive changes while the sine+noisepattern is representative of daily web patterns [48]

Fig 6 and Fig 7 show the results of these three dynamicload patterns for the energy proportionality and workloadconsolidation scenarios In each case the top figure mea-sures the observed throughput They are annotated with thecontrol loop events that add resources (green) or removethem (red) Empty triangles correspond to core allocationsand full triangles to DVFS changes The middle figure eval-uates the soundness of the algorithm and reports the 99thpercentile latency as observed by a client machine and re-ported every second Finally the bottom figures compare theoverall efficiency of our solution based on dynamic resourcecontrols with (i) the maximal static configuration using allcores and Turbo Boost and (ii) the ideal synthetic efficiencycomputed using the Pareto frontier of Fig 2Energy Proportionality Fig 6 shows the behavior for theenergy efficiency scenario In each case the workload tracksthe desired throughput of the pattern and exercises the en-tire sequence of configurations gradually adding cores en-abling hyperthreading increasing the frequency and finallyenabling Turbo Boost before doing it in reverse The steppattern is particularly challenging as the instant change inload level requires multiple back-to-back configurationschanges With a few exceptions the latencies remain wellbelow the 500micros SLO We further discuss the violations be-low Finally Fig 6 (bottom) compares the power dissipatedby the workload with the corresponding power levels as de-termined by the Pareto frontier (lower bound) or the max-imum static configuration (upper bound) This graph mea-

Smooth Step Sine+noiseEnergy Proportionality (W)

Max power 91 92 94Measured 42 (-54) 48 (-48) 53 (-44)Pareto bound 39 (-57) 41 (-55) 45 (-52)

Server consolidation opportunity ( of peak)Pareto bound 50 47 39Measured 46 39 32

Table 1 Energy Proportionality and Consolidation gains

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

max conf dynamic Pareto

0

20

40

60

80

100

0 50 100 150 200

Pow

er

(W)

Time (seconds)

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 6 Energy Proportionality of memcached running on IX for three load patterns

sures the effectiveness of the control loop to maximize en-ergy proportionality We observe that the dynamic (actuallymeasured) power curve tracks the Pareto (synthetic) curvewell which defines a bound on energy savings When the dy-namic resource controls enters Turbo Boost mode the mea-sured power in all three cases starts at the lower end of the4 W range and then gradually rises as expected Table 1shows that the three patterns have Pareto savings bounds of52 55 and 57 IXrsquos dynamic resource controls resultsin energy savings of 44 48 and 54 which is 85 87and 93 of the theoretical boundConsolidation Fig 7 shows the corresponding results forthe workload consolidation scenarios Fig 7 (top) measuresthe observed load which here as well tracks the desired loadRecall that the consolidation policy always operates at theprocessorrsquos nominal rate (or Turbo) which limits the numberof configuration changes Fig 7 (middle) similarly confirmsthat the system meets the SLO with few exceptions Fig 7(bottom) plots the throughput of the background batch appli-cation expressed as a percentage of its throughput on a dedi-cated processor at 24 Ghz We compare it only to the Paretooptimal upper bound as a maximum configuration wouldmonopolize cores and deliver zero background throughputTable 1 shows that for these three patterns our consolida-tion policy delivers 32ndash46 of the standalone throughput

of the background job which corresponds to 82ndash92 ofthe Pareto boundSLO violations A careful study of the SLO violations of the6 runs shows that they fall into two categories First there are16 violations caused by delays in packet processing due toflow group migrations resulting from the addition of a coreSecond there are 9 violations caused by abrupt increase ofthroughput mostly in the step pattern which occur beforeany flow migrations The control plane then reacts quickly(in sim100 ms) and accommodates to the new throughput byadjusting resources To further confirm the abrupt nature ofthroughput increase specific to the step pattern we note thatthe system performed up three consecutive increases in re-sources in order to resolve a single violation 23 of the 25total violations last a single second with the remaining twoviolations lasting two seconds We believe that the compli-ance with the SLO achieved by our system is more than ad-equate for any practical production deploymentFlow group migration analysis Table 2 measures the la-tency of the 552 flow group migrations that occur during the6 benchmarks as described in sect45 It also reports the totalnumber of packets whose processing is deferred during themigration (rather than dropped or reordered) We first ob-serve that migrations present distinct behaviors when scalingup and when scaling down the number of cores The differ-

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

0

20

40

60

80

100

0 50 100 150 200

o

f pea

k

Time (seconds)

dynamicPareto

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 7 Consolidation of memcached with a background batch task

ence can be intuitively explained since the migrations duringthe scale up are performed in a heavily loaded system whilethe system during the scale down is partially idle In absoluteterms migrations that occur when adding a core take 463microson average and less than 15 ms 95 of the time The outlierscan be explained by rare occurrences of longer preparationtimes or when processing up to 614 deferred packets

6 DiscussionUsing Pareto as a guide Even though the Pareto resultsare not used by the dynamic resource controller the Paretofrontier proved to be a valuable guide first to motivate andquantify the problem then to derive the configuration pol-icy sequence and finally to evaluate the effectiveness ofthe dynamic resource control by setting an upper bound onthe gains resulting from dynamic resource allocation Manyfactors such as software scalability hardware resource con-tention network and disk IO bottlenecks will influence thePareto frontier of any given application and therefore the de-rived control loop policies We will compare different work-loads in future workTradeoff between average latency and efficiency We usea one-dimensional SLO (99th pct le 500micros) and show thatwe can increase efficiency while maintaining that objectiveHowever this comes at a cost when considering the entirelatency distribution eg the average as well as the tail More

complex SLOs taking into account multiple aspects of la-tency distribution would define a different Pareto frontierand likely require adjustments to the control loopAdaptive flow-centric scheduling The new flow-groupmigration algorithm leads to a flow-centric approach to re-source scheduling where the network stack and applicationlogic always follow the steering decision POSIX applica-tions can balance flows by migrating file descriptors betweenthreads or processes but this tends to be inefficient becauseit is difficult for the kernel to match the flowrsquos destinationreceive queue to changes in CPU affinity Flow director canbe used by Linux to adjust the affinity of individual networkflows as a reactive measure to application-level and kernelthread migration rebalancing but the limited size of the redi-rection table prevents this mechanism from scaling to largeconnection counts By contrast our approach allows flowsto be migrated in entire groups improving efficiency and iscompatible with more scalable hardware flow steering mech-anisms based on RSSHardware trends Our experimental setup using one SandyBridge processor has a three-dimensional Pareto spaceClearly NUMA would add a new dimension Also thenewly released Haswell processors provide per-core DVFScontrols which further increases the Pareto space More-over hash filters for flow group steering could benefit fromrecent trends in NIC hardware For example Intelrsquos new

avg 95th pct max stddevad

dco

reprepare (micros) 67 231 521 93wait (micros) 135 541 983 190rpc (micros) 129 355 529 112deferred (micros) 101 401 1167 147total (micros) 463 1432 2870 459 packets 64 319 614 116

rem

ove

core prepare (micros) 25 54 398 32

wait (micros) 34 101 170 32rpc (micros) 12 25 49 7deferred (micros) 19 43 108 14total (micros) 93 160 456 47 packets 3 8 28 3

Table 2 Breakdown of flow group migration measured dur-ing the six benchmarks

XL710 chipset [19] has a 512 entry hash LUT (as well asindependent 64 entry LUTs for each VF) in contrast to the128 entries available in the 82599 chipset [17] used in ourevaluation This has the potential to reduce connection im-balances between cores especially with high core counts

7 Related WorkScheduling Scheduler activations [1] give applicationsgreater control over hardware threads and provide a mecha-nism for custom application-level scheduling Callisto [13]uses a similar strategy to improve the performance of co-located parallel runtime systems Our approach differs inthat an independent control plane manages the schedulingof hardware threads based on receive queuing latency indi-cators while the dataplane exposes a simple kernel thread-ing abstraction SEDA [51] also monitors queuing behaviorto make scheduling decisions such as thread pool sizingChronos [21] makes use of software-based flow steering butwith a focus on balancing load to reduce latency AffinityAccept [42] embraces a mixture of software and hardware-based flow steering in order to improve TCP connectionaffinity and increase throughput We focus instead on energyproportionality and workload consolidationEnergy Proportionality The energy proportionality prob-lem [3] has been well explored in previous work Somesystems have focused on solutions tailored to throughput-oriented workloads [33] or read-only workloads [23] Meis-ner et al [34] highlight unique challenges for low la-tency workloads and advocate full system active low-powermodes Similar to our system Pegasus [29] achieves CPUenergy proportionality for low latency workloads Our workexpands on Pegasus by exploring the elastic allocation ofhardware threads in combination with processor power man-agement states and by basing scheduling decisions on in-ternal latency metrics within a host endpoint instead of anexternal controller Niccolini et al show that a softwarerouter running on a dedicated machine can be made energy-

proportional [38] Similar to our approach queue lengthis used as a control signal to manage core allocation andDVFS settings However we focus on latency-sensitive ap-plications rather than middlebox traffic and consider theadditional case of workload consolidationCo-location Because host endpoints contain some com-ponents that are not energy proportional and thus are mostefficient when operating at 100 utilization co-location ofworkloads is also an important tool for improving energyefficiency At the cluster scheduler level BubbleUp [32] andParagon [10] make scheduling decisions that are interference-aware through efficient classification of the behavior ofworkload co-location Leverich et al [26] demonstratethat co-location of batch and low latency jobs is possi-ble on commodity operating systems Our approach ex-plores this issue at higher throughputs and with tighter la-tency SLOs Bubble-Flux [53] additionally controls back-ground threads we control background and latency-sensitivethreads CPI2 [54] detects performance interference by ob-serving changes in CPI and throttles offending jobs Thiswork is orthogonal to ours and could be a useful addi-tional signal for our control plane Heracles manages mul-tiple hardware and software isolation mechanisms includ-ing packet scheduling and cache partitioning to co-locatelatency-sensitive applications with batch tasks while main-taining millisecond SLOs [30] We limit our focus to DVFSand core assignment but target more aggressive SLOsLow-latency frameworks User-level networking stacks [2031 47] and RDMA applications [12 37 40] all rely onpolling typically by a fixed number of execution threadsto deliver micros-scale response times These stacks and appli-cations are evaluated based on their latency and throughputcharacteristics and do not take into account energy propor-tionality or workload consolidation considerations

8 ConclusionWe present the design and implementation of dynamic re-source controls for the IX dataplane which consists of acontroller that allocates cores and sets processor frequencyto adapt to changes in the load of latency-sensitive appli-cations The novel rebalancing mechanisms do not impactthe steady-state performance of the dataplane and can mi-grate a set of flow groups in milliseconds without droppingor reordering packets We develop two policies focused onoptimizing energy proportionality and workload consolida-tion Our results show that for three varying load patternsthe energy proportionality can save 44ndash54 of the pro-cessorrsquos energy or enable a background job to deliver 32ndash46 of its standalone throughput To evaluate the effective-ness of our approach we synthesize the Pareto frontier bycombining the behavior of all possible static configurationsOur policies deliver 85ndash93 of the Pareto optimal boundin terms of energy proportionality and 82ndash92 in termsof consolidation

AcknowledgementsThis work was supported by the MicrosoftEPFL Joint Re-search Center the Stanford Experimental Datacenter Laband NSF grant CNS-1422088 George Prekas is supportedby a Google Graduate Research Fellowship and Adam Be-lay by a Stanford Graduate Fellowship

References[1] T E Anderson B N Bershad E D Lazowska and H M

Levy Scheduler Activations Effective Kernel Support for theUser-Level Management of Parallelism ACM Trans ComputSyst 10(1)53ndash79 1992

[2] B Atikoglu Y Xu E Frachtenberg S Jiang andM Paleczny Workload Analysis of a Large-Scale Key-ValueStore In Proceedings of the 2012 ACM SIGMETRICS Inter-national Conference on Measurement and Modeling of Com-puter Systems pages 53ndash64 2012

[3] L A Barroso and U Holzle The Case for Energy-Proportional Computing IEEE Computer 40(12)33ndash372007

[4] L A Barroso J Clidaras and U Holzle The Datacenter as aComputer An Introduction to the Design of Warehouse-ScaleMachines Second Edition Synthesis Lectures on ComputerArchitecture Morgan amp Claypool Publishers 2013

[5] A Belay A Bittau A J Mashtizadeh D Terei D Mazieresand C Kozyrakis Dune Safe User-level Access to PrivilegedCPU Features In Proceedings of the 10th Symposium onOperating System Design and Implementation (OSDI) pages335ndash348 2012

[6] A Belay G Prekas A Klimovic S Grossman C Kozyrakisand E Bugnion IX A Protected Dataplane Operating Systemfor High Throughput and Low Latency In Proceedings of the11st Symposium on Operating System Design and Implemen-tation (OSDI) pages 49ndash65 2014

[7] J Bonwick and J Adams Magazines and Vmem Extendingthe Slab Allocator to Many CPUs and Arbitrary Resources InUSENIX Annual Technical Conference pages 15ndash33 2001

[8] H David E Gorbatov U R Hanebutte R Khanaa andC Le RAPL memory power estimation and capping InProceedings of the 2010 International Symposium on LowPower Electronics and Design pages 189ndash194 2010

[9] J Dean and L A Barroso The Tail at Scale Commun ACM56(2)74ndash80 2013

[10] C Delimitrou and C Kozyrakis Quality-of-Service-AwareScheduling in Heterogeneous Data centers with ParagonIEEE Micro 34(3)17ndash30 2014

[11] C Delimitrou and C Kozyrakis Quasar Resource-Efficientand QoS-Aware Cluster Management In Proceedings of the19th International Conference on Architectural Support forProgramming Languages and Operating Systems (ASPLOS-XIX) pages 127ndash144 2014

[12] A Dragojevic D Narayanan M Castro and O HodsonFaRM Fast Remote Memory In Proceedings of the 11stSymposium on Networked Systems Design and Implementa-tion (NSDI) pages 401ndash414 2014

[13] T Harris M Maas and V J Marathe Callisto co-schedulingparallel runtime systems In Proceedings of the 2014 EuroSysConference page 24 2014

[14] B Hindman A Konwinski M Zaharia A Ghodsi A DJoseph R H Katz S Shenker and I Stoica Mesos A Plat-form for Fine-Grained Resource Sharing in the Data CenterIn Proceedings of the 8th Symposium on Networked SystemsDesign and Implementation (NSDI) 2011

[15] C-H Hsu Y Zhang M A Laurenzano D Meisner T FWenisch J Mars L Tang and R G Dreslinski AdrenalinePinpointing and reining in tail queries with quick voltageboosting In Proceedings of the 21st IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages271ndash282 2015

[16] J Hwang K K Ramakrishnan and T Wood NetVM HighPerformance and Flexible Networking Using Virtualization onCommodity Platforms In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 445ndash458 2014

[17] Intel Corp Intel 82599 10 GbE Controller Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheets82599-10-gbe-controller-datasheetpdf2014

[18] Intel Corp Intel DPDK Data Plane Development Kit httpdpdkorg 2014

[19] Intel Corp Intel Ethernet Controller XL710 Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheetsxl710-10-40-controller-datasheetpdf2014

[20] E Jeong S Woo M A Jamshed H Jeong S Ihm D Hanand K Park mTCP a Highly Scalable User-level TCP Stackfor Multicore Systems In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 489ndash502 2014

[21] R Kapoor G Porter M Tewari G M Voelker and A Vah-dat Chronos Predictable Low Latency for Data Center Ap-plications In Proceedings of the 2012 ACM Symposium onCloud Computing (SOCC) page 9 2012

[22] W Kim M S Gupta G-Y Wei and D M Brooks Systemlevel analysis of fast per-core DVFS using on-chip switch-ing regulators In Proceedings of the 14th IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages123ndash134 2008

[23] A Krioukov P Mohan S Alspaugh L Keys D E Cullerand R H Katz NapSAC design and implementation of apower-proportional web cluster Computer CommunicationReview 41(1)102ndash108 2011

[24] K-C Leung V O K Li and D Yang An Overview ofPacket Reordering in Transmission Control Protocol (TCP)Problems Solutions and Challenges IEEE Trans ParallelDistrib Syst 18(4)522ndash535 2007

[25] J Leverich Mutilate High-Performance MemcachedLoad Generator httpsgithubcomleverichmutilate 2014

Such latency-critical services are challenging to run in ashared infrastructure environment They are particularly sen-sitive to resource allocation and frequency settings and theysuffer frequent tail latency violations when common powermanagement or consolidation approaches are used [26 27]As a result operators typically deploy them on dedicatedservers running in polling mode forgoing opportunities forworkload consolidation and for energy-proportional com-puting at below-peak utilization levels

To gain a principled understanding of the challenges forresource management in the presence of latency-critical ser-vices we performed an exhaustive analysis of static con-figurations for a latency-critical service (memcached [35])running on a modern server We explored up to 224 possi-ble settings for core allocation use of hyperthreads DVFSfrequencies and Turbo Boost While our experiments usea single application the implications have broad applica-bility because memcached has aggressive latency require-ments short service times and a large number of indepen-dent clients that are common among many latency-criticalapplications Our experiments reveal that there is an inher-ent tradeoff for any given static configuration between themaximum throughput and the overall efficiency when oper-ating below peak load Furthermore the experiments reveala Pareto-optimal frontier [41] in the efficiency of static con-figurations at any given load level which allows for close tolinear improvements in energy-proportionality and workloadconsolidation factors

Based on these insights we introduce a set of novel dy-namic resource management mechanisms and policies thatimprove energy proportionality and workload consolidationin the presence of latency-sensitive applications We inte-grate these techniques in IX a state-of-the-art dataplane op-erating system that optimizes both throughput and latencyfor latency-critical workloads [6]

Fig 1 illustrates our approach the dynamic controller(ixcp) adjusts the number of cores allocated to a latency-sensitive application running on top of IX and the DVFS set-tings for these cores The remaining cores can be placed inidle modes to reduce power consumption or can be safelyused to run background tasks The controller builds upontwo key mechanisms The first mechanism detects back-log and increases in queuing delays that exceed the allow-able upper bound for the specific latency-critical applica-tion It monitors CPU utilization and signals required adjust-ments in resource allocation The second mechanism quicklymigrates both network and application processing betweencores transparently and without dropping or reordering pack-ets

We evaluated our system with two control policies thatoptimize for energy proportionality and workload consolida-tion respectively A policy determines how resources (coreshyperthreads and DVFS settings) are adjusted to reduce un-derutilization or to restore violated SLO The two policies

IXCP

C C

ho

st

ap

plicati

on

C

Dune Linux

IX

latency-critical

workload

C

background

task

C

+ -

kern

el

user

C

dvfs

Figure 1 Dynamic resource controls with IX for a workloadconsolidation scenario with a latency-sensitive application(eg memcached) and a background batch task (eg an-alytics) The controller ixcp partitions cores among theapplications and adjusts the processorrsquos DVFS settings

are derived from the exhaustive analysis of the 224 staticconfigurations For the platform studied (a Xeon E5-2665)we conclude that for best energy proportionality (i) we startwith the lowest clock rate and allocate additional cores to thelatency-critical task as its load grows using at first only onehyperthread per core (ii) we enable the second hyperthreadonly when all cores are in use and finally (iii) we increasethe clock rate for the cores running the latency-critical taskFor best consolidation (i) we start at the nominal clock rateand add cores with both hyperthreads enabled as load in-creases and (ii) finally enable Turbo Boost as a last resort

This paper makes the following contributions We develop techniques for fine-grain resource manage-ment for latency-critical workloads This includes mecha-nisms for detection of load changes in sub-second timescalesand for rebalancing flow-groups between cores withoutcausing packet drops or reordered deliveries In our experi-ments this mechanism completes in less than 2 ms 95 ofthe time and in at most 35 ms We provide a methodology that uses the Pareto frontier of aset of static configurations to derive resource allocation poli-cies We derive two policies for memcached that respectthe SLO constraints of a latency-critical in-memory key-value store These policies lead to 44ndash54 energy sav-ings for a variety of load patterns and enable workload con-solidation with background jobs executing at 32ndash46 oftheir peak throughput on a standalone machine These gainsare close to the Pareto-optimal bounds for the server usedwithin 88 and 82ndash92 respectively We demonstrate that precise and fine-grain control ofcores hyperthreads and DVFS has a huge impact on bothenergy proportionality and workload consolidation espe-

cially when load is highly variable DVFS is a necessary butinsufficient mechanism to control latency-critical applica-tions that rely on polling

The rest of this paper is organized as follows sect2 describesthe experimental setup sect3 characterizes resource usage us-ing static configurations and derives insights for dynamic al-location policies sect4 describes the design and implementa-tion of the mechanisms and policies for dynamic resourcemanagement in IX sect5 evaluates the impact of the dynamiccontroller for energy proportionality and workload consol-idation We discuss insights and open issues in sect6 relatedwork in sect7 and conclude in sect8

2 Workloads and Experimental SetupOur experimental setup consists of a cluster of 11 clientsand one server connected by a low-latency 10 GbE switch10 clients generate load and the 11th measures latency ofrequests to the latency-sensitive application The client ma-chines are a mix of Xeon E5-2637 35 Ghz and XeonE5-2650 26 Ghz The server is a Xeon E5-2665 24 Ghz with 256 GB of DRAM typical in datacenters Eachclient and server socket has 8 cores and 16 hyperthreads Allmachines are configured with one Intel x520 10 GbE NIC(82599EB chipset) Our baseline configuration in each ma-chine is an Ubuntu LTS 1404 distribution updated to the3161 Linux kernel Although the server has two socketsthe foreground and background applications run on a singleprocessor to avoid any NUMA effects We run the controlplane on the otherwise empty second socket but do not ac-count for its energy draw

Our primary foreground workload is memcached awidely deployed in-memory key-value store built on topof the libevent framework [35] It is frequently usedas a high-throughput low-latency caching tier in front ofpersistent database servers memcached is a network-bound application with threads spending over 75 of ex-ecution time in kernel mode for network processing whenusing Linux [26] It is a difficult application to scale be-cause the common deployments involve high connectioncounts for memcached servers and small-sized requestsand replies [2 39] Combined with latency SLOs in the fewhundreds of microseconds these characteristics make mem-cached a challenging workload for our study Any short-comings in the mechanisms and policies for energy propor-tionality and workload consolidation will quickly translateto throughput and latency problems In contrast workloadswith SLOs in the millisecond range are more forgivingFurthermore memcached has well-known scalability lim-itations [28] To improve its scalability we configure mem-cached with a larger hash table size (-o hashpower=20)and use a random replacement policy instead of the built-inLRU which requires a global lock This last change pro-vides memcached with similar multicore scalability as state-

of-the-art key-value stores such as MICA [28] We configurememcached similarly for Linux and IX

Our primary background application is a simple syntheticproxy for computationally intense batch processing applica-tions it encrypts a memory stream using SHA-1 in parallelthreads that each run in an infinite loop and report the aggre-gate throughput The workload streams through a 40 MB ar-ray which exceeds the capacity of the processorrsquos L3-cacheand therefore interferes (by design) with the memory sub-system performance of the latency-sensitive application

We use the mutilate load-generator to place a se-lected load on the foreground application in terms of re-quests per second (RPS) and measure response latency [25]Mutilate coordinates a large number of client threadsacross 10 machines to generate the desired RPS load andone additional unloaded client to measure latency for atotal of 2752 connections to the memcached server Weconfigure mutilate to generate load representative of theFacebook USR workload [2] this is a large-scale deploy-ment dominated by GET requests (99) with short keys(lt20B) and 2B values Each request is issued separately (nomultiget operations) However clients are permitted topipeline up to four requests per connection if needed to keepup with their target request rate We enhance mutilateto support dynamically changing load levels used in theseexperiments As the memcached server is multi-threadedwe also enhance the latency-measuring client to randomlychoose among 32 open connections for each request as toensure statistically significant measurements

We compute metrics as follows the unloaded client mea-sures latency by issuing one request at the time chosen ran-domly among its 32 open connections The SLO is spec-ified so that the 99th percentile of requests issued by thatunloaded client return in le 500micros Latency distributionachieved requests and power metrics for all graphs are re-ported at one-second intervals Energy consumption is mea-sured using the Intel RAPL counters [8] for the processor inquestion these provide an accurate model of the power drawof the CPU but ignore platforms effectsA note on Turbo Boost For any given throughput levelwe observe that the reported power utilization is stable forall CPU frequencies except for Turbo Boost When runningin Turbo Boost the temperature of the CPU gradually risesover a few minutes from 58C to 78C and with it the dissi-pated energy rises by 4 W for the same level of performanceThe experiments in sect31 run for a long time in Turbo Boostmode with a hot processor we therefore report those resultsas an energy band of 4 W

3 Understanding Dynamic Resource UsageThe purpose of dynamic resource allocation is to adapt tochanges of load of a latency-critical workload by adding orremoving resources as needed The remaining resources canbe placed in idle modes to save energy or used by other con-

0

20

40

60

80

100

0 2

Pow

er

(W)

Memcached RPS x 106 at SLO

(a) Energy Proportionality (Linux)

0

20

40

60

80

100

0 2 4 6 8

Memcached RPS x 106 at SLO

cfg - all others

cfg at Turbo mode

cfg w 8 cores + HT

cfg at 12 GHz

Pareto frontier

(b) Energy Proportionality (IX)

0

20

40

60

80

100

0 2

o

f P

eak f

or

Best-

eff

ort

Task

Memcached RPS x 106 at SLO

(c) Consolidation (Linux)

0

20

40

60

80

100

0 2 4 6 8

Memcached RPS x 106 at SLO

(d) Server Consolidation (IX)

Figure 2 Pareto efficiency for energy proportionality and workload consolidation for Linux 3161 and IX The Paretoefficiency is in red while the various static configurations are color-coded according to their distinctive characteristics

solidated applications In general dynamic control can beapplied to any resource including processor cores mem-ory and IO We focus on managing CPU cores becausethey are the largest contributor to power consumption anddraw significant amounts of energy even when lightly uti-lized [4 29] Moreover the ability to dynamically reassigncores between latency-critical and background applicationsis key to meet latency SLOs for the former and to make rea-sonable progress using underutilized resources for the latter

Dynamic resource control is a multi-dimensional prob-lem on current CPUs multi-core hyperthreading and fre-quency each separately impact the workloadrsquos performanceenergy consumption and consolidation opportunities sect31uses a Pareto methodology to evaluate this three-dimensionalspace and sect32 discusses the derived allocation policies

31 Pareto-Optimal Static ConfigurationsStatic resource configurations allow for controlled experi-ments to quantify the tradeoff between an applicationrsquos per-formance and the resources consumed Our approach limitsbias by considering many possible static configurations inthe three-dimensional space of core hyperthread and fre-quency For each static configuration we characterize themaximum load that meets the SLO (le 500micros99th per-

centile) we then measure the energy draw and throughputof the background job for all load levels up to the maximumload supported From this large data set we derive the setof meaningful static configurations and build the Pareto effi-ciency frontier [41] The frontier specifies for any possibleload level the optimal static configuration and the resultingminimal energy draw or maximum background throughputdepending on the scenario

Fig 2 presents the frontier for four experiments theenergy proportionality and workload consolidation scenar-ios for both Linux and IX The four graphs each plot theobjectivemdashwhich is either to minimize energy or maximizebackground throughputmdashas a function of the foregroundthroughput provided that the SLO is met Except for the redlines each line corresponds to a distinct static configurationof the system the green curves correspond to configurationat the minimal clock rate of 12 Ghz the blue curves use allavailable cores and hyperthreads other configurations are inblack In Turbo Boost mode the energy drawn is reported asa band since it depends on operating temperature (see sect2)

Finally the red line is the Pareto frontier which corre-sponds for any load level to the optimal result using any

0

20

40

60

80

100

0 2 4 6 8

Pow

er (

W)

Memcached RPS x 106 at SLO

Pareto IX

DVFS-only IX

Pareto Linux

DVFS-only Linux

Figure 3 Energy-proportionality comparison between thePareto-optimal frontier considering only DVFS adjustmentsand the full Pareto frontier considering core allocation hy-perthread allocations and frequency

of the static configurations available Each graph only showsthe static configurations that participate in the frontierEnergy proportionality We evaluate 224 distinct combi-nations from one to eight cores using consistently eitherone or two threads per core for 14 different DVFS levelsfrom 12 Ghz to 24 Ghz as well as Turbo Boost Fig 2a andFig 2b show the 43 and 32 static configurations (out of 224)that build the Pareto frontier for energy proportionality Thefigures confirm the intuition that (i) various static configura-tions have very different dynamic ranges beyond which theyare no longer able to meet the SLO (ii) each static config-uration draws substantially different levels of energy for thesame amount of work (iii) at the low-end of the curve manydistinct configurations operate at the minimal frequency of12 Ghz obviously with a different number of cores andthreads and contribute to the frontier these are shown ingreen in the Figure (iv) at the high-end of the range manyconfigurations operate with the maximum of 8 cores withdifferent frequencies including Turbo BoostConsolidation The methodology here is a little differentWe first characterize the background job and observe that itdelivers energy-proportional throughput up to 24 Ghz butthat Turbo Boost came at an energythroughput premiumConsequently we restrict the Pareto configuration space at24 Ghz the objective function is the throughput of the back-ground job expressed as a fraction of the throughput of thatsame job without any foreground application Backgroundjobs run on all cores that are not used by the foreground ap-plication Fig 2c and Fig 2d show the background through-put expressed as a fraction of the standalone throughput asa function of the foreground throughput provided that theforeground application meets the SLO as the foregroundapplication requires additional cores to meet the SLO thebackground throughput decreases proportionally

Linux vs IX Fig 2a and Fig 2b show that Linux and IXbehave differently (i) IX improves the throughput of Linuxby 48times (ii) IX can clearly take advantage of hyperthread-ing and has its best performance with configurations with16 threads on 8 cores In contrast Linux cannot take ad-vantage of hyperthreading and has its best performance with8 threads of control running on 8 cores (iii) the efficiencyof individual static configurations of Linux which relieson interrupts is much more subject to load than individ-ual static configurations of IX which relies on a pollingmodel As a consequence an improperly configured over-sized IX dataplane will be extremely energy-inefficient atlow-to-moderate loads (iv) the comparison of the Paretofrontiers shows that IXrsquos frontier always dominates the fron-tier of Linux ie provides better energy efficiency for thesame load level This suggests that a properly configureddataplane design is always more energy-efficient than anyLinux-based configuration The difference is substantial in-cluding at low-to-moderate load levels IXrsquos frontier is lessthan half of Linuxrsquos when the throughput load ge 780 KRPS(sim54 of Linuxrsquos capacity)DVFS-only alternative Fig 3 further analyzes the data andcompares the Pareto frontiers with an alternate frontier thatonly considers changes in DVFS frequency The comparisonis limited to the energy proportionality scenario for consoli-dation available cores are necessary for the background ap-plication to run and a DVFS-only solution would not suffice

We observe that the impact of DVFS-only controls differsnoticeably between Linux and IX with Linux the DVFS-only alternate frontier is very close to the Pareto frontiermeaning that a DVFS-only approach such as Pegasus [29] orAdrenaline [15] would be adequate This is due to Linuxrsquosidling behavior which saves resources In the case of IXhowevermdashand likely for any polling-based dataplanemdashaDVFS-only scheduler would provide worse energy propor-tionality at low-moderate loads than a corresponding Linux-based solution As many datacenter servers operate in the10-30 range [4] we conclude that a dynamic resourceallocation scheme involving both DVFS and core allocationis necessary for dataplane architectures

32 Derived PolicyWe use the results from sect31 to derive a resource config-uration policy framework whose purpose is to determinethe sequence of configurations to be applied as a functionof the load on the foreground application to both the fore-ground (latency-sensitive) and background (batch) applica-tions Specifically given an ever-increasing (or -decreasing)load on the foreground applications the goal is to determinethe sequence of resource configurations minimizing energyconsumption or maximizing background throughput respec-tively

We observe that (i) the latency-sensitive application(memcached) can scale nearly linearly up to the 8 cores of

the processor (ii) it benefits from running a second thread oneach core with a consistent speedup of 13times (iii) it is mostenergy-efficient to first utilize the various cores and onlythen to enable the second hyperthread on each core ratherthan the other way around and (iv) it is least energy-efficientto increase the frequency

We observe that the background application (i) also scaleslinearly but (ii) does not benefit from the 2nd hyperthread(iii) is nearly energy-proportional across the frequency spec-trum with the exception of Turbo Boost From a total cost ofownership perspective the most efficient operating point forthe workload consolidation of the background task is there-fore to run the system at the processorrsquos nominal 24 Ghzfrequency whenever possible We combine these observa-tions with the data from the Pareto analysis and derive thefollowing policiesEnergy Proportional Policy As a base state run with onlyone core and hyperthread with the socket set at the mini-mal clock rate (12Ghz) To add resources first enable ad-ditional cores then enable hyperthreads on all cores (as asingle step) and only after that gradually increase the clockrate until reaching the nominal rate (24Ghz) finally enableTurbo Boost To remove resources do the opposite This pol-icy leads to a sequence of 22 different configurationsWorkload Consolidation Policy As a base state run thebackground jobs on all available cores with the processor atthe nominal clock rate To add resources to the foregroundapplication first shift cores from the background thread tothe foreground application one at a time This is done by firstsuspending the background threads use both hyperthreadsof the newly freed core for the foreground application Nextstop the background job entirely and allocate all cores to theforeground applications As a final step enable Turbo BoostThis policy leads to a sequence of 9 different configurations

These policies closely track the corresponding Paretofrontier For energy proportionality (i) the 32 different staticconfigurations of the frontier are a superset of the configura-tions enabled by the policy and (ii) the difference in overallimpact in terms of energy spent is marginal For consolida-tion Pareto and policy nearly identically overlap

Obviously these conclusions may vary with platformsand applications but the use of underlying Pareto method-ology remains valid In fact it should provide a robust guideto determine allocation policies for more complex workloads(eg with natural scalability limitations) and more flexibleplatforms (eg the newer Haswell processors can set theclock frequency separately for individual cores)

4 Dynamic Resource Controls of DataplanesWe now present the design of the mechanisms and poli-cies that are necessary to implement dynamic resource con-trols in dataplane operating systems Although the imple-mentation is specific to the IX dataplane operating systemthe design principles generally apply to all high-throughput

dataplane operating systems such as Arrakis [43] and user-level networking stacks running latency-sensitive applica-tions [18 20 31] Traditionally such environments have notfocused on dynamic resource controls instead they focusedsolely on throughput with statically allocated resourceswithout considerations for the energy consumed Addingresource controls in such environments poses the followingchallenges

1 How to rebalance flows across cores without impactingnormal performance This is particularly challenging asdataplane environments achieve high performance be-cause of their coherence-free execution model By thatwe mean a programming where all common case oper-ations execute without requiring communication of syn-chronization between threads

2 How to rebalance flows across processing cores withoutimpacting network performance given that dataplane op-erating systems implement their own TCPIP network-ing stack and that TCPIP performance is negatively im-pacted whenever packets are either lost or processed outof order [24] This is particularly challenging wheneverthe rebalancing operation also involves the reconfigura-tion of the NIC which generates inherent race conditions

3 How to determinemdashboth robustly and efficientlymdashwhetherresources need to be added or can be removed from thedataplane given the complexity of modern applicationsand the difficulty to estimate the impact of resource con-trols on application throughput

Our design addresses these three challenges We firstprovide in sect41 the necessary background on the IX data-plane and in sect42 the background on the few assumptionsmade on the hardware sect43 describes the implementationof the controller which relies exclusively on application-independent queuing delay metrics to adjust resources sect44describes how a dataplane operating system can have both acoherence-free execution model and at the same time allowfor rebalancing of flows Finally sect45 describes the mecha-nism that migrates flows without reordering packets

41 Background on IX

The IX system architecture separates the control plane re-sponsible for the allocation of resources from the dataplanewhich operates on them using an approach similar to Ar-rakis [43] IX relies on hardware virtualization to protect thekernel from applications it uses Dune [5] to load and exe-cute the IX dataplane instances and uses the Linux host asits control plane environment

The IX dynamic controller is a Python daemon that con-figures and launches the IX dataplane then monitors its ex-ecution makes policy decisions and communicates them tothe IX dataplane and the host operating system

Unlike conventional operating systems which decouplenetwork processing from application execution the IX data-

app

libix

tcpip tcpip

timer

adaptive batch

1

2

4

5

Event

ConditionsBatched

Syscalls

3

6

kernel

user

Figure 4 Adaptive batching in the IX data plane

plane relies on a run to completion with adaptive batchingscheme to deliver both low latency and high throughput toapplications

Fig 4 illustrates the key run-to-completion mechanismbuilt into the original IX dataplane [6] each elastic threadhas exclusive access to NIC queues In step (1) the data-plane polls the receive descriptor ring and moves all receivedpackets into an in-memory queue and potentially posts freshbuffer descriptors to the NIC for use with future incomingpackets The thread then executes steps (2) ndash (6) for an adap-tive batch of packets the TCPIP network stack processesthese packets (2) and the application then consumes the re-sulting events (3) Upon return from user-space the threadprocesses batched systems calls and in particular the onesthat direct outgoing TCPIP traffic (4) runs necessary timers(5) and places outgoing Ethernet frames in the NICrsquos trans-mit descriptor ring for transmission (6)

Adaptive batching is specifically defined as follows (i)we never wait to batch requests and batching only occurs inthe presence of congestion (ii) we set an upper bound on thenumber of batched packets Using batching only during con-gestion allows us to minimize the impact on latency whilebounding the batch size prevents the live set from exceed-ing cache capacities and avoids transmit queue starvationBatching improves packet rate because it amortizes systemcall transition overheads and improves instruction cache lo-cality prefetching effectiveness and branch prediction ac-curacy When applied adaptively batching also decreases la-tency because these same efficiencies reduce head-of-lineblocking

42 Dynamic NIC ConfigurationModern NICs provide hardware mechanisms such as VMDqFlow Director [17] and Receive Side Scaling (RSS) [36] todirect traffic to one of multiple queues as an essential mech-anism for multicore scalability For example the Intel x520

relies on an on-chip table (called the RSS Redirection Ta-ble or RETA [17]) to partition flows into multiple queueson the data path the NIC first computes the Toeplitz hashfunction of the packet 5-tuple header (as specified by RSS)and uses the lower 7 bits of the result as an index into theRETA table A flow group is the set of network flows with anidentical RETA index The host updates the table using PCIewrites on a distinct control path which can lead to packetreordering [52]

Such anomalies have severe consequences in generalon the performance of the TCPIP networking stack [24]IXrsquos coherence-free design exposes a further constraint be-cause there are no locks within the TCPIP stack two elasticthreads cannot process packets from the same flow simulta-neously For correctness packets received by the thread thatdoes not own the flow group must be dropped This furthernegatively impacts network performance

The IX dataplane implements the actual flow group mi-gration and relies on the RETA to change the mappings Thechallenge is to design a migration mechanism that updatesthe steering of packets to different queues (and by extensionto different threads) without (i) impacting the steady stateperformance of the dataplane and its coherence-free designor (ii) creating network anomalies during migration such asdropping packets or processing them out of order in the net-working stack

43 Control LoopWe first describe the control loop at the heart of IXrsquos dy-namic controller implemented within the control plane Thecontroller adjusts processor resources by suspending andresuming IX elastic threads and specifying the mappingbetween flow groups and threads It migrates flow groupsfrom multiple existing threads to a newly resumed threadand conversely from a thread to be suspended to the re-maining threads For the server consolidation scenario thecontrol loop also allocates resources dedicated to the back-ground applications In our experimental setup it simply is-sues SIGSTOP and SIGCONT signals to transparently con-trol the background application

The control loop does not depend on any detailed charac-terization of the Pareto frontier of sect31 Instead it merely im-plements the derived high-level policy of sect32 using queuingdelay estimates Specifically the user sets the upper boundon the acceptable queuing delay in our experiments withmemcached characterized by high packet rates and low ser-vice times we set the acceptable bound at 300micros to minimizelatency violations given our SLO of 500micros at the 99th per-centile as measured end-to-end by the client machine

For this we rely on a key side effect of IXrsquos use of adap-tive batching unprocessed packets that form the backlogare queued in a central location namely in step (1) in thepipeline of Fig 4 Packets are then processed in order inbounded batches to completion through both the networkingstack and the application logic In other words each IX core

Prepare Wait RPC Defer processIXCPpy

migrate(fgs)

CPU A

update HW

hold

finalize

done

NIC

first

packet

release

CPU B

prepare

timeout

(a) Thread-centric view

NICHWQ-A

HWQ-B

RETA

FGPoll

Poll

defaultQ-A

remoteQ-B

defaultQ-B

localQ-B

Thread-A

Thread-B

(b) Packet-centric view

Figure 5 Flow-group Migration Algorithm

operates like a simple FIFO queuing server onto which clas-sic queuing and control theory principles can be easily ap-plied In contrast conventional operating systems distributebuffers throughout the system in the NIC (because of coa-lesced interrupts) in the driver before networking process-ing and in the socket layer before being consumed by theapplication Furthermore these conventional systems pro-vide no ordering guarantees across flows which makes itdifficult to pinpoint congestion

To estimate queuing delays the controller monitors theiteration time τ and the queue depth Q With B the maximalbatch size the tail latency issimmax(delay) = dQBelowastτ Thedataplane computes each instantaneous metric every 10msfor the previous 10ms interval As these metrics are subjectto jitter the dataplane computes the exponential weightedmoving averages using multiple smoothing factors (α) inparallel For example we track the queue depth as Q(tα) =α lowastQnow +(1minusα) lowastQ(tminus 1α) The control loop executesat a frequency of 10 Hz which is sufficient to adapt to loadchanges

Deciding when to remove resources is trickier than de-ciding when to add them as shallow and near-empty queuesdo not provide reliable metrics Instead we measure idletime and observe that each change in the configuration addsor removes a predictable level of throughput (i) increas-ing cores from n to n+ 1 increases throughput by a factorof 1n (ii) enabling hyperthreads adds 30 (for our fore-ground workload at least) and (iii) the corersquos performanceis largely proportional to its clock rate We can thereforedefine the function threshold[ f rom to] as the ratio of maxthroughput between the f rom and to configurations Whenoperating at configuration f rom and the idle time exceedsthreshold[ f rom to] we can safely switch to configuration towithout loss of performance

These settings are not universal as a number of assump-tions such as load changes requests bursts and variationsin request service times will noticeably affect the metricsThese are policies not mechanisms In particular there is an

inherent tradeoff between aggressive energy saving policieswhich may lead to SLO violations and conservative poli-cies which will consume an excessive amount of energyFortunately the policy can be easily modified the dataplaneexposes naturally all relevant metrics and the control planedaemon issim500 lines of Python includingsim80 lines for thecontrol loop

44 Data Structures for Coherence-free ExecutionWe designed our memory management and data structurelayout to minimize the performance impact of flow groupmigration and to preserve the coherence-free property of ourdataplane in the steady-state Our memory allocator drawsinspiration from magazine-based SLAB allocation [7] Eachthread maintains a thread-local cache of every data typeIf the size of the thread-local cache exceeds a thresholdmemory items are returned using traditional locking to ashared memory pool Such rebalancing events are commonduring flow group migration but infrequent during normaloperation

We further organize our data structures so the pointerreferences building linked lists hash tables etc are onlybetween objects either both permanently associated with thesame thread (per-cpu) or both permanently associated withthe same flow group (per-fg) or both global in scope Asa result we can efficiently transfer ownership of an entireflow-grouprsquos data structures during migration while avoidinglocking or coherence traffic during regular operation

For example the event condition arrays batch syscallarrays timer wheels incoming and outgoing descriptorrings posted and pending mbufs are all per-cpu Theprotocol control blocks 5-tuple hash table listening sock-ets TW WAIT queues out-of-order fragments and unac-knowledged sequences are all per-fg The ARP table isglobal Some objects change scope during their lifetimebut only at well-defined junctures For example an mbuf ispart of the per-cpu queue while waiting to be processed(step 1 in Fig 4) but then becomes part of per-fg struc-tures during network processing

A global data structure the software flow group tablemanages the entry points into the per-fg structures IntelNICs conveniently store the RSS hash result as meta-dataalong with each received packet the bottom bits are used toindex into that table

We encountered one challenge in the management of TCPtimers We design our implementation to maximize normalexecution performance each thread independently managesits timers as a hierarchical timing wheel [49] ie as a per-cpu structure As a consequence a flow group migrationrequires a scan of the hierarchical wheels to identify andremove pending timers during the hold phase of Fig 5athose will be restored later in the destination thread at therelease phase of the flow group migration

45 Flow Group MigrationWhen adding or removing a thread the dynamic controllergenerates a set of migration requests Each individual re-quest is for a set of flow groups (fgs) currently handledby one elastic thread A to be handled by elastic thread B Tosimplify the implementation the controller serializes the mi-gration requests and the dataplane assumes that at most onesuch request is in progress at any point in time

Each thread has three queues that can hold incoming net-work packets the defaultQ contains packets received dur-ing steady state operation when no flow group migration ispending the remoteQ contains packets belonging to outgo-ing flow groups the localQ contains packets belonging toincoming flow groups The use of these two extra queuesis critical to ensure the processing of packets in the properorder by the networking stack

Fig 5 illustrates the migration steps in a thread-centricview (Fig 5a) and in a packet-centric view (Fig 5b) Thecontroller and the dataplane threads communicate via lock-free structures in shared memory First the controller signalsA to migrate fgs to B A first marks each flow group of theset fgs with a special tag to hold off normal processing onall threads moves packets which belong to the flow groupset fgs from defaultQ-A to remoteQ-B and stops alltimers belonging to the flow group set A then reprogramsthe RETA for all entries in fgs Packets still received byA will be appended to remoteQ-B packets received by Bwill go to localQ-B

Upon reception of the first packet whose flow group be-longs to fgs by B B signals A to initiate the final stageof migration Then B finalizes the migration by re-enablingfgsrsquos timers removing all migration tags and pre-pendingto its defaultQ-B the packets from remoteQ-B andthe packets from localQ-B Finally B notifies the con-trol plane that the operation is complete There are obviouslycorner-cases and subtleties in the algorithm For example Ainstalls a migration timer to finalize the migration even if Bhas not yet received any packets We set the timeout to 1 ms

5 EvaluationWe use three synthetic time-accelerated load patterns toevaluate the effectiveness of the control loop under stressfulconditions All three vary between nearly idle and maximumthroughput within a four minute period the slope patterngradually raises the target load from 0 and 62M RPS andthen reduces its load the step pattern increases load by 500KRPS every 10 seconds and finally the sine+noise patternis a basic sinusoidal pattern modified by randomly addingsharp noise that is uniformly distributed over [-250+250]KRPS and re-computed every 5 seconds The slope patternprovides a baseline to study smooth changes the step patternmodels abrupt and massive changes while the sine+noisepattern is representative of daily web patterns [48]

Fig 6 and Fig 7 show the results of these three dynamicload patterns for the energy proportionality and workloadconsolidation scenarios In each case the top figure mea-sures the observed throughput They are annotated with thecontrol loop events that add resources (green) or removethem (red) Empty triangles correspond to core allocationsand full triangles to DVFS changes The middle figure eval-uates the soundness of the algorithm and reports the 99thpercentile latency as observed by a client machine and re-ported every second Finally the bottom figures compare theoverall efficiency of our solution based on dynamic resourcecontrols with (i) the maximal static configuration using allcores and Turbo Boost and (ii) the ideal synthetic efficiencycomputed using the Pareto frontier of Fig 2Energy Proportionality Fig 6 shows the behavior for theenergy efficiency scenario In each case the workload tracksthe desired throughput of the pattern and exercises the en-tire sequence of configurations gradually adding cores en-abling hyperthreading increasing the frequency and finallyenabling Turbo Boost before doing it in reverse The steppattern is particularly challenging as the instant change inload level requires multiple back-to-back configurationschanges With a few exceptions the latencies remain wellbelow the 500micros SLO We further discuss the violations be-low Finally Fig 6 (bottom) compares the power dissipatedby the workload with the corresponding power levels as de-termined by the Pareto frontier (lower bound) or the max-imum static configuration (upper bound) This graph mea-

Smooth Step Sine+noiseEnergy Proportionality (W)

Max power 91 92 94Measured 42 (-54) 48 (-48) 53 (-44)Pareto bound 39 (-57) 41 (-55) 45 (-52)

Server consolidation opportunity ( of peak)Pareto bound 50 47 39Measured 46 39 32

Table 1 Energy Proportionality and Consolidation gains

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

max conf dynamic Pareto

0

20

40

60

80

100

0 50 100 150 200

Pow

er

(W)

Time (seconds)

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 6 Energy Proportionality of memcached running on IX for three load patterns

sures the effectiveness of the control loop to maximize en-ergy proportionality We observe that the dynamic (actuallymeasured) power curve tracks the Pareto (synthetic) curvewell which defines a bound on energy savings When the dy-namic resource controls enters Turbo Boost mode the mea-sured power in all three cases starts at the lower end of the4 W range and then gradually rises as expected Table 1shows that the three patterns have Pareto savings bounds of52 55 and 57 IXrsquos dynamic resource controls resultsin energy savings of 44 48 and 54 which is 85 87and 93 of the theoretical boundConsolidation Fig 7 shows the corresponding results forthe workload consolidation scenarios Fig 7 (top) measuresthe observed load which here as well tracks the desired loadRecall that the consolidation policy always operates at theprocessorrsquos nominal rate (or Turbo) which limits the numberof configuration changes Fig 7 (middle) similarly confirmsthat the system meets the SLO with few exceptions Fig 7(bottom) plots the throughput of the background batch appli-cation expressed as a percentage of its throughput on a dedi-cated processor at 24 Ghz We compare it only to the Paretooptimal upper bound as a maximum configuration wouldmonopolize cores and deliver zero background throughputTable 1 shows that for these three patterns our consolida-tion policy delivers 32ndash46 of the standalone throughput

of the background job which corresponds to 82ndash92 ofthe Pareto boundSLO violations A careful study of the SLO violations of the6 runs shows that they fall into two categories First there are16 violations caused by delays in packet processing due toflow group migrations resulting from the addition of a coreSecond there are 9 violations caused by abrupt increase ofthroughput mostly in the step pattern which occur beforeany flow migrations The control plane then reacts quickly(in sim100 ms) and accommodates to the new throughput byadjusting resources To further confirm the abrupt nature ofthroughput increase specific to the step pattern we note thatthe system performed up three consecutive increases in re-sources in order to resolve a single violation 23 of the 25total violations last a single second with the remaining twoviolations lasting two seconds We believe that the compli-ance with the SLO achieved by our system is more than ad-equate for any practical production deploymentFlow group migration analysis Table 2 measures the la-tency of the 552 flow group migrations that occur during the6 benchmarks as described in sect45 It also reports the totalnumber of packets whose processing is deferred during themigration (rather than dropped or reordered) We first ob-serve that migrations present distinct behaviors when scalingup and when scaling down the number of cores The differ-

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

0

20

40

60

80

100

0 50 100 150 200

o

f pea

k

Time (seconds)

dynamicPareto

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 7 Consolidation of memcached with a background batch task

ence can be intuitively explained since the migrations duringthe scale up are performed in a heavily loaded system whilethe system during the scale down is partially idle In absoluteterms migrations that occur when adding a core take 463microson average and less than 15 ms 95 of the time The outlierscan be explained by rare occurrences of longer preparationtimes or when processing up to 614 deferred packets

6 DiscussionUsing Pareto as a guide Even though the Pareto resultsare not used by the dynamic resource controller the Paretofrontier proved to be a valuable guide first to motivate andquantify the problem then to derive the configuration pol-icy sequence and finally to evaluate the effectiveness ofthe dynamic resource control by setting an upper bound onthe gains resulting from dynamic resource allocation Manyfactors such as software scalability hardware resource con-tention network and disk IO bottlenecks will influence thePareto frontier of any given application and therefore the de-rived control loop policies We will compare different work-loads in future workTradeoff between average latency and efficiency We usea one-dimensional SLO (99th pct le 500micros) and show thatwe can increase efficiency while maintaining that objectiveHowever this comes at a cost when considering the entirelatency distribution eg the average as well as the tail More

complex SLOs taking into account multiple aspects of la-tency distribution would define a different Pareto frontierand likely require adjustments to the control loopAdaptive flow-centric scheduling The new flow-groupmigration algorithm leads to a flow-centric approach to re-source scheduling where the network stack and applicationlogic always follow the steering decision POSIX applica-tions can balance flows by migrating file descriptors betweenthreads or processes but this tends to be inefficient becauseit is difficult for the kernel to match the flowrsquos destinationreceive queue to changes in CPU affinity Flow director canbe used by Linux to adjust the affinity of individual networkflows as a reactive measure to application-level and kernelthread migration rebalancing but the limited size of the redi-rection table prevents this mechanism from scaling to largeconnection counts By contrast our approach allows flowsto be migrated in entire groups improving efficiency and iscompatible with more scalable hardware flow steering mech-anisms based on RSSHardware trends Our experimental setup using one SandyBridge processor has a three-dimensional Pareto spaceClearly NUMA would add a new dimension Also thenewly released Haswell processors provide per-core DVFScontrols which further increases the Pareto space More-over hash filters for flow group steering could benefit fromrecent trends in NIC hardware For example Intelrsquos new

avg 95th pct max stddevad

dco

reprepare (micros) 67 231 521 93wait (micros) 135 541 983 190rpc (micros) 129 355 529 112deferred (micros) 101 401 1167 147total (micros) 463 1432 2870 459 packets 64 319 614 116

rem

ove

core prepare (micros) 25 54 398 32

wait (micros) 34 101 170 32rpc (micros) 12 25 49 7deferred (micros) 19 43 108 14total (micros) 93 160 456 47 packets 3 8 28 3

Table 2 Breakdown of flow group migration measured dur-ing the six benchmarks

XL710 chipset [19] has a 512 entry hash LUT (as well asindependent 64 entry LUTs for each VF) in contrast to the128 entries available in the 82599 chipset [17] used in ourevaluation This has the potential to reduce connection im-balances between cores especially with high core counts

7 Related WorkScheduling Scheduler activations [1] give applicationsgreater control over hardware threads and provide a mecha-nism for custom application-level scheduling Callisto [13]uses a similar strategy to improve the performance of co-located parallel runtime systems Our approach differs inthat an independent control plane manages the schedulingof hardware threads based on receive queuing latency indi-cators while the dataplane exposes a simple kernel thread-ing abstraction SEDA [51] also monitors queuing behaviorto make scheduling decisions such as thread pool sizingChronos [21] makes use of software-based flow steering butwith a focus on balancing load to reduce latency AffinityAccept [42] embraces a mixture of software and hardware-based flow steering in order to improve TCP connectionaffinity and increase throughput We focus instead on energyproportionality and workload consolidationEnergy Proportionality The energy proportionality prob-lem [3] has been well explored in previous work Somesystems have focused on solutions tailored to throughput-oriented workloads [33] or read-only workloads [23] Meis-ner et al [34] highlight unique challenges for low la-tency workloads and advocate full system active low-powermodes Similar to our system Pegasus [29] achieves CPUenergy proportionality for low latency workloads Our workexpands on Pegasus by exploring the elastic allocation ofhardware threads in combination with processor power man-agement states and by basing scheduling decisions on in-ternal latency metrics within a host endpoint instead of anexternal controller Niccolini et al show that a softwarerouter running on a dedicated machine can be made energy-

proportional [38] Similar to our approach queue lengthis used as a control signal to manage core allocation andDVFS settings However we focus on latency-sensitive ap-plications rather than middlebox traffic and consider theadditional case of workload consolidationCo-location Because host endpoints contain some com-ponents that are not energy proportional and thus are mostefficient when operating at 100 utilization co-location ofworkloads is also an important tool for improving energyefficiency At the cluster scheduler level BubbleUp [32] andParagon [10] make scheduling decisions that are interference-aware through efficient classification of the behavior ofworkload co-location Leverich et al [26] demonstratethat co-location of batch and low latency jobs is possi-ble on commodity operating systems Our approach ex-plores this issue at higher throughputs and with tighter la-tency SLOs Bubble-Flux [53] additionally controls back-ground threads we control background and latency-sensitivethreads CPI2 [54] detects performance interference by ob-serving changes in CPI and throttles offending jobs Thiswork is orthogonal to ours and could be a useful addi-tional signal for our control plane Heracles manages mul-tiple hardware and software isolation mechanisms includ-ing packet scheduling and cache partitioning to co-locatelatency-sensitive applications with batch tasks while main-taining millisecond SLOs [30] We limit our focus to DVFSand core assignment but target more aggressive SLOsLow-latency frameworks User-level networking stacks [2031 47] and RDMA applications [12 37 40] all rely onpolling typically by a fixed number of execution threadsto deliver micros-scale response times These stacks and appli-cations are evaluated based on their latency and throughputcharacteristics and do not take into account energy propor-tionality or workload consolidation considerations

8 ConclusionWe present the design and implementation of dynamic re-source controls for the IX dataplane which consists of acontroller that allocates cores and sets processor frequencyto adapt to changes in the load of latency-sensitive appli-cations The novel rebalancing mechanisms do not impactthe steady-state performance of the dataplane and can mi-grate a set of flow groups in milliseconds without droppingor reordering packets We develop two policies focused onoptimizing energy proportionality and workload consolida-tion Our results show that for three varying load patternsthe energy proportionality can save 44ndash54 of the pro-cessorrsquos energy or enable a background job to deliver 32ndash46 of its standalone throughput To evaluate the effective-ness of our approach we synthesize the Pareto frontier bycombining the behavior of all possible static configurationsOur policies deliver 85ndash93 of the Pareto optimal boundin terms of energy proportionality and 82ndash92 in termsof consolidation

AcknowledgementsThis work was supported by the MicrosoftEPFL Joint Re-search Center the Stanford Experimental Datacenter Laband NSF grant CNS-1422088 George Prekas is supportedby a Google Graduate Research Fellowship and Adam Be-lay by a Stanford Graduate Fellowship

References[1] T E Anderson B N Bershad E D Lazowska and H M

Levy Scheduler Activations Effective Kernel Support for theUser-Level Management of Parallelism ACM Trans ComputSyst 10(1)53ndash79 1992

[2] B Atikoglu Y Xu E Frachtenberg S Jiang andM Paleczny Workload Analysis of a Large-Scale Key-ValueStore In Proceedings of the 2012 ACM SIGMETRICS Inter-national Conference on Measurement and Modeling of Com-puter Systems pages 53ndash64 2012

[3] L A Barroso and U Holzle The Case for Energy-Proportional Computing IEEE Computer 40(12)33ndash372007

[4] L A Barroso J Clidaras and U Holzle The Datacenter as aComputer An Introduction to the Design of Warehouse-ScaleMachines Second Edition Synthesis Lectures on ComputerArchitecture Morgan amp Claypool Publishers 2013

[5] A Belay A Bittau A J Mashtizadeh D Terei D Mazieresand C Kozyrakis Dune Safe User-level Access to PrivilegedCPU Features In Proceedings of the 10th Symposium onOperating System Design and Implementation (OSDI) pages335ndash348 2012

[6] A Belay G Prekas A Klimovic S Grossman C Kozyrakisand E Bugnion IX A Protected Dataplane Operating Systemfor High Throughput and Low Latency In Proceedings of the11st Symposium on Operating System Design and Implemen-tation (OSDI) pages 49ndash65 2014

[7] J Bonwick and J Adams Magazines and Vmem Extendingthe Slab Allocator to Many CPUs and Arbitrary Resources InUSENIX Annual Technical Conference pages 15ndash33 2001

[8] H David E Gorbatov U R Hanebutte R Khanaa andC Le RAPL memory power estimation and capping InProceedings of the 2010 International Symposium on LowPower Electronics and Design pages 189ndash194 2010

[9] J Dean and L A Barroso The Tail at Scale Commun ACM56(2)74ndash80 2013

[10] C Delimitrou and C Kozyrakis Quality-of-Service-AwareScheduling in Heterogeneous Data centers with ParagonIEEE Micro 34(3)17ndash30 2014

[11] C Delimitrou and C Kozyrakis Quasar Resource-Efficientand QoS-Aware Cluster Management In Proceedings of the19th International Conference on Architectural Support forProgramming Languages and Operating Systems (ASPLOS-XIX) pages 127ndash144 2014

[12] A Dragojevic D Narayanan M Castro and O HodsonFaRM Fast Remote Memory In Proceedings of the 11stSymposium on Networked Systems Design and Implementa-tion (NSDI) pages 401ndash414 2014

[13] T Harris M Maas and V J Marathe Callisto co-schedulingparallel runtime systems In Proceedings of the 2014 EuroSysConference page 24 2014

[14] B Hindman A Konwinski M Zaharia A Ghodsi A DJoseph R H Katz S Shenker and I Stoica Mesos A Plat-form for Fine-Grained Resource Sharing in the Data CenterIn Proceedings of the 8th Symposium on Networked SystemsDesign and Implementation (NSDI) 2011

[15] C-H Hsu Y Zhang M A Laurenzano D Meisner T FWenisch J Mars L Tang and R G Dreslinski AdrenalinePinpointing and reining in tail queries with quick voltageboosting In Proceedings of the 21st IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages271ndash282 2015

[16] J Hwang K K Ramakrishnan and T Wood NetVM HighPerformance and Flexible Networking Using Virtualization onCommodity Platforms In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 445ndash458 2014

[17] Intel Corp Intel 82599 10 GbE Controller Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheets82599-10-gbe-controller-datasheetpdf2014

[18] Intel Corp Intel DPDK Data Plane Development Kit httpdpdkorg 2014

[19] Intel Corp Intel Ethernet Controller XL710 Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheetsxl710-10-40-controller-datasheetpdf2014

[20] E Jeong S Woo M A Jamshed H Jeong S Ihm D Hanand K Park mTCP a Highly Scalable User-level TCP Stackfor Multicore Systems In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 489ndash502 2014

[21] R Kapoor G Porter M Tewari G M Voelker and A Vah-dat Chronos Predictable Low Latency for Data Center Ap-plications In Proceedings of the 2012 ACM Symposium onCloud Computing (SOCC) page 9 2012

[22] W Kim M S Gupta G-Y Wei and D M Brooks Systemlevel analysis of fast per-core DVFS using on-chip switch-ing regulators In Proceedings of the 14th IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages123ndash134 2008

[23] A Krioukov P Mohan S Alspaugh L Keys D E Cullerand R H Katz NapSAC design and implementation of apower-proportional web cluster Computer CommunicationReview 41(1)102ndash108 2011

[24] K-C Leung V O K Li and D Yang An Overview ofPacket Reordering in Transmission Control Protocol (TCP)Problems Solutions and Challenges IEEE Trans ParallelDistrib Syst 18(4)522ndash535 2007

[25] J Leverich Mutilate High-Performance MemcachedLoad Generator httpsgithubcomleverichmutilate 2014

cially when load is highly variable DVFS is a necessary butinsufficient mechanism to control latency-critical applica-tions that rely on polling

The rest of this paper is organized as follows sect2 describesthe experimental setup sect3 characterizes resource usage us-ing static configurations and derives insights for dynamic al-location policies sect4 describes the design and implementa-tion of the mechanisms and policies for dynamic resourcemanagement in IX sect5 evaluates the impact of the dynamiccontroller for energy proportionality and workload consol-idation We discuss insights and open issues in sect6 relatedwork in sect7 and conclude in sect8

2 Workloads and Experimental SetupOur experimental setup consists of a cluster of 11 clientsand one server connected by a low-latency 10 GbE switch10 clients generate load and the 11th measures latency ofrequests to the latency-sensitive application The client ma-chines are a mix of Xeon E5-2637 35 Ghz and XeonE5-2650 26 Ghz The server is a Xeon E5-2665 24 Ghz with 256 GB of DRAM typical in datacenters Eachclient and server socket has 8 cores and 16 hyperthreads Allmachines are configured with one Intel x520 10 GbE NIC(82599EB chipset) Our baseline configuration in each ma-chine is an Ubuntu LTS 1404 distribution updated to the3161 Linux kernel Although the server has two socketsthe foreground and background applications run on a singleprocessor to avoid any NUMA effects We run the controlplane on the otherwise empty second socket but do not ac-count for its energy draw

Our primary foreground workload is memcached awidely deployed in-memory key-value store built on topof the libevent framework [35] It is frequently usedas a high-throughput low-latency caching tier in front ofpersistent database servers memcached is a network-bound application with threads spending over 75 of ex-ecution time in kernel mode for network processing whenusing Linux [26] It is a difficult application to scale be-cause the common deployments involve high connectioncounts for memcached servers and small-sized requestsand replies [2 39] Combined with latency SLOs in the fewhundreds of microseconds these characteristics make mem-cached a challenging workload for our study Any short-comings in the mechanisms and policies for energy propor-tionality and workload consolidation will quickly translateto throughput and latency problems In contrast workloadswith SLOs in the millisecond range are more forgivingFurthermore memcached has well-known scalability lim-itations [28] To improve its scalability we configure mem-cached with a larger hash table size (-o hashpower=20)and use a random replacement policy instead of the built-inLRU which requires a global lock This last change pro-vides memcached with similar multicore scalability as state-

of-the-art key-value stores such as MICA [28] We configurememcached similarly for Linux and IX

Our primary background application is a simple syntheticproxy for computationally intense batch processing applica-tions it encrypts a memory stream using SHA-1 in parallelthreads that each run in an infinite loop and report the aggre-gate throughput The workload streams through a 40 MB ar-ray which exceeds the capacity of the processorrsquos L3-cacheand therefore interferes (by design) with the memory sub-system performance of the latency-sensitive application

We use the mutilate load-generator to place a se-lected load on the foreground application in terms of re-quests per second (RPS) and measure response latency [25]Mutilate coordinates a large number of client threadsacross 10 machines to generate the desired RPS load andone additional unloaded client to measure latency for atotal of 2752 connections to the memcached server Weconfigure mutilate to generate load representative of theFacebook USR workload [2] this is a large-scale deploy-ment dominated by GET requests (99) with short keys(lt20B) and 2B values Each request is issued separately (nomultiget operations) However clients are permitted topipeline up to four requests per connection if needed to keepup with their target request rate We enhance mutilateto support dynamically changing load levels used in theseexperiments As the memcached server is multi-threadedwe also enhance the latency-measuring client to randomlychoose among 32 open connections for each request as toensure statistically significant measurements

We compute metrics as follows the unloaded client mea-sures latency by issuing one request at the time chosen ran-domly among its 32 open connections The SLO is spec-ified so that the 99th percentile of requests issued by thatunloaded client return in le 500micros Latency distributionachieved requests and power metrics for all graphs are re-ported at one-second intervals Energy consumption is mea-sured using the Intel RAPL counters [8] for the processor inquestion these provide an accurate model of the power drawof the CPU but ignore platforms effectsA note on Turbo Boost For any given throughput levelwe observe that the reported power utilization is stable forall CPU frequencies except for Turbo Boost When runningin Turbo Boost the temperature of the CPU gradually risesover a few minutes from 58C to 78C and with it the dissi-pated energy rises by 4 W for the same level of performanceThe experiments in sect31 run for a long time in Turbo Boostmode with a hot processor we therefore report those resultsas an energy band of 4 W

3 Understanding Dynamic Resource UsageThe purpose of dynamic resource allocation is to adapt tochanges of load of a latency-critical workload by adding orremoving resources as needed The remaining resources canbe placed in idle modes to save energy or used by other con-

0

20

40

60

80

100

0 2

Pow

er

(W)

Memcached RPS x 106 at SLO

(a) Energy Proportionality (Linux)

0

20

40

60

80

100

0 2 4 6 8

Memcached RPS x 106 at SLO

cfg - all others

cfg at Turbo mode

cfg w 8 cores + HT

cfg at 12 GHz

Pareto frontier

(b) Energy Proportionality (IX)

0

20

40

60

80

100

0 2

o

f P

eak f

or

Best-

eff

ort

Task

Memcached RPS x 106 at SLO

(c) Consolidation (Linux)

0

20

40

60

80

100

0 2 4 6 8

Memcached RPS x 106 at SLO

(d) Server Consolidation (IX)

Figure 2 Pareto efficiency for energy proportionality and workload consolidation for Linux 3161 and IX The Paretoefficiency is in red while the various static configurations are color-coded according to their distinctive characteristics

solidated applications In general dynamic control can beapplied to any resource including processor cores mem-ory and IO We focus on managing CPU cores becausethey are the largest contributor to power consumption anddraw significant amounts of energy even when lightly uti-lized [4 29] Moreover the ability to dynamically reassigncores between latency-critical and background applicationsis key to meet latency SLOs for the former and to make rea-sonable progress using underutilized resources for the latter

Dynamic resource control is a multi-dimensional prob-lem on current CPUs multi-core hyperthreading and fre-quency each separately impact the workloadrsquos performanceenergy consumption and consolidation opportunities sect31uses a Pareto methodology to evaluate this three-dimensionalspace and sect32 discusses the derived allocation policies

31 Pareto-Optimal Static ConfigurationsStatic resource configurations allow for controlled experi-ments to quantify the tradeoff between an applicationrsquos per-formance and the resources consumed Our approach limitsbias by considering many possible static configurations inthe three-dimensional space of core hyperthread and fre-quency For each static configuration we characterize themaximum load that meets the SLO (le 500micros99th per-

centile) we then measure the energy draw and throughputof the background job for all load levels up to the maximumload supported From this large data set we derive the setof meaningful static configurations and build the Pareto effi-ciency frontier [41] The frontier specifies for any possibleload level the optimal static configuration and the resultingminimal energy draw or maximum background throughputdepending on the scenario

Fig 2 presents the frontier for four experiments theenergy proportionality and workload consolidation scenar-ios for both Linux and IX The four graphs each plot theobjectivemdashwhich is either to minimize energy or maximizebackground throughputmdashas a function of the foregroundthroughput provided that the SLO is met Except for the redlines each line corresponds to a distinct static configurationof the system the green curves correspond to configurationat the minimal clock rate of 12 Ghz the blue curves use allavailable cores and hyperthreads other configurations are inblack In Turbo Boost mode the energy drawn is reported asa band since it depends on operating temperature (see sect2)

Finally the red line is the Pareto frontier which corre-sponds for any load level to the optimal result using any

0

20

40

60

80

100

0 2 4 6 8

Pow

er (

W)

Memcached RPS x 106 at SLO

Pareto IX

DVFS-only IX

Pareto Linux

DVFS-only Linux

Figure 3 Energy-proportionality comparison between thePareto-optimal frontier considering only DVFS adjustmentsand the full Pareto frontier considering core allocation hy-perthread allocations and frequency

of the static configurations available Each graph only showsthe static configurations that participate in the frontierEnergy proportionality We evaluate 224 distinct combi-nations from one to eight cores using consistently eitherone or two threads per core for 14 different DVFS levelsfrom 12 Ghz to 24 Ghz as well as Turbo Boost Fig 2a andFig 2b show the 43 and 32 static configurations (out of 224)that build the Pareto frontier for energy proportionality Thefigures confirm the intuition that (i) various static configura-tions have very different dynamic ranges beyond which theyare no longer able to meet the SLO (ii) each static config-uration draws substantially different levels of energy for thesame amount of work (iii) at the low-end of the curve manydistinct configurations operate at the minimal frequency of12 Ghz obviously with a different number of cores andthreads and contribute to the frontier these are shown ingreen in the Figure (iv) at the high-end of the range manyconfigurations operate with the maximum of 8 cores withdifferent frequencies including Turbo BoostConsolidation The methodology here is a little differentWe first characterize the background job and observe that itdelivers energy-proportional throughput up to 24 Ghz butthat Turbo Boost came at an energythroughput premiumConsequently we restrict the Pareto configuration space at24 Ghz the objective function is the throughput of the back-ground job expressed as a fraction of the throughput of thatsame job without any foreground application Backgroundjobs run on all cores that are not used by the foreground ap-plication Fig 2c and Fig 2d show the background through-put expressed as a fraction of the standalone throughput asa function of the foreground throughput provided that theforeground application meets the SLO as the foregroundapplication requires additional cores to meet the SLO thebackground throughput decreases proportionally

Linux vs IX Fig 2a and Fig 2b show that Linux and IXbehave differently (i) IX improves the throughput of Linuxby 48times (ii) IX can clearly take advantage of hyperthread-ing and has its best performance with configurations with16 threads on 8 cores In contrast Linux cannot take ad-vantage of hyperthreading and has its best performance with8 threads of control running on 8 cores (iii) the efficiencyof individual static configurations of Linux which relieson interrupts is much more subject to load than individ-ual static configurations of IX which relies on a pollingmodel As a consequence an improperly configured over-sized IX dataplane will be extremely energy-inefficient atlow-to-moderate loads (iv) the comparison of the Paretofrontiers shows that IXrsquos frontier always dominates the fron-tier of Linux ie provides better energy efficiency for thesame load level This suggests that a properly configureddataplane design is always more energy-efficient than anyLinux-based configuration The difference is substantial in-cluding at low-to-moderate load levels IXrsquos frontier is lessthan half of Linuxrsquos when the throughput load ge 780 KRPS(sim54 of Linuxrsquos capacity)DVFS-only alternative Fig 3 further analyzes the data andcompares the Pareto frontiers with an alternate frontier thatonly considers changes in DVFS frequency The comparisonis limited to the energy proportionality scenario for consoli-dation available cores are necessary for the background ap-plication to run and a DVFS-only solution would not suffice

We observe that the impact of DVFS-only controls differsnoticeably between Linux and IX with Linux the DVFS-only alternate frontier is very close to the Pareto frontiermeaning that a DVFS-only approach such as Pegasus [29] orAdrenaline [15] would be adequate This is due to Linuxrsquosidling behavior which saves resources In the case of IXhowevermdashand likely for any polling-based dataplanemdashaDVFS-only scheduler would provide worse energy propor-tionality at low-moderate loads than a corresponding Linux-based solution As many datacenter servers operate in the10-30 range [4] we conclude that a dynamic resourceallocation scheme involving both DVFS and core allocationis necessary for dataplane architectures

32 Derived PolicyWe use the results from sect31 to derive a resource config-uration policy framework whose purpose is to determinethe sequence of configurations to be applied as a functionof the load on the foreground application to both the fore-ground (latency-sensitive) and background (batch) applica-tions Specifically given an ever-increasing (or -decreasing)load on the foreground applications the goal is to determinethe sequence of resource configurations minimizing energyconsumption or maximizing background throughput respec-tively

We observe that (i) the latency-sensitive application(memcached) can scale nearly linearly up to the 8 cores of

the processor (ii) it benefits from running a second thread oneach core with a consistent speedup of 13times (iii) it is mostenergy-efficient to first utilize the various cores and onlythen to enable the second hyperthread on each core ratherthan the other way around and (iv) it is least energy-efficientto increase the frequency

We observe that the background application (i) also scaleslinearly but (ii) does not benefit from the 2nd hyperthread(iii) is nearly energy-proportional across the frequency spec-trum with the exception of Turbo Boost From a total cost ofownership perspective the most efficient operating point forthe workload consolidation of the background task is there-fore to run the system at the processorrsquos nominal 24 Ghzfrequency whenever possible We combine these observa-tions with the data from the Pareto analysis and derive thefollowing policiesEnergy Proportional Policy As a base state run with onlyone core and hyperthread with the socket set at the mini-mal clock rate (12Ghz) To add resources first enable ad-ditional cores then enable hyperthreads on all cores (as asingle step) and only after that gradually increase the clockrate until reaching the nominal rate (24Ghz) finally enableTurbo Boost To remove resources do the opposite This pol-icy leads to a sequence of 22 different configurationsWorkload Consolidation Policy As a base state run thebackground jobs on all available cores with the processor atthe nominal clock rate To add resources to the foregroundapplication first shift cores from the background thread tothe foreground application one at a time This is done by firstsuspending the background threads use both hyperthreadsof the newly freed core for the foreground application Nextstop the background job entirely and allocate all cores to theforeground applications As a final step enable Turbo BoostThis policy leads to a sequence of 9 different configurations

These policies closely track the corresponding Paretofrontier For energy proportionality (i) the 32 different staticconfigurations of the frontier are a superset of the configura-tions enabled by the policy and (ii) the difference in overallimpact in terms of energy spent is marginal For consolida-tion Pareto and policy nearly identically overlap

Obviously these conclusions may vary with platformsand applications but the use of underlying Pareto method-ology remains valid In fact it should provide a robust guideto determine allocation policies for more complex workloads(eg with natural scalability limitations) and more flexibleplatforms (eg the newer Haswell processors can set theclock frequency separately for individual cores)

4 Dynamic Resource Controls of DataplanesWe now present the design of the mechanisms and poli-cies that are necessary to implement dynamic resource con-trols in dataplane operating systems Although the imple-mentation is specific to the IX dataplane operating systemthe design principles generally apply to all high-throughput

dataplane operating systems such as Arrakis [43] and user-level networking stacks running latency-sensitive applica-tions [18 20 31] Traditionally such environments have notfocused on dynamic resource controls instead they focusedsolely on throughput with statically allocated resourceswithout considerations for the energy consumed Addingresource controls in such environments poses the followingchallenges

1 How to rebalance flows across cores without impactingnormal performance This is particularly challenging asdataplane environments achieve high performance be-cause of their coherence-free execution model By thatwe mean a programming where all common case oper-ations execute without requiring communication of syn-chronization between threads

2 How to rebalance flows across processing cores withoutimpacting network performance given that dataplane op-erating systems implement their own TCPIP network-ing stack and that TCPIP performance is negatively im-pacted whenever packets are either lost or processed outof order [24] This is particularly challenging wheneverthe rebalancing operation also involves the reconfigura-tion of the NIC which generates inherent race conditions

3 How to determinemdashboth robustly and efficientlymdashwhetherresources need to be added or can be removed from thedataplane given the complexity of modern applicationsand the difficulty to estimate the impact of resource con-trols on application throughput

Our design addresses these three challenges We firstprovide in sect41 the necessary background on the IX data-plane and in sect42 the background on the few assumptionsmade on the hardware sect43 describes the implementationof the controller which relies exclusively on application-independent queuing delay metrics to adjust resources sect44describes how a dataplane operating system can have both acoherence-free execution model and at the same time allowfor rebalancing of flows Finally sect45 describes the mecha-nism that migrates flows without reordering packets

41 Background on IX

The IX system architecture separates the control plane re-sponsible for the allocation of resources from the dataplanewhich operates on them using an approach similar to Ar-rakis [43] IX relies on hardware virtualization to protect thekernel from applications it uses Dune [5] to load and exe-cute the IX dataplane instances and uses the Linux host asits control plane environment

The IX dynamic controller is a Python daemon that con-figures and launches the IX dataplane then monitors its ex-ecution makes policy decisions and communicates them tothe IX dataplane and the host operating system

Unlike conventional operating systems which decouplenetwork processing from application execution the IX data-

app

libix

tcpip tcpip

timer

adaptive batch

1

2

4

5

Event

ConditionsBatched

Syscalls

3

6

kernel

user

Figure 4 Adaptive batching in the IX data plane

plane relies on a run to completion with adaptive batchingscheme to deliver both low latency and high throughput toapplications

Fig 4 illustrates the key run-to-completion mechanismbuilt into the original IX dataplane [6] each elastic threadhas exclusive access to NIC queues In step (1) the data-plane polls the receive descriptor ring and moves all receivedpackets into an in-memory queue and potentially posts freshbuffer descriptors to the NIC for use with future incomingpackets The thread then executes steps (2) ndash (6) for an adap-tive batch of packets the TCPIP network stack processesthese packets (2) and the application then consumes the re-sulting events (3) Upon return from user-space the threadprocesses batched systems calls and in particular the onesthat direct outgoing TCPIP traffic (4) runs necessary timers(5) and places outgoing Ethernet frames in the NICrsquos trans-mit descriptor ring for transmission (6)

Adaptive batching is specifically defined as follows (i)we never wait to batch requests and batching only occurs inthe presence of congestion (ii) we set an upper bound on thenumber of batched packets Using batching only during con-gestion allows us to minimize the impact on latency whilebounding the batch size prevents the live set from exceed-ing cache capacities and avoids transmit queue starvationBatching improves packet rate because it amortizes systemcall transition overheads and improves instruction cache lo-cality prefetching effectiveness and branch prediction ac-curacy When applied adaptively batching also decreases la-tency because these same efficiencies reduce head-of-lineblocking

42 Dynamic NIC ConfigurationModern NICs provide hardware mechanisms such as VMDqFlow Director [17] and Receive Side Scaling (RSS) [36] todirect traffic to one of multiple queues as an essential mech-anism for multicore scalability For example the Intel x520

relies on an on-chip table (called the RSS Redirection Ta-ble or RETA [17]) to partition flows into multiple queueson the data path the NIC first computes the Toeplitz hashfunction of the packet 5-tuple header (as specified by RSS)and uses the lower 7 bits of the result as an index into theRETA table A flow group is the set of network flows with anidentical RETA index The host updates the table using PCIewrites on a distinct control path which can lead to packetreordering [52]

Such anomalies have severe consequences in generalon the performance of the TCPIP networking stack [24]IXrsquos coherence-free design exposes a further constraint be-cause there are no locks within the TCPIP stack two elasticthreads cannot process packets from the same flow simulta-neously For correctness packets received by the thread thatdoes not own the flow group must be dropped This furthernegatively impacts network performance

The IX dataplane implements the actual flow group mi-gration and relies on the RETA to change the mappings Thechallenge is to design a migration mechanism that updatesthe steering of packets to different queues (and by extensionto different threads) without (i) impacting the steady stateperformance of the dataplane and its coherence-free designor (ii) creating network anomalies during migration such asdropping packets or processing them out of order in the net-working stack

43 Control LoopWe first describe the control loop at the heart of IXrsquos dy-namic controller implemented within the control plane Thecontroller adjusts processor resources by suspending andresuming IX elastic threads and specifying the mappingbetween flow groups and threads It migrates flow groupsfrom multiple existing threads to a newly resumed threadand conversely from a thread to be suspended to the re-maining threads For the server consolidation scenario thecontrol loop also allocates resources dedicated to the back-ground applications In our experimental setup it simply is-sues SIGSTOP and SIGCONT signals to transparently con-trol the background application

The control loop does not depend on any detailed charac-terization of the Pareto frontier of sect31 Instead it merely im-plements the derived high-level policy of sect32 using queuingdelay estimates Specifically the user sets the upper boundon the acceptable queuing delay in our experiments withmemcached characterized by high packet rates and low ser-vice times we set the acceptable bound at 300micros to minimizelatency violations given our SLO of 500micros at the 99th per-centile as measured end-to-end by the client machine

For this we rely on a key side effect of IXrsquos use of adap-tive batching unprocessed packets that form the backlogare queued in a central location namely in step (1) in thepipeline of Fig 4 Packets are then processed in order inbounded batches to completion through both the networkingstack and the application logic In other words each IX core

Prepare Wait RPC Defer processIXCPpy

migrate(fgs)

CPU A

update HW

hold

finalize

done

NIC

first

packet

release

CPU B

prepare

timeout

(a) Thread-centric view

NICHWQ-A

HWQ-B

RETA

FGPoll

Poll

defaultQ-A

remoteQ-B

defaultQ-B

localQ-B

Thread-A

Thread-B

(b) Packet-centric view

Figure 5 Flow-group Migration Algorithm

operates like a simple FIFO queuing server onto which clas-sic queuing and control theory principles can be easily ap-plied In contrast conventional operating systems distributebuffers throughout the system in the NIC (because of coa-lesced interrupts) in the driver before networking process-ing and in the socket layer before being consumed by theapplication Furthermore these conventional systems pro-vide no ordering guarantees across flows which makes itdifficult to pinpoint congestion

To estimate queuing delays the controller monitors theiteration time τ and the queue depth Q With B the maximalbatch size the tail latency issimmax(delay) = dQBelowastτ Thedataplane computes each instantaneous metric every 10msfor the previous 10ms interval As these metrics are subjectto jitter the dataplane computes the exponential weightedmoving averages using multiple smoothing factors (α) inparallel For example we track the queue depth as Q(tα) =α lowastQnow +(1minusα) lowastQ(tminus 1α) The control loop executesat a frequency of 10 Hz which is sufficient to adapt to loadchanges

Deciding when to remove resources is trickier than de-ciding when to add them as shallow and near-empty queuesdo not provide reliable metrics Instead we measure idletime and observe that each change in the configuration addsor removes a predictable level of throughput (i) increas-ing cores from n to n+ 1 increases throughput by a factorof 1n (ii) enabling hyperthreads adds 30 (for our fore-ground workload at least) and (iii) the corersquos performanceis largely proportional to its clock rate We can thereforedefine the function threshold[ f rom to] as the ratio of maxthroughput between the f rom and to configurations Whenoperating at configuration f rom and the idle time exceedsthreshold[ f rom to] we can safely switch to configuration towithout loss of performance

These settings are not universal as a number of assump-tions such as load changes requests bursts and variationsin request service times will noticeably affect the metricsThese are policies not mechanisms In particular there is an

inherent tradeoff between aggressive energy saving policieswhich may lead to SLO violations and conservative poli-cies which will consume an excessive amount of energyFortunately the policy can be easily modified the dataplaneexposes naturally all relevant metrics and the control planedaemon issim500 lines of Python includingsim80 lines for thecontrol loop

44 Data Structures for Coherence-free ExecutionWe designed our memory management and data structurelayout to minimize the performance impact of flow groupmigration and to preserve the coherence-free property of ourdataplane in the steady-state Our memory allocator drawsinspiration from magazine-based SLAB allocation [7] Eachthread maintains a thread-local cache of every data typeIf the size of the thread-local cache exceeds a thresholdmemory items are returned using traditional locking to ashared memory pool Such rebalancing events are commonduring flow group migration but infrequent during normaloperation

We further organize our data structures so the pointerreferences building linked lists hash tables etc are onlybetween objects either both permanently associated with thesame thread (per-cpu) or both permanently associated withthe same flow group (per-fg) or both global in scope Asa result we can efficiently transfer ownership of an entireflow-grouprsquos data structures during migration while avoidinglocking or coherence traffic during regular operation

For example the event condition arrays batch syscallarrays timer wheels incoming and outgoing descriptorrings posted and pending mbufs are all per-cpu Theprotocol control blocks 5-tuple hash table listening sock-ets TW WAIT queues out-of-order fragments and unac-knowledged sequences are all per-fg The ARP table isglobal Some objects change scope during their lifetimebut only at well-defined junctures For example an mbuf ispart of the per-cpu queue while waiting to be processed(step 1 in Fig 4) but then becomes part of per-fg struc-tures during network processing

A global data structure the software flow group tablemanages the entry points into the per-fg structures IntelNICs conveniently store the RSS hash result as meta-dataalong with each received packet the bottom bits are used toindex into that table

We encountered one challenge in the management of TCPtimers We design our implementation to maximize normalexecution performance each thread independently managesits timers as a hierarchical timing wheel [49] ie as a per-cpu structure As a consequence a flow group migrationrequires a scan of the hierarchical wheels to identify andremove pending timers during the hold phase of Fig 5athose will be restored later in the destination thread at therelease phase of the flow group migration

45 Flow Group MigrationWhen adding or removing a thread the dynamic controllergenerates a set of migration requests Each individual re-quest is for a set of flow groups (fgs) currently handledby one elastic thread A to be handled by elastic thread B Tosimplify the implementation the controller serializes the mi-gration requests and the dataplane assumes that at most onesuch request is in progress at any point in time

Each thread has three queues that can hold incoming net-work packets the defaultQ contains packets received dur-ing steady state operation when no flow group migration ispending the remoteQ contains packets belonging to outgo-ing flow groups the localQ contains packets belonging toincoming flow groups The use of these two extra queuesis critical to ensure the processing of packets in the properorder by the networking stack

Fig 5 illustrates the migration steps in a thread-centricview (Fig 5a) and in a packet-centric view (Fig 5b) Thecontroller and the dataplane threads communicate via lock-free structures in shared memory First the controller signalsA to migrate fgs to B A first marks each flow group of theset fgs with a special tag to hold off normal processing onall threads moves packets which belong to the flow groupset fgs from defaultQ-A to remoteQ-B and stops alltimers belonging to the flow group set A then reprogramsthe RETA for all entries in fgs Packets still received byA will be appended to remoteQ-B packets received by Bwill go to localQ-B

Upon reception of the first packet whose flow group be-longs to fgs by B B signals A to initiate the final stageof migration Then B finalizes the migration by re-enablingfgsrsquos timers removing all migration tags and pre-pendingto its defaultQ-B the packets from remoteQ-B andthe packets from localQ-B Finally B notifies the con-trol plane that the operation is complete There are obviouslycorner-cases and subtleties in the algorithm For example Ainstalls a migration timer to finalize the migration even if Bhas not yet received any packets We set the timeout to 1 ms

5 EvaluationWe use three synthetic time-accelerated load patterns toevaluate the effectiveness of the control loop under stressfulconditions All three vary between nearly idle and maximumthroughput within a four minute period the slope patterngradually raises the target load from 0 and 62M RPS andthen reduces its load the step pattern increases load by 500KRPS every 10 seconds and finally the sine+noise patternis a basic sinusoidal pattern modified by randomly addingsharp noise that is uniformly distributed over [-250+250]KRPS and re-computed every 5 seconds The slope patternprovides a baseline to study smooth changes the step patternmodels abrupt and massive changes while the sine+noisepattern is representative of daily web patterns [48]

Fig 6 and Fig 7 show the results of these three dynamicload patterns for the energy proportionality and workloadconsolidation scenarios In each case the top figure mea-sures the observed throughput They are annotated with thecontrol loop events that add resources (green) or removethem (red) Empty triangles correspond to core allocationsand full triangles to DVFS changes The middle figure eval-uates the soundness of the algorithm and reports the 99thpercentile latency as observed by a client machine and re-ported every second Finally the bottom figures compare theoverall efficiency of our solution based on dynamic resourcecontrols with (i) the maximal static configuration using allcores and Turbo Boost and (ii) the ideal synthetic efficiencycomputed using the Pareto frontier of Fig 2Energy Proportionality Fig 6 shows the behavior for theenergy efficiency scenario In each case the workload tracksthe desired throughput of the pattern and exercises the en-tire sequence of configurations gradually adding cores en-abling hyperthreading increasing the frequency and finallyenabling Turbo Boost before doing it in reverse The steppattern is particularly challenging as the instant change inload level requires multiple back-to-back configurationschanges With a few exceptions the latencies remain wellbelow the 500micros SLO We further discuss the violations be-low Finally Fig 6 (bottom) compares the power dissipatedby the workload with the corresponding power levels as de-termined by the Pareto frontier (lower bound) or the max-imum static configuration (upper bound) This graph mea-

Smooth Step Sine+noiseEnergy Proportionality (W)

Max power 91 92 94Measured 42 (-54) 48 (-48) 53 (-44)Pareto bound 39 (-57) 41 (-55) 45 (-52)

Server consolidation opportunity ( of peak)Pareto bound 50 47 39Measured 46 39 32

Table 1 Energy Proportionality and Consolidation gains

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

max conf dynamic Pareto

0

20

40

60

80

100

0 50 100 150 200

Pow

er

(W)

Time (seconds)

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 6 Energy Proportionality of memcached running on IX for three load patterns

sures the effectiveness of the control loop to maximize en-ergy proportionality We observe that the dynamic (actuallymeasured) power curve tracks the Pareto (synthetic) curvewell which defines a bound on energy savings When the dy-namic resource controls enters Turbo Boost mode the mea-sured power in all three cases starts at the lower end of the4 W range and then gradually rises as expected Table 1shows that the three patterns have Pareto savings bounds of52 55 and 57 IXrsquos dynamic resource controls resultsin energy savings of 44 48 and 54 which is 85 87and 93 of the theoretical boundConsolidation Fig 7 shows the corresponding results forthe workload consolidation scenarios Fig 7 (top) measuresthe observed load which here as well tracks the desired loadRecall that the consolidation policy always operates at theprocessorrsquos nominal rate (or Turbo) which limits the numberof configuration changes Fig 7 (middle) similarly confirmsthat the system meets the SLO with few exceptions Fig 7(bottom) plots the throughput of the background batch appli-cation expressed as a percentage of its throughput on a dedi-cated processor at 24 Ghz We compare it only to the Paretooptimal upper bound as a maximum configuration wouldmonopolize cores and deliver zero background throughputTable 1 shows that for these three patterns our consolida-tion policy delivers 32ndash46 of the standalone throughput

of the background job which corresponds to 82ndash92 ofthe Pareto boundSLO violations A careful study of the SLO violations of the6 runs shows that they fall into two categories First there are16 violations caused by delays in packet processing due toflow group migrations resulting from the addition of a coreSecond there are 9 violations caused by abrupt increase ofthroughput mostly in the step pattern which occur beforeany flow migrations The control plane then reacts quickly(in sim100 ms) and accommodates to the new throughput byadjusting resources To further confirm the abrupt nature ofthroughput increase specific to the step pattern we note thatthe system performed up three consecutive increases in re-sources in order to resolve a single violation 23 of the 25total violations last a single second with the remaining twoviolations lasting two seconds We believe that the compli-ance with the SLO achieved by our system is more than ad-equate for any practical production deploymentFlow group migration analysis Table 2 measures the la-tency of the 552 flow group migrations that occur during the6 benchmarks as described in sect45 It also reports the totalnumber of packets whose processing is deferred during themigration (rather than dropped or reordered) We first ob-serve that migrations present distinct behaviors when scalingup and when scaling down the number of cores The differ-

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

0

20

40

60

80

100

0 50 100 150 200

o

f pea

k

Time (seconds)

dynamicPareto

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 7 Consolidation of memcached with a background batch task

ence can be intuitively explained since the migrations duringthe scale up are performed in a heavily loaded system whilethe system during the scale down is partially idle In absoluteterms migrations that occur when adding a core take 463microson average and less than 15 ms 95 of the time The outlierscan be explained by rare occurrences of longer preparationtimes or when processing up to 614 deferred packets

6 DiscussionUsing Pareto as a guide Even though the Pareto resultsare not used by the dynamic resource controller the Paretofrontier proved to be a valuable guide first to motivate andquantify the problem then to derive the configuration pol-icy sequence and finally to evaluate the effectiveness ofthe dynamic resource control by setting an upper bound onthe gains resulting from dynamic resource allocation Manyfactors such as software scalability hardware resource con-tention network and disk IO bottlenecks will influence thePareto frontier of any given application and therefore the de-rived control loop policies We will compare different work-loads in future workTradeoff between average latency and efficiency We usea one-dimensional SLO (99th pct le 500micros) and show thatwe can increase efficiency while maintaining that objectiveHowever this comes at a cost when considering the entirelatency distribution eg the average as well as the tail More

complex SLOs taking into account multiple aspects of la-tency distribution would define a different Pareto frontierand likely require adjustments to the control loopAdaptive flow-centric scheduling The new flow-groupmigration algorithm leads to a flow-centric approach to re-source scheduling where the network stack and applicationlogic always follow the steering decision POSIX applica-tions can balance flows by migrating file descriptors betweenthreads or processes but this tends to be inefficient becauseit is difficult for the kernel to match the flowrsquos destinationreceive queue to changes in CPU affinity Flow director canbe used by Linux to adjust the affinity of individual networkflows as a reactive measure to application-level and kernelthread migration rebalancing but the limited size of the redi-rection table prevents this mechanism from scaling to largeconnection counts By contrast our approach allows flowsto be migrated in entire groups improving efficiency and iscompatible with more scalable hardware flow steering mech-anisms based on RSSHardware trends Our experimental setup using one SandyBridge processor has a three-dimensional Pareto spaceClearly NUMA would add a new dimension Also thenewly released Haswell processors provide per-core DVFScontrols which further increases the Pareto space More-over hash filters for flow group steering could benefit fromrecent trends in NIC hardware For example Intelrsquos new

avg 95th pct max stddevad

dco

reprepare (micros) 67 231 521 93wait (micros) 135 541 983 190rpc (micros) 129 355 529 112deferred (micros) 101 401 1167 147total (micros) 463 1432 2870 459 packets 64 319 614 116

rem

ove

core prepare (micros) 25 54 398 32

wait (micros) 34 101 170 32rpc (micros) 12 25 49 7deferred (micros) 19 43 108 14total (micros) 93 160 456 47 packets 3 8 28 3

Table 2 Breakdown of flow group migration measured dur-ing the six benchmarks

XL710 chipset [19] has a 512 entry hash LUT (as well asindependent 64 entry LUTs for each VF) in contrast to the128 entries available in the 82599 chipset [17] used in ourevaluation This has the potential to reduce connection im-balances between cores especially with high core counts

7 Related WorkScheduling Scheduler activations [1] give applicationsgreater control over hardware threads and provide a mecha-nism for custom application-level scheduling Callisto [13]uses a similar strategy to improve the performance of co-located parallel runtime systems Our approach differs inthat an independent control plane manages the schedulingof hardware threads based on receive queuing latency indi-cators while the dataplane exposes a simple kernel thread-ing abstraction SEDA [51] also monitors queuing behaviorto make scheduling decisions such as thread pool sizingChronos [21] makes use of software-based flow steering butwith a focus on balancing load to reduce latency AffinityAccept [42] embraces a mixture of software and hardware-based flow steering in order to improve TCP connectionaffinity and increase throughput We focus instead on energyproportionality and workload consolidationEnergy Proportionality The energy proportionality prob-lem [3] has been well explored in previous work Somesystems have focused on solutions tailored to throughput-oriented workloads [33] or read-only workloads [23] Meis-ner et al [34] highlight unique challenges for low la-tency workloads and advocate full system active low-powermodes Similar to our system Pegasus [29] achieves CPUenergy proportionality for low latency workloads Our workexpands on Pegasus by exploring the elastic allocation ofhardware threads in combination with processor power man-agement states and by basing scheduling decisions on in-ternal latency metrics within a host endpoint instead of anexternal controller Niccolini et al show that a softwarerouter running on a dedicated machine can be made energy-

proportional [38] Similar to our approach queue lengthis used as a control signal to manage core allocation andDVFS settings However we focus on latency-sensitive ap-plications rather than middlebox traffic and consider theadditional case of workload consolidationCo-location Because host endpoints contain some com-ponents that are not energy proportional and thus are mostefficient when operating at 100 utilization co-location ofworkloads is also an important tool for improving energyefficiency At the cluster scheduler level BubbleUp [32] andParagon [10] make scheduling decisions that are interference-aware through efficient classification of the behavior ofworkload co-location Leverich et al [26] demonstratethat co-location of batch and low latency jobs is possi-ble on commodity operating systems Our approach ex-plores this issue at higher throughputs and with tighter la-tency SLOs Bubble-Flux [53] additionally controls back-ground threads we control background and latency-sensitivethreads CPI2 [54] detects performance interference by ob-serving changes in CPI and throttles offending jobs Thiswork is orthogonal to ours and could be a useful addi-tional signal for our control plane Heracles manages mul-tiple hardware and software isolation mechanisms includ-ing packet scheduling and cache partitioning to co-locatelatency-sensitive applications with batch tasks while main-taining millisecond SLOs [30] We limit our focus to DVFSand core assignment but target more aggressive SLOsLow-latency frameworks User-level networking stacks [2031 47] and RDMA applications [12 37 40] all rely onpolling typically by a fixed number of execution threadsto deliver micros-scale response times These stacks and appli-cations are evaluated based on their latency and throughputcharacteristics and do not take into account energy propor-tionality or workload consolidation considerations

8 ConclusionWe present the design and implementation of dynamic re-source controls for the IX dataplane which consists of acontroller that allocates cores and sets processor frequencyto adapt to changes in the load of latency-sensitive appli-cations The novel rebalancing mechanisms do not impactthe steady-state performance of the dataplane and can mi-grate a set of flow groups in milliseconds without droppingor reordering packets We develop two policies focused onoptimizing energy proportionality and workload consolida-tion Our results show that for three varying load patternsthe energy proportionality can save 44ndash54 of the pro-cessorrsquos energy or enable a background job to deliver 32ndash46 of its standalone throughput To evaluate the effective-ness of our approach we synthesize the Pareto frontier bycombining the behavior of all possible static configurationsOur policies deliver 85ndash93 of the Pareto optimal boundin terms of energy proportionality and 82ndash92 in termsof consolidation

AcknowledgementsThis work was supported by the MicrosoftEPFL Joint Re-search Center the Stanford Experimental Datacenter Laband NSF grant CNS-1422088 George Prekas is supportedby a Google Graduate Research Fellowship and Adam Be-lay by a Stanford Graduate Fellowship

References[1] T E Anderson B N Bershad E D Lazowska and H M

Levy Scheduler Activations Effective Kernel Support for theUser-Level Management of Parallelism ACM Trans ComputSyst 10(1)53ndash79 1992

[2] B Atikoglu Y Xu E Frachtenberg S Jiang andM Paleczny Workload Analysis of a Large-Scale Key-ValueStore In Proceedings of the 2012 ACM SIGMETRICS Inter-national Conference on Measurement and Modeling of Com-puter Systems pages 53ndash64 2012

[3] L A Barroso and U Holzle The Case for Energy-Proportional Computing IEEE Computer 40(12)33ndash372007

[4] L A Barroso J Clidaras and U Holzle The Datacenter as aComputer An Introduction to the Design of Warehouse-ScaleMachines Second Edition Synthesis Lectures on ComputerArchitecture Morgan amp Claypool Publishers 2013

[5] A Belay A Bittau A J Mashtizadeh D Terei D Mazieresand C Kozyrakis Dune Safe User-level Access to PrivilegedCPU Features In Proceedings of the 10th Symposium onOperating System Design and Implementation (OSDI) pages335ndash348 2012

[6] A Belay G Prekas A Klimovic S Grossman C Kozyrakisand E Bugnion IX A Protected Dataplane Operating Systemfor High Throughput and Low Latency In Proceedings of the11st Symposium on Operating System Design and Implemen-tation (OSDI) pages 49ndash65 2014

[7] J Bonwick and J Adams Magazines and Vmem Extendingthe Slab Allocator to Many CPUs and Arbitrary Resources InUSENIX Annual Technical Conference pages 15ndash33 2001

[8] H David E Gorbatov U R Hanebutte R Khanaa andC Le RAPL memory power estimation and capping InProceedings of the 2010 International Symposium on LowPower Electronics and Design pages 189ndash194 2010

[9] J Dean and L A Barroso The Tail at Scale Commun ACM56(2)74ndash80 2013

[10] C Delimitrou and C Kozyrakis Quality-of-Service-AwareScheduling in Heterogeneous Data centers with ParagonIEEE Micro 34(3)17ndash30 2014

[11] C Delimitrou and C Kozyrakis Quasar Resource-Efficientand QoS-Aware Cluster Management In Proceedings of the19th International Conference on Architectural Support forProgramming Languages and Operating Systems (ASPLOS-XIX) pages 127ndash144 2014

[12] A Dragojevic D Narayanan M Castro and O HodsonFaRM Fast Remote Memory In Proceedings of the 11stSymposium on Networked Systems Design and Implementa-tion (NSDI) pages 401ndash414 2014

[13] T Harris M Maas and V J Marathe Callisto co-schedulingparallel runtime systems In Proceedings of the 2014 EuroSysConference page 24 2014

[14] B Hindman A Konwinski M Zaharia A Ghodsi A DJoseph R H Katz S Shenker and I Stoica Mesos A Plat-form for Fine-Grained Resource Sharing in the Data CenterIn Proceedings of the 8th Symposium on Networked SystemsDesign and Implementation (NSDI) 2011

[15] C-H Hsu Y Zhang M A Laurenzano D Meisner T FWenisch J Mars L Tang and R G Dreslinski AdrenalinePinpointing and reining in tail queries with quick voltageboosting In Proceedings of the 21st IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages271ndash282 2015

[16] J Hwang K K Ramakrishnan and T Wood NetVM HighPerformance and Flexible Networking Using Virtualization onCommodity Platforms In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 445ndash458 2014

[17] Intel Corp Intel 82599 10 GbE Controller Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheets82599-10-gbe-controller-datasheetpdf2014

[18] Intel Corp Intel DPDK Data Plane Development Kit httpdpdkorg 2014

[19] Intel Corp Intel Ethernet Controller XL710 Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheetsxl710-10-40-controller-datasheetpdf2014

[20] E Jeong S Woo M A Jamshed H Jeong S Ihm D Hanand K Park mTCP a Highly Scalable User-level TCP Stackfor Multicore Systems In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 489ndash502 2014

[21] R Kapoor G Porter M Tewari G M Voelker and A Vah-dat Chronos Predictable Low Latency for Data Center Ap-plications In Proceedings of the 2012 ACM Symposium onCloud Computing (SOCC) page 9 2012

[22] W Kim M S Gupta G-Y Wei and D M Brooks Systemlevel analysis of fast per-core DVFS using on-chip switch-ing regulators In Proceedings of the 14th IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages123ndash134 2008

[23] A Krioukov P Mohan S Alspaugh L Keys D E Cullerand R H Katz NapSAC design and implementation of apower-proportional web cluster Computer CommunicationReview 41(1)102ndash108 2011

[24] K-C Leung V O K Li and D Yang An Overview ofPacket Reordering in Transmission Control Protocol (TCP)Problems Solutions and Challenges IEEE Trans ParallelDistrib Syst 18(4)522ndash535 2007

[25] J Leverich Mutilate High-Performance MemcachedLoad Generator httpsgithubcomleverichmutilate 2014

0

20

40

60

80

100

0 2

Pow

er

(W)

Memcached RPS x 106 at SLO

(a) Energy Proportionality (Linux)

0

20

40

60

80

100

0 2 4 6 8

Memcached RPS x 106 at SLO

cfg - all others

cfg at Turbo mode

cfg w 8 cores + HT

cfg at 12 GHz

Pareto frontier

(b) Energy Proportionality (IX)

0

20

40

60

80

100

0 2

o

f P

eak f

or

Best-

eff

ort

Task

Memcached RPS x 106 at SLO

(c) Consolidation (Linux)

0

20

40

60

80

100

0 2 4 6 8

Memcached RPS x 106 at SLO

(d) Server Consolidation (IX)

Figure 2 Pareto efficiency for energy proportionality and workload consolidation for Linux 3161 and IX The Paretoefficiency is in red while the various static configurations are color-coded according to their distinctive characteristics

solidated applications In general dynamic control can beapplied to any resource including processor cores mem-ory and IO We focus on managing CPU cores becausethey are the largest contributor to power consumption anddraw significant amounts of energy even when lightly uti-lized [4 29] Moreover the ability to dynamically reassigncores between latency-critical and background applicationsis key to meet latency SLOs for the former and to make rea-sonable progress using underutilized resources for the latter

Dynamic resource control is a multi-dimensional prob-lem on current CPUs multi-core hyperthreading and fre-quency each separately impact the workloadrsquos performanceenergy consumption and consolidation opportunities sect31uses a Pareto methodology to evaluate this three-dimensionalspace and sect32 discusses the derived allocation policies

31 Pareto-Optimal Static ConfigurationsStatic resource configurations allow for controlled experi-ments to quantify the tradeoff between an applicationrsquos per-formance and the resources consumed Our approach limitsbias by considering many possible static configurations inthe three-dimensional space of core hyperthread and fre-quency For each static configuration we characterize themaximum load that meets the SLO (le 500micros99th per-

centile) we then measure the energy draw and throughputof the background job for all load levels up to the maximumload supported From this large data set we derive the setof meaningful static configurations and build the Pareto effi-ciency frontier [41] The frontier specifies for any possibleload level the optimal static configuration and the resultingminimal energy draw or maximum background throughputdepending on the scenario

Fig 2 presents the frontier for four experiments theenergy proportionality and workload consolidation scenar-ios for both Linux and IX The four graphs each plot theobjectivemdashwhich is either to minimize energy or maximizebackground throughputmdashas a function of the foregroundthroughput provided that the SLO is met Except for the redlines each line corresponds to a distinct static configurationof the system the green curves correspond to configurationat the minimal clock rate of 12 Ghz the blue curves use allavailable cores and hyperthreads other configurations are inblack In Turbo Boost mode the energy drawn is reported asa band since it depends on operating temperature (see sect2)

Finally the red line is the Pareto frontier which corre-sponds for any load level to the optimal result using any

0

20

40

60

80

100

0 2 4 6 8

Pow

er (

W)

Memcached RPS x 106 at SLO

Pareto IX

DVFS-only IX

Pareto Linux

DVFS-only Linux

Figure 3 Energy-proportionality comparison between thePareto-optimal frontier considering only DVFS adjustmentsand the full Pareto frontier considering core allocation hy-perthread allocations and frequency

of the static configurations available Each graph only showsthe static configurations that participate in the frontierEnergy proportionality We evaluate 224 distinct combi-nations from one to eight cores using consistently eitherone or two threads per core for 14 different DVFS levelsfrom 12 Ghz to 24 Ghz as well as Turbo Boost Fig 2a andFig 2b show the 43 and 32 static configurations (out of 224)that build the Pareto frontier for energy proportionality Thefigures confirm the intuition that (i) various static configura-tions have very different dynamic ranges beyond which theyare no longer able to meet the SLO (ii) each static config-uration draws substantially different levels of energy for thesame amount of work (iii) at the low-end of the curve manydistinct configurations operate at the minimal frequency of12 Ghz obviously with a different number of cores andthreads and contribute to the frontier these are shown ingreen in the Figure (iv) at the high-end of the range manyconfigurations operate with the maximum of 8 cores withdifferent frequencies including Turbo BoostConsolidation The methodology here is a little differentWe first characterize the background job and observe that itdelivers energy-proportional throughput up to 24 Ghz butthat Turbo Boost came at an energythroughput premiumConsequently we restrict the Pareto configuration space at24 Ghz the objective function is the throughput of the back-ground job expressed as a fraction of the throughput of thatsame job without any foreground application Backgroundjobs run on all cores that are not used by the foreground ap-plication Fig 2c and Fig 2d show the background through-put expressed as a fraction of the standalone throughput asa function of the foreground throughput provided that theforeground application meets the SLO as the foregroundapplication requires additional cores to meet the SLO thebackground throughput decreases proportionally

Linux vs IX Fig 2a and Fig 2b show that Linux and IXbehave differently (i) IX improves the throughput of Linuxby 48times (ii) IX can clearly take advantage of hyperthread-ing and has its best performance with configurations with16 threads on 8 cores In contrast Linux cannot take ad-vantage of hyperthreading and has its best performance with8 threads of control running on 8 cores (iii) the efficiencyof individual static configurations of Linux which relieson interrupts is much more subject to load than individ-ual static configurations of IX which relies on a pollingmodel As a consequence an improperly configured over-sized IX dataplane will be extremely energy-inefficient atlow-to-moderate loads (iv) the comparison of the Paretofrontiers shows that IXrsquos frontier always dominates the fron-tier of Linux ie provides better energy efficiency for thesame load level This suggests that a properly configureddataplane design is always more energy-efficient than anyLinux-based configuration The difference is substantial in-cluding at low-to-moderate load levels IXrsquos frontier is lessthan half of Linuxrsquos when the throughput load ge 780 KRPS(sim54 of Linuxrsquos capacity)DVFS-only alternative Fig 3 further analyzes the data andcompares the Pareto frontiers with an alternate frontier thatonly considers changes in DVFS frequency The comparisonis limited to the energy proportionality scenario for consoli-dation available cores are necessary for the background ap-plication to run and a DVFS-only solution would not suffice

We observe that the impact of DVFS-only controls differsnoticeably between Linux and IX with Linux the DVFS-only alternate frontier is very close to the Pareto frontiermeaning that a DVFS-only approach such as Pegasus [29] orAdrenaline [15] would be adequate This is due to Linuxrsquosidling behavior which saves resources In the case of IXhowevermdashand likely for any polling-based dataplanemdashaDVFS-only scheduler would provide worse energy propor-tionality at low-moderate loads than a corresponding Linux-based solution As many datacenter servers operate in the10-30 range [4] we conclude that a dynamic resourceallocation scheme involving both DVFS and core allocationis necessary for dataplane architectures

32 Derived PolicyWe use the results from sect31 to derive a resource config-uration policy framework whose purpose is to determinethe sequence of configurations to be applied as a functionof the load on the foreground application to both the fore-ground (latency-sensitive) and background (batch) applica-tions Specifically given an ever-increasing (or -decreasing)load on the foreground applications the goal is to determinethe sequence of resource configurations minimizing energyconsumption or maximizing background throughput respec-tively

We observe that (i) the latency-sensitive application(memcached) can scale nearly linearly up to the 8 cores of

the processor (ii) it benefits from running a second thread oneach core with a consistent speedup of 13times (iii) it is mostenergy-efficient to first utilize the various cores and onlythen to enable the second hyperthread on each core ratherthan the other way around and (iv) it is least energy-efficientto increase the frequency

We observe that the background application (i) also scaleslinearly but (ii) does not benefit from the 2nd hyperthread(iii) is nearly energy-proportional across the frequency spec-trum with the exception of Turbo Boost From a total cost ofownership perspective the most efficient operating point forthe workload consolidation of the background task is there-fore to run the system at the processorrsquos nominal 24 Ghzfrequency whenever possible We combine these observa-tions with the data from the Pareto analysis and derive thefollowing policiesEnergy Proportional Policy As a base state run with onlyone core and hyperthread with the socket set at the mini-mal clock rate (12Ghz) To add resources first enable ad-ditional cores then enable hyperthreads on all cores (as asingle step) and only after that gradually increase the clockrate until reaching the nominal rate (24Ghz) finally enableTurbo Boost To remove resources do the opposite This pol-icy leads to a sequence of 22 different configurationsWorkload Consolidation Policy As a base state run thebackground jobs on all available cores with the processor atthe nominal clock rate To add resources to the foregroundapplication first shift cores from the background thread tothe foreground application one at a time This is done by firstsuspending the background threads use both hyperthreadsof the newly freed core for the foreground application Nextstop the background job entirely and allocate all cores to theforeground applications As a final step enable Turbo BoostThis policy leads to a sequence of 9 different configurations

These policies closely track the corresponding Paretofrontier For energy proportionality (i) the 32 different staticconfigurations of the frontier are a superset of the configura-tions enabled by the policy and (ii) the difference in overallimpact in terms of energy spent is marginal For consolida-tion Pareto and policy nearly identically overlap

Obviously these conclusions may vary with platformsand applications but the use of underlying Pareto method-ology remains valid In fact it should provide a robust guideto determine allocation policies for more complex workloads(eg with natural scalability limitations) and more flexibleplatforms (eg the newer Haswell processors can set theclock frequency separately for individual cores)

4 Dynamic Resource Controls of DataplanesWe now present the design of the mechanisms and poli-cies that are necessary to implement dynamic resource con-trols in dataplane operating systems Although the imple-mentation is specific to the IX dataplane operating systemthe design principles generally apply to all high-throughput

dataplane operating systems such as Arrakis [43] and user-level networking stacks running latency-sensitive applica-tions [18 20 31] Traditionally such environments have notfocused on dynamic resource controls instead they focusedsolely on throughput with statically allocated resourceswithout considerations for the energy consumed Addingresource controls in such environments poses the followingchallenges

1 How to rebalance flows across cores without impactingnormal performance This is particularly challenging asdataplane environments achieve high performance be-cause of their coherence-free execution model By thatwe mean a programming where all common case oper-ations execute without requiring communication of syn-chronization between threads

2 How to rebalance flows across processing cores withoutimpacting network performance given that dataplane op-erating systems implement their own TCPIP network-ing stack and that TCPIP performance is negatively im-pacted whenever packets are either lost or processed outof order [24] This is particularly challenging wheneverthe rebalancing operation also involves the reconfigura-tion of the NIC which generates inherent race conditions

3 How to determinemdashboth robustly and efficientlymdashwhetherresources need to be added or can be removed from thedataplane given the complexity of modern applicationsand the difficulty to estimate the impact of resource con-trols on application throughput

Our design addresses these three challenges We firstprovide in sect41 the necessary background on the IX data-plane and in sect42 the background on the few assumptionsmade on the hardware sect43 describes the implementationof the controller which relies exclusively on application-independent queuing delay metrics to adjust resources sect44describes how a dataplane operating system can have both acoherence-free execution model and at the same time allowfor rebalancing of flows Finally sect45 describes the mecha-nism that migrates flows without reordering packets

41 Background on IX

The IX system architecture separates the control plane re-sponsible for the allocation of resources from the dataplanewhich operates on them using an approach similar to Ar-rakis [43] IX relies on hardware virtualization to protect thekernel from applications it uses Dune [5] to load and exe-cute the IX dataplane instances and uses the Linux host asits control plane environment

The IX dynamic controller is a Python daemon that con-figures and launches the IX dataplane then monitors its ex-ecution makes policy decisions and communicates them tothe IX dataplane and the host operating system

Unlike conventional operating systems which decouplenetwork processing from application execution the IX data-

app

libix

tcpip tcpip

timer

adaptive batch

1

2

4

5

Event

ConditionsBatched

Syscalls

3

6

kernel

user

Figure 4 Adaptive batching in the IX data plane

plane relies on a run to completion with adaptive batchingscheme to deliver both low latency and high throughput toapplications

Fig 4 illustrates the key run-to-completion mechanismbuilt into the original IX dataplane [6] each elastic threadhas exclusive access to NIC queues In step (1) the data-plane polls the receive descriptor ring and moves all receivedpackets into an in-memory queue and potentially posts freshbuffer descriptors to the NIC for use with future incomingpackets The thread then executes steps (2) ndash (6) for an adap-tive batch of packets the TCPIP network stack processesthese packets (2) and the application then consumes the re-sulting events (3) Upon return from user-space the threadprocesses batched systems calls and in particular the onesthat direct outgoing TCPIP traffic (4) runs necessary timers(5) and places outgoing Ethernet frames in the NICrsquos trans-mit descriptor ring for transmission (6)

Adaptive batching is specifically defined as follows (i)we never wait to batch requests and batching only occurs inthe presence of congestion (ii) we set an upper bound on thenumber of batched packets Using batching only during con-gestion allows us to minimize the impact on latency whilebounding the batch size prevents the live set from exceed-ing cache capacities and avoids transmit queue starvationBatching improves packet rate because it amortizes systemcall transition overheads and improves instruction cache lo-cality prefetching effectiveness and branch prediction ac-curacy When applied adaptively batching also decreases la-tency because these same efficiencies reduce head-of-lineblocking

42 Dynamic NIC ConfigurationModern NICs provide hardware mechanisms such as VMDqFlow Director [17] and Receive Side Scaling (RSS) [36] todirect traffic to one of multiple queues as an essential mech-anism for multicore scalability For example the Intel x520

relies on an on-chip table (called the RSS Redirection Ta-ble or RETA [17]) to partition flows into multiple queueson the data path the NIC first computes the Toeplitz hashfunction of the packet 5-tuple header (as specified by RSS)and uses the lower 7 bits of the result as an index into theRETA table A flow group is the set of network flows with anidentical RETA index The host updates the table using PCIewrites on a distinct control path which can lead to packetreordering [52]

Such anomalies have severe consequences in generalon the performance of the TCPIP networking stack [24]IXrsquos coherence-free design exposes a further constraint be-cause there are no locks within the TCPIP stack two elasticthreads cannot process packets from the same flow simulta-neously For correctness packets received by the thread thatdoes not own the flow group must be dropped This furthernegatively impacts network performance

The IX dataplane implements the actual flow group mi-gration and relies on the RETA to change the mappings Thechallenge is to design a migration mechanism that updatesthe steering of packets to different queues (and by extensionto different threads) without (i) impacting the steady stateperformance of the dataplane and its coherence-free designor (ii) creating network anomalies during migration such asdropping packets or processing them out of order in the net-working stack

43 Control LoopWe first describe the control loop at the heart of IXrsquos dy-namic controller implemented within the control plane Thecontroller adjusts processor resources by suspending andresuming IX elastic threads and specifying the mappingbetween flow groups and threads It migrates flow groupsfrom multiple existing threads to a newly resumed threadand conversely from a thread to be suspended to the re-maining threads For the server consolidation scenario thecontrol loop also allocates resources dedicated to the back-ground applications In our experimental setup it simply is-sues SIGSTOP and SIGCONT signals to transparently con-trol the background application

The control loop does not depend on any detailed charac-terization of the Pareto frontier of sect31 Instead it merely im-plements the derived high-level policy of sect32 using queuingdelay estimates Specifically the user sets the upper boundon the acceptable queuing delay in our experiments withmemcached characterized by high packet rates and low ser-vice times we set the acceptable bound at 300micros to minimizelatency violations given our SLO of 500micros at the 99th per-centile as measured end-to-end by the client machine

For this we rely on a key side effect of IXrsquos use of adap-tive batching unprocessed packets that form the backlogare queued in a central location namely in step (1) in thepipeline of Fig 4 Packets are then processed in order inbounded batches to completion through both the networkingstack and the application logic In other words each IX core

Prepare Wait RPC Defer processIXCPpy

migrate(fgs)

CPU A

update HW

hold

finalize

done

NIC

first

packet

release

CPU B

prepare

timeout

(a) Thread-centric view

NICHWQ-A

HWQ-B

RETA

FGPoll

Poll

defaultQ-A

remoteQ-B

defaultQ-B

localQ-B

Thread-A

Thread-B

(b) Packet-centric view

Figure 5 Flow-group Migration Algorithm

operates like a simple FIFO queuing server onto which clas-sic queuing and control theory principles can be easily ap-plied In contrast conventional operating systems distributebuffers throughout the system in the NIC (because of coa-lesced interrupts) in the driver before networking process-ing and in the socket layer before being consumed by theapplication Furthermore these conventional systems pro-vide no ordering guarantees across flows which makes itdifficult to pinpoint congestion

To estimate queuing delays the controller monitors theiteration time τ and the queue depth Q With B the maximalbatch size the tail latency issimmax(delay) = dQBelowastτ Thedataplane computes each instantaneous metric every 10msfor the previous 10ms interval As these metrics are subjectto jitter the dataplane computes the exponential weightedmoving averages using multiple smoothing factors (α) inparallel For example we track the queue depth as Q(tα) =α lowastQnow +(1minusα) lowastQ(tminus 1α) The control loop executesat a frequency of 10 Hz which is sufficient to adapt to loadchanges

Deciding when to remove resources is trickier than de-ciding when to add them as shallow and near-empty queuesdo not provide reliable metrics Instead we measure idletime and observe that each change in the configuration addsor removes a predictable level of throughput (i) increas-ing cores from n to n+ 1 increases throughput by a factorof 1n (ii) enabling hyperthreads adds 30 (for our fore-ground workload at least) and (iii) the corersquos performanceis largely proportional to its clock rate We can thereforedefine the function threshold[ f rom to] as the ratio of maxthroughput between the f rom and to configurations Whenoperating at configuration f rom and the idle time exceedsthreshold[ f rom to] we can safely switch to configuration towithout loss of performance

These settings are not universal as a number of assump-tions such as load changes requests bursts and variationsin request service times will noticeably affect the metricsThese are policies not mechanisms In particular there is an

inherent tradeoff between aggressive energy saving policieswhich may lead to SLO violations and conservative poli-cies which will consume an excessive amount of energyFortunately the policy can be easily modified the dataplaneexposes naturally all relevant metrics and the control planedaemon issim500 lines of Python includingsim80 lines for thecontrol loop

44 Data Structures for Coherence-free ExecutionWe designed our memory management and data structurelayout to minimize the performance impact of flow groupmigration and to preserve the coherence-free property of ourdataplane in the steady-state Our memory allocator drawsinspiration from magazine-based SLAB allocation [7] Eachthread maintains a thread-local cache of every data typeIf the size of the thread-local cache exceeds a thresholdmemory items are returned using traditional locking to ashared memory pool Such rebalancing events are commonduring flow group migration but infrequent during normaloperation

We further organize our data structures so the pointerreferences building linked lists hash tables etc are onlybetween objects either both permanently associated with thesame thread (per-cpu) or both permanently associated withthe same flow group (per-fg) or both global in scope Asa result we can efficiently transfer ownership of an entireflow-grouprsquos data structures during migration while avoidinglocking or coherence traffic during regular operation

For example the event condition arrays batch syscallarrays timer wheels incoming and outgoing descriptorrings posted and pending mbufs are all per-cpu Theprotocol control blocks 5-tuple hash table listening sock-ets TW WAIT queues out-of-order fragments and unac-knowledged sequences are all per-fg The ARP table isglobal Some objects change scope during their lifetimebut only at well-defined junctures For example an mbuf ispart of the per-cpu queue while waiting to be processed(step 1 in Fig 4) but then becomes part of per-fg struc-tures during network processing

A global data structure the software flow group tablemanages the entry points into the per-fg structures IntelNICs conveniently store the RSS hash result as meta-dataalong with each received packet the bottom bits are used toindex into that table

We encountered one challenge in the management of TCPtimers We design our implementation to maximize normalexecution performance each thread independently managesits timers as a hierarchical timing wheel [49] ie as a per-cpu structure As a consequence a flow group migrationrequires a scan of the hierarchical wheels to identify andremove pending timers during the hold phase of Fig 5athose will be restored later in the destination thread at therelease phase of the flow group migration

45 Flow Group MigrationWhen adding or removing a thread the dynamic controllergenerates a set of migration requests Each individual re-quest is for a set of flow groups (fgs) currently handledby one elastic thread A to be handled by elastic thread B Tosimplify the implementation the controller serializes the mi-gration requests and the dataplane assumes that at most onesuch request is in progress at any point in time

Each thread has three queues that can hold incoming net-work packets the defaultQ contains packets received dur-ing steady state operation when no flow group migration ispending the remoteQ contains packets belonging to outgo-ing flow groups the localQ contains packets belonging toincoming flow groups The use of these two extra queuesis critical to ensure the processing of packets in the properorder by the networking stack

Fig 5 illustrates the migration steps in a thread-centricview (Fig 5a) and in a packet-centric view (Fig 5b) Thecontroller and the dataplane threads communicate via lock-free structures in shared memory First the controller signalsA to migrate fgs to B A first marks each flow group of theset fgs with a special tag to hold off normal processing onall threads moves packets which belong to the flow groupset fgs from defaultQ-A to remoteQ-B and stops alltimers belonging to the flow group set A then reprogramsthe RETA for all entries in fgs Packets still received byA will be appended to remoteQ-B packets received by Bwill go to localQ-B

Upon reception of the first packet whose flow group be-longs to fgs by B B signals A to initiate the final stageof migration Then B finalizes the migration by re-enablingfgsrsquos timers removing all migration tags and pre-pendingto its defaultQ-B the packets from remoteQ-B andthe packets from localQ-B Finally B notifies the con-trol plane that the operation is complete There are obviouslycorner-cases and subtleties in the algorithm For example Ainstalls a migration timer to finalize the migration even if Bhas not yet received any packets We set the timeout to 1 ms

5 EvaluationWe use three synthetic time-accelerated load patterns toevaluate the effectiveness of the control loop under stressfulconditions All three vary between nearly idle and maximumthroughput within a four minute period the slope patterngradually raises the target load from 0 and 62M RPS andthen reduces its load the step pattern increases load by 500KRPS every 10 seconds and finally the sine+noise patternis a basic sinusoidal pattern modified by randomly addingsharp noise that is uniformly distributed over [-250+250]KRPS and re-computed every 5 seconds The slope patternprovides a baseline to study smooth changes the step patternmodels abrupt and massive changes while the sine+noisepattern is representative of daily web patterns [48]

Fig 6 and Fig 7 show the results of these three dynamicload patterns for the energy proportionality and workloadconsolidation scenarios In each case the top figure mea-sures the observed throughput They are annotated with thecontrol loop events that add resources (green) or removethem (red) Empty triangles correspond to core allocationsand full triangles to DVFS changes The middle figure eval-uates the soundness of the algorithm and reports the 99thpercentile latency as observed by a client machine and re-ported every second Finally the bottom figures compare theoverall efficiency of our solution based on dynamic resourcecontrols with (i) the maximal static configuration using allcores and Turbo Boost and (ii) the ideal synthetic efficiencycomputed using the Pareto frontier of Fig 2Energy Proportionality Fig 6 shows the behavior for theenergy efficiency scenario In each case the workload tracksthe desired throughput of the pattern and exercises the en-tire sequence of configurations gradually adding cores en-abling hyperthreading increasing the frequency and finallyenabling Turbo Boost before doing it in reverse The steppattern is particularly challenging as the instant change inload level requires multiple back-to-back configurationschanges With a few exceptions the latencies remain wellbelow the 500micros SLO We further discuss the violations be-low Finally Fig 6 (bottom) compares the power dissipatedby the workload with the corresponding power levels as de-termined by the Pareto frontier (lower bound) or the max-imum static configuration (upper bound) This graph mea-

Smooth Step Sine+noiseEnergy Proportionality (W)

Max power 91 92 94Measured 42 (-54) 48 (-48) 53 (-44)Pareto bound 39 (-57) 41 (-55) 45 (-52)

Server consolidation opportunity ( of peak)Pareto bound 50 47 39Measured 46 39 32

Table 1 Energy Proportionality and Consolidation gains

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

max conf dynamic Pareto

0

20

40

60

80

100

0 50 100 150 200

Pow

er

(W)

Time (seconds)

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 6 Energy Proportionality of memcached running on IX for three load patterns

sures the effectiveness of the control loop to maximize en-ergy proportionality We observe that the dynamic (actuallymeasured) power curve tracks the Pareto (synthetic) curvewell which defines a bound on energy savings When the dy-namic resource controls enters Turbo Boost mode the mea-sured power in all three cases starts at the lower end of the4 W range and then gradually rises as expected Table 1shows that the three patterns have Pareto savings bounds of52 55 and 57 IXrsquos dynamic resource controls resultsin energy savings of 44 48 and 54 which is 85 87and 93 of the theoretical boundConsolidation Fig 7 shows the corresponding results forthe workload consolidation scenarios Fig 7 (top) measuresthe observed load which here as well tracks the desired loadRecall that the consolidation policy always operates at theprocessorrsquos nominal rate (or Turbo) which limits the numberof configuration changes Fig 7 (middle) similarly confirmsthat the system meets the SLO with few exceptions Fig 7(bottom) plots the throughput of the background batch appli-cation expressed as a percentage of its throughput on a dedi-cated processor at 24 Ghz We compare it only to the Paretooptimal upper bound as a maximum configuration wouldmonopolize cores and deliver zero background throughputTable 1 shows that for these three patterns our consolida-tion policy delivers 32ndash46 of the standalone throughput

of the background job which corresponds to 82ndash92 ofthe Pareto boundSLO violations A careful study of the SLO violations of the6 runs shows that they fall into two categories First there are16 violations caused by delays in packet processing due toflow group migrations resulting from the addition of a coreSecond there are 9 violations caused by abrupt increase ofthroughput mostly in the step pattern which occur beforeany flow migrations The control plane then reacts quickly(in sim100 ms) and accommodates to the new throughput byadjusting resources To further confirm the abrupt nature ofthroughput increase specific to the step pattern we note thatthe system performed up three consecutive increases in re-sources in order to resolve a single violation 23 of the 25total violations last a single second with the remaining twoviolations lasting two seconds We believe that the compli-ance with the SLO achieved by our system is more than ad-equate for any practical production deploymentFlow group migration analysis Table 2 measures the la-tency of the 552 flow group migrations that occur during the6 benchmarks as described in sect45 It also reports the totalnumber of packets whose processing is deferred during themigration (rather than dropped or reordered) We first ob-serve that migrations present distinct behaviors when scalingup and when scaling down the number of cores The differ-

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

0

20

40

60

80

100

0 50 100 150 200

o

f pea

k

Time (seconds)

dynamicPareto

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 7 Consolidation of memcached with a background batch task

ence can be intuitively explained since the migrations duringthe scale up are performed in a heavily loaded system whilethe system during the scale down is partially idle In absoluteterms migrations that occur when adding a core take 463microson average and less than 15 ms 95 of the time The outlierscan be explained by rare occurrences of longer preparationtimes or when processing up to 614 deferred packets

6 DiscussionUsing Pareto as a guide Even though the Pareto resultsare not used by the dynamic resource controller the Paretofrontier proved to be a valuable guide first to motivate andquantify the problem then to derive the configuration pol-icy sequence and finally to evaluate the effectiveness ofthe dynamic resource control by setting an upper bound onthe gains resulting from dynamic resource allocation Manyfactors such as software scalability hardware resource con-tention network and disk IO bottlenecks will influence thePareto frontier of any given application and therefore the de-rived control loop policies We will compare different work-loads in future workTradeoff between average latency and efficiency We usea one-dimensional SLO (99th pct le 500micros) and show thatwe can increase efficiency while maintaining that objectiveHowever this comes at a cost when considering the entirelatency distribution eg the average as well as the tail More

complex SLOs taking into account multiple aspects of la-tency distribution would define a different Pareto frontierand likely require adjustments to the control loopAdaptive flow-centric scheduling The new flow-groupmigration algorithm leads to a flow-centric approach to re-source scheduling where the network stack and applicationlogic always follow the steering decision POSIX applica-tions can balance flows by migrating file descriptors betweenthreads or processes but this tends to be inefficient becauseit is difficult for the kernel to match the flowrsquos destinationreceive queue to changes in CPU affinity Flow director canbe used by Linux to adjust the affinity of individual networkflows as a reactive measure to application-level and kernelthread migration rebalancing but the limited size of the redi-rection table prevents this mechanism from scaling to largeconnection counts By contrast our approach allows flowsto be migrated in entire groups improving efficiency and iscompatible with more scalable hardware flow steering mech-anisms based on RSSHardware trends Our experimental setup using one SandyBridge processor has a three-dimensional Pareto spaceClearly NUMA would add a new dimension Also thenewly released Haswell processors provide per-core DVFScontrols which further increases the Pareto space More-over hash filters for flow group steering could benefit fromrecent trends in NIC hardware For example Intelrsquos new

avg 95th pct max stddevad

dco

reprepare (micros) 67 231 521 93wait (micros) 135 541 983 190rpc (micros) 129 355 529 112deferred (micros) 101 401 1167 147total (micros) 463 1432 2870 459 packets 64 319 614 116

rem

ove

core prepare (micros) 25 54 398 32

wait (micros) 34 101 170 32rpc (micros) 12 25 49 7deferred (micros) 19 43 108 14total (micros) 93 160 456 47 packets 3 8 28 3

Table 2 Breakdown of flow group migration measured dur-ing the six benchmarks

XL710 chipset [19] has a 512 entry hash LUT (as well asindependent 64 entry LUTs for each VF) in contrast to the128 entries available in the 82599 chipset [17] used in ourevaluation This has the potential to reduce connection im-balances between cores especially with high core counts

7 Related WorkScheduling Scheduler activations [1] give applicationsgreater control over hardware threads and provide a mecha-nism for custom application-level scheduling Callisto [13]uses a similar strategy to improve the performance of co-located parallel runtime systems Our approach differs inthat an independent control plane manages the schedulingof hardware threads based on receive queuing latency indi-cators while the dataplane exposes a simple kernel thread-ing abstraction SEDA [51] also monitors queuing behaviorto make scheduling decisions such as thread pool sizingChronos [21] makes use of software-based flow steering butwith a focus on balancing load to reduce latency AffinityAccept [42] embraces a mixture of software and hardware-based flow steering in order to improve TCP connectionaffinity and increase throughput We focus instead on energyproportionality and workload consolidationEnergy Proportionality The energy proportionality prob-lem [3] has been well explored in previous work Somesystems have focused on solutions tailored to throughput-oriented workloads [33] or read-only workloads [23] Meis-ner et al [34] highlight unique challenges for low la-tency workloads and advocate full system active low-powermodes Similar to our system Pegasus [29] achieves CPUenergy proportionality for low latency workloads Our workexpands on Pegasus by exploring the elastic allocation ofhardware threads in combination with processor power man-agement states and by basing scheduling decisions on in-ternal latency metrics within a host endpoint instead of anexternal controller Niccolini et al show that a softwarerouter running on a dedicated machine can be made energy-

proportional [38] Similar to our approach queue lengthis used as a control signal to manage core allocation andDVFS settings However we focus on latency-sensitive ap-plications rather than middlebox traffic and consider theadditional case of workload consolidationCo-location Because host endpoints contain some com-ponents that are not energy proportional and thus are mostefficient when operating at 100 utilization co-location ofworkloads is also an important tool for improving energyefficiency At the cluster scheduler level BubbleUp [32] andParagon [10] make scheduling decisions that are interference-aware through efficient classification of the behavior ofworkload co-location Leverich et al [26] demonstratethat co-location of batch and low latency jobs is possi-ble on commodity operating systems Our approach ex-plores this issue at higher throughputs and with tighter la-tency SLOs Bubble-Flux [53] additionally controls back-ground threads we control background and latency-sensitivethreads CPI2 [54] detects performance interference by ob-serving changes in CPI and throttles offending jobs Thiswork is orthogonal to ours and could be a useful addi-tional signal for our control plane Heracles manages mul-tiple hardware and software isolation mechanisms includ-ing packet scheduling and cache partitioning to co-locatelatency-sensitive applications with batch tasks while main-taining millisecond SLOs [30] We limit our focus to DVFSand core assignment but target more aggressive SLOsLow-latency frameworks User-level networking stacks [2031 47] and RDMA applications [12 37 40] all rely onpolling typically by a fixed number of execution threadsto deliver micros-scale response times These stacks and appli-cations are evaluated based on their latency and throughputcharacteristics and do not take into account energy propor-tionality or workload consolidation considerations

8 ConclusionWe present the design and implementation of dynamic re-source controls for the IX dataplane which consists of acontroller that allocates cores and sets processor frequencyto adapt to changes in the load of latency-sensitive appli-cations The novel rebalancing mechanisms do not impactthe steady-state performance of the dataplane and can mi-grate a set of flow groups in milliseconds without droppingor reordering packets We develop two policies focused onoptimizing energy proportionality and workload consolida-tion Our results show that for three varying load patternsthe energy proportionality can save 44ndash54 of the pro-cessorrsquos energy or enable a background job to deliver 32ndash46 of its standalone throughput To evaluate the effective-ness of our approach we synthesize the Pareto frontier bycombining the behavior of all possible static configurationsOur policies deliver 85ndash93 of the Pareto optimal boundin terms of energy proportionality and 82ndash92 in termsof consolidation

AcknowledgementsThis work was supported by the MicrosoftEPFL Joint Re-search Center the Stanford Experimental Datacenter Laband NSF grant CNS-1422088 George Prekas is supportedby a Google Graduate Research Fellowship and Adam Be-lay by a Stanford Graduate Fellowship

References[1] T E Anderson B N Bershad E D Lazowska and H M

Levy Scheduler Activations Effective Kernel Support for theUser-Level Management of Parallelism ACM Trans ComputSyst 10(1)53ndash79 1992

[2] B Atikoglu Y Xu E Frachtenberg S Jiang andM Paleczny Workload Analysis of a Large-Scale Key-ValueStore In Proceedings of the 2012 ACM SIGMETRICS Inter-national Conference on Measurement and Modeling of Com-puter Systems pages 53ndash64 2012

[3] L A Barroso and U Holzle The Case for Energy-Proportional Computing IEEE Computer 40(12)33ndash372007

[4] L A Barroso J Clidaras and U Holzle The Datacenter as aComputer An Introduction to the Design of Warehouse-ScaleMachines Second Edition Synthesis Lectures on ComputerArchitecture Morgan amp Claypool Publishers 2013

[5] A Belay A Bittau A J Mashtizadeh D Terei D Mazieresand C Kozyrakis Dune Safe User-level Access to PrivilegedCPU Features In Proceedings of the 10th Symposium onOperating System Design and Implementation (OSDI) pages335ndash348 2012

[6] A Belay G Prekas A Klimovic S Grossman C Kozyrakisand E Bugnion IX A Protected Dataplane Operating Systemfor High Throughput and Low Latency In Proceedings of the11st Symposium on Operating System Design and Implemen-tation (OSDI) pages 49ndash65 2014

[7] J Bonwick and J Adams Magazines and Vmem Extendingthe Slab Allocator to Many CPUs and Arbitrary Resources InUSENIX Annual Technical Conference pages 15ndash33 2001

[8] H David E Gorbatov U R Hanebutte R Khanaa andC Le RAPL memory power estimation and capping InProceedings of the 2010 International Symposium on LowPower Electronics and Design pages 189ndash194 2010

[9] J Dean and L A Barroso The Tail at Scale Commun ACM56(2)74ndash80 2013

[10] C Delimitrou and C Kozyrakis Quality-of-Service-AwareScheduling in Heterogeneous Data centers with ParagonIEEE Micro 34(3)17ndash30 2014

[11] C Delimitrou and C Kozyrakis Quasar Resource-Efficientand QoS-Aware Cluster Management In Proceedings of the19th International Conference on Architectural Support forProgramming Languages and Operating Systems (ASPLOS-XIX) pages 127ndash144 2014

[12] A Dragojevic D Narayanan M Castro and O HodsonFaRM Fast Remote Memory In Proceedings of the 11stSymposium on Networked Systems Design and Implementa-tion (NSDI) pages 401ndash414 2014

[13] T Harris M Maas and V J Marathe Callisto co-schedulingparallel runtime systems In Proceedings of the 2014 EuroSysConference page 24 2014

[14] B Hindman A Konwinski M Zaharia A Ghodsi A DJoseph R H Katz S Shenker and I Stoica Mesos A Plat-form for Fine-Grained Resource Sharing in the Data CenterIn Proceedings of the 8th Symposium on Networked SystemsDesign and Implementation (NSDI) 2011

[15] C-H Hsu Y Zhang M A Laurenzano D Meisner T FWenisch J Mars L Tang and R G Dreslinski AdrenalinePinpointing and reining in tail queries with quick voltageboosting In Proceedings of the 21st IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages271ndash282 2015

[16] J Hwang K K Ramakrishnan and T Wood NetVM HighPerformance and Flexible Networking Using Virtualization onCommodity Platforms In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 445ndash458 2014

[17] Intel Corp Intel 82599 10 GbE Controller Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheets82599-10-gbe-controller-datasheetpdf2014

[18] Intel Corp Intel DPDK Data Plane Development Kit httpdpdkorg 2014

[19] Intel Corp Intel Ethernet Controller XL710 Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheetsxl710-10-40-controller-datasheetpdf2014

[20] E Jeong S Woo M A Jamshed H Jeong S Ihm D Hanand K Park mTCP a Highly Scalable User-level TCP Stackfor Multicore Systems In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 489ndash502 2014

[21] R Kapoor G Porter M Tewari G M Voelker and A Vah-dat Chronos Predictable Low Latency for Data Center Ap-plications In Proceedings of the 2012 ACM Symposium onCloud Computing (SOCC) page 9 2012

[22] W Kim M S Gupta G-Y Wei and D M Brooks Systemlevel analysis of fast per-core DVFS using on-chip switch-ing regulators In Proceedings of the 14th IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages123ndash134 2008

[23] A Krioukov P Mohan S Alspaugh L Keys D E Cullerand R H Katz NapSAC design and implementation of apower-proportional web cluster Computer CommunicationReview 41(1)102ndash108 2011

[24] K-C Leung V O K Li and D Yang An Overview ofPacket Reordering in Transmission Control Protocol (TCP)Problems Solutions and Challenges IEEE Trans ParallelDistrib Syst 18(4)522ndash535 2007

[25] J Leverich Mutilate High-Performance MemcachedLoad Generator httpsgithubcomleverichmutilate 2014

0

20

40

60

80

100

0 2 4 6 8

Pow

er (

W)

Memcached RPS x 106 at SLO

Pareto IX

DVFS-only IX

Pareto Linux

DVFS-only Linux

Figure 3 Energy-proportionality comparison between thePareto-optimal frontier considering only DVFS adjustmentsand the full Pareto frontier considering core allocation hy-perthread allocations and frequency

of the static configurations available Each graph only showsthe static configurations that participate in the frontierEnergy proportionality We evaluate 224 distinct combi-nations from one to eight cores using consistently eitherone or two threads per core for 14 different DVFS levelsfrom 12 Ghz to 24 Ghz as well as Turbo Boost Fig 2a andFig 2b show the 43 and 32 static configurations (out of 224)that build the Pareto frontier for energy proportionality Thefigures confirm the intuition that (i) various static configura-tions have very different dynamic ranges beyond which theyare no longer able to meet the SLO (ii) each static config-uration draws substantially different levels of energy for thesame amount of work (iii) at the low-end of the curve manydistinct configurations operate at the minimal frequency of12 Ghz obviously with a different number of cores andthreads and contribute to the frontier these are shown ingreen in the Figure (iv) at the high-end of the range manyconfigurations operate with the maximum of 8 cores withdifferent frequencies including Turbo BoostConsolidation The methodology here is a little differentWe first characterize the background job and observe that itdelivers energy-proportional throughput up to 24 Ghz butthat Turbo Boost came at an energythroughput premiumConsequently we restrict the Pareto configuration space at24 Ghz the objective function is the throughput of the back-ground job expressed as a fraction of the throughput of thatsame job without any foreground application Backgroundjobs run on all cores that are not used by the foreground ap-plication Fig 2c and Fig 2d show the background through-put expressed as a fraction of the standalone throughput asa function of the foreground throughput provided that theforeground application meets the SLO as the foregroundapplication requires additional cores to meet the SLO thebackground throughput decreases proportionally

Linux vs IX Fig 2a and Fig 2b show that Linux and IXbehave differently (i) IX improves the throughput of Linuxby 48times (ii) IX can clearly take advantage of hyperthread-ing and has its best performance with configurations with16 threads on 8 cores In contrast Linux cannot take ad-vantage of hyperthreading and has its best performance with8 threads of control running on 8 cores (iii) the efficiencyof individual static configurations of Linux which relieson interrupts is much more subject to load than individ-ual static configurations of IX which relies on a pollingmodel As a consequence an improperly configured over-sized IX dataplane will be extremely energy-inefficient atlow-to-moderate loads (iv) the comparison of the Paretofrontiers shows that IXrsquos frontier always dominates the fron-tier of Linux ie provides better energy efficiency for thesame load level This suggests that a properly configureddataplane design is always more energy-efficient than anyLinux-based configuration The difference is substantial in-cluding at low-to-moderate load levels IXrsquos frontier is lessthan half of Linuxrsquos when the throughput load ge 780 KRPS(sim54 of Linuxrsquos capacity)DVFS-only alternative Fig 3 further analyzes the data andcompares the Pareto frontiers with an alternate frontier thatonly considers changes in DVFS frequency The comparisonis limited to the energy proportionality scenario for consoli-dation available cores are necessary for the background ap-plication to run and a DVFS-only solution would not suffice

We observe that the impact of DVFS-only controls differsnoticeably between Linux and IX with Linux the DVFS-only alternate frontier is very close to the Pareto frontiermeaning that a DVFS-only approach such as Pegasus [29] orAdrenaline [15] would be adequate This is due to Linuxrsquosidling behavior which saves resources In the case of IXhowevermdashand likely for any polling-based dataplanemdashaDVFS-only scheduler would provide worse energy propor-tionality at low-moderate loads than a corresponding Linux-based solution As many datacenter servers operate in the10-30 range [4] we conclude that a dynamic resourceallocation scheme involving both DVFS and core allocationis necessary for dataplane architectures

32 Derived PolicyWe use the results from sect31 to derive a resource config-uration policy framework whose purpose is to determinethe sequence of configurations to be applied as a functionof the load on the foreground application to both the fore-ground (latency-sensitive) and background (batch) applica-tions Specifically given an ever-increasing (or -decreasing)load on the foreground applications the goal is to determinethe sequence of resource configurations minimizing energyconsumption or maximizing background throughput respec-tively

We observe that (i) the latency-sensitive application(memcached) can scale nearly linearly up to the 8 cores of

the processor (ii) it benefits from running a second thread oneach core with a consistent speedup of 13times (iii) it is mostenergy-efficient to first utilize the various cores and onlythen to enable the second hyperthread on each core ratherthan the other way around and (iv) it is least energy-efficientto increase the frequency

We observe that the background application (i) also scaleslinearly but (ii) does not benefit from the 2nd hyperthread(iii) is nearly energy-proportional across the frequency spec-trum with the exception of Turbo Boost From a total cost ofownership perspective the most efficient operating point forthe workload consolidation of the background task is there-fore to run the system at the processorrsquos nominal 24 Ghzfrequency whenever possible We combine these observa-tions with the data from the Pareto analysis and derive thefollowing policiesEnergy Proportional Policy As a base state run with onlyone core and hyperthread with the socket set at the mini-mal clock rate (12Ghz) To add resources first enable ad-ditional cores then enable hyperthreads on all cores (as asingle step) and only after that gradually increase the clockrate until reaching the nominal rate (24Ghz) finally enableTurbo Boost To remove resources do the opposite This pol-icy leads to a sequence of 22 different configurationsWorkload Consolidation Policy As a base state run thebackground jobs on all available cores with the processor atthe nominal clock rate To add resources to the foregroundapplication first shift cores from the background thread tothe foreground application one at a time This is done by firstsuspending the background threads use both hyperthreadsof the newly freed core for the foreground application Nextstop the background job entirely and allocate all cores to theforeground applications As a final step enable Turbo BoostThis policy leads to a sequence of 9 different configurations

These policies closely track the corresponding Paretofrontier For energy proportionality (i) the 32 different staticconfigurations of the frontier are a superset of the configura-tions enabled by the policy and (ii) the difference in overallimpact in terms of energy spent is marginal For consolida-tion Pareto and policy nearly identically overlap

Obviously these conclusions may vary with platformsand applications but the use of underlying Pareto method-ology remains valid In fact it should provide a robust guideto determine allocation policies for more complex workloads(eg with natural scalability limitations) and more flexibleplatforms (eg the newer Haswell processors can set theclock frequency separately for individual cores)

4 Dynamic Resource Controls of DataplanesWe now present the design of the mechanisms and poli-cies that are necessary to implement dynamic resource con-trols in dataplane operating systems Although the imple-mentation is specific to the IX dataplane operating systemthe design principles generally apply to all high-throughput

dataplane operating systems such as Arrakis [43] and user-level networking stacks running latency-sensitive applica-tions [18 20 31] Traditionally such environments have notfocused on dynamic resource controls instead they focusedsolely on throughput with statically allocated resourceswithout considerations for the energy consumed Addingresource controls in such environments poses the followingchallenges

1 How to rebalance flows across cores without impactingnormal performance This is particularly challenging asdataplane environments achieve high performance be-cause of their coherence-free execution model By thatwe mean a programming where all common case oper-ations execute without requiring communication of syn-chronization between threads

2 How to rebalance flows across processing cores withoutimpacting network performance given that dataplane op-erating systems implement their own TCPIP network-ing stack and that TCPIP performance is negatively im-pacted whenever packets are either lost or processed outof order [24] This is particularly challenging wheneverthe rebalancing operation also involves the reconfigura-tion of the NIC which generates inherent race conditions

3 How to determinemdashboth robustly and efficientlymdashwhetherresources need to be added or can be removed from thedataplane given the complexity of modern applicationsand the difficulty to estimate the impact of resource con-trols on application throughput

Our design addresses these three challenges We firstprovide in sect41 the necessary background on the IX data-plane and in sect42 the background on the few assumptionsmade on the hardware sect43 describes the implementationof the controller which relies exclusively on application-independent queuing delay metrics to adjust resources sect44describes how a dataplane operating system can have both acoherence-free execution model and at the same time allowfor rebalancing of flows Finally sect45 describes the mecha-nism that migrates flows without reordering packets

41 Background on IX

The IX system architecture separates the control plane re-sponsible for the allocation of resources from the dataplanewhich operates on them using an approach similar to Ar-rakis [43] IX relies on hardware virtualization to protect thekernel from applications it uses Dune [5] to load and exe-cute the IX dataplane instances and uses the Linux host asits control plane environment

The IX dynamic controller is a Python daemon that con-figures and launches the IX dataplane then monitors its ex-ecution makes policy decisions and communicates them tothe IX dataplane and the host operating system

Unlike conventional operating systems which decouplenetwork processing from application execution the IX data-

app

libix

tcpip tcpip

timer

adaptive batch

1

2

4

5

Event

ConditionsBatched

Syscalls

3

6

kernel

user

Figure 4 Adaptive batching in the IX data plane

plane relies on a run to completion with adaptive batchingscheme to deliver both low latency and high throughput toapplications

Fig 4 illustrates the key run-to-completion mechanismbuilt into the original IX dataplane [6] each elastic threadhas exclusive access to NIC queues In step (1) the data-plane polls the receive descriptor ring and moves all receivedpackets into an in-memory queue and potentially posts freshbuffer descriptors to the NIC for use with future incomingpackets The thread then executes steps (2) ndash (6) for an adap-tive batch of packets the TCPIP network stack processesthese packets (2) and the application then consumes the re-sulting events (3) Upon return from user-space the threadprocesses batched systems calls and in particular the onesthat direct outgoing TCPIP traffic (4) runs necessary timers(5) and places outgoing Ethernet frames in the NICrsquos trans-mit descriptor ring for transmission (6)

Adaptive batching is specifically defined as follows (i)we never wait to batch requests and batching only occurs inthe presence of congestion (ii) we set an upper bound on thenumber of batched packets Using batching only during con-gestion allows us to minimize the impact on latency whilebounding the batch size prevents the live set from exceed-ing cache capacities and avoids transmit queue starvationBatching improves packet rate because it amortizes systemcall transition overheads and improves instruction cache lo-cality prefetching effectiveness and branch prediction ac-curacy When applied adaptively batching also decreases la-tency because these same efficiencies reduce head-of-lineblocking

42 Dynamic NIC ConfigurationModern NICs provide hardware mechanisms such as VMDqFlow Director [17] and Receive Side Scaling (RSS) [36] todirect traffic to one of multiple queues as an essential mech-anism for multicore scalability For example the Intel x520

relies on an on-chip table (called the RSS Redirection Ta-ble or RETA [17]) to partition flows into multiple queueson the data path the NIC first computes the Toeplitz hashfunction of the packet 5-tuple header (as specified by RSS)and uses the lower 7 bits of the result as an index into theRETA table A flow group is the set of network flows with anidentical RETA index The host updates the table using PCIewrites on a distinct control path which can lead to packetreordering [52]

Such anomalies have severe consequences in generalon the performance of the TCPIP networking stack [24]IXrsquos coherence-free design exposes a further constraint be-cause there are no locks within the TCPIP stack two elasticthreads cannot process packets from the same flow simulta-neously For correctness packets received by the thread thatdoes not own the flow group must be dropped This furthernegatively impacts network performance

The IX dataplane implements the actual flow group mi-gration and relies on the RETA to change the mappings Thechallenge is to design a migration mechanism that updatesthe steering of packets to different queues (and by extensionto different threads) without (i) impacting the steady stateperformance of the dataplane and its coherence-free designor (ii) creating network anomalies during migration such asdropping packets or processing them out of order in the net-working stack

43 Control LoopWe first describe the control loop at the heart of IXrsquos dy-namic controller implemented within the control plane Thecontroller adjusts processor resources by suspending andresuming IX elastic threads and specifying the mappingbetween flow groups and threads It migrates flow groupsfrom multiple existing threads to a newly resumed threadand conversely from a thread to be suspended to the re-maining threads For the server consolidation scenario thecontrol loop also allocates resources dedicated to the back-ground applications In our experimental setup it simply is-sues SIGSTOP and SIGCONT signals to transparently con-trol the background application

The control loop does not depend on any detailed charac-terization of the Pareto frontier of sect31 Instead it merely im-plements the derived high-level policy of sect32 using queuingdelay estimates Specifically the user sets the upper boundon the acceptable queuing delay in our experiments withmemcached characterized by high packet rates and low ser-vice times we set the acceptable bound at 300micros to minimizelatency violations given our SLO of 500micros at the 99th per-centile as measured end-to-end by the client machine

For this we rely on a key side effect of IXrsquos use of adap-tive batching unprocessed packets that form the backlogare queued in a central location namely in step (1) in thepipeline of Fig 4 Packets are then processed in order inbounded batches to completion through both the networkingstack and the application logic In other words each IX core

Prepare Wait RPC Defer processIXCPpy

migrate(fgs)

CPU A

update HW

hold

finalize

done

NIC

first

packet

release

CPU B

prepare

timeout

(a) Thread-centric view

NICHWQ-A

HWQ-B

RETA

FGPoll

Poll

defaultQ-A

remoteQ-B

defaultQ-B

localQ-B

Thread-A

Thread-B

(b) Packet-centric view

Figure 5 Flow-group Migration Algorithm

operates like a simple FIFO queuing server onto which clas-sic queuing and control theory principles can be easily ap-plied In contrast conventional operating systems distributebuffers throughout the system in the NIC (because of coa-lesced interrupts) in the driver before networking process-ing and in the socket layer before being consumed by theapplication Furthermore these conventional systems pro-vide no ordering guarantees across flows which makes itdifficult to pinpoint congestion

To estimate queuing delays the controller monitors theiteration time τ and the queue depth Q With B the maximalbatch size the tail latency issimmax(delay) = dQBelowastτ Thedataplane computes each instantaneous metric every 10msfor the previous 10ms interval As these metrics are subjectto jitter the dataplane computes the exponential weightedmoving averages using multiple smoothing factors (α) inparallel For example we track the queue depth as Q(tα) =α lowastQnow +(1minusα) lowastQ(tminus 1α) The control loop executesat a frequency of 10 Hz which is sufficient to adapt to loadchanges

Deciding when to remove resources is trickier than de-ciding when to add them as shallow and near-empty queuesdo not provide reliable metrics Instead we measure idletime and observe that each change in the configuration addsor removes a predictable level of throughput (i) increas-ing cores from n to n+ 1 increases throughput by a factorof 1n (ii) enabling hyperthreads adds 30 (for our fore-ground workload at least) and (iii) the corersquos performanceis largely proportional to its clock rate We can thereforedefine the function threshold[ f rom to] as the ratio of maxthroughput between the f rom and to configurations Whenoperating at configuration f rom and the idle time exceedsthreshold[ f rom to] we can safely switch to configuration towithout loss of performance

These settings are not universal as a number of assump-tions such as load changes requests bursts and variationsin request service times will noticeably affect the metricsThese are policies not mechanisms In particular there is an

inherent tradeoff between aggressive energy saving policieswhich may lead to SLO violations and conservative poli-cies which will consume an excessive amount of energyFortunately the policy can be easily modified the dataplaneexposes naturally all relevant metrics and the control planedaemon issim500 lines of Python includingsim80 lines for thecontrol loop

44 Data Structures for Coherence-free ExecutionWe designed our memory management and data structurelayout to minimize the performance impact of flow groupmigration and to preserve the coherence-free property of ourdataplane in the steady-state Our memory allocator drawsinspiration from magazine-based SLAB allocation [7] Eachthread maintains a thread-local cache of every data typeIf the size of the thread-local cache exceeds a thresholdmemory items are returned using traditional locking to ashared memory pool Such rebalancing events are commonduring flow group migration but infrequent during normaloperation

We further organize our data structures so the pointerreferences building linked lists hash tables etc are onlybetween objects either both permanently associated with thesame thread (per-cpu) or both permanently associated withthe same flow group (per-fg) or both global in scope Asa result we can efficiently transfer ownership of an entireflow-grouprsquos data structures during migration while avoidinglocking or coherence traffic during regular operation

For example the event condition arrays batch syscallarrays timer wheels incoming and outgoing descriptorrings posted and pending mbufs are all per-cpu Theprotocol control blocks 5-tuple hash table listening sock-ets TW WAIT queues out-of-order fragments and unac-knowledged sequences are all per-fg The ARP table isglobal Some objects change scope during their lifetimebut only at well-defined junctures For example an mbuf ispart of the per-cpu queue while waiting to be processed(step 1 in Fig 4) but then becomes part of per-fg struc-tures during network processing

A global data structure the software flow group tablemanages the entry points into the per-fg structures IntelNICs conveniently store the RSS hash result as meta-dataalong with each received packet the bottom bits are used toindex into that table

We encountered one challenge in the management of TCPtimers We design our implementation to maximize normalexecution performance each thread independently managesits timers as a hierarchical timing wheel [49] ie as a per-cpu structure As a consequence a flow group migrationrequires a scan of the hierarchical wheels to identify andremove pending timers during the hold phase of Fig 5athose will be restored later in the destination thread at therelease phase of the flow group migration

45 Flow Group MigrationWhen adding or removing a thread the dynamic controllergenerates a set of migration requests Each individual re-quest is for a set of flow groups (fgs) currently handledby one elastic thread A to be handled by elastic thread B Tosimplify the implementation the controller serializes the mi-gration requests and the dataplane assumes that at most onesuch request is in progress at any point in time

Each thread has three queues that can hold incoming net-work packets the defaultQ contains packets received dur-ing steady state operation when no flow group migration ispending the remoteQ contains packets belonging to outgo-ing flow groups the localQ contains packets belonging toincoming flow groups The use of these two extra queuesis critical to ensure the processing of packets in the properorder by the networking stack

Fig 5 illustrates the migration steps in a thread-centricview (Fig 5a) and in a packet-centric view (Fig 5b) Thecontroller and the dataplane threads communicate via lock-free structures in shared memory First the controller signalsA to migrate fgs to B A first marks each flow group of theset fgs with a special tag to hold off normal processing onall threads moves packets which belong to the flow groupset fgs from defaultQ-A to remoteQ-B and stops alltimers belonging to the flow group set A then reprogramsthe RETA for all entries in fgs Packets still received byA will be appended to remoteQ-B packets received by Bwill go to localQ-B

Upon reception of the first packet whose flow group be-longs to fgs by B B signals A to initiate the final stageof migration Then B finalizes the migration by re-enablingfgsrsquos timers removing all migration tags and pre-pendingto its defaultQ-B the packets from remoteQ-B andthe packets from localQ-B Finally B notifies the con-trol plane that the operation is complete There are obviouslycorner-cases and subtleties in the algorithm For example Ainstalls a migration timer to finalize the migration even if Bhas not yet received any packets We set the timeout to 1 ms

5 EvaluationWe use three synthetic time-accelerated load patterns toevaluate the effectiveness of the control loop under stressfulconditions All three vary between nearly idle and maximumthroughput within a four minute period the slope patterngradually raises the target load from 0 and 62M RPS andthen reduces its load the step pattern increases load by 500KRPS every 10 seconds and finally the sine+noise patternis a basic sinusoidal pattern modified by randomly addingsharp noise that is uniformly distributed over [-250+250]KRPS and re-computed every 5 seconds The slope patternprovides a baseline to study smooth changes the step patternmodels abrupt and massive changes while the sine+noisepattern is representative of daily web patterns [48]

Fig 6 and Fig 7 show the results of these three dynamicload patterns for the energy proportionality and workloadconsolidation scenarios In each case the top figure mea-sures the observed throughput They are annotated with thecontrol loop events that add resources (green) or removethem (red) Empty triangles correspond to core allocationsand full triangles to DVFS changes The middle figure eval-uates the soundness of the algorithm and reports the 99thpercentile latency as observed by a client machine and re-ported every second Finally the bottom figures compare theoverall efficiency of our solution based on dynamic resourcecontrols with (i) the maximal static configuration using allcores and Turbo Boost and (ii) the ideal synthetic efficiencycomputed using the Pareto frontier of Fig 2Energy Proportionality Fig 6 shows the behavior for theenergy efficiency scenario In each case the workload tracksthe desired throughput of the pattern and exercises the en-tire sequence of configurations gradually adding cores en-abling hyperthreading increasing the frequency and finallyenabling Turbo Boost before doing it in reverse The steppattern is particularly challenging as the instant change inload level requires multiple back-to-back configurationschanges With a few exceptions the latencies remain wellbelow the 500micros SLO We further discuss the violations be-low Finally Fig 6 (bottom) compares the power dissipatedby the workload with the corresponding power levels as de-termined by the Pareto frontier (lower bound) or the max-imum static configuration (upper bound) This graph mea-

Smooth Step Sine+noiseEnergy Proportionality (W)

Max power 91 92 94Measured 42 (-54) 48 (-48) 53 (-44)Pareto bound 39 (-57) 41 (-55) 45 (-52)

Server consolidation opportunity ( of peak)Pareto bound 50 47 39Measured 46 39 32

Table 1 Energy Proportionality and Consolidation gains

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

max conf dynamic Pareto

0

20

40

60

80

100

0 50 100 150 200

Pow

er

(W)

Time (seconds)

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 6 Energy Proportionality of memcached running on IX for three load patterns

sures the effectiveness of the control loop to maximize en-ergy proportionality We observe that the dynamic (actuallymeasured) power curve tracks the Pareto (synthetic) curvewell which defines a bound on energy savings When the dy-namic resource controls enters Turbo Boost mode the mea-sured power in all three cases starts at the lower end of the4 W range and then gradually rises as expected Table 1shows that the three patterns have Pareto savings bounds of52 55 and 57 IXrsquos dynamic resource controls resultsin energy savings of 44 48 and 54 which is 85 87and 93 of the theoretical boundConsolidation Fig 7 shows the corresponding results forthe workload consolidation scenarios Fig 7 (top) measuresthe observed load which here as well tracks the desired loadRecall that the consolidation policy always operates at theprocessorrsquos nominal rate (or Turbo) which limits the numberof configuration changes Fig 7 (middle) similarly confirmsthat the system meets the SLO with few exceptions Fig 7(bottom) plots the throughput of the background batch appli-cation expressed as a percentage of its throughput on a dedi-cated processor at 24 Ghz We compare it only to the Paretooptimal upper bound as a maximum configuration wouldmonopolize cores and deliver zero background throughputTable 1 shows that for these three patterns our consolida-tion policy delivers 32ndash46 of the standalone throughput

of the background job which corresponds to 82ndash92 ofthe Pareto boundSLO violations A careful study of the SLO violations of the6 runs shows that they fall into two categories First there are16 violations caused by delays in packet processing due toflow group migrations resulting from the addition of a coreSecond there are 9 violations caused by abrupt increase ofthroughput mostly in the step pattern which occur beforeany flow migrations The control plane then reacts quickly(in sim100 ms) and accommodates to the new throughput byadjusting resources To further confirm the abrupt nature ofthroughput increase specific to the step pattern we note thatthe system performed up three consecutive increases in re-sources in order to resolve a single violation 23 of the 25total violations last a single second with the remaining twoviolations lasting two seconds We believe that the compli-ance with the SLO achieved by our system is more than ad-equate for any practical production deploymentFlow group migration analysis Table 2 measures the la-tency of the 552 flow group migrations that occur during the6 benchmarks as described in sect45 It also reports the totalnumber of packets whose processing is deferred during themigration (rather than dropped or reordered) We first ob-serve that migrations present distinct behaviors when scalingup and when scaling down the number of cores The differ-

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

0

20

40

60

80

100

0 50 100 150 200

o

f pea

k

Time (seconds)

dynamicPareto

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 7 Consolidation of memcached with a background batch task

ence can be intuitively explained since the migrations duringthe scale up are performed in a heavily loaded system whilethe system during the scale down is partially idle In absoluteterms migrations that occur when adding a core take 463microson average and less than 15 ms 95 of the time The outlierscan be explained by rare occurrences of longer preparationtimes or when processing up to 614 deferred packets

6 DiscussionUsing Pareto as a guide Even though the Pareto resultsare not used by the dynamic resource controller the Paretofrontier proved to be a valuable guide first to motivate andquantify the problem then to derive the configuration pol-icy sequence and finally to evaluate the effectiveness ofthe dynamic resource control by setting an upper bound onthe gains resulting from dynamic resource allocation Manyfactors such as software scalability hardware resource con-tention network and disk IO bottlenecks will influence thePareto frontier of any given application and therefore the de-rived control loop policies We will compare different work-loads in future workTradeoff between average latency and efficiency We usea one-dimensional SLO (99th pct le 500micros) and show thatwe can increase efficiency while maintaining that objectiveHowever this comes at a cost when considering the entirelatency distribution eg the average as well as the tail More

complex SLOs taking into account multiple aspects of la-tency distribution would define a different Pareto frontierand likely require adjustments to the control loopAdaptive flow-centric scheduling The new flow-groupmigration algorithm leads to a flow-centric approach to re-source scheduling where the network stack and applicationlogic always follow the steering decision POSIX applica-tions can balance flows by migrating file descriptors betweenthreads or processes but this tends to be inefficient becauseit is difficult for the kernel to match the flowrsquos destinationreceive queue to changes in CPU affinity Flow director canbe used by Linux to adjust the affinity of individual networkflows as a reactive measure to application-level and kernelthread migration rebalancing but the limited size of the redi-rection table prevents this mechanism from scaling to largeconnection counts By contrast our approach allows flowsto be migrated in entire groups improving efficiency and iscompatible with more scalable hardware flow steering mech-anisms based on RSSHardware trends Our experimental setup using one SandyBridge processor has a three-dimensional Pareto spaceClearly NUMA would add a new dimension Also thenewly released Haswell processors provide per-core DVFScontrols which further increases the Pareto space More-over hash filters for flow group steering could benefit fromrecent trends in NIC hardware For example Intelrsquos new

avg 95th pct max stddevad

dco

reprepare (micros) 67 231 521 93wait (micros) 135 541 983 190rpc (micros) 129 355 529 112deferred (micros) 101 401 1167 147total (micros) 463 1432 2870 459 packets 64 319 614 116

rem

ove

core prepare (micros) 25 54 398 32

wait (micros) 34 101 170 32rpc (micros) 12 25 49 7deferred (micros) 19 43 108 14total (micros) 93 160 456 47 packets 3 8 28 3

Table 2 Breakdown of flow group migration measured dur-ing the six benchmarks

XL710 chipset [19] has a 512 entry hash LUT (as well asindependent 64 entry LUTs for each VF) in contrast to the128 entries available in the 82599 chipset [17] used in ourevaluation This has the potential to reduce connection im-balances between cores especially with high core counts

7 Related WorkScheduling Scheduler activations [1] give applicationsgreater control over hardware threads and provide a mecha-nism for custom application-level scheduling Callisto [13]uses a similar strategy to improve the performance of co-located parallel runtime systems Our approach differs inthat an independent control plane manages the schedulingof hardware threads based on receive queuing latency indi-cators while the dataplane exposes a simple kernel thread-ing abstraction SEDA [51] also monitors queuing behaviorto make scheduling decisions such as thread pool sizingChronos [21] makes use of software-based flow steering butwith a focus on balancing load to reduce latency AffinityAccept [42] embraces a mixture of software and hardware-based flow steering in order to improve TCP connectionaffinity and increase throughput We focus instead on energyproportionality and workload consolidationEnergy Proportionality The energy proportionality prob-lem [3] has been well explored in previous work Somesystems have focused on solutions tailored to throughput-oriented workloads [33] or read-only workloads [23] Meis-ner et al [34] highlight unique challenges for low la-tency workloads and advocate full system active low-powermodes Similar to our system Pegasus [29] achieves CPUenergy proportionality for low latency workloads Our workexpands on Pegasus by exploring the elastic allocation ofhardware threads in combination with processor power man-agement states and by basing scheduling decisions on in-ternal latency metrics within a host endpoint instead of anexternal controller Niccolini et al show that a softwarerouter running on a dedicated machine can be made energy-

proportional [38] Similar to our approach queue lengthis used as a control signal to manage core allocation andDVFS settings However we focus on latency-sensitive ap-plications rather than middlebox traffic and consider theadditional case of workload consolidationCo-location Because host endpoints contain some com-ponents that are not energy proportional and thus are mostefficient when operating at 100 utilization co-location ofworkloads is also an important tool for improving energyefficiency At the cluster scheduler level BubbleUp [32] andParagon [10] make scheduling decisions that are interference-aware through efficient classification of the behavior ofworkload co-location Leverich et al [26] demonstratethat co-location of batch and low latency jobs is possi-ble on commodity operating systems Our approach ex-plores this issue at higher throughputs and with tighter la-tency SLOs Bubble-Flux [53] additionally controls back-ground threads we control background and latency-sensitivethreads CPI2 [54] detects performance interference by ob-serving changes in CPI and throttles offending jobs Thiswork is orthogonal to ours and could be a useful addi-tional signal for our control plane Heracles manages mul-tiple hardware and software isolation mechanisms includ-ing packet scheduling and cache partitioning to co-locatelatency-sensitive applications with batch tasks while main-taining millisecond SLOs [30] We limit our focus to DVFSand core assignment but target more aggressive SLOsLow-latency frameworks User-level networking stacks [2031 47] and RDMA applications [12 37 40] all rely onpolling typically by a fixed number of execution threadsto deliver micros-scale response times These stacks and appli-cations are evaluated based on their latency and throughputcharacteristics and do not take into account energy propor-tionality or workload consolidation considerations

8 ConclusionWe present the design and implementation of dynamic re-source controls for the IX dataplane which consists of acontroller that allocates cores and sets processor frequencyto adapt to changes in the load of latency-sensitive appli-cations The novel rebalancing mechanisms do not impactthe steady-state performance of the dataplane and can mi-grate a set of flow groups in milliseconds without droppingor reordering packets We develop two policies focused onoptimizing energy proportionality and workload consolida-tion Our results show that for three varying load patternsthe energy proportionality can save 44ndash54 of the pro-cessorrsquos energy or enable a background job to deliver 32ndash46 of its standalone throughput To evaluate the effective-ness of our approach we synthesize the Pareto frontier bycombining the behavior of all possible static configurationsOur policies deliver 85ndash93 of the Pareto optimal boundin terms of energy proportionality and 82ndash92 in termsof consolidation

AcknowledgementsThis work was supported by the MicrosoftEPFL Joint Re-search Center the Stanford Experimental Datacenter Laband NSF grant CNS-1422088 George Prekas is supportedby a Google Graduate Research Fellowship and Adam Be-lay by a Stanford Graduate Fellowship

References[1] T E Anderson B N Bershad E D Lazowska and H M

Levy Scheduler Activations Effective Kernel Support for theUser-Level Management of Parallelism ACM Trans ComputSyst 10(1)53ndash79 1992

[2] B Atikoglu Y Xu E Frachtenberg S Jiang andM Paleczny Workload Analysis of a Large-Scale Key-ValueStore In Proceedings of the 2012 ACM SIGMETRICS Inter-national Conference on Measurement and Modeling of Com-puter Systems pages 53ndash64 2012

[3] L A Barroso and U Holzle The Case for Energy-Proportional Computing IEEE Computer 40(12)33ndash372007

[4] L A Barroso J Clidaras and U Holzle The Datacenter as aComputer An Introduction to the Design of Warehouse-ScaleMachines Second Edition Synthesis Lectures on ComputerArchitecture Morgan amp Claypool Publishers 2013

[5] A Belay A Bittau A J Mashtizadeh D Terei D Mazieresand C Kozyrakis Dune Safe User-level Access to PrivilegedCPU Features In Proceedings of the 10th Symposium onOperating System Design and Implementation (OSDI) pages335ndash348 2012

[6] A Belay G Prekas A Klimovic S Grossman C Kozyrakisand E Bugnion IX A Protected Dataplane Operating Systemfor High Throughput and Low Latency In Proceedings of the11st Symposium on Operating System Design and Implemen-tation (OSDI) pages 49ndash65 2014

[7] J Bonwick and J Adams Magazines and Vmem Extendingthe Slab Allocator to Many CPUs and Arbitrary Resources InUSENIX Annual Technical Conference pages 15ndash33 2001

[8] H David E Gorbatov U R Hanebutte R Khanaa andC Le RAPL memory power estimation and capping InProceedings of the 2010 International Symposium on LowPower Electronics and Design pages 189ndash194 2010

[9] J Dean and L A Barroso The Tail at Scale Commun ACM56(2)74ndash80 2013

[10] C Delimitrou and C Kozyrakis Quality-of-Service-AwareScheduling in Heterogeneous Data centers with ParagonIEEE Micro 34(3)17ndash30 2014

[11] C Delimitrou and C Kozyrakis Quasar Resource-Efficientand QoS-Aware Cluster Management In Proceedings of the19th International Conference on Architectural Support forProgramming Languages and Operating Systems (ASPLOS-XIX) pages 127ndash144 2014

[12] A Dragojevic D Narayanan M Castro and O HodsonFaRM Fast Remote Memory In Proceedings of the 11stSymposium on Networked Systems Design and Implementa-tion (NSDI) pages 401ndash414 2014

[13] T Harris M Maas and V J Marathe Callisto co-schedulingparallel runtime systems In Proceedings of the 2014 EuroSysConference page 24 2014

[14] B Hindman A Konwinski M Zaharia A Ghodsi A DJoseph R H Katz S Shenker and I Stoica Mesos A Plat-form for Fine-Grained Resource Sharing in the Data CenterIn Proceedings of the 8th Symposium on Networked SystemsDesign and Implementation (NSDI) 2011

[15] C-H Hsu Y Zhang M A Laurenzano D Meisner T FWenisch J Mars L Tang and R G Dreslinski AdrenalinePinpointing and reining in tail queries with quick voltageboosting In Proceedings of the 21st IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages271ndash282 2015

[16] J Hwang K K Ramakrishnan and T Wood NetVM HighPerformance and Flexible Networking Using Virtualization onCommodity Platforms In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 445ndash458 2014

[17] Intel Corp Intel 82599 10 GbE Controller Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheets82599-10-gbe-controller-datasheetpdf2014

[18] Intel Corp Intel DPDK Data Plane Development Kit httpdpdkorg 2014

[19] Intel Corp Intel Ethernet Controller XL710 Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheetsxl710-10-40-controller-datasheetpdf2014

[20] E Jeong S Woo M A Jamshed H Jeong S Ihm D Hanand K Park mTCP a Highly Scalable User-level TCP Stackfor Multicore Systems In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 489ndash502 2014

[21] R Kapoor G Porter M Tewari G M Voelker and A Vah-dat Chronos Predictable Low Latency for Data Center Ap-plications In Proceedings of the 2012 ACM Symposium onCloud Computing (SOCC) page 9 2012

[22] W Kim M S Gupta G-Y Wei and D M Brooks Systemlevel analysis of fast per-core DVFS using on-chip switch-ing regulators In Proceedings of the 14th IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages123ndash134 2008

[23] A Krioukov P Mohan S Alspaugh L Keys D E Cullerand R H Katz NapSAC design and implementation of apower-proportional web cluster Computer CommunicationReview 41(1)102ndash108 2011

[24] K-C Leung V O K Li and D Yang An Overview ofPacket Reordering in Transmission Control Protocol (TCP)Problems Solutions and Challenges IEEE Trans ParallelDistrib Syst 18(4)522ndash535 2007

[25] J Leverich Mutilate High-Performance MemcachedLoad Generator httpsgithubcomleverichmutilate 2014

the processor (ii) it benefits from running a second thread oneach core with a consistent speedup of 13times (iii) it is mostenergy-efficient to first utilize the various cores and onlythen to enable the second hyperthread on each core ratherthan the other way around and (iv) it is least energy-efficientto increase the frequency

We observe that the background application (i) also scaleslinearly but (ii) does not benefit from the 2nd hyperthread(iii) is nearly energy-proportional across the frequency spec-trum with the exception of Turbo Boost From a total cost ofownership perspective the most efficient operating point forthe workload consolidation of the background task is there-fore to run the system at the processorrsquos nominal 24 Ghzfrequency whenever possible We combine these observa-tions with the data from the Pareto analysis and derive thefollowing policiesEnergy Proportional Policy As a base state run with onlyone core and hyperthread with the socket set at the mini-mal clock rate (12Ghz) To add resources first enable ad-ditional cores then enable hyperthreads on all cores (as asingle step) and only after that gradually increase the clockrate until reaching the nominal rate (24Ghz) finally enableTurbo Boost To remove resources do the opposite This pol-icy leads to a sequence of 22 different configurationsWorkload Consolidation Policy As a base state run thebackground jobs on all available cores with the processor atthe nominal clock rate To add resources to the foregroundapplication first shift cores from the background thread tothe foreground application one at a time This is done by firstsuspending the background threads use both hyperthreadsof the newly freed core for the foreground application Nextstop the background job entirely and allocate all cores to theforeground applications As a final step enable Turbo BoostThis policy leads to a sequence of 9 different configurations

These policies closely track the corresponding Paretofrontier For energy proportionality (i) the 32 different staticconfigurations of the frontier are a superset of the configura-tions enabled by the policy and (ii) the difference in overallimpact in terms of energy spent is marginal For consolida-tion Pareto and policy nearly identically overlap

Obviously these conclusions may vary with platformsand applications but the use of underlying Pareto method-ology remains valid In fact it should provide a robust guideto determine allocation policies for more complex workloads(eg with natural scalability limitations) and more flexibleplatforms (eg the newer Haswell processors can set theclock frequency separately for individual cores)

4 Dynamic Resource Controls of DataplanesWe now present the design of the mechanisms and poli-cies that are necessary to implement dynamic resource con-trols in dataplane operating systems Although the imple-mentation is specific to the IX dataplane operating systemthe design principles generally apply to all high-throughput

dataplane operating systems such as Arrakis [43] and user-level networking stacks running latency-sensitive applica-tions [18 20 31] Traditionally such environments have notfocused on dynamic resource controls instead they focusedsolely on throughput with statically allocated resourceswithout considerations for the energy consumed Addingresource controls in such environments poses the followingchallenges

1 How to rebalance flows across cores without impactingnormal performance This is particularly challenging asdataplane environments achieve high performance be-cause of their coherence-free execution model By thatwe mean a programming where all common case oper-ations execute without requiring communication of syn-chronization between threads

2 How to rebalance flows across processing cores withoutimpacting network performance given that dataplane op-erating systems implement their own TCPIP network-ing stack and that TCPIP performance is negatively im-pacted whenever packets are either lost or processed outof order [24] This is particularly challenging wheneverthe rebalancing operation also involves the reconfigura-tion of the NIC which generates inherent race conditions

3 How to determinemdashboth robustly and efficientlymdashwhetherresources need to be added or can be removed from thedataplane given the complexity of modern applicationsand the difficulty to estimate the impact of resource con-trols on application throughput

Our design addresses these three challenges We firstprovide in sect41 the necessary background on the IX data-plane and in sect42 the background on the few assumptionsmade on the hardware sect43 describes the implementationof the controller which relies exclusively on application-independent queuing delay metrics to adjust resources sect44describes how a dataplane operating system can have both acoherence-free execution model and at the same time allowfor rebalancing of flows Finally sect45 describes the mecha-nism that migrates flows without reordering packets

41 Background on IX

The IX system architecture separates the control plane re-sponsible for the allocation of resources from the dataplanewhich operates on them using an approach similar to Ar-rakis [43] IX relies on hardware virtualization to protect thekernel from applications it uses Dune [5] to load and exe-cute the IX dataplane instances and uses the Linux host asits control plane environment

The IX dynamic controller is a Python daemon that con-figures and launches the IX dataplane then monitors its ex-ecution makes policy decisions and communicates them tothe IX dataplane and the host operating system

Unlike conventional operating systems which decouplenetwork processing from application execution the IX data-

app

libix

tcpip tcpip

timer

adaptive batch

1

2

4

5

Event

ConditionsBatched

Syscalls

3

6

kernel

user

Figure 4 Adaptive batching in the IX data plane

plane relies on a run to completion with adaptive batchingscheme to deliver both low latency and high throughput toapplications

Fig 4 illustrates the key run-to-completion mechanismbuilt into the original IX dataplane [6] each elastic threadhas exclusive access to NIC queues In step (1) the data-plane polls the receive descriptor ring and moves all receivedpackets into an in-memory queue and potentially posts freshbuffer descriptors to the NIC for use with future incomingpackets The thread then executes steps (2) ndash (6) for an adap-tive batch of packets the TCPIP network stack processesthese packets (2) and the application then consumes the re-sulting events (3) Upon return from user-space the threadprocesses batched systems calls and in particular the onesthat direct outgoing TCPIP traffic (4) runs necessary timers(5) and places outgoing Ethernet frames in the NICrsquos trans-mit descriptor ring for transmission (6)

Adaptive batching is specifically defined as follows (i)we never wait to batch requests and batching only occurs inthe presence of congestion (ii) we set an upper bound on thenumber of batched packets Using batching only during con-gestion allows us to minimize the impact on latency whilebounding the batch size prevents the live set from exceed-ing cache capacities and avoids transmit queue starvationBatching improves packet rate because it amortizes systemcall transition overheads and improves instruction cache lo-cality prefetching effectiveness and branch prediction ac-curacy When applied adaptively batching also decreases la-tency because these same efficiencies reduce head-of-lineblocking

42 Dynamic NIC ConfigurationModern NICs provide hardware mechanisms such as VMDqFlow Director [17] and Receive Side Scaling (RSS) [36] todirect traffic to one of multiple queues as an essential mech-anism for multicore scalability For example the Intel x520

relies on an on-chip table (called the RSS Redirection Ta-ble or RETA [17]) to partition flows into multiple queueson the data path the NIC first computes the Toeplitz hashfunction of the packet 5-tuple header (as specified by RSS)and uses the lower 7 bits of the result as an index into theRETA table A flow group is the set of network flows with anidentical RETA index The host updates the table using PCIewrites on a distinct control path which can lead to packetreordering [52]

Such anomalies have severe consequences in generalon the performance of the TCPIP networking stack [24]IXrsquos coherence-free design exposes a further constraint be-cause there are no locks within the TCPIP stack two elasticthreads cannot process packets from the same flow simulta-neously For correctness packets received by the thread thatdoes not own the flow group must be dropped This furthernegatively impacts network performance

The IX dataplane implements the actual flow group mi-gration and relies on the RETA to change the mappings Thechallenge is to design a migration mechanism that updatesthe steering of packets to different queues (and by extensionto different threads) without (i) impacting the steady stateperformance of the dataplane and its coherence-free designor (ii) creating network anomalies during migration such asdropping packets or processing them out of order in the net-working stack

43 Control LoopWe first describe the control loop at the heart of IXrsquos dy-namic controller implemented within the control plane Thecontroller adjusts processor resources by suspending andresuming IX elastic threads and specifying the mappingbetween flow groups and threads It migrates flow groupsfrom multiple existing threads to a newly resumed threadand conversely from a thread to be suspended to the re-maining threads For the server consolidation scenario thecontrol loop also allocates resources dedicated to the back-ground applications In our experimental setup it simply is-sues SIGSTOP and SIGCONT signals to transparently con-trol the background application

The control loop does not depend on any detailed charac-terization of the Pareto frontier of sect31 Instead it merely im-plements the derived high-level policy of sect32 using queuingdelay estimates Specifically the user sets the upper boundon the acceptable queuing delay in our experiments withmemcached characterized by high packet rates and low ser-vice times we set the acceptable bound at 300micros to minimizelatency violations given our SLO of 500micros at the 99th per-centile as measured end-to-end by the client machine

For this we rely on a key side effect of IXrsquos use of adap-tive batching unprocessed packets that form the backlogare queued in a central location namely in step (1) in thepipeline of Fig 4 Packets are then processed in order inbounded batches to completion through both the networkingstack and the application logic In other words each IX core

Prepare Wait RPC Defer processIXCPpy

migrate(fgs)

CPU A

update HW

hold

finalize

done

NIC

first

packet

release

CPU B

prepare

timeout

(a) Thread-centric view

NICHWQ-A

HWQ-B

RETA

FGPoll

Poll

defaultQ-A

remoteQ-B

defaultQ-B

localQ-B

Thread-A

Thread-B

(b) Packet-centric view

Figure 5 Flow-group Migration Algorithm

operates like a simple FIFO queuing server onto which clas-sic queuing and control theory principles can be easily ap-plied In contrast conventional operating systems distributebuffers throughout the system in the NIC (because of coa-lesced interrupts) in the driver before networking process-ing and in the socket layer before being consumed by theapplication Furthermore these conventional systems pro-vide no ordering guarantees across flows which makes itdifficult to pinpoint congestion

To estimate queuing delays the controller monitors theiteration time τ and the queue depth Q With B the maximalbatch size the tail latency issimmax(delay) = dQBelowastτ Thedataplane computes each instantaneous metric every 10msfor the previous 10ms interval As these metrics are subjectto jitter the dataplane computes the exponential weightedmoving averages using multiple smoothing factors (α) inparallel For example we track the queue depth as Q(tα) =α lowastQnow +(1minusα) lowastQ(tminus 1α) The control loop executesat a frequency of 10 Hz which is sufficient to adapt to loadchanges

Deciding when to remove resources is trickier than de-ciding when to add them as shallow and near-empty queuesdo not provide reliable metrics Instead we measure idletime and observe that each change in the configuration addsor removes a predictable level of throughput (i) increas-ing cores from n to n+ 1 increases throughput by a factorof 1n (ii) enabling hyperthreads adds 30 (for our fore-ground workload at least) and (iii) the corersquos performanceis largely proportional to its clock rate We can thereforedefine the function threshold[ f rom to] as the ratio of maxthroughput between the f rom and to configurations Whenoperating at configuration f rom and the idle time exceedsthreshold[ f rom to] we can safely switch to configuration towithout loss of performance

These settings are not universal as a number of assump-tions such as load changes requests bursts and variationsin request service times will noticeably affect the metricsThese are policies not mechanisms In particular there is an

inherent tradeoff between aggressive energy saving policieswhich may lead to SLO violations and conservative poli-cies which will consume an excessive amount of energyFortunately the policy can be easily modified the dataplaneexposes naturally all relevant metrics and the control planedaemon issim500 lines of Python includingsim80 lines for thecontrol loop

44 Data Structures for Coherence-free ExecutionWe designed our memory management and data structurelayout to minimize the performance impact of flow groupmigration and to preserve the coherence-free property of ourdataplane in the steady-state Our memory allocator drawsinspiration from magazine-based SLAB allocation [7] Eachthread maintains a thread-local cache of every data typeIf the size of the thread-local cache exceeds a thresholdmemory items are returned using traditional locking to ashared memory pool Such rebalancing events are commonduring flow group migration but infrequent during normaloperation

We further organize our data structures so the pointerreferences building linked lists hash tables etc are onlybetween objects either both permanently associated with thesame thread (per-cpu) or both permanently associated withthe same flow group (per-fg) or both global in scope Asa result we can efficiently transfer ownership of an entireflow-grouprsquos data structures during migration while avoidinglocking or coherence traffic during regular operation

For example the event condition arrays batch syscallarrays timer wheels incoming and outgoing descriptorrings posted and pending mbufs are all per-cpu Theprotocol control blocks 5-tuple hash table listening sock-ets TW WAIT queues out-of-order fragments and unac-knowledged sequences are all per-fg The ARP table isglobal Some objects change scope during their lifetimebut only at well-defined junctures For example an mbuf ispart of the per-cpu queue while waiting to be processed(step 1 in Fig 4) but then becomes part of per-fg struc-tures during network processing

A global data structure the software flow group tablemanages the entry points into the per-fg structures IntelNICs conveniently store the RSS hash result as meta-dataalong with each received packet the bottom bits are used toindex into that table

We encountered one challenge in the management of TCPtimers We design our implementation to maximize normalexecution performance each thread independently managesits timers as a hierarchical timing wheel [49] ie as a per-cpu structure As a consequence a flow group migrationrequires a scan of the hierarchical wheels to identify andremove pending timers during the hold phase of Fig 5athose will be restored later in the destination thread at therelease phase of the flow group migration

45 Flow Group MigrationWhen adding or removing a thread the dynamic controllergenerates a set of migration requests Each individual re-quest is for a set of flow groups (fgs) currently handledby one elastic thread A to be handled by elastic thread B Tosimplify the implementation the controller serializes the mi-gration requests and the dataplane assumes that at most onesuch request is in progress at any point in time

Each thread has three queues that can hold incoming net-work packets the defaultQ contains packets received dur-ing steady state operation when no flow group migration ispending the remoteQ contains packets belonging to outgo-ing flow groups the localQ contains packets belonging toincoming flow groups The use of these two extra queuesis critical to ensure the processing of packets in the properorder by the networking stack

Fig 5 illustrates the migration steps in a thread-centricview (Fig 5a) and in a packet-centric view (Fig 5b) Thecontroller and the dataplane threads communicate via lock-free structures in shared memory First the controller signalsA to migrate fgs to B A first marks each flow group of theset fgs with a special tag to hold off normal processing onall threads moves packets which belong to the flow groupset fgs from defaultQ-A to remoteQ-B and stops alltimers belonging to the flow group set A then reprogramsthe RETA for all entries in fgs Packets still received byA will be appended to remoteQ-B packets received by Bwill go to localQ-B

Upon reception of the first packet whose flow group be-longs to fgs by B B signals A to initiate the final stageof migration Then B finalizes the migration by re-enablingfgsrsquos timers removing all migration tags and pre-pendingto its defaultQ-B the packets from remoteQ-B andthe packets from localQ-B Finally B notifies the con-trol plane that the operation is complete There are obviouslycorner-cases and subtleties in the algorithm For example Ainstalls a migration timer to finalize the migration even if Bhas not yet received any packets We set the timeout to 1 ms

5 EvaluationWe use three synthetic time-accelerated load patterns toevaluate the effectiveness of the control loop under stressfulconditions All three vary between nearly idle and maximumthroughput within a four minute period the slope patterngradually raises the target load from 0 and 62M RPS andthen reduces its load the step pattern increases load by 500KRPS every 10 seconds and finally the sine+noise patternis a basic sinusoidal pattern modified by randomly addingsharp noise that is uniformly distributed over [-250+250]KRPS and re-computed every 5 seconds The slope patternprovides a baseline to study smooth changes the step patternmodels abrupt and massive changes while the sine+noisepattern is representative of daily web patterns [48]

Fig 6 and Fig 7 show the results of these three dynamicload patterns for the energy proportionality and workloadconsolidation scenarios In each case the top figure mea-sures the observed throughput They are annotated with thecontrol loop events that add resources (green) or removethem (red) Empty triangles correspond to core allocationsand full triangles to DVFS changes The middle figure eval-uates the soundness of the algorithm and reports the 99thpercentile latency as observed by a client machine and re-ported every second Finally the bottom figures compare theoverall efficiency of our solution based on dynamic resourcecontrols with (i) the maximal static configuration using allcores and Turbo Boost and (ii) the ideal synthetic efficiencycomputed using the Pareto frontier of Fig 2Energy Proportionality Fig 6 shows the behavior for theenergy efficiency scenario In each case the workload tracksthe desired throughput of the pattern and exercises the en-tire sequence of configurations gradually adding cores en-abling hyperthreading increasing the frequency and finallyenabling Turbo Boost before doing it in reverse The steppattern is particularly challenging as the instant change inload level requires multiple back-to-back configurationschanges With a few exceptions the latencies remain wellbelow the 500micros SLO We further discuss the violations be-low Finally Fig 6 (bottom) compares the power dissipatedby the workload with the corresponding power levels as de-termined by the Pareto frontier (lower bound) or the max-imum static configuration (upper bound) This graph mea-

Smooth Step Sine+noiseEnergy Proportionality (W)

Max power 91 92 94Measured 42 (-54) 48 (-48) 53 (-44)Pareto bound 39 (-57) 41 (-55) 45 (-52)

Server consolidation opportunity ( of peak)Pareto bound 50 47 39Measured 46 39 32

Table 1 Energy Proportionality and Consolidation gains

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

max conf dynamic Pareto

0

20

40

60

80

100

0 50 100 150 200

Pow

er

(W)

Time (seconds)

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 6 Energy Proportionality of memcached running on IX for three load patterns

sures the effectiveness of the control loop to maximize en-ergy proportionality We observe that the dynamic (actuallymeasured) power curve tracks the Pareto (synthetic) curvewell which defines a bound on energy savings When the dy-namic resource controls enters Turbo Boost mode the mea-sured power in all three cases starts at the lower end of the4 W range and then gradually rises as expected Table 1shows that the three patterns have Pareto savings bounds of52 55 and 57 IXrsquos dynamic resource controls resultsin energy savings of 44 48 and 54 which is 85 87and 93 of the theoretical boundConsolidation Fig 7 shows the corresponding results forthe workload consolidation scenarios Fig 7 (top) measuresthe observed load which here as well tracks the desired loadRecall that the consolidation policy always operates at theprocessorrsquos nominal rate (or Turbo) which limits the numberof configuration changes Fig 7 (middle) similarly confirmsthat the system meets the SLO with few exceptions Fig 7(bottom) plots the throughput of the background batch appli-cation expressed as a percentage of its throughput on a dedi-cated processor at 24 Ghz We compare it only to the Paretooptimal upper bound as a maximum configuration wouldmonopolize cores and deliver zero background throughputTable 1 shows that for these three patterns our consolida-tion policy delivers 32ndash46 of the standalone throughput

of the background job which corresponds to 82ndash92 ofthe Pareto boundSLO violations A careful study of the SLO violations of the6 runs shows that they fall into two categories First there are16 violations caused by delays in packet processing due toflow group migrations resulting from the addition of a coreSecond there are 9 violations caused by abrupt increase ofthroughput mostly in the step pattern which occur beforeany flow migrations The control plane then reacts quickly(in sim100 ms) and accommodates to the new throughput byadjusting resources To further confirm the abrupt nature ofthroughput increase specific to the step pattern we note thatthe system performed up three consecutive increases in re-sources in order to resolve a single violation 23 of the 25total violations last a single second with the remaining twoviolations lasting two seconds We believe that the compli-ance with the SLO achieved by our system is more than ad-equate for any practical production deploymentFlow group migration analysis Table 2 measures the la-tency of the 552 flow group migrations that occur during the6 benchmarks as described in sect45 It also reports the totalnumber of packets whose processing is deferred during themigration (rather than dropped or reordered) We first ob-serve that migrations present distinct behaviors when scalingup and when scaling down the number of cores The differ-

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

0

20

40

60

80

100

0 50 100 150 200

o

f pea

k

Time (seconds)

dynamicPareto

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 7 Consolidation of memcached with a background batch task

ence can be intuitively explained since the migrations duringthe scale up are performed in a heavily loaded system whilethe system during the scale down is partially idle In absoluteterms migrations that occur when adding a core take 463microson average and less than 15 ms 95 of the time The outlierscan be explained by rare occurrences of longer preparationtimes or when processing up to 614 deferred packets

6 DiscussionUsing Pareto as a guide Even though the Pareto resultsare not used by the dynamic resource controller the Paretofrontier proved to be a valuable guide first to motivate andquantify the problem then to derive the configuration pol-icy sequence and finally to evaluate the effectiveness ofthe dynamic resource control by setting an upper bound onthe gains resulting from dynamic resource allocation Manyfactors such as software scalability hardware resource con-tention network and disk IO bottlenecks will influence thePareto frontier of any given application and therefore the de-rived control loop policies We will compare different work-loads in future workTradeoff between average latency and efficiency We usea one-dimensional SLO (99th pct le 500micros) and show thatwe can increase efficiency while maintaining that objectiveHowever this comes at a cost when considering the entirelatency distribution eg the average as well as the tail More

complex SLOs taking into account multiple aspects of la-tency distribution would define a different Pareto frontierand likely require adjustments to the control loopAdaptive flow-centric scheduling The new flow-groupmigration algorithm leads to a flow-centric approach to re-source scheduling where the network stack and applicationlogic always follow the steering decision POSIX applica-tions can balance flows by migrating file descriptors betweenthreads or processes but this tends to be inefficient becauseit is difficult for the kernel to match the flowrsquos destinationreceive queue to changes in CPU affinity Flow director canbe used by Linux to adjust the affinity of individual networkflows as a reactive measure to application-level and kernelthread migration rebalancing but the limited size of the redi-rection table prevents this mechanism from scaling to largeconnection counts By contrast our approach allows flowsto be migrated in entire groups improving efficiency and iscompatible with more scalable hardware flow steering mech-anisms based on RSSHardware trends Our experimental setup using one SandyBridge processor has a three-dimensional Pareto spaceClearly NUMA would add a new dimension Also thenewly released Haswell processors provide per-core DVFScontrols which further increases the Pareto space More-over hash filters for flow group steering could benefit fromrecent trends in NIC hardware For example Intelrsquos new

avg 95th pct max stddevad

dco

reprepare (micros) 67 231 521 93wait (micros) 135 541 983 190rpc (micros) 129 355 529 112deferred (micros) 101 401 1167 147total (micros) 463 1432 2870 459 packets 64 319 614 116

rem

ove

core prepare (micros) 25 54 398 32

wait (micros) 34 101 170 32rpc (micros) 12 25 49 7deferred (micros) 19 43 108 14total (micros) 93 160 456 47 packets 3 8 28 3

Table 2 Breakdown of flow group migration measured dur-ing the six benchmarks

XL710 chipset [19] has a 512 entry hash LUT (as well asindependent 64 entry LUTs for each VF) in contrast to the128 entries available in the 82599 chipset [17] used in ourevaluation This has the potential to reduce connection im-balances between cores especially with high core counts

7 Related WorkScheduling Scheduler activations [1] give applicationsgreater control over hardware threads and provide a mecha-nism for custom application-level scheduling Callisto [13]uses a similar strategy to improve the performance of co-located parallel runtime systems Our approach differs inthat an independent control plane manages the schedulingof hardware threads based on receive queuing latency indi-cators while the dataplane exposes a simple kernel thread-ing abstraction SEDA [51] also monitors queuing behaviorto make scheduling decisions such as thread pool sizingChronos [21] makes use of software-based flow steering butwith a focus on balancing load to reduce latency AffinityAccept [42] embraces a mixture of software and hardware-based flow steering in order to improve TCP connectionaffinity and increase throughput We focus instead on energyproportionality and workload consolidationEnergy Proportionality The energy proportionality prob-lem [3] has been well explored in previous work Somesystems have focused on solutions tailored to throughput-oriented workloads [33] or read-only workloads [23] Meis-ner et al [34] highlight unique challenges for low la-tency workloads and advocate full system active low-powermodes Similar to our system Pegasus [29] achieves CPUenergy proportionality for low latency workloads Our workexpands on Pegasus by exploring the elastic allocation ofhardware threads in combination with processor power man-agement states and by basing scheduling decisions on in-ternal latency metrics within a host endpoint instead of anexternal controller Niccolini et al show that a softwarerouter running on a dedicated machine can be made energy-

proportional [38] Similar to our approach queue lengthis used as a control signal to manage core allocation andDVFS settings However we focus on latency-sensitive ap-plications rather than middlebox traffic and consider theadditional case of workload consolidationCo-location Because host endpoints contain some com-ponents that are not energy proportional and thus are mostefficient when operating at 100 utilization co-location ofworkloads is also an important tool for improving energyefficiency At the cluster scheduler level BubbleUp [32] andParagon [10] make scheduling decisions that are interference-aware through efficient classification of the behavior ofworkload co-location Leverich et al [26] demonstratethat co-location of batch and low latency jobs is possi-ble on commodity operating systems Our approach ex-plores this issue at higher throughputs and with tighter la-tency SLOs Bubble-Flux [53] additionally controls back-ground threads we control background and latency-sensitivethreads CPI2 [54] detects performance interference by ob-serving changes in CPI and throttles offending jobs Thiswork is orthogonal to ours and could be a useful addi-tional signal for our control plane Heracles manages mul-tiple hardware and software isolation mechanisms includ-ing packet scheduling and cache partitioning to co-locatelatency-sensitive applications with batch tasks while main-taining millisecond SLOs [30] We limit our focus to DVFSand core assignment but target more aggressive SLOsLow-latency frameworks User-level networking stacks [2031 47] and RDMA applications [12 37 40] all rely onpolling typically by a fixed number of execution threadsto deliver micros-scale response times These stacks and appli-cations are evaluated based on their latency and throughputcharacteristics and do not take into account energy propor-tionality or workload consolidation considerations

8 ConclusionWe present the design and implementation of dynamic re-source controls for the IX dataplane which consists of acontroller that allocates cores and sets processor frequencyto adapt to changes in the load of latency-sensitive appli-cations The novel rebalancing mechanisms do not impactthe steady-state performance of the dataplane and can mi-grate a set of flow groups in milliseconds without droppingor reordering packets We develop two policies focused onoptimizing energy proportionality and workload consolida-tion Our results show that for three varying load patternsthe energy proportionality can save 44ndash54 of the pro-cessorrsquos energy or enable a background job to deliver 32ndash46 of its standalone throughput To evaluate the effective-ness of our approach we synthesize the Pareto frontier bycombining the behavior of all possible static configurationsOur policies deliver 85ndash93 of the Pareto optimal boundin terms of energy proportionality and 82ndash92 in termsof consolidation

AcknowledgementsThis work was supported by the MicrosoftEPFL Joint Re-search Center the Stanford Experimental Datacenter Laband NSF grant CNS-1422088 George Prekas is supportedby a Google Graduate Research Fellowship and Adam Be-lay by a Stanford Graduate Fellowship

References[1] T E Anderson B N Bershad E D Lazowska and H M

Levy Scheduler Activations Effective Kernel Support for theUser-Level Management of Parallelism ACM Trans ComputSyst 10(1)53ndash79 1992

[2] B Atikoglu Y Xu E Frachtenberg S Jiang andM Paleczny Workload Analysis of a Large-Scale Key-ValueStore In Proceedings of the 2012 ACM SIGMETRICS Inter-national Conference on Measurement and Modeling of Com-puter Systems pages 53ndash64 2012

[3] L A Barroso and U Holzle The Case for Energy-Proportional Computing IEEE Computer 40(12)33ndash372007

[4] L A Barroso J Clidaras and U Holzle The Datacenter as aComputer An Introduction to the Design of Warehouse-ScaleMachines Second Edition Synthesis Lectures on ComputerArchitecture Morgan amp Claypool Publishers 2013

[5] A Belay A Bittau A J Mashtizadeh D Terei D Mazieresand C Kozyrakis Dune Safe User-level Access to PrivilegedCPU Features In Proceedings of the 10th Symposium onOperating System Design and Implementation (OSDI) pages335ndash348 2012

[6] A Belay G Prekas A Klimovic S Grossman C Kozyrakisand E Bugnion IX A Protected Dataplane Operating Systemfor High Throughput and Low Latency In Proceedings of the11st Symposium on Operating System Design and Implemen-tation (OSDI) pages 49ndash65 2014

[7] J Bonwick and J Adams Magazines and Vmem Extendingthe Slab Allocator to Many CPUs and Arbitrary Resources InUSENIX Annual Technical Conference pages 15ndash33 2001

[8] H David E Gorbatov U R Hanebutte R Khanaa andC Le RAPL memory power estimation and capping InProceedings of the 2010 International Symposium on LowPower Electronics and Design pages 189ndash194 2010

[9] J Dean and L A Barroso The Tail at Scale Commun ACM56(2)74ndash80 2013

[10] C Delimitrou and C Kozyrakis Quality-of-Service-AwareScheduling in Heterogeneous Data centers with ParagonIEEE Micro 34(3)17ndash30 2014

[11] C Delimitrou and C Kozyrakis Quasar Resource-Efficientand QoS-Aware Cluster Management In Proceedings of the19th International Conference on Architectural Support forProgramming Languages and Operating Systems (ASPLOS-XIX) pages 127ndash144 2014

[12] A Dragojevic D Narayanan M Castro and O HodsonFaRM Fast Remote Memory In Proceedings of the 11stSymposium on Networked Systems Design and Implementa-tion (NSDI) pages 401ndash414 2014

[13] T Harris M Maas and V J Marathe Callisto co-schedulingparallel runtime systems In Proceedings of the 2014 EuroSysConference page 24 2014

[14] B Hindman A Konwinski M Zaharia A Ghodsi A DJoseph R H Katz S Shenker and I Stoica Mesos A Plat-form for Fine-Grained Resource Sharing in the Data CenterIn Proceedings of the 8th Symposium on Networked SystemsDesign and Implementation (NSDI) 2011

[15] C-H Hsu Y Zhang M A Laurenzano D Meisner T FWenisch J Mars L Tang and R G Dreslinski AdrenalinePinpointing and reining in tail queries with quick voltageboosting In Proceedings of the 21st IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages271ndash282 2015

[16] J Hwang K K Ramakrishnan and T Wood NetVM HighPerformance and Flexible Networking Using Virtualization onCommodity Platforms In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 445ndash458 2014

[17] Intel Corp Intel 82599 10 GbE Controller Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheets82599-10-gbe-controller-datasheetpdf2014

[18] Intel Corp Intel DPDK Data Plane Development Kit httpdpdkorg 2014

[19] Intel Corp Intel Ethernet Controller XL710 Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheetsxl710-10-40-controller-datasheetpdf2014

[20] E Jeong S Woo M A Jamshed H Jeong S Ihm D Hanand K Park mTCP a Highly Scalable User-level TCP Stackfor Multicore Systems In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 489ndash502 2014

[21] R Kapoor G Porter M Tewari G M Voelker and A Vah-dat Chronos Predictable Low Latency for Data Center Ap-plications In Proceedings of the 2012 ACM Symposium onCloud Computing (SOCC) page 9 2012

[22] W Kim M S Gupta G-Y Wei and D M Brooks Systemlevel analysis of fast per-core DVFS using on-chip switch-ing regulators In Proceedings of the 14th IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages123ndash134 2008

[23] A Krioukov P Mohan S Alspaugh L Keys D E Cullerand R H Katz NapSAC design and implementation of apower-proportional web cluster Computer CommunicationReview 41(1)102ndash108 2011

[24] K-C Leung V O K Li and D Yang An Overview ofPacket Reordering in Transmission Control Protocol (TCP)Problems Solutions and Challenges IEEE Trans ParallelDistrib Syst 18(4)522ndash535 2007

[25] J Leverich Mutilate High-Performance MemcachedLoad Generator httpsgithubcomleverichmutilate 2014

app

libix

tcpip tcpip

timer

adaptive batch

1

2

4

5

Event

ConditionsBatched

Syscalls

3

6

kernel

user

Figure 4 Adaptive batching in the IX data plane

plane relies on a run to completion with adaptive batchingscheme to deliver both low latency and high throughput toapplications

Fig 4 illustrates the key run-to-completion mechanismbuilt into the original IX dataplane [6] each elastic threadhas exclusive access to NIC queues In step (1) the data-plane polls the receive descriptor ring and moves all receivedpackets into an in-memory queue and potentially posts freshbuffer descriptors to the NIC for use with future incomingpackets The thread then executes steps (2) ndash (6) for an adap-tive batch of packets the TCPIP network stack processesthese packets (2) and the application then consumes the re-sulting events (3) Upon return from user-space the threadprocesses batched systems calls and in particular the onesthat direct outgoing TCPIP traffic (4) runs necessary timers(5) and places outgoing Ethernet frames in the NICrsquos trans-mit descriptor ring for transmission (6)

Adaptive batching is specifically defined as follows (i)we never wait to batch requests and batching only occurs inthe presence of congestion (ii) we set an upper bound on thenumber of batched packets Using batching only during con-gestion allows us to minimize the impact on latency whilebounding the batch size prevents the live set from exceed-ing cache capacities and avoids transmit queue starvationBatching improves packet rate because it amortizes systemcall transition overheads and improves instruction cache lo-cality prefetching effectiveness and branch prediction ac-curacy When applied adaptively batching also decreases la-tency because these same efficiencies reduce head-of-lineblocking

42 Dynamic NIC ConfigurationModern NICs provide hardware mechanisms such as VMDqFlow Director [17] and Receive Side Scaling (RSS) [36] todirect traffic to one of multiple queues as an essential mech-anism for multicore scalability For example the Intel x520

relies on an on-chip table (called the RSS Redirection Ta-ble or RETA [17]) to partition flows into multiple queueson the data path the NIC first computes the Toeplitz hashfunction of the packet 5-tuple header (as specified by RSS)and uses the lower 7 bits of the result as an index into theRETA table A flow group is the set of network flows with anidentical RETA index The host updates the table using PCIewrites on a distinct control path which can lead to packetreordering [52]

Such anomalies have severe consequences in generalon the performance of the TCPIP networking stack [24]IXrsquos coherence-free design exposes a further constraint be-cause there are no locks within the TCPIP stack two elasticthreads cannot process packets from the same flow simulta-neously For correctness packets received by the thread thatdoes not own the flow group must be dropped This furthernegatively impacts network performance

The IX dataplane implements the actual flow group mi-gration and relies on the RETA to change the mappings Thechallenge is to design a migration mechanism that updatesthe steering of packets to different queues (and by extensionto different threads) without (i) impacting the steady stateperformance of the dataplane and its coherence-free designor (ii) creating network anomalies during migration such asdropping packets or processing them out of order in the net-working stack

43 Control LoopWe first describe the control loop at the heart of IXrsquos dy-namic controller implemented within the control plane Thecontroller adjusts processor resources by suspending andresuming IX elastic threads and specifying the mappingbetween flow groups and threads It migrates flow groupsfrom multiple existing threads to a newly resumed threadand conversely from a thread to be suspended to the re-maining threads For the server consolidation scenario thecontrol loop also allocates resources dedicated to the back-ground applications In our experimental setup it simply is-sues SIGSTOP and SIGCONT signals to transparently con-trol the background application

The control loop does not depend on any detailed charac-terization of the Pareto frontier of sect31 Instead it merely im-plements the derived high-level policy of sect32 using queuingdelay estimates Specifically the user sets the upper boundon the acceptable queuing delay in our experiments withmemcached characterized by high packet rates and low ser-vice times we set the acceptable bound at 300micros to minimizelatency violations given our SLO of 500micros at the 99th per-centile as measured end-to-end by the client machine

For this we rely on a key side effect of IXrsquos use of adap-tive batching unprocessed packets that form the backlogare queued in a central location namely in step (1) in thepipeline of Fig 4 Packets are then processed in order inbounded batches to completion through both the networkingstack and the application logic In other words each IX core

Prepare Wait RPC Defer processIXCPpy

migrate(fgs)

CPU A

update HW

hold

finalize

done

NIC

first

packet

release

CPU B

prepare

timeout

(a) Thread-centric view

NICHWQ-A

HWQ-B

RETA

FGPoll

Poll

defaultQ-A

remoteQ-B

defaultQ-B

localQ-B

Thread-A

Thread-B

(b) Packet-centric view

Figure 5 Flow-group Migration Algorithm

operates like a simple FIFO queuing server onto which clas-sic queuing and control theory principles can be easily ap-plied In contrast conventional operating systems distributebuffers throughout the system in the NIC (because of coa-lesced interrupts) in the driver before networking process-ing and in the socket layer before being consumed by theapplication Furthermore these conventional systems pro-vide no ordering guarantees across flows which makes itdifficult to pinpoint congestion

To estimate queuing delays the controller monitors theiteration time τ and the queue depth Q With B the maximalbatch size the tail latency issimmax(delay) = dQBelowastτ Thedataplane computes each instantaneous metric every 10msfor the previous 10ms interval As these metrics are subjectto jitter the dataplane computes the exponential weightedmoving averages using multiple smoothing factors (α) inparallel For example we track the queue depth as Q(tα) =α lowastQnow +(1minusα) lowastQ(tminus 1α) The control loop executesat a frequency of 10 Hz which is sufficient to adapt to loadchanges

Deciding when to remove resources is trickier than de-ciding when to add them as shallow and near-empty queuesdo not provide reliable metrics Instead we measure idletime and observe that each change in the configuration addsor removes a predictable level of throughput (i) increas-ing cores from n to n+ 1 increases throughput by a factorof 1n (ii) enabling hyperthreads adds 30 (for our fore-ground workload at least) and (iii) the corersquos performanceis largely proportional to its clock rate We can thereforedefine the function threshold[ f rom to] as the ratio of maxthroughput between the f rom and to configurations Whenoperating at configuration f rom and the idle time exceedsthreshold[ f rom to] we can safely switch to configuration towithout loss of performance

These settings are not universal as a number of assump-tions such as load changes requests bursts and variationsin request service times will noticeably affect the metricsThese are policies not mechanisms In particular there is an

inherent tradeoff between aggressive energy saving policieswhich may lead to SLO violations and conservative poli-cies which will consume an excessive amount of energyFortunately the policy can be easily modified the dataplaneexposes naturally all relevant metrics and the control planedaemon issim500 lines of Python includingsim80 lines for thecontrol loop

44 Data Structures for Coherence-free ExecutionWe designed our memory management and data structurelayout to minimize the performance impact of flow groupmigration and to preserve the coherence-free property of ourdataplane in the steady-state Our memory allocator drawsinspiration from magazine-based SLAB allocation [7] Eachthread maintains a thread-local cache of every data typeIf the size of the thread-local cache exceeds a thresholdmemory items are returned using traditional locking to ashared memory pool Such rebalancing events are commonduring flow group migration but infrequent during normaloperation

We further organize our data structures so the pointerreferences building linked lists hash tables etc are onlybetween objects either both permanently associated with thesame thread (per-cpu) or both permanently associated withthe same flow group (per-fg) or both global in scope Asa result we can efficiently transfer ownership of an entireflow-grouprsquos data structures during migration while avoidinglocking or coherence traffic during regular operation

For example the event condition arrays batch syscallarrays timer wheels incoming and outgoing descriptorrings posted and pending mbufs are all per-cpu Theprotocol control blocks 5-tuple hash table listening sock-ets TW WAIT queues out-of-order fragments and unac-knowledged sequences are all per-fg The ARP table isglobal Some objects change scope during their lifetimebut only at well-defined junctures For example an mbuf ispart of the per-cpu queue while waiting to be processed(step 1 in Fig 4) but then becomes part of per-fg struc-tures during network processing

A global data structure the software flow group tablemanages the entry points into the per-fg structures IntelNICs conveniently store the RSS hash result as meta-dataalong with each received packet the bottom bits are used toindex into that table

We encountered one challenge in the management of TCPtimers We design our implementation to maximize normalexecution performance each thread independently managesits timers as a hierarchical timing wheel [49] ie as a per-cpu structure As a consequence a flow group migrationrequires a scan of the hierarchical wheels to identify andremove pending timers during the hold phase of Fig 5athose will be restored later in the destination thread at therelease phase of the flow group migration

45 Flow Group MigrationWhen adding or removing a thread the dynamic controllergenerates a set of migration requests Each individual re-quest is for a set of flow groups (fgs) currently handledby one elastic thread A to be handled by elastic thread B Tosimplify the implementation the controller serializes the mi-gration requests and the dataplane assumes that at most onesuch request is in progress at any point in time

Each thread has three queues that can hold incoming net-work packets the defaultQ contains packets received dur-ing steady state operation when no flow group migration ispending the remoteQ contains packets belonging to outgo-ing flow groups the localQ contains packets belonging toincoming flow groups The use of these two extra queuesis critical to ensure the processing of packets in the properorder by the networking stack

Fig 5 illustrates the migration steps in a thread-centricview (Fig 5a) and in a packet-centric view (Fig 5b) Thecontroller and the dataplane threads communicate via lock-free structures in shared memory First the controller signalsA to migrate fgs to B A first marks each flow group of theset fgs with a special tag to hold off normal processing onall threads moves packets which belong to the flow groupset fgs from defaultQ-A to remoteQ-B and stops alltimers belonging to the flow group set A then reprogramsthe RETA for all entries in fgs Packets still received byA will be appended to remoteQ-B packets received by Bwill go to localQ-B

Upon reception of the first packet whose flow group be-longs to fgs by B B signals A to initiate the final stageof migration Then B finalizes the migration by re-enablingfgsrsquos timers removing all migration tags and pre-pendingto its defaultQ-B the packets from remoteQ-B andthe packets from localQ-B Finally B notifies the con-trol plane that the operation is complete There are obviouslycorner-cases and subtleties in the algorithm For example Ainstalls a migration timer to finalize the migration even if Bhas not yet received any packets We set the timeout to 1 ms

5 EvaluationWe use three synthetic time-accelerated load patterns toevaluate the effectiveness of the control loop under stressfulconditions All three vary between nearly idle and maximumthroughput within a four minute period the slope patterngradually raises the target load from 0 and 62M RPS andthen reduces its load the step pattern increases load by 500KRPS every 10 seconds and finally the sine+noise patternis a basic sinusoidal pattern modified by randomly addingsharp noise that is uniformly distributed over [-250+250]KRPS and re-computed every 5 seconds The slope patternprovides a baseline to study smooth changes the step patternmodels abrupt and massive changes while the sine+noisepattern is representative of daily web patterns [48]

Fig 6 and Fig 7 show the results of these three dynamicload patterns for the energy proportionality and workloadconsolidation scenarios In each case the top figure mea-sures the observed throughput They are annotated with thecontrol loop events that add resources (green) or removethem (red) Empty triangles correspond to core allocationsand full triangles to DVFS changes The middle figure eval-uates the soundness of the algorithm and reports the 99thpercentile latency as observed by a client machine and re-ported every second Finally the bottom figures compare theoverall efficiency of our solution based on dynamic resourcecontrols with (i) the maximal static configuration using allcores and Turbo Boost and (ii) the ideal synthetic efficiencycomputed using the Pareto frontier of Fig 2Energy Proportionality Fig 6 shows the behavior for theenergy efficiency scenario In each case the workload tracksthe desired throughput of the pattern and exercises the en-tire sequence of configurations gradually adding cores en-abling hyperthreading increasing the frequency and finallyenabling Turbo Boost before doing it in reverse The steppattern is particularly challenging as the instant change inload level requires multiple back-to-back configurationschanges With a few exceptions the latencies remain wellbelow the 500micros SLO We further discuss the violations be-low Finally Fig 6 (bottom) compares the power dissipatedby the workload with the corresponding power levels as de-termined by the Pareto frontier (lower bound) or the max-imum static configuration (upper bound) This graph mea-

Smooth Step Sine+noiseEnergy Proportionality (W)

Max power 91 92 94Measured 42 (-54) 48 (-48) 53 (-44)Pareto bound 39 (-57) 41 (-55) 45 (-52)

Server consolidation opportunity ( of peak)Pareto bound 50 47 39Measured 46 39 32

Table 1 Energy Proportionality and Consolidation gains

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

max conf dynamic Pareto

0

20

40

60

80

100

0 50 100 150 200

Pow

er

(W)

Time (seconds)

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 6 Energy Proportionality of memcached running on IX for three load patterns

sures the effectiveness of the control loop to maximize en-ergy proportionality We observe that the dynamic (actuallymeasured) power curve tracks the Pareto (synthetic) curvewell which defines a bound on energy savings When the dy-namic resource controls enters Turbo Boost mode the mea-sured power in all three cases starts at the lower end of the4 W range and then gradually rises as expected Table 1shows that the three patterns have Pareto savings bounds of52 55 and 57 IXrsquos dynamic resource controls resultsin energy savings of 44 48 and 54 which is 85 87and 93 of the theoretical boundConsolidation Fig 7 shows the corresponding results forthe workload consolidation scenarios Fig 7 (top) measuresthe observed load which here as well tracks the desired loadRecall that the consolidation policy always operates at theprocessorrsquos nominal rate (or Turbo) which limits the numberof configuration changes Fig 7 (middle) similarly confirmsthat the system meets the SLO with few exceptions Fig 7(bottom) plots the throughput of the background batch appli-cation expressed as a percentage of its throughput on a dedi-cated processor at 24 Ghz We compare it only to the Paretooptimal upper bound as a maximum configuration wouldmonopolize cores and deliver zero background throughputTable 1 shows that for these three patterns our consolida-tion policy delivers 32ndash46 of the standalone throughput

of the background job which corresponds to 82ndash92 ofthe Pareto boundSLO violations A careful study of the SLO violations of the6 runs shows that they fall into two categories First there are16 violations caused by delays in packet processing due toflow group migrations resulting from the addition of a coreSecond there are 9 violations caused by abrupt increase ofthroughput mostly in the step pattern which occur beforeany flow migrations The control plane then reacts quickly(in sim100 ms) and accommodates to the new throughput byadjusting resources To further confirm the abrupt nature ofthroughput increase specific to the step pattern we note thatthe system performed up three consecutive increases in re-sources in order to resolve a single violation 23 of the 25total violations last a single second with the remaining twoviolations lasting two seconds We believe that the compli-ance with the SLO achieved by our system is more than ad-equate for any practical production deploymentFlow group migration analysis Table 2 measures the la-tency of the 552 flow group migrations that occur during the6 benchmarks as described in sect45 It also reports the totalnumber of packets whose processing is deferred during themigration (rather than dropped or reordered) We first ob-serve that migrations present distinct behaviors when scalingup and when scaling down the number of cores The differ-

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

0

20

40

60

80

100

0 50 100 150 200

o

f pea

k

Time (seconds)

dynamicPareto

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 7 Consolidation of memcached with a background batch task

ence can be intuitively explained since the migrations duringthe scale up are performed in a heavily loaded system whilethe system during the scale down is partially idle In absoluteterms migrations that occur when adding a core take 463microson average and less than 15 ms 95 of the time The outlierscan be explained by rare occurrences of longer preparationtimes or when processing up to 614 deferred packets

6 DiscussionUsing Pareto as a guide Even though the Pareto resultsare not used by the dynamic resource controller the Paretofrontier proved to be a valuable guide first to motivate andquantify the problem then to derive the configuration pol-icy sequence and finally to evaluate the effectiveness ofthe dynamic resource control by setting an upper bound onthe gains resulting from dynamic resource allocation Manyfactors such as software scalability hardware resource con-tention network and disk IO bottlenecks will influence thePareto frontier of any given application and therefore the de-rived control loop policies We will compare different work-loads in future workTradeoff between average latency and efficiency We usea one-dimensional SLO (99th pct le 500micros) and show thatwe can increase efficiency while maintaining that objectiveHowever this comes at a cost when considering the entirelatency distribution eg the average as well as the tail More

complex SLOs taking into account multiple aspects of la-tency distribution would define a different Pareto frontierand likely require adjustments to the control loopAdaptive flow-centric scheduling The new flow-groupmigration algorithm leads to a flow-centric approach to re-source scheduling where the network stack and applicationlogic always follow the steering decision POSIX applica-tions can balance flows by migrating file descriptors betweenthreads or processes but this tends to be inefficient becauseit is difficult for the kernel to match the flowrsquos destinationreceive queue to changes in CPU affinity Flow director canbe used by Linux to adjust the affinity of individual networkflows as a reactive measure to application-level and kernelthread migration rebalancing but the limited size of the redi-rection table prevents this mechanism from scaling to largeconnection counts By contrast our approach allows flowsto be migrated in entire groups improving efficiency and iscompatible with more scalable hardware flow steering mech-anisms based on RSSHardware trends Our experimental setup using one SandyBridge processor has a three-dimensional Pareto spaceClearly NUMA would add a new dimension Also thenewly released Haswell processors provide per-core DVFScontrols which further increases the Pareto space More-over hash filters for flow group steering could benefit fromrecent trends in NIC hardware For example Intelrsquos new

avg 95th pct max stddevad

dco

reprepare (micros) 67 231 521 93wait (micros) 135 541 983 190rpc (micros) 129 355 529 112deferred (micros) 101 401 1167 147total (micros) 463 1432 2870 459 packets 64 319 614 116

rem

ove

core prepare (micros) 25 54 398 32

wait (micros) 34 101 170 32rpc (micros) 12 25 49 7deferred (micros) 19 43 108 14total (micros) 93 160 456 47 packets 3 8 28 3

Table 2 Breakdown of flow group migration measured dur-ing the six benchmarks

XL710 chipset [19] has a 512 entry hash LUT (as well asindependent 64 entry LUTs for each VF) in contrast to the128 entries available in the 82599 chipset [17] used in ourevaluation This has the potential to reduce connection im-balances between cores especially with high core counts

7 Related WorkScheduling Scheduler activations [1] give applicationsgreater control over hardware threads and provide a mecha-nism for custom application-level scheduling Callisto [13]uses a similar strategy to improve the performance of co-located parallel runtime systems Our approach differs inthat an independent control plane manages the schedulingof hardware threads based on receive queuing latency indi-cators while the dataplane exposes a simple kernel thread-ing abstraction SEDA [51] also monitors queuing behaviorto make scheduling decisions such as thread pool sizingChronos [21] makes use of software-based flow steering butwith a focus on balancing load to reduce latency AffinityAccept [42] embraces a mixture of software and hardware-based flow steering in order to improve TCP connectionaffinity and increase throughput We focus instead on energyproportionality and workload consolidationEnergy Proportionality The energy proportionality prob-lem [3] has been well explored in previous work Somesystems have focused on solutions tailored to throughput-oriented workloads [33] or read-only workloads [23] Meis-ner et al [34] highlight unique challenges for low la-tency workloads and advocate full system active low-powermodes Similar to our system Pegasus [29] achieves CPUenergy proportionality for low latency workloads Our workexpands on Pegasus by exploring the elastic allocation ofhardware threads in combination with processor power man-agement states and by basing scheduling decisions on in-ternal latency metrics within a host endpoint instead of anexternal controller Niccolini et al show that a softwarerouter running on a dedicated machine can be made energy-

proportional [38] Similar to our approach queue lengthis used as a control signal to manage core allocation andDVFS settings However we focus on latency-sensitive ap-plications rather than middlebox traffic and consider theadditional case of workload consolidationCo-location Because host endpoints contain some com-ponents that are not energy proportional and thus are mostefficient when operating at 100 utilization co-location ofworkloads is also an important tool for improving energyefficiency At the cluster scheduler level BubbleUp [32] andParagon [10] make scheduling decisions that are interference-aware through efficient classification of the behavior ofworkload co-location Leverich et al [26] demonstratethat co-location of batch and low latency jobs is possi-ble on commodity operating systems Our approach ex-plores this issue at higher throughputs and with tighter la-tency SLOs Bubble-Flux [53] additionally controls back-ground threads we control background and latency-sensitivethreads CPI2 [54] detects performance interference by ob-serving changes in CPI and throttles offending jobs Thiswork is orthogonal to ours and could be a useful addi-tional signal for our control plane Heracles manages mul-tiple hardware and software isolation mechanisms includ-ing packet scheduling and cache partitioning to co-locatelatency-sensitive applications with batch tasks while main-taining millisecond SLOs [30] We limit our focus to DVFSand core assignment but target more aggressive SLOsLow-latency frameworks User-level networking stacks [2031 47] and RDMA applications [12 37 40] all rely onpolling typically by a fixed number of execution threadsto deliver micros-scale response times These stacks and appli-cations are evaluated based on their latency and throughputcharacteristics and do not take into account energy propor-tionality or workload consolidation considerations

8 ConclusionWe present the design and implementation of dynamic re-source controls for the IX dataplane which consists of acontroller that allocates cores and sets processor frequencyto adapt to changes in the load of latency-sensitive appli-cations The novel rebalancing mechanisms do not impactthe steady-state performance of the dataplane and can mi-grate a set of flow groups in milliseconds without droppingor reordering packets We develop two policies focused onoptimizing energy proportionality and workload consolida-tion Our results show that for three varying load patternsthe energy proportionality can save 44ndash54 of the pro-cessorrsquos energy or enable a background job to deliver 32ndash46 of its standalone throughput To evaluate the effective-ness of our approach we synthesize the Pareto frontier bycombining the behavior of all possible static configurationsOur policies deliver 85ndash93 of the Pareto optimal boundin terms of energy proportionality and 82ndash92 in termsof consolidation

AcknowledgementsThis work was supported by the MicrosoftEPFL Joint Re-search Center the Stanford Experimental Datacenter Laband NSF grant CNS-1422088 George Prekas is supportedby a Google Graduate Research Fellowship and Adam Be-lay by a Stanford Graduate Fellowship

References[1] T E Anderson B N Bershad E D Lazowska and H M

Levy Scheduler Activations Effective Kernel Support for theUser-Level Management of Parallelism ACM Trans ComputSyst 10(1)53ndash79 1992

[2] B Atikoglu Y Xu E Frachtenberg S Jiang andM Paleczny Workload Analysis of a Large-Scale Key-ValueStore In Proceedings of the 2012 ACM SIGMETRICS Inter-national Conference on Measurement and Modeling of Com-puter Systems pages 53ndash64 2012

[3] L A Barroso and U Holzle The Case for Energy-Proportional Computing IEEE Computer 40(12)33ndash372007

[4] L A Barroso J Clidaras and U Holzle The Datacenter as aComputer An Introduction to the Design of Warehouse-ScaleMachines Second Edition Synthesis Lectures on ComputerArchitecture Morgan amp Claypool Publishers 2013

[5] A Belay A Bittau A J Mashtizadeh D Terei D Mazieresand C Kozyrakis Dune Safe User-level Access to PrivilegedCPU Features In Proceedings of the 10th Symposium onOperating System Design and Implementation (OSDI) pages335ndash348 2012

[6] A Belay G Prekas A Klimovic S Grossman C Kozyrakisand E Bugnion IX A Protected Dataplane Operating Systemfor High Throughput and Low Latency In Proceedings of the11st Symposium on Operating System Design and Implemen-tation (OSDI) pages 49ndash65 2014

[7] J Bonwick and J Adams Magazines and Vmem Extendingthe Slab Allocator to Many CPUs and Arbitrary Resources InUSENIX Annual Technical Conference pages 15ndash33 2001

[8] H David E Gorbatov U R Hanebutte R Khanaa andC Le RAPL memory power estimation and capping InProceedings of the 2010 International Symposium on LowPower Electronics and Design pages 189ndash194 2010

[9] J Dean and L A Barroso The Tail at Scale Commun ACM56(2)74ndash80 2013

[10] C Delimitrou and C Kozyrakis Quality-of-Service-AwareScheduling in Heterogeneous Data centers with ParagonIEEE Micro 34(3)17ndash30 2014

[11] C Delimitrou and C Kozyrakis Quasar Resource-Efficientand QoS-Aware Cluster Management In Proceedings of the19th International Conference on Architectural Support forProgramming Languages and Operating Systems (ASPLOS-XIX) pages 127ndash144 2014

[12] A Dragojevic D Narayanan M Castro and O HodsonFaRM Fast Remote Memory In Proceedings of the 11stSymposium on Networked Systems Design and Implementa-tion (NSDI) pages 401ndash414 2014

[13] T Harris M Maas and V J Marathe Callisto co-schedulingparallel runtime systems In Proceedings of the 2014 EuroSysConference page 24 2014

[14] B Hindman A Konwinski M Zaharia A Ghodsi A DJoseph R H Katz S Shenker and I Stoica Mesos A Plat-form for Fine-Grained Resource Sharing in the Data CenterIn Proceedings of the 8th Symposium on Networked SystemsDesign and Implementation (NSDI) 2011

[15] C-H Hsu Y Zhang M A Laurenzano D Meisner T FWenisch J Mars L Tang and R G Dreslinski AdrenalinePinpointing and reining in tail queries with quick voltageboosting In Proceedings of the 21st IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages271ndash282 2015

[16] J Hwang K K Ramakrishnan and T Wood NetVM HighPerformance and Flexible Networking Using Virtualization onCommodity Platforms In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 445ndash458 2014

[17] Intel Corp Intel 82599 10 GbE Controller Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheets82599-10-gbe-controller-datasheetpdf2014

[18] Intel Corp Intel DPDK Data Plane Development Kit httpdpdkorg 2014

[19] Intel Corp Intel Ethernet Controller XL710 Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheetsxl710-10-40-controller-datasheetpdf2014

[20] E Jeong S Woo M A Jamshed H Jeong S Ihm D Hanand K Park mTCP a Highly Scalable User-level TCP Stackfor Multicore Systems In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 489ndash502 2014

[21] R Kapoor G Porter M Tewari G M Voelker and A Vah-dat Chronos Predictable Low Latency for Data Center Ap-plications In Proceedings of the 2012 ACM Symposium onCloud Computing (SOCC) page 9 2012

[22] W Kim M S Gupta G-Y Wei and D M Brooks Systemlevel analysis of fast per-core DVFS using on-chip switch-ing regulators In Proceedings of the 14th IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages123ndash134 2008

[23] A Krioukov P Mohan S Alspaugh L Keys D E Cullerand R H Katz NapSAC design and implementation of apower-proportional web cluster Computer CommunicationReview 41(1)102ndash108 2011

[24] K-C Leung V O K Li and D Yang An Overview ofPacket Reordering in Transmission Control Protocol (TCP)Problems Solutions and Challenges IEEE Trans ParallelDistrib Syst 18(4)522ndash535 2007

[25] J Leverich Mutilate High-Performance MemcachedLoad Generator httpsgithubcomleverichmutilate 2014

Prepare Wait RPC Defer processIXCPpy

migrate(fgs)

CPU A

update HW

hold

finalize

done

NIC

first

packet

release

CPU B

prepare

timeout

(a) Thread-centric view

NICHWQ-A

HWQ-B

RETA

FGPoll

Poll

defaultQ-A

remoteQ-B

defaultQ-B

localQ-B

Thread-A

Thread-B

(b) Packet-centric view

Figure 5 Flow-group Migration Algorithm

operates like a simple FIFO queuing server onto which clas-sic queuing and control theory principles can be easily ap-plied In contrast conventional operating systems distributebuffers throughout the system in the NIC (because of coa-lesced interrupts) in the driver before networking process-ing and in the socket layer before being consumed by theapplication Furthermore these conventional systems pro-vide no ordering guarantees across flows which makes itdifficult to pinpoint congestion

To estimate queuing delays the controller monitors theiteration time τ and the queue depth Q With B the maximalbatch size the tail latency issimmax(delay) = dQBelowastτ Thedataplane computes each instantaneous metric every 10msfor the previous 10ms interval As these metrics are subjectto jitter the dataplane computes the exponential weightedmoving averages using multiple smoothing factors (α) inparallel For example we track the queue depth as Q(tα) =α lowastQnow +(1minusα) lowastQ(tminus 1α) The control loop executesat a frequency of 10 Hz which is sufficient to adapt to loadchanges

Deciding when to remove resources is trickier than de-ciding when to add them as shallow and near-empty queuesdo not provide reliable metrics Instead we measure idletime and observe that each change in the configuration addsor removes a predictable level of throughput (i) increas-ing cores from n to n+ 1 increases throughput by a factorof 1n (ii) enabling hyperthreads adds 30 (for our fore-ground workload at least) and (iii) the corersquos performanceis largely proportional to its clock rate We can thereforedefine the function threshold[ f rom to] as the ratio of maxthroughput between the f rom and to configurations Whenoperating at configuration f rom and the idle time exceedsthreshold[ f rom to] we can safely switch to configuration towithout loss of performance

These settings are not universal as a number of assump-tions such as load changes requests bursts and variationsin request service times will noticeably affect the metricsThese are policies not mechanisms In particular there is an

inherent tradeoff between aggressive energy saving policieswhich may lead to SLO violations and conservative poli-cies which will consume an excessive amount of energyFortunately the policy can be easily modified the dataplaneexposes naturally all relevant metrics and the control planedaemon issim500 lines of Python includingsim80 lines for thecontrol loop

44 Data Structures for Coherence-free ExecutionWe designed our memory management and data structurelayout to minimize the performance impact of flow groupmigration and to preserve the coherence-free property of ourdataplane in the steady-state Our memory allocator drawsinspiration from magazine-based SLAB allocation [7] Eachthread maintains a thread-local cache of every data typeIf the size of the thread-local cache exceeds a thresholdmemory items are returned using traditional locking to ashared memory pool Such rebalancing events are commonduring flow group migration but infrequent during normaloperation

We further organize our data structures so the pointerreferences building linked lists hash tables etc are onlybetween objects either both permanently associated with thesame thread (per-cpu) or both permanently associated withthe same flow group (per-fg) or both global in scope Asa result we can efficiently transfer ownership of an entireflow-grouprsquos data structures during migration while avoidinglocking or coherence traffic during regular operation

For example the event condition arrays batch syscallarrays timer wheels incoming and outgoing descriptorrings posted and pending mbufs are all per-cpu Theprotocol control blocks 5-tuple hash table listening sock-ets TW WAIT queues out-of-order fragments and unac-knowledged sequences are all per-fg The ARP table isglobal Some objects change scope during their lifetimebut only at well-defined junctures For example an mbuf ispart of the per-cpu queue while waiting to be processed(step 1 in Fig 4) but then becomes part of per-fg struc-tures during network processing

A global data structure the software flow group tablemanages the entry points into the per-fg structures IntelNICs conveniently store the RSS hash result as meta-dataalong with each received packet the bottom bits are used toindex into that table

We encountered one challenge in the management of TCPtimers We design our implementation to maximize normalexecution performance each thread independently managesits timers as a hierarchical timing wheel [49] ie as a per-cpu structure As a consequence a flow group migrationrequires a scan of the hierarchical wheels to identify andremove pending timers during the hold phase of Fig 5athose will be restored later in the destination thread at therelease phase of the flow group migration

45 Flow Group MigrationWhen adding or removing a thread the dynamic controllergenerates a set of migration requests Each individual re-quest is for a set of flow groups (fgs) currently handledby one elastic thread A to be handled by elastic thread B Tosimplify the implementation the controller serializes the mi-gration requests and the dataplane assumes that at most onesuch request is in progress at any point in time

Each thread has three queues that can hold incoming net-work packets the defaultQ contains packets received dur-ing steady state operation when no flow group migration ispending the remoteQ contains packets belonging to outgo-ing flow groups the localQ contains packets belonging toincoming flow groups The use of these two extra queuesis critical to ensure the processing of packets in the properorder by the networking stack

Fig 5 illustrates the migration steps in a thread-centricview (Fig 5a) and in a packet-centric view (Fig 5b) Thecontroller and the dataplane threads communicate via lock-free structures in shared memory First the controller signalsA to migrate fgs to B A first marks each flow group of theset fgs with a special tag to hold off normal processing onall threads moves packets which belong to the flow groupset fgs from defaultQ-A to remoteQ-B and stops alltimers belonging to the flow group set A then reprogramsthe RETA for all entries in fgs Packets still received byA will be appended to remoteQ-B packets received by Bwill go to localQ-B

Upon reception of the first packet whose flow group be-longs to fgs by B B signals A to initiate the final stageof migration Then B finalizes the migration by re-enablingfgsrsquos timers removing all migration tags and pre-pendingto its defaultQ-B the packets from remoteQ-B andthe packets from localQ-B Finally B notifies the con-trol plane that the operation is complete There are obviouslycorner-cases and subtleties in the algorithm For example Ainstalls a migration timer to finalize the migration even if Bhas not yet received any packets We set the timeout to 1 ms

5 EvaluationWe use three synthetic time-accelerated load patterns toevaluate the effectiveness of the control loop under stressfulconditions All three vary between nearly idle and maximumthroughput within a four minute period the slope patterngradually raises the target load from 0 and 62M RPS andthen reduces its load the step pattern increases load by 500KRPS every 10 seconds and finally the sine+noise patternis a basic sinusoidal pattern modified by randomly addingsharp noise that is uniformly distributed over [-250+250]KRPS and re-computed every 5 seconds The slope patternprovides a baseline to study smooth changes the step patternmodels abrupt and massive changes while the sine+noisepattern is representative of daily web patterns [48]

Fig 6 and Fig 7 show the results of these three dynamicload patterns for the energy proportionality and workloadconsolidation scenarios In each case the top figure mea-sures the observed throughput They are annotated with thecontrol loop events that add resources (green) or removethem (red) Empty triangles correspond to core allocationsand full triangles to DVFS changes The middle figure eval-uates the soundness of the algorithm and reports the 99thpercentile latency as observed by a client machine and re-ported every second Finally the bottom figures compare theoverall efficiency of our solution based on dynamic resourcecontrols with (i) the maximal static configuration using allcores and Turbo Boost and (ii) the ideal synthetic efficiencycomputed using the Pareto frontier of Fig 2Energy Proportionality Fig 6 shows the behavior for theenergy efficiency scenario In each case the workload tracksthe desired throughput of the pattern and exercises the en-tire sequence of configurations gradually adding cores en-abling hyperthreading increasing the frequency and finallyenabling Turbo Boost before doing it in reverse The steppattern is particularly challenging as the instant change inload level requires multiple back-to-back configurationschanges With a few exceptions the latencies remain wellbelow the 500micros SLO We further discuss the violations be-low Finally Fig 6 (bottom) compares the power dissipatedby the workload with the corresponding power levels as de-termined by the Pareto frontier (lower bound) or the max-imum static configuration (upper bound) This graph mea-

Smooth Step Sine+noiseEnergy Proportionality (W)

Max power 91 92 94Measured 42 (-54) 48 (-48) 53 (-44)Pareto bound 39 (-57) 41 (-55) 45 (-52)

Server consolidation opportunity ( of peak)Pareto bound 50 47 39Measured 46 39 32

Table 1 Energy Proportionality and Consolidation gains

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

max conf dynamic Pareto

0

20

40

60

80

100

0 50 100 150 200

Pow

er

(W)

Time (seconds)

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 6 Energy Proportionality of memcached running on IX for three load patterns

sures the effectiveness of the control loop to maximize en-ergy proportionality We observe that the dynamic (actuallymeasured) power curve tracks the Pareto (synthetic) curvewell which defines a bound on energy savings When the dy-namic resource controls enters Turbo Boost mode the mea-sured power in all three cases starts at the lower end of the4 W range and then gradually rises as expected Table 1shows that the three patterns have Pareto savings bounds of52 55 and 57 IXrsquos dynamic resource controls resultsin energy savings of 44 48 and 54 which is 85 87and 93 of the theoretical boundConsolidation Fig 7 shows the corresponding results forthe workload consolidation scenarios Fig 7 (top) measuresthe observed load which here as well tracks the desired loadRecall that the consolidation policy always operates at theprocessorrsquos nominal rate (or Turbo) which limits the numberof configuration changes Fig 7 (middle) similarly confirmsthat the system meets the SLO with few exceptions Fig 7(bottom) plots the throughput of the background batch appli-cation expressed as a percentage of its throughput on a dedi-cated processor at 24 Ghz We compare it only to the Paretooptimal upper bound as a maximum configuration wouldmonopolize cores and deliver zero background throughputTable 1 shows that for these three patterns our consolida-tion policy delivers 32ndash46 of the standalone throughput

of the background job which corresponds to 82ndash92 ofthe Pareto boundSLO violations A careful study of the SLO violations of the6 runs shows that they fall into two categories First there are16 violations caused by delays in packet processing due toflow group migrations resulting from the addition of a coreSecond there are 9 violations caused by abrupt increase ofthroughput mostly in the step pattern which occur beforeany flow migrations The control plane then reacts quickly(in sim100 ms) and accommodates to the new throughput byadjusting resources To further confirm the abrupt nature ofthroughput increase specific to the step pattern we note thatthe system performed up three consecutive increases in re-sources in order to resolve a single violation 23 of the 25total violations last a single second with the remaining twoviolations lasting two seconds We believe that the compli-ance with the SLO achieved by our system is more than ad-equate for any practical production deploymentFlow group migration analysis Table 2 measures the la-tency of the 552 flow group migrations that occur during the6 benchmarks as described in sect45 It also reports the totalnumber of packets whose processing is deferred during themigration (rather than dropped or reordered) We first ob-serve that migrations present distinct behaviors when scalingup and when scaling down the number of cores The differ-

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

0

20

40

60

80

100

0 50 100 150 200

o

f pea

k

Time (seconds)

dynamicPareto

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 7 Consolidation of memcached with a background batch task

ence can be intuitively explained since the migrations duringthe scale up are performed in a heavily loaded system whilethe system during the scale down is partially idle In absoluteterms migrations that occur when adding a core take 463microson average and less than 15 ms 95 of the time The outlierscan be explained by rare occurrences of longer preparationtimes or when processing up to 614 deferred packets

6 DiscussionUsing Pareto as a guide Even though the Pareto resultsare not used by the dynamic resource controller the Paretofrontier proved to be a valuable guide first to motivate andquantify the problem then to derive the configuration pol-icy sequence and finally to evaluate the effectiveness ofthe dynamic resource control by setting an upper bound onthe gains resulting from dynamic resource allocation Manyfactors such as software scalability hardware resource con-tention network and disk IO bottlenecks will influence thePareto frontier of any given application and therefore the de-rived control loop policies We will compare different work-loads in future workTradeoff between average latency and efficiency We usea one-dimensional SLO (99th pct le 500micros) and show thatwe can increase efficiency while maintaining that objectiveHowever this comes at a cost when considering the entirelatency distribution eg the average as well as the tail More

complex SLOs taking into account multiple aspects of la-tency distribution would define a different Pareto frontierand likely require adjustments to the control loopAdaptive flow-centric scheduling The new flow-groupmigration algorithm leads to a flow-centric approach to re-source scheduling where the network stack and applicationlogic always follow the steering decision POSIX applica-tions can balance flows by migrating file descriptors betweenthreads or processes but this tends to be inefficient becauseit is difficult for the kernel to match the flowrsquos destinationreceive queue to changes in CPU affinity Flow director canbe used by Linux to adjust the affinity of individual networkflows as a reactive measure to application-level and kernelthread migration rebalancing but the limited size of the redi-rection table prevents this mechanism from scaling to largeconnection counts By contrast our approach allows flowsto be migrated in entire groups improving efficiency and iscompatible with more scalable hardware flow steering mech-anisms based on RSSHardware trends Our experimental setup using one SandyBridge processor has a three-dimensional Pareto spaceClearly NUMA would add a new dimension Also thenewly released Haswell processors provide per-core DVFScontrols which further increases the Pareto space More-over hash filters for flow group steering could benefit fromrecent trends in NIC hardware For example Intelrsquos new

avg 95th pct max stddevad

dco

reprepare (micros) 67 231 521 93wait (micros) 135 541 983 190rpc (micros) 129 355 529 112deferred (micros) 101 401 1167 147total (micros) 463 1432 2870 459 packets 64 319 614 116

rem

ove

core prepare (micros) 25 54 398 32

wait (micros) 34 101 170 32rpc (micros) 12 25 49 7deferred (micros) 19 43 108 14total (micros) 93 160 456 47 packets 3 8 28 3

Table 2 Breakdown of flow group migration measured dur-ing the six benchmarks

XL710 chipset [19] has a 512 entry hash LUT (as well asindependent 64 entry LUTs for each VF) in contrast to the128 entries available in the 82599 chipset [17] used in ourevaluation This has the potential to reduce connection im-balances between cores especially with high core counts

7 Related WorkScheduling Scheduler activations [1] give applicationsgreater control over hardware threads and provide a mecha-nism for custom application-level scheduling Callisto [13]uses a similar strategy to improve the performance of co-located parallel runtime systems Our approach differs inthat an independent control plane manages the schedulingof hardware threads based on receive queuing latency indi-cators while the dataplane exposes a simple kernel thread-ing abstraction SEDA [51] also monitors queuing behaviorto make scheduling decisions such as thread pool sizingChronos [21] makes use of software-based flow steering butwith a focus on balancing load to reduce latency AffinityAccept [42] embraces a mixture of software and hardware-based flow steering in order to improve TCP connectionaffinity and increase throughput We focus instead on energyproportionality and workload consolidationEnergy Proportionality The energy proportionality prob-lem [3] has been well explored in previous work Somesystems have focused on solutions tailored to throughput-oriented workloads [33] or read-only workloads [23] Meis-ner et al [34] highlight unique challenges for low la-tency workloads and advocate full system active low-powermodes Similar to our system Pegasus [29] achieves CPUenergy proportionality for low latency workloads Our workexpands on Pegasus by exploring the elastic allocation ofhardware threads in combination with processor power man-agement states and by basing scheduling decisions on in-ternal latency metrics within a host endpoint instead of anexternal controller Niccolini et al show that a softwarerouter running on a dedicated machine can be made energy-

proportional [38] Similar to our approach queue lengthis used as a control signal to manage core allocation andDVFS settings However we focus on latency-sensitive ap-plications rather than middlebox traffic and consider theadditional case of workload consolidationCo-location Because host endpoints contain some com-ponents that are not energy proportional and thus are mostefficient when operating at 100 utilization co-location ofworkloads is also an important tool for improving energyefficiency At the cluster scheduler level BubbleUp [32] andParagon [10] make scheduling decisions that are interference-aware through efficient classification of the behavior ofworkload co-location Leverich et al [26] demonstratethat co-location of batch and low latency jobs is possi-ble on commodity operating systems Our approach ex-plores this issue at higher throughputs and with tighter la-tency SLOs Bubble-Flux [53] additionally controls back-ground threads we control background and latency-sensitivethreads CPI2 [54] detects performance interference by ob-serving changes in CPI and throttles offending jobs Thiswork is orthogonal to ours and could be a useful addi-tional signal for our control plane Heracles manages mul-tiple hardware and software isolation mechanisms includ-ing packet scheduling and cache partitioning to co-locatelatency-sensitive applications with batch tasks while main-taining millisecond SLOs [30] We limit our focus to DVFSand core assignment but target more aggressive SLOsLow-latency frameworks User-level networking stacks [2031 47] and RDMA applications [12 37 40] all rely onpolling typically by a fixed number of execution threadsto deliver micros-scale response times These stacks and appli-cations are evaluated based on their latency and throughputcharacteristics and do not take into account energy propor-tionality or workload consolidation considerations

8 ConclusionWe present the design and implementation of dynamic re-source controls for the IX dataplane which consists of acontroller that allocates cores and sets processor frequencyto adapt to changes in the load of latency-sensitive appli-cations The novel rebalancing mechanisms do not impactthe steady-state performance of the dataplane and can mi-grate a set of flow groups in milliseconds without droppingor reordering packets We develop two policies focused onoptimizing energy proportionality and workload consolida-tion Our results show that for three varying load patternsthe energy proportionality can save 44ndash54 of the pro-cessorrsquos energy or enable a background job to deliver 32ndash46 of its standalone throughput To evaluate the effective-ness of our approach we synthesize the Pareto frontier bycombining the behavior of all possible static configurationsOur policies deliver 85ndash93 of the Pareto optimal boundin terms of energy proportionality and 82ndash92 in termsof consolidation

AcknowledgementsThis work was supported by the MicrosoftEPFL Joint Re-search Center the Stanford Experimental Datacenter Laband NSF grant CNS-1422088 George Prekas is supportedby a Google Graduate Research Fellowship and Adam Be-lay by a Stanford Graduate Fellowship

References[1] T E Anderson B N Bershad E D Lazowska and H M

Levy Scheduler Activations Effective Kernel Support for theUser-Level Management of Parallelism ACM Trans ComputSyst 10(1)53ndash79 1992

[2] B Atikoglu Y Xu E Frachtenberg S Jiang andM Paleczny Workload Analysis of a Large-Scale Key-ValueStore In Proceedings of the 2012 ACM SIGMETRICS Inter-national Conference on Measurement and Modeling of Com-puter Systems pages 53ndash64 2012

[3] L A Barroso and U Holzle The Case for Energy-Proportional Computing IEEE Computer 40(12)33ndash372007

[4] L A Barroso J Clidaras and U Holzle The Datacenter as aComputer An Introduction to the Design of Warehouse-ScaleMachines Second Edition Synthesis Lectures on ComputerArchitecture Morgan amp Claypool Publishers 2013

[5] A Belay A Bittau A J Mashtizadeh D Terei D Mazieresand C Kozyrakis Dune Safe User-level Access to PrivilegedCPU Features In Proceedings of the 10th Symposium onOperating System Design and Implementation (OSDI) pages335ndash348 2012

[6] A Belay G Prekas A Klimovic S Grossman C Kozyrakisand E Bugnion IX A Protected Dataplane Operating Systemfor High Throughput and Low Latency In Proceedings of the11st Symposium on Operating System Design and Implemen-tation (OSDI) pages 49ndash65 2014

[7] J Bonwick and J Adams Magazines and Vmem Extendingthe Slab Allocator to Many CPUs and Arbitrary Resources InUSENIX Annual Technical Conference pages 15ndash33 2001

[8] H David E Gorbatov U R Hanebutte R Khanaa andC Le RAPL memory power estimation and capping InProceedings of the 2010 International Symposium on LowPower Electronics and Design pages 189ndash194 2010

[9] J Dean and L A Barroso The Tail at Scale Commun ACM56(2)74ndash80 2013

[10] C Delimitrou and C Kozyrakis Quality-of-Service-AwareScheduling in Heterogeneous Data centers with ParagonIEEE Micro 34(3)17ndash30 2014

[11] C Delimitrou and C Kozyrakis Quasar Resource-Efficientand QoS-Aware Cluster Management In Proceedings of the19th International Conference on Architectural Support forProgramming Languages and Operating Systems (ASPLOS-XIX) pages 127ndash144 2014

[12] A Dragojevic D Narayanan M Castro and O HodsonFaRM Fast Remote Memory In Proceedings of the 11stSymposium on Networked Systems Design and Implementa-tion (NSDI) pages 401ndash414 2014

[13] T Harris M Maas and V J Marathe Callisto co-schedulingparallel runtime systems In Proceedings of the 2014 EuroSysConference page 24 2014

[14] B Hindman A Konwinski M Zaharia A Ghodsi A DJoseph R H Katz S Shenker and I Stoica Mesos A Plat-form for Fine-Grained Resource Sharing in the Data CenterIn Proceedings of the 8th Symposium on Networked SystemsDesign and Implementation (NSDI) 2011

[15] C-H Hsu Y Zhang M A Laurenzano D Meisner T FWenisch J Mars L Tang and R G Dreslinski AdrenalinePinpointing and reining in tail queries with quick voltageboosting In Proceedings of the 21st IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages271ndash282 2015

[16] J Hwang K K Ramakrishnan and T Wood NetVM HighPerformance and Flexible Networking Using Virtualization onCommodity Platforms In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 445ndash458 2014

[17] Intel Corp Intel 82599 10 GbE Controller Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheets82599-10-gbe-controller-datasheetpdf2014

[18] Intel Corp Intel DPDK Data Plane Development Kit httpdpdkorg 2014

[19] Intel Corp Intel Ethernet Controller XL710 Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheetsxl710-10-40-controller-datasheetpdf2014

[20] E Jeong S Woo M A Jamshed H Jeong S Ihm D Hanand K Park mTCP a Highly Scalable User-level TCP Stackfor Multicore Systems In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 489ndash502 2014

[21] R Kapoor G Porter M Tewari G M Voelker and A Vah-dat Chronos Predictable Low Latency for Data Center Ap-plications In Proceedings of the 2012 ACM Symposium onCloud Computing (SOCC) page 9 2012

[22] W Kim M S Gupta G-Y Wei and D M Brooks Systemlevel analysis of fast per-core DVFS using on-chip switch-ing regulators In Proceedings of the 14th IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages123ndash134 2008

[23] A Krioukov P Mohan S Alspaugh L Keys D E Cullerand R H Katz NapSAC design and implementation of apower-proportional web cluster Computer CommunicationReview 41(1)102ndash108 2011

[24] K-C Leung V O K Li and D Yang An Overview ofPacket Reordering in Transmission Control Protocol (TCP)Problems Solutions and Challenges IEEE Trans ParallelDistrib Syst 18(4)522ndash535 2007

[25] J Leverich Mutilate High-Performance MemcachedLoad Generator httpsgithubcomleverichmutilate 2014

A global data structure the software flow group tablemanages the entry points into the per-fg structures IntelNICs conveniently store the RSS hash result as meta-dataalong with each received packet the bottom bits are used toindex into that table

We encountered one challenge in the management of TCPtimers We design our implementation to maximize normalexecution performance each thread independently managesits timers as a hierarchical timing wheel [49] ie as a per-cpu structure As a consequence a flow group migrationrequires a scan of the hierarchical wheels to identify andremove pending timers during the hold phase of Fig 5athose will be restored later in the destination thread at therelease phase of the flow group migration

45 Flow Group MigrationWhen adding or removing a thread the dynamic controllergenerates a set of migration requests Each individual re-quest is for a set of flow groups (fgs) currently handledby one elastic thread A to be handled by elastic thread B Tosimplify the implementation the controller serializes the mi-gration requests and the dataplane assumes that at most onesuch request is in progress at any point in time

Each thread has three queues that can hold incoming net-work packets the defaultQ contains packets received dur-ing steady state operation when no flow group migration ispending the remoteQ contains packets belonging to outgo-ing flow groups the localQ contains packets belonging toincoming flow groups The use of these two extra queuesis critical to ensure the processing of packets in the properorder by the networking stack

Fig 5 illustrates the migration steps in a thread-centricview (Fig 5a) and in a packet-centric view (Fig 5b) Thecontroller and the dataplane threads communicate via lock-free structures in shared memory First the controller signalsA to migrate fgs to B A first marks each flow group of theset fgs with a special tag to hold off normal processing onall threads moves packets which belong to the flow groupset fgs from defaultQ-A to remoteQ-B and stops alltimers belonging to the flow group set A then reprogramsthe RETA for all entries in fgs Packets still received byA will be appended to remoteQ-B packets received by Bwill go to localQ-B

Upon reception of the first packet whose flow group be-longs to fgs by B B signals A to initiate the final stageof migration Then B finalizes the migration by re-enablingfgsrsquos timers removing all migration tags and pre-pendingto its defaultQ-B the packets from remoteQ-B andthe packets from localQ-B Finally B notifies the con-trol plane that the operation is complete There are obviouslycorner-cases and subtleties in the algorithm For example Ainstalls a migration timer to finalize the migration even if Bhas not yet received any packets We set the timeout to 1 ms

5 EvaluationWe use three synthetic time-accelerated load patterns toevaluate the effectiveness of the control loop under stressfulconditions All three vary between nearly idle and maximumthroughput within a four minute period the slope patterngradually raises the target load from 0 and 62M RPS andthen reduces its load the step pattern increases load by 500KRPS every 10 seconds and finally the sine+noise patternis a basic sinusoidal pattern modified by randomly addingsharp noise that is uniformly distributed over [-250+250]KRPS and re-computed every 5 seconds The slope patternprovides a baseline to study smooth changes the step patternmodels abrupt and massive changes while the sine+noisepattern is representative of daily web patterns [48]

Fig 6 and Fig 7 show the results of these three dynamicload patterns for the energy proportionality and workloadconsolidation scenarios In each case the top figure mea-sures the observed throughput They are annotated with thecontrol loop events that add resources (green) or removethem (red) Empty triangles correspond to core allocationsand full triangles to DVFS changes The middle figure eval-uates the soundness of the algorithm and reports the 99thpercentile latency as observed by a client machine and re-ported every second Finally the bottom figures compare theoverall efficiency of our solution based on dynamic resourcecontrols with (i) the maximal static configuration using allcores and Turbo Boost and (ii) the ideal synthetic efficiencycomputed using the Pareto frontier of Fig 2Energy Proportionality Fig 6 shows the behavior for theenergy efficiency scenario In each case the workload tracksthe desired throughput of the pattern and exercises the en-tire sequence of configurations gradually adding cores en-abling hyperthreading increasing the frequency and finallyenabling Turbo Boost before doing it in reverse The steppattern is particularly challenging as the instant change inload level requires multiple back-to-back configurationschanges With a few exceptions the latencies remain wellbelow the 500micros SLO We further discuss the violations be-low Finally Fig 6 (bottom) compares the power dissipatedby the workload with the corresponding power levels as de-termined by the Pareto frontier (lower bound) or the max-imum static configuration (upper bound) This graph mea-

Smooth Step Sine+noiseEnergy Proportionality (W)

Max power 91 92 94Measured 42 (-54) 48 (-48) 53 (-44)Pareto bound 39 (-57) 41 (-55) 45 (-52)

Server consolidation opportunity ( of peak)Pareto bound 50 47 39Measured 46 39 32

Table 1 Energy Proportionality and Consolidation gains

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

max conf dynamic Pareto

0

20

40

60

80

100

0 50 100 150 200

Pow

er

(W)

Time (seconds)

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 6 Energy Proportionality of memcached running on IX for three load patterns

sures the effectiveness of the control loop to maximize en-ergy proportionality We observe that the dynamic (actuallymeasured) power curve tracks the Pareto (synthetic) curvewell which defines a bound on energy savings When the dy-namic resource controls enters Turbo Boost mode the mea-sured power in all three cases starts at the lower end of the4 W range and then gradually rises as expected Table 1shows that the three patterns have Pareto savings bounds of52 55 and 57 IXrsquos dynamic resource controls resultsin energy savings of 44 48 and 54 which is 85 87and 93 of the theoretical boundConsolidation Fig 7 shows the corresponding results forthe workload consolidation scenarios Fig 7 (top) measuresthe observed load which here as well tracks the desired loadRecall that the consolidation policy always operates at theprocessorrsquos nominal rate (or Turbo) which limits the numberof configuration changes Fig 7 (middle) similarly confirmsthat the system meets the SLO with few exceptions Fig 7(bottom) plots the throughput of the background batch appli-cation expressed as a percentage of its throughput on a dedi-cated processor at 24 Ghz We compare it only to the Paretooptimal upper bound as a maximum configuration wouldmonopolize cores and deliver zero background throughputTable 1 shows that for these three patterns our consolida-tion policy delivers 32ndash46 of the standalone throughput

of the background job which corresponds to 82ndash92 ofthe Pareto boundSLO violations A careful study of the SLO violations of the6 runs shows that they fall into two categories First there are16 violations caused by delays in packet processing due toflow group migrations resulting from the addition of a coreSecond there are 9 violations caused by abrupt increase ofthroughput mostly in the step pattern which occur beforeany flow migrations The control plane then reacts quickly(in sim100 ms) and accommodates to the new throughput byadjusting resources To further confirm the abrupt nature ofthroughput increase specific to the step pattern we note thatthe system performed up three consecutive increases in re-sources in order to resolve a single violation 23 of the 25total violations last a single second with the remaining twoviolations lasting two seconds We believe that the compli-ance with the SLO achieved by our system is more than ad-equate for any practical production deploymentFlow group migration analysis Table 2 measures the la-tency of the 552 flow group migrations that occur during the6 benchmarks as described in sect45 It also reports the totalnumber of packets whose processing is deferred during themigration (rather than dropped or reordered) We first ob-serve that migrations present distinct behaviors when scalingup and when scaling down the number of cores The differ-

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

0

20

40

60

80

100

0 50 100 150 200

o

f pea

k

Time (seconds)

dynamicPareto

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 7 Consolidation of memcached with a background batch task

ence can be intuitively explained since the migrations duringthe scale up are performed in a heavily loaded system whilethe system during the scale down is partially idle In absoluteterms migrations that occur when adding a core take 463microson average and less than 15 ms 95 of the time The outlierscan be explained by rare occurrences of longer preparationtimes or when processing up to 614 deferred packets

6 DiscussionUsing Pareto as a guide Even though the Pareto resultsare not used by the dynamic resource controller the Paretofrontier proved to be a valuable guide first to motivate andquantify the problem then to derive the configuration pol-icy sequence and finally to evaluate the effectiveness ofthe dynamic resource control by setting an upper bound onthe gains resulting from dynamic resource allocation Manyfactors such as software scalability hardware resource con-tention network and disk IO bottlenecks will influence thePareto frontier of any given application and therefore the de-rived control loop policies We will compare different work-loads in future workTradeoff between average latency and efficiency We usea one-dimensional SLO (99th pct le 500micros) and show thatwe can increase efficiency while maintaining that objectiveHowever this comes at a cost when considering the entirelatency distribution eg the average as well as the tail More

complex SLOs taking into account multiple aspects of la-tency distribution would define a different Pareto frontierand likely require adjustments to the control loopAdaptive flow-centric scheduling The new flow-groupmigration algorithm leads to a flow-centric approach to re-source scheduling where the network stack and applicationlogic always follow the steering decision POSIX applica-tions can balance flows by migrating file descriptors betweenthreads or processes but this tends to be inefficient becauseit is difficult for the kernel to match the flowrsquos destinationreceive queue to changes in CPU affinity Flow director canbe used by Linux to adjust the affinity of individual networkflows as a reactive measure to application-level and kernelthread migration rebalancing but the limited size of the redi-rection table prevents this mechanism from scaling to largeconnection counts By contrast our approach allows flowsto be migrated in entire groups improving efficiency and iscompatible with more scalable hardware flow steering mech-anisms based on RSSHardware trends Our experimental setup using one SandyBridge processor has a three-dimensional Pareto spaceClearly NUMA would add a new dimension Also thenewly released Haswell processors provide per-core DVFScontrols which further increases the Pareto space More-over hash filters for flow group steering could benefit fromrecent trends in NIC hardware For example Intelrsquos new

avg 95th pct max stddevad

dco

reprepare (micros) 67 231 521 93wait (micros) 135 541 983 190rpc (micros) 129 355 529 112deferred (micros) 101 401 1167 147total (micros) 463 1432 2870 459 packets 64 319 614 116

rem

ove

core prepare (micros) 25 54 398 32

wait (micros) 34 101 170 32rpc (micros) 12 25 49 7deferred (micros) 19 43 108 14total (micros) 93 160 456 47 packets 3 8 28 3

Table 2 Breakdown of flow group migration measured dur-ing the six benchmarks

XL710 chipset [19] has a 512 entry hash LUT (as well asindependent 64 entry LUTs for each VF) in contrast to the128 entries available in the 82599 chipset [17] used in ourevaluation This has the potential to reduce connection im-balances between cores especially with high core counts

7 Related WorkScheduling Scheduler activations [1] give applicationsgreater control over hardware threads and provide a mecha-nism for custom application-level scheduling Callisto [13]uses a similar strategy to improve the performance of co-located parallel runtime systems Our approach differs inthat an independent control plane manages the schedulingof hardware threads based on receive queuing latency indi-cators while the dataplane exposes a simple kernel thread-ing abstraction SEDA [51] also monitors queuing behaviorto make scheduling decisions such as thread pool sizingChronos [21] makes use of software-based flow steering butwith a focus on balancing load to reduce latency AffinityAccept [42] embraces a mixture of software and hardware-based flow steering in order to improve TCP connectionaffinity and increase throughput We focus instead on energyproportionality and workload consolidationEnergy Proportionality The energy proportionality prob-lem [3] has been well explored in previous work Somesystems have focused on solutions tailored to throughput-oriented workloads [33] or read-only workloads [23] Meis-ner et al [34] highlight unique challenges for low la-tency workloads and advocate full system active low-powermodes Similar to our system Pegasus [29] achieves CPUenergy proportionality for low latency workloads Our workexpands on Pegasus by exploring the elastic allocation ofhardware threads in combination with processor power man-agement states and by basing scheduling decisions on in-ternal latency metrics within a host endpoint instead of anexternal controller Niccolini et al show that a softwarerouter running on a dedicated machine can be made energy-

proportional [38] Similar to our approach queue lengthis used as a control signal to manage core allocation andDVFS settings However we focus on latency-sensitive ap-plications rather than middlebox traffic and consider theadditional case of workload consolidationCo-location Because host endpoints contain some com-ponents that are not energy proportional and thus are mostefficient when operating at 100 utilization co-location ofworkloads is also an important tool for improving energyefficiency At the cluster scheduler level BubbleUp [32] andParagon [10] make scheduling decisions that are interference-aware through efficient classification of the behavior ofworkload co-location Leverich et al [26] demonstratethat co-location of batch and low latency jobs is possi-ble on commodity operating systems Our approach ex-plores this issue at higher throughputs and with tighter la-tency SLOs Bubble-Flux [53] additionally controls back-ground threads we control background and latency-sensitivethreads CPI2 [54] detects performance interference by ob-serving changes in CPI and throttles offending jobs Thiswork is orthogonal to ours and could be a useful addi-tional signal for our control plane Heracles manages mul-tiple hardware and software isolation mechanisms includ-ing packet scheduling and cache partitioning to co-locatelatency-sensitive applications with batch tasks while main-taining millisecond SLOs [30] We limit our focus to DVFSand core assignment but target more aggressive SLOsLow-latency frameworks User-level networking stacks [2031 47] and RDMA applications [12 37 40] all rely onpolling typically by a fixed number of execution threadsto deliver micros-scale response times These stacks and appli-cations are evaluated based on their latency and throughputcharacteristics and do not take into account energy propor-tionality or workload consolidation considerations

8 ConclusionWe present the design and implementation of dynamic re-source controls for the IX dataplane which consists of acontroller that allocates cores and sets processor frequencyto adapt to changes in the load of latency-sensitive appli-cations The novel rebalancing mechanisms do not impactthe steady-state performance of the dataplane and can mi-grate a set of flow groups in milliseconds without droppingor reordering packets We develop two policies focused onoptimizing energy proportionality and workload consolida-tion Our results show that for three varying load patternsthe energy proportionality can save 44ndash54 of the pro-cessorrsquos energy or enable a background job to deliver 32ndash46 of its standalone throughput To evaluate the effective-ness of our approach we synthesize the Pareto frontier bycombining the behavior of all possible static configurationsOur policies deliver 85ndash93 of the Pareto optimal boundin terms of energy proportionality and 82ndash92 in termsof consolidation

AcknowledgementsThis work was supported by the MicrosoftEPFL Joint Re-search Center the Stanford Experimental Datacenter Laband NSF grant CNS-1422088 George Prekas is supportedby a Google Graduate Research Fellowship and Adam Be-lay by a Stanford Graduate Fellowship

References[1] T E Anderson B N Bershad E D Lazowska and H M

Levy Scheduler Activations Effective Kernel Support for theUser-Level Management of Parallelism ACM Trans ComputSyst 10(1)53ndash79 1992

[2] B Atikoglu Y Xu E Frachtenberg S Jiang andM Paleczny Workload Analysis of a Large-Scale Key-ValueStore In Proceedings of the 2012 ACM SIGMETRICS Inter-national Conference on Measurement and Modeling of Com-puter Systems pages 53ndash64 2012

[3] L A Barroso and U Holzle The Case for Energy-Proportional Computing IEEE Computer 40(12)33ndash372007

[4] L A Barroso J Clidaras and U Holzle The Datacenter as aComputer An Introduction to the Design of Warehouse-ScaleMachines Second Edition Synthesis Lectures on ComputerArchitecture Morgan amp Claypool Publishers 2013

[5] A Belay A Bittau A J Mashtizadeh D Terei D Mazieresand C Kozyrakis Dune Safe User-level Access to PrivilegedCPU Features In Proceedings of the 10th Symposium onOperating System Design and Implementation (OSDI) pages335ndash348 2012

[6] A Belay G Prekas A Klimovic S Grossman C Kozyrakisand E Bugnion IX A Protected Dataplane Operating Systemfor High Throughput and Low Latency In Proceedings of the11st Symposium on Operating System Design and Implemen-tation (OSDI) pages 49ndash65 2014

[7] J Bonwick and J Adams Magazines and Vmem Extendingthe Slab Allocator to Many CPUs and Arbitrary Resources InUSENIX Annual Technical Conference pages 15ndash33 2001

[8] H David E Gorbatov U R Hanebutte R Khanaa andC Le RAPL memory power estimation and capping InProceedings of the 2010 International Symposium on LowPower Electronics and Design pages 189ndash194 2010

[9] J Dean and L A Barroso The Tail at Scale Commun ACM56(2)74ndash80 2013

[10] C Delimitrou and C Kozyrakis Quality-of-Service-AwareScheduling in Heterogeneous Data centers with ParagonIEEE Micro 34(3)17ndash30 2014

[11] C Delimitrou and C Kozyrakis Quasar Resource-Efficientand QoS-Aware Cluster Management In Proceedings of the19th International Conference on Architectural Support forProgramming Languages and Operating Systems (ASPLOS-XIX) pages 127ndash144 2014

[12] A Dragojevic D Narayanan M Castro and O HodsonFaRM Fast Remote Memory In Proceedings of the 11stSymposium on Networked Systems Design and Implementa-tion (NSDI) pages 401ndash414 2014

[13] T Harris M Maas and V J Marathe Callisto co-schedulingparallel runtime systems In Proceedings of the 2014 EuroSysConference page 24 2014

[14] B Hindman A Konwinski M Zaharia A Ghodsi A DJoseph R H Katz S Shenker and I Stoica Mesos A Plat-form for Fine-Grained Resource Sharing in the Data CenterIn Proceedings of the 8th Symposium on Networked SystemsDesign and Implementation (NSDI) 2011

[15] C-H Hsu Y Zhang M A Laurenzano D Meisner T FWenisch J Mars L Tang and R G Dreslinski AdrenalinePinpointing and reining in tail queries with quick voltageboosting In Proceedings of the 21st IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages271ndash282 2015

[16] J Hwang K K Ramakrishnan and T Wood NetVM HighPerformance and Flexible Networking Using Virtualization onCommodity Platforms In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 445ndash458 2014

[17] Intel Corp Intel 82599 10 GbE Controller Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheets82599-10-gbe-controller-datasheetpdf2014

[18] Intel Corp Intel DPDK Data Plane Development Kit httpdpdkorg 2014

[19] Intel Corp Intel Ethernet Controller XL710 Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheetsxl710-10-40-controller-datasheetpdf2014

[20] E Jeong S Woo M A Jamshed H Jeong S Ihm D Hanand K Park mTCP a Highly Scalable User-level TCP Stackfor Multicore Systems In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 489ndash502 2014

[21] R Kapoor G Porter M Tewari G M Voelker and A Vah-dat Chronos Predictable Low Latency for Data Center Ap-plications In Proceedings of the 2012 ACM Symposium onCloud Computing (SOCC) page 9 2012

[22] W Kim M S Gupta G-Y Wei and D M Brooks Systemlevel analysis of fast per-core DVFS using on-chip switch-ing regulators In Proceedings of the 14th IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages123ndash134 2008

[23] A Krioukov P Mohan S Alspaugh L Keys D E Cullerand R H Katz NapSAC design and implementation of apower-proportional web cluster Computer CommunicationReview 41(1)102ndash108 2011

[24] K-C Leung V O K Li and D Yang An Overview ofPacket Reordering in Transmission Control Protocol (TCP)Problems Solutions and Challenges IEEE Trans ParallelDistrib Syst 18(4)522ndash535 2007

[25] J Leverich Mutilate High-Performance MemcachedLoad Generator httpsgithubcomleverichmutilate 2014

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

max conf dynamic Pareto

0

20

40

60

80

100

0 50 100 150 200

Pow

er

(W)

Time (seconds)

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 6 Energy Proportionality of memcached running on IX for three load patterns

sures the effectiveness of the control loop to maximize en-ergy proportionality We observe that the dynamic (actuallymeasured) power curve tracks the Pareto (synthetic) curvewell which defines a bound on energy savings When the dy-namic resource controls enters Turbo Boost mode the mea-sured power in all three cases starts at the lower end of the4 W range and then gradually rises as expected Table 1shows that the three patterns have Pareto savings bounds of52 55 and 57 IXrsquos dynamic resource controls resultsin energy savings of 44 48 and 54 which is 85 87and 93 of the theoretical boundConsolidation Fig 7 shows the corresponding results forthe workload consolidation scenarios Fig 7 (top) measuresthe observed load which here as well tracks the desired loadRecall that the consolidation policy always operates at theprocessorrsquos nominal rate (or Turbo) which limits the numberof configuration changes Fig 7 (middle) similarly confirmsthat the system meets the SLO with few exceptions Fig 7(bottom) plots the throughput of the background batch appli-cation expressed as a percentage of its throughput on a dedi-cated processor at 24 Ghz We compare it only to the Paretooptimal upper bound as a maximum configuration wouldmonopolize cores and deliver zero background throughputTable 1 shows that for these three patterns our consolida-tion policy delivers 32ndash46 of the standalone throughput

of the background job which corresponds to 82ndash92 ofthe Pareto boundSLO violations A careful study of the SLO violations of the6 runs shows that they fall into two categories First there are16 violations caused by delays in packet processing due toflow group migrations resulting from the addition of a coreSecond there are 9 violations caused by abrupt increase ofthroughput mostly in the step pattern which occur beforeany flow migrations The control plane then reacts quickly(in sim100 ms) and accommodates to the new throughput byadjusting resources To further confirm the abrupt nature ofthroughput increase specific to the step pattern we note thatthe system performed up three consecutive increases in re-sources in order to resolve a single violation 23 of the 25total violations last a single second with the remaining twoviolations lasting two seconds We believe that the compli-ance with the SLO achieved by our system is more than ad-equate for any practical production deploymentFlow group migration analysis Table 2 measures the la-tency of the 552 flow group migrations that occur during the6 benchmarks as described in sect45 It also reports the totalnumber of packets whose processing is deferred during themigration (rather than dropped or reordered) We first ob-serve that migrations present distinct behaviors when scalingup and when scaling down the number of cores The differ-

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

0

20

40

60

80

100

0 50 100 150 200

o

f pea

k

Time (seconds)

dynamicPareto

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 7 Consolidation of memcached with a background batch task

ence can be intuitively explained since the migrations duringthe scale up are performed in a heavily loaded system whilethe system during the scale down is partially idle In absoluteterms migrations that occur when adding a core take 463microson average and less than 15 ms 95 of the time The outlierscan be explained by rare occurrences of longer preparationtimes or when processing up to 614 deferred packets

6 DiscussionUsing Pareto as a guide Even though the Pareto resultsare not used by the dynamic resource controller the Paretofrontier proved to be a valuable guide first to motivate andquantify the problem then to derive the configuration pol-icy sequence and finally to evaluate the effectiveness ofthe dynamic resource control by setting an upper bound onthe gains resulting from dynamic resource allocation Manyfactors such as software scalability hardware resource con-tention network and disk IO bottlenecks will influence thePareto frontier of any given application and therefore the de-rived control loop policies We will compare different work-loads in future workTradeoff between average latency and efficiency We usea one-dimensional SLO (99th pct le 500micros) and show thatwe can increase efficiency while maintaining that objectiveHowever this comes at a cost when considering the entirelatency distribution eg the average as well as the tail More

complex SLOs taking into account multiple aspects of la-tency distribution would define a different Pareto frontierand likely require adjustments to the control loopAdaptive flow-centric scheduling The new flow-groupmigration algorithm leads to a flow-centric approach to re-source scheduling where the network stack and applicationlogic always follow the steering decision POSIX applica-tions can balance flows by migrating file descriptors betweenthreads or processes but this tends to be inefficient becauseit is difficult for the kernel to match the flowrsquos destinationreceive queue to changes in CPU affinity Flow director canbe used by Linux to adjust the affinity of individual networkflows as a reactive measure to application-level and kernelthread migration rebalancing but the limited size of the redi-rection table prevents this mechanism from scaling to largeconnection counts By contrast our approach allows flowsto be migrated in entire groups improving efficiency and iscompatible with more scalable hardware flow steering mech-anisms based on RSSHardware trends Our experimental setup using one SandyBridge processor has a three-dimensional Pareto spaceClearly NUMA would add a new dimension Also thenewly released Haswell processors provide per-core DVFScontrols which further increases the Pareto space More-over hash filters for flow group steering could benefit fromrecent trends in NIC hardware For example Intelrsquos new

avg 95th pct max stddevad

dco

reprepare (micros) 67 231 521 93wait (micros) 135 541 983 190rpc (micros) 129 355 529 112deferred (micros) 101 401 1167 147total (micros) 463 1432 2870 459 packets 64 319 614 116

rem

ove

core prepare (micros) 25 54 398 32

wait (micros) 34 101 170 32rpc (micros) 12 25 49 7deferred (micros) 19 43 108 14total (micros) 93 160 456 47 packets 3 8 28 3

Table 2 Breakdown of flow group migration measured dur-ing the six benchmarks

XL710 chipset [19] has a 512 entry hash LUT (as well asindependent 64 entry LUTs for each VF) in contrast to the128 entries available in the 82599 chipset [17] used in ourevaluation This has the potential to reduce connection im-balances between cores especially with high core counts

7 Related WorkScheduling Scheduler activations [1] give applicationsgreater control over hardware threads and provide a mecha-nism for custom application-level scheduling Callisto [13]uses a similar strategy to improve the performance of co-located parallel runtime systems Our approach differs inthat an independent control plane manages the schedulingof hardware threads based on receive queuing latency indi-cators while the dataplane exposes a simple kernel thread-ing abstraction SEDA [51] also monitors queuing behaviorto make scheduling decisions such as thread pool sizingChronos [21] makes use of software-based flow steering butwith a focus on balancing load to reduce latency AffinityAccept [42] embraces a mixture of software and hardware-based flow steering in order to improve TCP connectionaffinity and increase throughput We focus instead on energyproportionality and workload consolidationEnergy Proportionality The energy proportionality prob-lem [3] has been well explored in previous work Somesystems have focused on solutions tailored to throughput-oriented workloads [33] or read-only workloads [23] Meis-ner et al [34] highlight unique challenges for low la-tency workloads and advocate full system active low-powermodes Similar to our system Pegasus [29] achieves CPUenergy proportionality for low latency workloads Our workexpands on Pegasus by exploring the elastic allocation ofhardware threads in combination with processor power man-agement states and by basing scheduling decisions on in-ternal latency metrics within a host endpoint instead of anexternal controller Niccolini et al show that a softwarerouter running on a dedicated machine can be made energy-

proportional [38] Similar to our approach queue lengthis used as a control signal to manage core allocation andDVFS settings However we focus on latency-sensitive ap-plications rather than middlebox traffic and consider theadditional case of workload consolidationCo-location Because host endpoints contain some com-ponents that are not energy proportional and thus are mostefficient when operating at 100 utilization co-location ofworkloads is also an important tool for improving energyefficiency At the cluster scheduler level BubbleUp [32] andParagon [10] make scheduling decisions that are interference-aware through efficient classification of the behavior ofworkload co-location Leverich et al [26] demonstratethat co-location of batch and low latency jobs is possi-ble on commodity operating systems Our approach ex-plores this issue at higher throughputs and with tighter la-tency SLOs Bubble-Flux [53] additionally controls back-ground threads we control background and latency-sensitivethreads CPI2 [54] detects performance interference by ob-serving changes in CPI and throttles offending jobs Thiswork is orthogonal to ours and could be a useful addi-tional signal for our control plane Heracles manages mul-tiple hardware and software isolation mechanisms includ-ing packet scheduling and cache partitioning to co-locatelatency-sensitive applications with batch tasks while main-taining millisecond SLOs [30] We limit our focus to DVFSand core assignment but target more aggressive SLOsLow-latency frameworks User-level networking stacks [2031 47] and RDMA applications [12 37 40] all rely onpolling typically by a fixed number of execution threadsto deliver micros-scale response times These stacks and appli-cations are evaluated based on their latency and throughputcharacteristics and do not take into account energy propor-tionality or workload consolidation considerations

8 ConclusionWe present the design and implementation of dynamic re-source controls for the IX dataplane which consists of acontroller that allocates cores and sets processor frequencyto adapt to changes in the load of latency-sensitive appli-cations The novel rebalancing mechanisms do not impactthe steady-state performance of the dataplane and can mi-grate a set of flow groups in milliseconds without droppingor reordering packets We develop two policies focused onoptimizing energy proportionality and workload consolida-tion Our results show that for three varying load patternsthe energy proportionality can save 44ndash54 of the pro-cessorrsquos energy or enable a background job to deliver 32ndash46 of its standalone throughput To evaluate the effective-ness of our approach we synthesize the Pareto frontier bycombining the behavior of all possible static configurationsOur policies deliver 85ndash93 of the Pareto optimal boundin terms of energy proportionality and 82ndash92 in termsof consolidation

AcknowledgementsThis work was supported by the MicrosoftEPFL Joint Re-search Center the Stanford Experimental Datacenter Laband NSF grant CNS-1422088 George Prekas is supportedby a Google Graduate Research Fellowship and Adam Be-lay by a Stanford Graduate Fellowship

References[1] T E Anderson B N Bershad E D Lazowska and H M

Levy Scheduler Activations Effective Kernel Support for theUser-Level Management of Parallelism ACM Trans ComputSyst 10(1)53ndash79 1992

[2] B Atikoglu Y Xu E Frachtenberg S Jiang andM Paleczny Workload Analysis of a Large-Scale Key-ValueStore In Proceedings of the 2012 ACM SIGMETRICS Inter-national Conference on Measurement and Modeling of Com-puter Systems pages 53ndash64 2012

[3] L A Barroso and U Holzle The Case for Energy-Proportional Computing IEEE Computer 40(12)33ndash372007

[4] L A Barroso J Clidaras and U Holzle The Datacenter as aComputer An Introduction to the Design of Warehouse-ScaleMachines Second Edition Synthesis Lectures on ComputerArchitecture Morgan amp Claypool Publishers 2013

[5] A Belay A Bittau A J Mashtizadeh D Terei D Mazieresand C Kozyrakis Dune Safe User-level Access to PrivilegedCPU Features In Proceedings of the 10th Symposium onOperating System Design and Implementation (OSDI) pages335ndash348 2012

[6] A Belay G Prekas A Klimovic S Grossman C Kozyrakisand E Bugnion IX A Protected Dataplane Operating Systemfor High Throughput and Low Latency In Proceedings of the11st Symposium on Operating System Design and Implemen-tation (OSDI) pages 49ndash65 2014

[7] J Bonwick and J Adams Magazines and Vmem Extendingthe Slab Allocator to Many CPUs and Arbitrary Resources InUSENIX Annual Technical Conference pages 15ndash33 2001

[8] H David E Gorbatov U R Hanebutte R Khanaa andC Le RAPL memory power estimation and capping InProceedings of the 2010 International Symposium on LowPower Electronics and Design pages 189ndash194 2010

[9] J Dean and L A Barroso The Tail at Scale Commun ACM56(2)74ndash80 2013

[10] C Delimitrou and C Kozyrakis Quality-of-Service-AwareScheduling in Heterogeneous Data centers with ParagonIEEE Micro 34(3)17ndash30 2014

[11] C Delimitrou and C Kozyrakis Quasar Resource-Efficientand QoS-Aware Cluster Management In Proceedings of the19th International Conference on Architectural Support forProgramming Languages and Operating Systems (ASPLOS-XIX) pages 127ndash144 2014

[12] A Dragojevic D Narayanan M Castro and O HodsonFaRM Fast Remote Memory In Proceedings of the 11stSymposium on Networked Systems Design and Implementa-tion (NSDI) pages 401ndash414 2014

[13] T Harris M Maas and V J Marathe Callisto co-schedulingparallel runtime systems In Proceedings of the 2014 EuroSysConference page 24 2014

[14] B Hindman A Konwinski M Zaharia A Ghodsi A DJoseph R H Katz S Shenker and I Stoica Mesos A Plat-form for Fine-Grained Resource Sharing in the Data CenterIn Proceedings of the 8th Symposium on Networked SystemsDesign and Implementation (NSDI) 2011

[15] C-H Hsu Y Zhang M A Laurenzano D Meisner T FWenisch J Mars L Tang and R G Dreslinski AdrenalinePinpointing and reining in tail queries with quick voltageboosting In Proceedings of the 21st IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages271ndash282 2015

[16] J Hwang K K Ramakrishnan and T Wood NetVM HighPerformance and Flexible Networking Using Virtualization onCommodity Platforms In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 445ndash458 2014

[17] Intel Corp Intel 82599 10 GbE Controller Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheets82599-10-gbe-controller-datasheetpdf2014

[18] Intel Corp Intel DPDK Data Plane Development Kit httpdpdkorg 2014

[19] Intel Corp Intel Ethernet Controller XL710 Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheetsxl710-10-40-controller-datasheetpdf2014

[20] E Jeong S Woo M A Jamshed H Jeong S Ihm D Hanand K Park mTCP a Highly Scalable User-level TCP Stackfor Multicore Systems In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 489ndash502 2014

[21] R Kapoor G Porter M Tewari G M Voelker and A Vah-dat Chronos Predictable Low Latency for Data Center Ap-plications In Proceedings of the 2012 ACM Symposium onCloud Computing (SOCC) page 9 2012

[22] W Kim M S Gupta G-Y Wei and D M Brooks Systemlevel analysis of fast per-core DVFS using on-chip switch-ing regulators In Proceedings of the 14th IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages123ndash134 2008

[23] A Krioukov P Mohan S Alspaugh L Keys D E Cullerand R H Katz NapSAC design and implementation of apower-proportional web cluster Computer CommunicationReview 41(1)102ndash108 2011

[24] K-C Leung V O K Li and D Yang An Overview ofPacket Reordering in Transmission Control Protocol (TCP)Problems Solutions and Challenges IEEE Trans ParallelDistrib Syst 18(4)522ndash535 2007

[25] J Leverich Mutilate High-Performance MemcachedLoad Generator httpsgithubcomleverichmutilate 2014

0

1

2

3

4

5

6

0 50 100 150 200

Achie

ved R

PS

(x 1

06)

0 50 100 150 200 0 50 100 150 200

0

200

400

600

800

1000

0 50 100 150 200

99

th p

ct l

aten

cy (

micros)

0 50 100 150 200

SLO

0 50 100 150 200

0

20

40

60

80

100

0 50 100 150 200

o

f pea

k

Time (seconds)

dynamicPareto

0 50 100 150 200

Time (seconds)

0 50 100 150 200

Time (seconds)

Figure 7 Consolidation of memcached with a background batch task

ence can be intuitively explained since the migrations duringthe scale up are performed in a heavily loaded system whilethe system during the scale down is partially idle In absoluteterms migrations that occur when adding a core take 463microson average and less than 15 ms 95 of the time The outlierscan be explained by rare occurrences of longer preparationtimes or when processing up to 614 deferred packets

6 DiscussionUsing Pareto as a guide Even though the Pareto resultsare not used by the dynamic resource controller the Paretofrontier proved to be a valuable guide first to motivate andquantify the problem then to derive the configuration pol-icy sequence and finally to evaluate the effectiveness ofthe dynamic resource control by setting an upper bound onthe gains resulting from dynamic resource allocation Manyfactors such as software scalability hardware resource con-tention network and disk IO bottlenecks will influence thePareto frontier of any given application and therefore the de-rived control loop policies We will compare different work-loads in future workTradeoff between average latency and efficiency We usea one-dimensional SLO (99th pct le 500micros) and show thatwe can increase efficiency while maintaining that objectiveHowever this comes at a cost when considering the entirelatency distribution eg the average as well as the tail More

complex SLOs taking into account multiple aspects of la-tency distribution would define a different Pareto frontierand likely require adjustments to the control loopAdaptive flow-centric scheduling The new flow-groupmigration algorithm leads to a flow-centric approach to re-source scheduling where the network stack and applicationlogic always follow the steering decision POSIX applica-tions can balance flows by migrating file descriptors betweenthreads or processes but this tends to be inefficient becauseit is difficult for the kernel to match the flowrsquos destinationreceive queue to changes in CPU affinity Flow director canbe used by Linux to adjust the affinity of individual networkflows as a reactive measure to application-level and kernelthread migration rebalancing but the limited size of the redi-rection table prevents this mechanism from scaling to largeconnection counts By contrast our approach allows flowsto be migrated in entire groups improving efficiency and iscompatible with more scalable hardware flow steering mech-anisms based on RSSHardware trends Our experimental setup using one SandyBridge processor has a three-dimensional Pareto spaceClearly NUMA would add a new dimension Also thenewly released Haswell processors provide per-core DVFScontrols which further increases the Pareto space More-over hash filters for flow group steering could benefit fromrecent trends in NIC hardware For example Intelrsquos new

avg 95th pct max stddevad

dco

reprepare (micros) 67 231 521 93wait (micros) 135 541 983 190rpc (micros) 129 355 529 112deferred (micros) 101 401 1167 147total (micros) 463 1432 2870 459 packets 64 319 614 116

rem

ove

core prepare (micros) 25 54 398 32

wait (micros) 34 101 170 32rpc (micros) 12 25 49 7deferred (micros) 19 43 108 14total (micros) 93 160 456 47 packets 3 8 28 3

Table 2 Breakdown of flow group migration measured dur-ing the six benchmarks

XL710 chipset [19] has a 512 entry hash LUT (as well asindependent 64 entry LUTs for each VF) in contrast to the128 entries available in the 82599 chipset [17] used in ourevaluation This has the potential to reduce connection im-balances between cores especially with high core counts

7 Related WorkScheduling Scheduler activations [1] give applicationsgreater control over hardware threads and provide a mecha-nism for custom application-level scheduling Callisto [13]uses a similar strategy to improve the performance of co-located parallel runtime systems Our approach differs inthat an independent control plane manages the schedulingof hardware threads based on receive queuing latency indi-cators while the dataplane exposes a simple kernel thread-ing abstraction SEDA [51] also monitors queuing behaviorto make scheduling decisions such as thread pool sizingChronos [21] makes use of software-based flow steering butwith a focus on balancing load to reduce latency AffinityAccept [42] embraces a mixture of software and hardware-based flow steering in order to improve TCP connectionaffinity and increase throughput We focus instead on energyproportionality and workload consolidationEnergy Proportionality The energy proportionality prob-lem [3] has been well explored in previous work Somesystems have focused on solutions tailored to throughput-oriented workloads [33] or read-only workloads [23] Meis-ner et al [34] highlight unique challenges for low la-tency workloads and advocate full system active low-powermodes Similar to our system Pegasus [29] achieves CPUenergy proportionality for low latency workloads Our workexpands on Pegasus by exploring the elastic allocation ofhardware threads in combination with processor power man-agement states and by basing scheduling decisions on in-ternal latency metrics within a host endpoint instead of anexternal controller Niccolini et al show that a softwarerouter running on a dedicated machine can be made energy-

proportional [38] Similar to our approach queue lengthis used as a control signal to manage core allocation andDVFS settings However we focus on latency-sensitive ap-plications rather than middlebox traffic and consider theadditional case of workload consolidationCo-location Because host endpoints contain some com-ponents that are not energy proportional and thus are mostefficient when operating at 100 utilization co-location ofworkloads is also an important tool for improving energyefficiency At the cluster scheduler level BubbleUp [32] andParagon [10] make scheduling decisions that are interference-aware through efficient classification of the behavior ofworkload co-location Leverich et al [26] demonstratethat co-location of batch and low latency jobs is possi-ble on commodity operating systems Our approach ex-plores this issue at higher throughputs and with tighter la-tency SLOs Bubble-Flux [53] additionally controls back-ground threads we control background and latency-sensitivethreads CPI2 [54] detects performance interference by ob-serving changes in CPI and throttles offending jobs Thiswork is orthogonal to ours and could be a useful addi-tional signal for our control plane Heracles manages mul-tiple hardware and software isolation mechanisms includ-ing packet scheduling and cache partitioning to co-locatelatency-sensitive applications with batch tasks while main-taining millisecond SLOs [30] We limit our focus to DVFSand core assignment but target more aggressive SLOsLow-latency frameworks User-level networking stacks [2031 47] and RDMA applications [12 37 40] all rely onpolling typically by a fixed number of execution threadsto deliver micros-scale response times These stacks and appli-cations are evaluated based on their latency and throughputcharacteristics and do not take into account energy propor-tionality or workload consolidation considerations

8 ConclusionWe present the design and implementation of dynamic re-source controls for the IX dataplane which consists of acontroller that allocates cores and sets processor frequencyto adapt to changes in the load of latency-sensitive appli-cations The novel rebalancing mechanisms do not impactthe steady-state performance of the dataplane and can mi-grate a set of flow groups in milliseconds without droppingor reordering packets We develop two policies focused onoptimizing energy proportionality and workload consolida-tion Our results show that for three varying load patternsthe energy proportionality can save 44ndash54 of the pro-cessorrsquos energy or enable a background job to deliver 32ndash46 of its standalone throughput To evaluate the effective-ness of our approach we synthesize the Pareto frontier bycombining the behavior of all possible static configurationsOur policies deliver 85ndash93 of the Pareto optimal boundin terms of energy proportionality and 82ndash92 in termsof consolidation

AcknowledgementsThis work was supported by the MicrosoftEPFL Joint Re-search Center the Stanford Experimental Datacenter Laband NSF grant CNS-1422088 George Prekas is supportedby a Google Graduate Research Fellowship and Adam Be-lay by a Stanford Graduate Fellowship

References[1] T E Anderson B N Bershad E D Lazowska and H M

Levy Scheduler Activations Effective Kernel Support for theUser-Level Management of Parallelism ACM Trans ComputSyst 10(1)53ndash79 1992

[2] B Atikoglu Y Xu E Frachtenberg S Jiang andM Paleczny Workload Analysis of a Large-Scale Key-ValueStore In Proceedings of the 2012 ACM SIGMETRICS Inter-national Conference on Measurement and Modeling of Com-puter Systems pages 53ndash64 2012

[3] L A Barroso and U Holzle The Case for Energy-Proportional Computing IEEE Computer 40(12)33ndash372007

[4] L A Barroso J Clidaras and U Holzle The Datacenter as aComputer An Introduction to the Design of Warehouse-ScaleMachines Second Edition Synthesis Lectures on ComputerArchitecture Morgan amp Claypool Publishers 2013

[5] A Belay A Bittau A J Mashtizadeh D Terei D Mazieresand C Kozyrakis Dune Safe User-level Access to PrivilegedCPU Features In Proceedings of the 10th Symposium onOperating System Design and Implementation (OSDI) pages335ndash348 2012

[6] A Belay G Prekas A Klimovic S Grossman C Kozyrakisand E Bugnion IX A Protected Dataplane Operating Systemfor High Throughput and Low Latency In Proceedings of the11st Symposium on Operating System Design and Implemen-tation (OSDI) pages 49ndash65 2014

[7] J Bonwick and J Adams Magazines and Vmem Extendingthe Slab Allocator to Many CPUs and Arbitrary Resources InUSENIX Annual Technical Conference pages 15ndash33 2001

[8] H David E Gorbatov U R Hanebutte R Khanaa andC Le RAPL memory power estimation and capping InProceedings of the 2010 International Symposium on LowPower Electronics and Design pages 189ndash194 2010

[9] J Dean and L A Barroso The Tail at Scale Commun ACM56(2)74ndash80 2013

[10] C Delimitrou and C Kozyrakis Quality-of-Service-AwareScheduling in Heterogeneous Data centers with ParagonIEEE Micro 34(3)17ndash30 2014

[11] C Delimitrou and C Kozyrakis Quasar Resource-Efficientand QoS-Aware Cluster Management In Proceedings of the19th International Conference on Architectural Support forProgramming Languages and Operating Systems (ASPLOS-XIX) pages 127ndash144 2014

[12] A Dragojevic D Narayanan M Castro and O HodsonFaRM Fast Remote Memory In Proceedings of the 11stSymposium on Networked Systems Design and Implementa-tion (NSDI) pages 401ndash414 2014

[13] T Harris M Maas and V J Marathe Callisto co-schedulingparallel runtime systems In Proceedings of the 2014 EuroSysConference page 24 2014

[14] B Hindman A Konwinski M Zaharia A Ghodsi A DJoseph R H Katz S Shenker and I Stoica Mesos A Plat-form for Fine-Grained Resource Sharing in the Data CenterIn Proceedings of the 8th Symposium on Networked SystemsDesign and Implementation (NSDI) 2011

[15] C-H Hsu Y Zhang M A Laurenzano D Meisner T FWenisch J Mars L Tang and R G Dreslinski AdrenalinePinpointing and reining in tail queries with quick voltageboosting In Proceedings of the 21st IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages271ndash282 2015

[16] J Hwang K K Ramakrishnan and T Wood NetVM HighPerformance and Flexible Networking Using Virtualization onCommodity Platforms In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 445ndash458 2014

[17] Intel Corp Intel 82599 10 GbE Controller Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheets82599-10-gbe-controller-datasheetpdf2014

[18] Intel Corp Intel DPDK Data Plane Development Kit httpdpdkorg 2014

[19] Intel Corp Intel Ethernet Controller XL710 Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheetsxl710-10-40-controller-datasheetpdf2014

[20] E Jeong S Woo M A Jamshed H Jeong S Ihm D Hanand K Park mTCP a Highly Scalable User-level TCP Stackfor Multicore Systems In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 489ndash502 2014

[21] R Kapoor G Porter M Tewari G M Voelker and A Vah-dat Chronos Predictable Low Latency for Data Center Ap-plications In Proceedings of the 2012 ACM Symposium onCloud Computing (SOCC) page 9 2012

[22] W Kim M S Gupta G-Y Wei and D M Brooks Systemlevel analysis of fast per-core DVFS using on-chip switch-ing regulators In Proceedings of the 14th IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages123ndash134 2008

[23] A Krioukov P Mohan S Alspaugh L Keys D E Cullerand R H Katz NapSAC design and implementation of apower-proportional web cluster Computer CommunicationReview 41(1)102ndash108 2011

[24] K-C Leung V O K Li and D Yang An Overview ofPacket Reordering in Transmission Control Protocol (TCP)Problems Solutions and Challenges IEEE Trans ParallelDistrib Syst 18(4)522ndash535 2007

[25] J Leverich Mutilate High-Performance MemcachedLoad Generator httpsgithubcomleverichmutilate 2014

avg 95th pct max stddevad

dco

reprepare (micros) 67 231 521 93wait (micros) 135 541 983 190rpc (micros) 129 355 529 112deferred (micros) 101 401 1167 147total (micros) 463 1432 2870 459 packets 64 319 614 116

rem

ove

core prepare (micros) 25 54 398 32

wait (micros) 34 101 170 32rpc (micros) 12 25 49 7deferred (micros) 19 43 108 14total (micros) 93 160 456 47 packets 3 8 28 3

Table 2 Breakdown of flow group migration measured dur-ing the six benchmarks

XL710 chipset [19] has a 512 entry hash LUT (as well asindependent 64 entry LUTs for each VF) in contrast to the128 entries available in the 82599 chipset [17] used in ourevaluation This has the potential to reduce connection im-balances between cores especially with high core counts

7 Related WorkScheduling Scheduler activations [1] give applicationsgreater control over hardware threads and provide a mecha-nism for custom application-level scheduling Callisto [13]uses a similar strategy to improve the performance of co-located parallel runtime systems Our approach differs inthat an independent control plane manages the schedulingof hardware threads based on receive queuing latency indi-cators while the dataplane exposes a simple kernel thread-ing abstraction SEDA [51] also monitors queuing behaviorto make scheduling decisions such as thread pool sizingChronos [21] makes use of software-based flow steering butwith a focus on balancing load to reduce latency AffinityAccept [42] embraces a mixture of software and hardware-based flow steering in order to improve TCP connectionaffinity and increase throughput We focus instead on energyproportionality and workload consolidationEnergy Proportionality The energy proportionality prob-lem [3] has been well explored in previous work Somesystems have focused on solutions tailored to throughput-oriented workloads [33] or read-only workloads [23] Meis-ner et al [34] highlight unique challenges for low la-tency workloads and advocate full system active low-powermodes Similar to our system Pegasus [29] achieves CPUenergy proportionality for low latency workloads Our workexpands on Pegasus by exploring the elastic allocation ofhardware threads in combination with processor power man-agement states and by basing scheduling decisions on in-ternal latency metrics within a host endpoint instead of anexternal controller Niccolini et al show that a softwarerouter running on a dedicated machine can be made energy-

proportional [38] Similar to our approach queue lengthis used as a control signal to manage core allocation andDVFS settings However we focus on latency-sensitive ap-plications rather than middlebox traffic and consider theadditional case of workload consolidationCo-location Because host endpoints contain some com-ponents that are not energy proportional and thus are mostefficient when operating at 100 utilization co-location ofworkloads is also an important tool for improving energyefficiency At the cluster scheduler level BubbleUp [32] andParagon [10] make scheduling decisions that are interference-aware through efficient classification of the behavior ofworkload co-location Leverich et al [26] demonstratethat co-location of batch and low latency jobs is possi-ble on commodity operating systems Our approach ex-plores this issue at higher throughputs and with tighter la-tency SLOs Bubble-Flux [53] additionally controls back-ground threads we control background and latency-sensitivethreads CPI2 [54] detects performance interference by ob-serving changes in CPI and throttles offending jobs Thiswork is orthogonal to ours and could be a useful addi-tional signal for our control plane Heracles manages mul-tiple hardware and software isolation mechanisms includ-ing packet scheduling and cache partitioning to co-locatelatency-sensitive applications with batch tasks while main-taining millisecond SLOs [30] We limit our focus to DVFSand core assignment but target more aggressive SLOsLow-latency frameworks User-level networking stacks [2031 47] and RDMA applications [12 37 40] all rely onpolling typically by a fixed number of execution threadsto deliver micros-scale response times These stacks and appli-cations are evaluated based on their latency and throughputcharacteristics and do not take into account energy propor-tionality or workload consolidation considerations

8 ConclusionWe present the design and implementation of dynamic re-source controls for the IX dataplane which consists of acontroller that allocates cores and sets processor frequencyto adapt to changes in the load of latency-sensitive appli-cations The novel rebalancing mechanisms do not impactthe steady-state performance of the dataplane and can mi-grate a set of flow groups in milliseconds without droppingor reordering packets We develop two policies focused onoptimizing energy proportionality and workload consolida-tion Our results show that for three varying load patternsthe energy proportionality can save 44ndash54 of the pro-cessorrsquos energy or enable a background job to deliver 32ndash46 of its standalone throughput To evaluate the effective-ness of our approach we synthesize the Pareto frontier bycombining the behavior of all possible static configurationsOur policies deliver 85ndash93 of the Pareto optimal boundin terms of energy proportionality and 82ndash92 in termsof consolidation

AcknowledgementsThis work was supported by the MicrosoftEPFL Joint Re-search Center the Stanford Experimental Datacenter Laband NSF grant CNS-1422088 George Prekas is supportedby a Google Graduate Research Fellowship and Adam Be-lay by a Stanford Graduate Fellowship

References[1] T E Anderson B N Bershad E D Lazowska and H M

Levy Scheduler Activations Effective Kernel Support for theUser-Level Management of Parallelism ACM Trans ComputSyst 10(1)53ndash79 1992

[2] B Atikoglu Y Xu E Frachtenberg S Jiang andM Paleczny Workload Analysis of a Large-Scale Key-ValueStore In Proceedings of the 2012 ACM SIGMETRICS Inter-national Conference on Measurement and Modeling of Com-puter Systems pages 53ndash64 2012

[3] L A Barroso and U Holzle The Case for Energy-Proportional Computing IEEE Computer 40(12)33ndash372007

[4] L A Barroso J Clidaras and U Holzle The Datacenter as aComputer An Introduction to the Design of Warehouse-ScaleMachines Second Edition Synthesis Lectures on ComputerArchitecture Morgan amp Claypool Publishers 2013

[5] A Belay A Bittau A J Mashtizadeh D Terei D Mazieresand C Kozyrakis Dune Safe User-level Access to PrivilegedCPU Features In Proceedings of the 10th Symposium onOperating System Design and Implementation (OSDI) pages335ndash348 2012

[6] A Belay G Prekas A Klimovic S Grossman C Kozyrakisand E Bugnion IX A Protected Dataplane Operating Systemfor High Throughput and Low Latency In Proceedings of the11st Symposium on Operating System Design and Implemen-tation (OSDI) pages 49ndash65 2014

[7] J Bonwick and J Adams Magazines and Vmem Extendingthe Slab Allocator to Many CPUs and Arbitrary Resources InUSENIX Annual Technical Conference pages 15ndash33 2001

[8] H David E Gorbatov U R Hanebutte R Khanaa andC Le RAPL memory power estimation and capping InProceedings of the 2010 International Symposium on LowPower Electronics and Design pages 189ndash194 2010

[9] J Dean and L A Barroso The Tail at Scale Commun ACM56(2)74ndash80 2013

[10] C Delimitrou and C Kozyrakis Quality-of-Service-AwareScheduling in Heterogeneous Data centers with ParagonIEEE Micro 34(3)17ndash30 2014

[11] C Delimitrou and C Kozyrakis Quasar Resource-Efficientand QoS-Aware Cluster Management In Proceedings of the19th International Conference on Architectural Support forProgramming Languages and Operating Systems (ASPLOS-XIX) pages 127ndash144 2014

[12] A Dragojevic D Narayanan M Castro and O HodsonFaRM Fast Remote Memory In Proceedings of the 11stSymposium on Networked Systems Design and Implementa-tion (NSDI) pages 401ndash414 2014

[13] T Harris M Maas and V J Marathe Callisto co-schedulingparallel runtime systems In Proceedings of the 2014 EuroSysConference page 24 2014

[14] B Hindman A Konwinski M Zaharia A Ghodsi A DJoseph R H Katz S Shenker and I Stoica Mesos A Plat-form for Fine-Grained Resource Sharing in the Data CenterIn Proceedings of the 8th Symposium on Networked SystemsDesign and Implementation (NSDI) 2011

[15] C-H Hsu Y Zhang M A Laurenzano D Meisner T FWenisch J Mars L Tang and R G Dreslinski AdrenalinePinpointing and reining in tail queries with quick voltageboosting In Proceedings of the 21st IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages271ndash282 2015

[16] J Hwang K K Ramakrishnan and T Wood NetVM HighPerformance and Flexible Networking Using Virtualization onCommodity Platforms In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 445ndash458 2014

[17] Intel Corp Intel 82599 10 GbE Controller Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheets82599-10-gbe-controller-datasheetpdf2014

[18] Intel Corp Intel DPDK Data Plane Development Kit httpdpdkorg 2014

[19] Intel Corp Intel Ethernet Controller XL710 Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheetsxl710-10-40-controller-datasheetpdf2014

[20] E Jeong S Woo M A Jamshed H Jeong S Ihm D Hanand K Park mTCP a Highly Scalable User-level TCP Stackfor Multicore Systems In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 489ndash502 2014

[21] R Kapoor G Porter M Tewari G M Voelker and A Vah-dat Chronos Predictable Low Latency for Data Center Ap-plications In Proceedings of the 2012 ACM Symposium onCloud Computing (SOCC) page 9 2012

[22] W Kim M S Gupta G-Y Wei and D M Brooks Systemlevel analysis of fast per-core DVFS using on-chip switch-ing regulators In Proceedings of the 14th IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages123ndash134 2008

[23] A Krioukov P Mohan S Alspaugh L Keys D E Cullerand R H Katz NapSAC design and implementation of apower-proportional web cluster Computer CommunicationReview 41(1)102ndash108 2011

[24] K-C Leung V O K Li and D Yang An Overview ofPacket Reordering in Transmission Control Protocol (TCP)Problems Solutions and Challenges IEEE Trans ParallelDistrib Syst 18(4)522ndash535 2007

[25] J Leverich Mutilate High-Performance MemcachedLoad Generator httpsgithubcomleverichmutilate 2014

AcknowledgementsThis work was supported by the MicrosoftEPFL Joint Re-search Center the Stanford Experimental Datacenter Laband NSF grant CNS-1422088 George Prekas is supportedby a Google Graduate Research Fellowship and Adam Be-lay by a Stanford Graduate Fellowship

References[1] T E Anderson B N Bershad E D Lazowska and H M

Levy Scheduler Activations Effective Kernel Support for theUser-Level Management of Parallelism ACM Trans ComputSyst 10(1)53ndash79 1992

[2] B Atikoglu Y Xu E Frachtenberg S Jiang andM Paleczny Workload Analysis of a Large-Scale Key-ValueStore In Proceedings of the 2012 ACM SIGMETRICS Inter-national Conference on Measurement and Modeling of Com-puter Systems pages 53ndash64 2012

[3] L A Barroso and U Holzle The Case for Energy-Proportional Computing IEEE Computer 40(12)33ndash372007

[4] L A Barroso J Clidaras and U Holzle The Datacenter as aComputer An Introduction to the Design of Warehouse-ScaleMachines Second Edition Synthesis Lectures on ComputerArchitecture Morgan amp Claypool Publishers 2013

[5] A Belay A Bittau A J Mashtizadeh D Terei D Mazieresand C Kozyrakis Dune Safe User-level Access to PrivilegedCPU Features In Proceedings of the 10th Symposium onOperating System Design and Implementation (OSDI) pages335ndash348 2012

[6] A Belay G Prekas A Klimovic S Grossman C Kozyrakisand E Bugnion IX A Protected Dataplane Operating Systemfor High Throughput and Low Latency In Proceedings of the11st Symposium on Operating System Design and Implemen-tation (OSDI) pages 49ndash65 2014

[7] J Bonwick and J Adams Magazines and Vmem Extendingthe Slab Allocator to Many CPUs and Arbitrary Resources InUSENIX Annual Technical Conference pages 15ndash33 2001

[8] H David E Gorbatov U R Hanebutte R Khanaa andC Le RAPL memory power estimation and capping InProceedings of the 2010 International Symposium on LowPower Electronics and Design pages 189ndash194 2010

[9] J Dean and L A Barroso The Tail at Scale Commun ACM56(2)74ndash80 2013

[10] C Delimitrou and C Kozyrakis Quality-of-Service-AwareScheduling in Heterogeneous Data centers with ParagonIEEE Micro 34(3)17ndash30 2014

[11] C Delimitrou and C Kozyrakis Quasar Resource-Efficientand QoS-Aware Cluster Management In Proceedings of the19th International Conference on Architectural Support forProgramming Languages and Operating Systems (ASPLOS-XIX) pages 127ndash144 2014

[12] A Dragojevic D Narayanan M Castro and O HodsonFaRM Fast Remote Memory In Proceedings of the 11stSymposium on Networked Systems Design and Implementa-tion (NSDI) pages 401ndash414 2014

[13] T Harris M Maas and V J Marathe Callisto co-schedulingparallel runtime systems In Proceedings of the 2014 EuroSysConference page 24 2014

[14] B Hindman A Konwinski M Zaharia A Ghodsi A DJoseph R H Katz S Shenker and I Stoica Mesos A Plat-form for Fine-Grained Resource Sharing in the Data CenterIn Proceedings of the 8th Symposium on Networked SystemsDesign and Implementation (NSDI) 2011

[15] C-H Hsu Y Zhang M A Laurenzano D Meisner T FWenisch J Mars L Tang and R G Dreslinski AdrenalinePinpointing and reining in tail queries with quick voltageboosting In Proceedings of the 21st IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages271ndash282 2015

[16] J Hwang K K Ramakrishnan and T Wood NetVM HighPerformance and Flexible Networking Using Virtualization onCommodity Platforms In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 445ndash458 2014

[17] Intel Corp Intel 82599 10 GbE Controller Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheets82599-10-gbe-controller-datasheetpdf2014

[18] Intel Corp Intel DPDK Data Plane Development Kit httpdpdkorg 2014

[19] Intel Corp Intel Ethernet Controller XL710 Datasheethttpwwwintelcomcontentdamwwwpublicusendocumentsdatasheetsxl710-10-40-controller-datasheetpdf2014

[20] E Jeong S Woo M A Jamshed H Jeong S Ihm D Hanand K Park mTCP a Highly Scalable User-level TCP Stackfor Multicore Systems In Proceedings of the 11st Symposiumon Networked Systems Design and Implementation (NSDI)pages 489ndash502 2014

[21] R Kapoor G Porter M Tewari G M Voelker and A Vah-dat Chronos Predictable Low Latency for Data Center Ap-plications In Proceedings of the 2012 ACM Symposium onCloud Computing (SOCC) page 9 2012

[22] W Kim M S Gupta G-Y Wei and D M Brooks Systemlevel analysis of fast per-core DVFS using on-chip switch-ing regulators In Proceedings of the 14th IEEE Symposiumon High-Performance Computer Architecture (HPCA) pages123ndash134 2008

[23] A Krioukov P Mohan S Alspaugh L Keys D E Cullerand R H Katz NapSAC design and implementation of apower-proportional web cluster Computer CommunicationReview 41(1)102ndash108 2011

[24] K-C Leung V O K Li and D Yang An Overview ofPacket Reordering in Transmission Control Protocol (TCP)Problems Solutions and Challenges IEEE Trans ParallelDistrib Syst 18(4)522ndash535 2007

[25] J Leverich Mutilate High-Performance MemcachedLoad Generator httpsgithubcomleverichmutilate 2014

[26] J Leverich and C Kozyrakis Reconciling High Server Uti-lization and Sub-Millisecond Quality-of-Service In Proceed-ings of the 2014 EuroSys Conference page 4 2014

[27] J Li N K Sharma D R K Ports and S D Gribble Talesof the Tail Hardware OS and Application-level Sources ofTail Latency In Proceedings of the 2014 ACM Symposium onCloud Computing (SOCC) pages 1ndash14 2014

[28] H Lim D Han D G Andersen and M Kaminsky MICAA Holistic Approach to Fast In-Memory Key-Value StorageIn Proceedings of the 11st Symposium on Networked SystemsDesign and Implementation (NSDI) pages 429ndash444 2014

[29] D Lo L Cheng R Govindaraju L A Barroso andC Kozyrakis Towards Energy Proportionality for Large-Scale Latency-Critical Workloads In Proceedings of the 41stInternational Symposium on Computer Architecture (ISCA)pages 301ndash312 2014

[30] D Lo L Cheng R Govindaraju P Ranganathan andC Kozyrakis Heracles improving resource efficiency atscale In Proceedings of the 42nd International Symposiumon Computer Architecture (ISCA) pages 450ndash462 2015

[31] I Marinos R N M Watson and M Handley Network StackSpecialization for Performance In Proceedings of the ACMSIGCOMM 2014 Conference pages 175ndash186 2014

[32] J Mars L Tang K Skadron M L Soffa and R Hundt In-creasing Utilization in Modern Warehouse-Scale ComputersUsing Bubble-Up IEEE Micro 32(3)88ndash99 2012

[33] D Meisner B T Gold and T F Wenisch The PowerNapServer Architecture ACM Trans Comput Syst 29(1)32011

[34] D Meisner C M Sadler L A Barroso W-D Weber andT F Wenisch Power management of online data-intensiveservices In Proceedings of the 38th International Symposiumon Computer Architecture (ISCA) pages 319ndash330 2011

[35] memcached memcached ndash a distributed memory objectcaching system httpmemcachedorg 2014

[36] Microsoft Corp Receive Side Scaling httpmsdnmicrosoftcomlibrarywindowshardwareff556942aspx 2014

[37] C Mitchell Y Geng and J Li Using One-Sided RDMAReads to Build a Fast CPU-Efficient Key-Value Store InProceedings of the 2013 USENIX Annual Technical Confer-ence (ATC) pages 103ndash114 2013

[38] L Niccolini G Iannaccone S Ratnasamy J Chandrashekarand L Rizzo Building a Power-Proportional Software RouterIn Proceedings of the 2012 USENIX Annual Technical Confer-ence (ATC) pages 89ndash100 2012

[39] R Nishtala H Fugal S Grimm M Kwiatkowski H LeeH C Li R McElroy M Paleczny D Peek P SaabD Stafford T Tung and V Venkataramani Scaling Mem-cache at Facebook In Proceedings of the 10th Symposiumon Networked Systems Design and Implementation (NSDI)pages 385ndash398 2013

[40] D Ongaro S M Rumble R Stutsman J K Ousterhoutand M Rosenblum Fast Crash Recovery in RAMCloudIn Proceedings of the 23rd ACM Symposium on OperatingSystems Principles (SOSP) pages 29ndash41 2011

[41] Pareto Pareto efficiency httphttpenwikipediaorgwikiPareto_efficiency

[42] A Pesterev J Strauss N Zeldovich and R T Morris Im-proving Network Connection Locality on Multicore SystemsIn Proceedings of the 2012 EuroSys Conference pages 337ndash350 2012

[43] S Peter J Li I Zhang D R K Ports D Woos A Krishna-murthy T E Anderson and T Roscoe Arrakis The Operat-ing System is the Control Plane In Proceedings of the 11stSymposium on Operating System Design and Implementation(OSDI) pages 1ndash16 2014

[44] L Rizzo netmap A Novel Framework for Fast Packet IOIn Proceedings of the 2012 USENIX Annual Technical Con-ference (ATC) pages 101ndash112 2012

[45] E Rotem A Naveh A Ananthakrishnan E Weissmann andD Rajwan Power-Management Architecture of the IntelMicroarchitecture Code-Named Sandy Bridge IEEE Micro32(2)20ndash27 2012

[46] M Schwarzkopf A Konwinski M Abd-El-Malek andJ Wilkes Omega flexible scalable schedulers for large com-pute clusters In Proceedings of the 2013 EuroSys Conferencepages 351ndash364 2013

[47] Solarflare Communications Introduction to OpenOn-load Building Application Transparency and Pro-tocol Conformance into Application AccelerationMiddleware httpwwwsolarflarecomcontentuserfilesdocumentssolarflare_openonload_intropaperpdf 2011

[48] G Urdaneta G Pierre and M van Steen Wikipedia work-load analysis for decentralized hosting Computer Networks53(11)1830ndash1845 2009

[49] G Varghese and A Lauck Hashed and Hierarchical TimingWheels Data Structures for the Efficient Implementation of aTimer Facility In Proceedings of the 11st ACM Symposiumon Operating Systems Principles (SOSP) pages 25ndash38 1987

[50] W Vogels Beyond Server Consolidation ACM Queue 6(1)20ndash26 2008

[51] M Welsh D E Culler and E A Brewer SEDA An Archi-tecture for Well-Conditioned Scalable Internet Services InProceedings of the 18th ACM Symposium on Operating Sys-tems Principles (SOSP) pages 230ndash243 2001

[52] W Wu P DeMar and M Crawford Why Can Some Ad-vanced Ethernet NICs Cause Packet Reordering IEEE Com-munications Letters 15(2)253ndash255 2011

[53] H Yang A D Breslow J Mars and L Tang Bubble-flux precise online QoS management for increased utilizationin warehouse scale computers In Proceedings of the 40thInternational Symposium on Computer Architecture (ISCA)pages 607ndash618 2013

[54] X Zhang E Tune R Hagmann R Jnagal V Gokhale andJ Wilkes CPI2 CPU performance isolation for shared com-pute clusters In Proceedings of the 2013 EuroSys Conferencepages 379ndash391 2013