Maestro: Orchestrating Lifetime Reliability in ChipMultiprocessors

Shuguang Feng Shantanu Gupta Amin Ansari Scott Mahlke

Advanced Computer Architecture LaboratoryUniversity of MichiganAnn Arbor, MI 48109

{shoe, shangupt, ansary, mahlke}@umich.edu

ABSTRACTAs CMOS feature sizes venture deep into the nanometer regime,wearout mechanisms including negative-bias temperature instabil-ity and time-dependent dielectric breakdown can severely reduceprocessor operating lifetimes and performance. This paperpresentsan introspective reliability management system, Maestro,to tacklereliability challenges in future chip multiprocessors (CMPs) head-on. Unlike traditional approaches, Maestro relies on low-level sen-sors to monitor the CMP as it ages (introspection). Leveragingthis real-time assessment of CMP health, runtime heuristics iden-tify wearout-centric job assignments (management). By exploit-ing the complementary effects of the natural heterogeneity(due toprocess variation and wearout) that exists in CMPs and the diver-sity found in system workloads, Maestro composes job schedulesthat intelligently control the aging process. Monte Carlo experi-ments show that Maestro significantly enhances lifetime reliabilitythrough intelligent wear-leveling, increasing the expected servicelife of a population of 16-core CMPs by as much as 38% comparedto a naive, round-robin scheduler. Furthermore, in the presenceof process variation, Maestro’s wearout-centric scheduling outper-formed both performance counter and temperature sensor basedschedulers, achieving an order of magnitude more improvement inlifetime throughput – the amount of useful work done by a systemprior to failure.

1. INTRODUCTIONIn recent years, computer architects have accepted the factthat

transistors become less reliable with each new technology gener-ation [4]. As technology scaling leads to higher device counts,power densities and operating temperatures will continue to rise atan alarming pace. With an exponential dependence on temperature,faults due to failure mechanisms like negative-bias temperature in-stability (NBTI) and time-dependent dielectric breakdown(TDDB)will result in ever-shrinking device lifetimes. Furthermore, as pro-cess variation (random + systematic) and wearout gain more promi-nence in future technology nodes, fundamental design assumptionswill become increasingly less accurate. For example, the charac-teristics of a core on one part of a chip multiprocessor (CMP)may,due to manufacturing defects, only loosely resemble anidenticallydesigned core on a different part of the CMP [26, 23]. Even thebehavior of the same core can be expected to change over time as aresult of age-dependent degradation [18, 25].

In light of this uncertain landscape, researchers have begun in-vestigating dynamic thermal and reliability management (DTM andDRM). Such techniques hope to sustain current performance im-provement trends deep into the nanometer regime, while maintain-ing the levels of reliability and life-expectancy that consumers have

come to expect, by hiding a processor’s inherent susceptibility tofailures and hotspots. Some recent proposals rely on a combina-tion of thread scheduling and dynamic voltage and frequencyscal-ing (DVFS) to recover performance lost to process variation[23,26]. Others implement intelligent thermal management policiesthat can extend processor lifetimes and alleviate hotspotsby min-imizing and bounding the overall thermal stress experienced by acore [16, 17, 9, 7]. There have also been efforts to design sophisti-cated circuits that tolerate faults and adaptive pipelineswith flexibletiming constraints [10, 24]. Although many DTM schemes activelymanipulate job-to-core assignments to avoid thermal emergencies,most existing DRM approaches onlyreact to faults, tolerating themas they develop.

In contrast, Maestro takes a proactive approach to reliability. Tothe first order, Maestro performs fine-grained, module-level wear-leveling for many-core CMPs. Although analogous to wear-levelingin flash devices, the challenge of achieving successful wear-levelingtransparently in CMPs is considerably more difficult. Left unchecked,wearout causes all structures within a core to age and eventuallyfail. However, due to process variation, not all cores (or structures)will be created equal. Every core will invariably possess some mi-croarchitectural structures that are more “damaged” (moresuscep-tible to wearout) than others [24, 23]. Performing post-mortems onfailed cores (in simulations) often reveals that a single microarchi-tectural module, which varies from core to core, breaks downlongbefore the rest. Maestro extends the life of these “weak” structures,their corresponding cores, and ultimately the CMP by ensuring uni-form aging with scheduling-driven wear-leveling across all levelsof the hierarchy.

Maestro dynamically formulates wearout-centric schedules, wherejobs are assigned to cores such that cores do not execute work-loads that apply excessive stress to their weakest modules (i.e., afloating-point intensive thread is not bound to a core with a weak-ened floating-point adder). This accomplisheslocal wear-levelingat the core level, avoiding failures induced by a single weakstruc-ture. When two cores both have a strong affinity for the same job,a heuristic, which enforcesglobal wear-leveling at the CMP leveldetermines which core is given priority. Typically, unlessthere isa substantial negative impact on local wear-leveling, deference isgiven to the weaker of the two cores. This ensures that, when nec-essary, stronger cores are allowed to execute less desirable jobs inorder to postpone failures in weaker cores (details in Section 3.2).

By leveraging the natural, module-level diversity in applicationthermal footprints (Section 2.1), Maestro has finer-grained controlover the aging process than a standard core-level DVFS approach,without any of the attendant hardware/design overheads. Given thecomplex nature of wearout degradation, Maestro departs from the

conventional reliance on static analysis to project optimized sched-ules. Instead, the condition of the underlying CMP hardwareiscontinuously monitored, allowing Maestro to dynamically refineand adapt scheduling algorithms as the system ages. Architectureslike those envisioned in [22], with low-level circuit sensors, canreadily supply this real-time “health” monitoring.

Maestro offers two key benefits for future CMP systems. First,the fine-grained, local wear-leveling prevents unnecessary core fail-ures, maximizing the life ofindividual cores. Longer lasting corestranslates to more work that can be done over the life of the system.Second, it improves the ability of the system to sustain heavy work-loads despite the effects of aging. Enforcing global wear-levelingmaximizes thenumber of functional cores (throughout its usefullife), which in turn maximizes the computational horsepower avail-able to meet peak demands. With higher degrees of process vari-ation on the horizon, premature core failures will make it increas-ingly more difficult to design and qualify future CMPs. However,by harnessing the potential of Maestro, proactive management willenable semiconductor manufacturers to provide chips with longerlifetimes as well as ensure that system performance targetsare con-sistently met throughout that lifetime. The central contributions ofthis paper include:

• An evaluation of workload variability and its impact on reli-ability/wearout.

• An introspective system, Maestro, that utilizes low-levelsen-sor feedback andapplication-driven wear-leveling to proactively manage life-time reliability.

• The design and evaluation of two reliability-centric job schedul-ing algorithms.

2. SCHEDULING FOR DAMAGED CORESAND DYNAMIC WORKLOADS

Scheduling, in the context of this paper, refers to the processof assigning jobs to cores in a CMP, and is conceptually decou-pled from the operating system (OS) scheduler. The schedulersproposed by microarchitects in the past typically resided in a virtu-alization layer (i.e., system firmware) that sits between the OS andthe underlying hardware. At each scheduling interval, the OS sup-plies a set of jobs,J , to this virtualization layer, and it is the task ofthe low-level scheduler to bind the jobs to cores. Prior works haveinvestigated techniques that leverage intelligent job scheduling tomanage on-core temperatures or cope with process variation. How-ever, none have studied the impact that wearout-centric schedulingalone can have on the evolution of aging within a core.

Embracing process variation and workload diversity, Maestrocan enhance lifetime reliability without the extensive hardware sup-port for adaptive body biasing (ABB) and adaptive supply voltage(ASV) required by other approaches [25]. The remainder of thispaper targets TDDB and NBTI, which are expected to be the twoleading causes of wearout-related failures in future technologies,but can be easily extended to address any progressive failure mech-anisms that may emerge. Since both TDDB and NBTI are highlydependent on temperature, it is important to understand thether-mal footprints of typical applications in order to appreciate the po-tential for reliability-centric scheduling. Section 2.1 examines themodule-level thermal diversity seen across a set of SPEC2000 ap-plications and Section 2.2 presents preliminary results quantifyingthe impact of this variation on processor lifetimes.

Figure 1: Variation of module temperatures across SPEC2000workloads. All temperatures are normalized toTmax, the peaktemperature seen across all benchmarks and modules (83◦C).

2.1 Workload VariationFigure 1 shows the range of temperatures experienced by differ-

ent structures within an Alpha21364-like processor [1] across a setof 8 SPECINT (bzip2, gcc, gzip, mcf, perlbmk, twolf, vortex, vpr)and 9 SPECFP benchmarks (ammp, applu, apsi, art, equake, gal-gel, lucas, sixtrack, swim, wupwise). All temperatures are normal-ized to the peak temperature,Tmax, seen across all modules andbenchmarks, which corresponds to the temperature of theFPAddmodule when runninglucas (83◦C). Notice the significant variationin temperature within nearly every module. Apart from the morethan 40% variation seen inFPAdd (a 37◦C swing), other structures(whose utilizations are not as strongly correlated with theexecutionof floating-point and integer benchmarks) also exhibit significanttemperature shifts, 10-15% forBpred andIntReg. These largetemperature ranges suggest that scheduling alone can be a powerfultool for manipulating aging rates.

Figure 2 selects a few representative applications and examinesthem in greater detail. Figures 2(a) and 2(b) highlight how the tra-ditional view of “hot” and “cold” applications is perhaps too sim-plistic. Without accounting for the module-level variation in tem-peratures, one could incorrectly assume thatapplu is more taxing,from a reliability perspective, thanvpr or wupwise simply becauseit exhibits a higher peak operating temperature (FPMul). However,this would neglect the fact that for many structures, likeIntReg,temperatures forapplu are actually much lower than the other twoapplications. For completeness, Figure 2(c) is included toshowthat variations in module temperatures exist even between applica-tions with comparable peak temperatures. All things considered,deciding where on the CMP to schedule a particular application,to achieve the least reliability impact, requires additional informa-tion about the strength of individual structures within every core.Although the magnitude of the temperature differences may notseem impressive at first, with peak deltas in module temperaturesaround 10-20% in Figure 2(a), these modest variations in tempera-ture can have dramatic impacts on a processor’s mean time to fail-ure (MTTF).

2.2 Implications for Mean Time to FailureFrom Figure 2, one could expect a core consistently runningap-

plu to fail because of a fault in theFPMul unit due to its high oper-ating temperatures. However, in the presence of process variationother structures within the core could have been manufactured withmore defects (or tighter timing margins), and therefore even moresusceptible to failure despite not ever realizing the same peak tem-peratures asFPMul. In this environment, a reliability-centric jobscheduler must take into consideration the extent of damagepresentwithin a core in addition to the per-module thermal footprint of run-ning applications. Figure 3 presents the expected lifetimeof a corerunningapplu or vpr as a function of the module identified as the

(a) SPECFP v. SPECINT (b) SPECFP v. SPECFP (c) Variation despite comparable peak tem-peratures

Figure 2: Head-to-head comparisons of applu (SPECFP), vpr (SPECINT), and wupwise (SPECFP). No one benchmark in (a), (b),or (c) strictly dominates the other (with respect to temperature) across all modules.

Figure 3: Projected core lifetime based on execution ofappluand vpr as a function of the module identified as the weakeststructure. Values are normalized to the best achievable MTTF.

weakest structure. The lifetimes are projected based on well-knownMTTF equations for NBTI and TDDB [15, 21]. The values are nor-malized to the best achievable MTTF, which in this comparison isattained ifFPMap is the weakest module in the core and the coreis runningvpr. The optimal job to schedule on a particular core tomaximize its lifetime is dependent not just on the application mixcurrently available, but also on the strengths of individual structureswithin that core. Schedulingapplu on a core with a weakIntRegcan nearly triple its operating lifetime compared to naively forcingit to run vpr. Similarly, schedulingvpr instead ofapplu on a corewith a weakFPAdd improves its projected lifetime by more than4x.

To further highlight the need to address process and workloadvariation, a quick examination of the processors simulatedin Sec-tion 4.1 reveals that 35% of core failures are the result of failingstructures that never experience peak on-chip temperatures. Fur-thermore, 22% of core failures are caused by modules that do notrank among the top three most thermally active. By accounting forthe impact of process variation and module-level thermal variationof applications, Maestro can prevent premature core failures andreap the opportunity left on the table by previous schedulers.

3. MAESTROFigure 4 presents a block diagram of Maestro, which consists

of two main components: 1) a health monitoring system (intro-spection) and 2) a virtualization layer that implements wearout-centric job scheduling (management). Although this paper targetsreliability-centric scheduling, a broader vision of introspective re-liability management could use online sensor feedback to guidea range of solutions from traditional DVFS to more radical ap-proaches like system-level reconfiguration [14].

3.1 Health MonitoringTracking the evolution of wearout damage within a CMP (i.e.,

health monitoring) is essential to forming intelligent reliability-centric schedules. Maestro assumes that the underlying CMPisprovisioned with circuit-level sensors like those described in [22].Recognizing that the two mechanisms addressed in this work,NBTIand TDDB, both impact physical device parameters as they evolvehas led researchers to actively develop circuit-level sensors that cantrack these changes. NBTI is known to shift threshold voltage (Vt)leading to slower devices and increased subthreshold/standby leak-age current (Iddq), while TDDB increases gate currents (Igs andIgd). Both result in statistically measurable degradation in timingpaths at the microarchitectural-level [3, 6].

A runtime system collects raw data streams from the array ofcircuit-level sensors and applies statistical filtering and trend anal-ysis (similar to what is described in [3]) to convert these streamsinto descriptions of system characteristics including, delay profiles,leakage currents, and operating temperatures. These individualchannels of information are then processed to generate a compre-hensive microarchitectural-level reliability assessment of the CMP.This is shown in Figure 4 as a vector of per-module damage val-ues (relative to the maximum damage sustainable prior to failure).Introducing the additional analysis step allows the healthmonitor-ing system to account for things like the presence of redundantdevices within a structure, the influence of shifting environmen-tal conditions on sensor readings, and the interaction between dif-ferent wearout mechanisms. Ultimately, this allows the low-levelsensor feedback to be abstracted with each vector representing theeffective damage profile for a particular core.

3.2 Maestro Virtualization LayerThe second portion of the Maestro framework resides in system

firmware that serves as the interface between the OS and the un-derlying hardware. The OS provides the virtualization layer witha set of jobs that need to run on the CMP and other meta-data(optional) that can guide Maestro in refining its schedulingpoli-cies (Section 3.2.3). Online profiling of system workloads identi-fies application-specific thermal footprints, shown in Figure 4 as avector of per-module temperatures for each application. This ther-mal footprint can either be generated by brief exploratory execu-tion of jobs on the available cores, similar to what is done in[26],or projected by correlating thermal behavior with program phases(leveraging the existing body of work on runtime phase monitoringand prediction). Given the prevalence of on-chip temperature sen-sors [13], Maestro assumes low-overhead exploration is performed

Figure 4: A high-level block diagram of the Maestro introspective reliability management system. Dynamic monitoring of sensorfeedback and detailed characterization of workload behavior enables Maestro to improve lifetime system reliability with wearout-centric scheduling.

during each scheduling interval. Coupled with the real-time healthassessments, this detailed module-level application characteriza-tion enables Maestro to create wearout-centric job schedules thatintelligently manage CMP aging.

As previously defined, scheduling in this paper will refer totheact of mapping threads to cores and is initiated by two main events,1) the OS issues new jobs for Maestro to execute (pushes into aFIFO queue) or 2) the damage profile of the underlying CMP haschanged sufficiently (taking on the order of days/weeks) to warrantthread migration. The two reliability-centric schedulingpoliciesevaluated in this work illustrate two approaches to lifetime relia-bility. The greedy policy (Section 3.2.2) takes the position thatall core failures are unacceptable and aggressively preserves eventhe weakest cores. The adaptive policy (Section 3.2.3) champi-ons a more unconventional philosophy that claims individual corefailures are tolerable provided the lifetime reliability of the CMPsystem is maximized.

Both wearout-centric policies, and the naive baseline scheduler,are presented below along with corresponding pseudocode. Unlessotherwise indicated, the following definitions are common to allpolicies: m, a microarchitectural module (i.e.,FPMul, IntReg,etc.);LiveCores, the set of functional cores in the CMP,{ c0, c1, ..., cN }; JobQueue, the set ofall pending, uncompletedjobs issued from the OS;ActiveJobs, the set of theN oldest, un-completed, jobs, {j0, j1, ..., jN }; Dmg(m), the entry in the CMPdamage profile for modulem; Temp(j,m), the entry for modulem in the temperature footprint for jobj.

3.2.1 Naive SchedulerA standard round-robin scheduler is used as the baseline pol-

icy. The least-recently-used (LRU) core in the set ofLiveCores

is assigned the oldest job from the set ofActiveJobs. This pro-cess is repeated until all jobs inActiveJobs have been sched-uled. This policy maintains high-level load balancing by distribut-ing jobs uniformly across the cores. However, without accountingfor core damage profiles or application thermal footprints,the re-sulting schedule is effectively a random mapping (from a reliabilityperspective).

Algorithm 1: Greedy wearout-centric schedulerStep 1:

foreach c ∈ LiveCores dofind cdmg , the damage present in corec , where

cdmg ←− Dmg(m′) | m′ ∈ c ∧ Dmg(m′) ≥Dmg(m), ∀m ∈ c

endsort LiveCores based oncdmg

endStep 2:

until ActiveJobs is emptycw ←− weakest core inLiveCores based oncdmg

mw ←− m′ |m′ ∈ cw ∧Dmg(m′) ≥ Dmg(m), ∀m ∈cw

foreach j ∈ ActiveJobs dofind costj,cw

, the cost of executing jobj on corecw ,where

costj,cw←− Temp(j, mw)

endjopt ←− j′ | j′ ∈ ActiveJobs ∧ costj′,cw

≤ costj,cw,

∀j ∈ ActiveJobsAssign jobjopt to corecw

Removecw from LiveCores andjopt from ActiveJobs

endend

3.2.2 Greedy SchedulerThis policy attempts to minimize the number of premature core

failures by greedily favoring the weakest cores (Algorithm1). Coresare sorted based upon their damage profiles and priority is given tothe cores whose weakest modules possess the most damage (Step 1of Algorithm 1). These “weak” cores are greedily assigned jobswith the most favorable thermal footprints with respect to theirdamage profiles (Step 2 of Algorithm 1), minimizing their effec-tive thermal stress. Thislocal wear-leveling reduces the probabil-ity that these weak cores will fail due to asingle damaged struc-ture. Scheduling the weak cores first maximizes the probability offinding jobs with favorable thermal footprints with respectto eachweak core since there is a larger application mix to choose from.However, this also forces the stronger cores to execute the remain-ing, potentially less desirable, jobs. In practice, this means that

Algorithm 2: Adaptive wearout-centric schedulerlet GA(J, C) be the optimal schedule generated by the GA for jobsJ and coresCStep 1:

foreach c ∈ LiveCores dofind cdmg , the damage present in corec , where

cdmg ←−Pc

miαiDmg(mi) andαi is a scaling factor

biased toward modules with more damage

endsort LiveCores in increasing order ofcdmg

PrimaryCores ←− first n coreswhere n is set by the userthrough the OSSecondaryCores←− remainingN − n cores

endStep 2:

let Sprimary, be the set of job-to-core assignments,(j, c), ∀c ∈PrimaryCoresSprimary ←− GA(ActiveJobs, P rimaryCores)Assign jobs forPrimaryCores according toSprimary

Remove assigned jobs fromActiveJobs

endStep 3:

let Ssecondary, be the set of job-to-core assignments,(j, c),∀c ∈ SecondaryCoresSsecondary ←− GA(ActiveJobs, SecondaryCores)Assign jobs forSecondaryCores according toSsecondary

end

the stronger cores in the CMP actually sacrifice a portion of theirlifetime to lighten the burden on their weaker counterparts(globalwear-leveling).

3.2.3 Adaptive SchedulerThe adaptive scheduler recognizes that many CMP systems are

often underutilized, provisioned with more cores than theytypi-cally have jobs to run (see Section 4.3). The scheduler exploitsthis fact by allowing a few weak cores to be sacrificed in orderto preserve the remaining stronger cores (Algorithm 2). Althoughbeing complicit in core failures may seem non-intuitive, insys-tems that are underutilized, the greedy scheduler can lead to CMPsthat are overprovisioned early in the CMP’s life (LiveCores >>

JobQueue) while not assuring enough available throughput(LiveCores < JobQueue) later on. This insight forms the basisof the adaptive policy.

Promoting a survival-of-the-fittest environment, this policy max-imizes the functional life of the strongest subset of cores(PrimaryCores in Step 1 of Algorithm 2), those with the leastamount of initial damage and the potential to have the longest life-times. By assigning jobs to thePrimaryCores first, Maestro en-sures that they execute applications with the most appropriate ther-mal footprints (Step 2 of Algorithm 2). The remaining jobs areassigned amongst theSecondaryCores (Step 3 of Algorithm 2).This can lead to some weak cores failing sooner than under a greedypolicy. Note, however, in Step 3 of Algorithm 2, the scheduler isstill looking amongst the remaining jobs for the one with thebestthermal footprint given a core’s damage profile. Thislocal wear-leveling, common to both the greedy and adaptive policies, ensuresthat the weaker cores even under the adaptive policy survivelongerthan they would under the naive policy. Ultimately, over thelife-time of the CMP, ifPrimaryCores ≥ JobQueue consistently,while avoiding periods whenPrimaryCores >> JobQueue orPrimaryCores < JobQueue, then Maestro has maximized thetotal amount of computation performed by the system. The propersize ofPrimaryCores, n, is exposed to the OS so that the be-

havior of the scheduler can be customized to the needs of the enduser.

Finally, note in Step 2 and Step 3 of Algorithm 2, the scheduleruses an optimization scheme based on a genetic algorithm (GA) toidentify the least-cost schedules for both thePrimaryCores andSecondaryCores. This allows the adaptive scheduler to considerthe effect scheduling a job has on all structures within a core (unlikethe greedy scheduler which only looks at the weakest structure) formore effectivelocal wear-leveling. The optimization used in thiswork is derived from [8], a standard solution of the generalizedassignment problem, and is described below1.

Chromosome definition: The chromosome modeled is a job-to-core mapping of a set ofn jobs,J = {j0, j1, ..., jn}, to a set ofmcoresC = {c0, c1, ..., cn}. It is represented as a one-dimensionalarray where the value stored at indexi, ji, is the job that has beenassigned to corei. The example in Figure 5 has jobsj1 mapped tocore 0,jn−1 mapped to core 1, andj0 mapped to corem. DuringStep 2of the adaptive scheduling algorithmn > m, while for theoptimization performed inStep 3m = n.

Figure 5: Chromosome structure

Cost function: The cost function used by the GA is recalculated ateach scheduling interval, based on the CMP damage profile andap-plication thermal footprints, according to Equation 1, whereCost(S)= the cost of scheduleS andCost(j, c) = the cost of scheduling jobj on corec.

Cost(S) =S

X

j,c

Cost(j, c)

=

SX

j,c

“

cX

m

Dmg(m) · Temp(j,m)”

(1)

The individual steps of the GA are enumerated below:

1. Generate initial population: An initial population of solu-tions (schedules) is created by randomly enumerating a sub-set of the possible job-to-core mappings.

2. Evaluate fitness:Calculate the fitness (cost) of all membersof the population using Equation 1.

3. Reproduction: Two parents are identified, each using a sim-ple binary tournament where two candidates are selected ran-domly from the population and the one with the best fitness(smallest cost) is chosen for reproduction (Figure 6(a)). Achild is generated by applying a one-point crossover operatoron the parent chromosomes, where a random crossover pointi ∈ [0, m] is selected, wherem is the size of the chromo-somes. The child chromosome is formed by combining thefirst i genes from one parent with the lastm − i genes fromthe second parent (Figure 6(b)) . Note that this newly formedchromosome could have the same job assigned to two dif-ferent cores. For this case to arise there must also be a set ofjobsJ ′ ⊂ J that are unassigned sincen ≥ m. To resolve the

1The runtime overhead of the GA is negligible for long-running scientificand server workloads. However, for shorter-running applications the GAoptimization can be replaced by a greedy version without severely impact-ing the effectiveness of the adaptive scheduler.

(a) Parent selection (b) Crossover operation

(c) Conflict resolution (d) Mutation

Figure 6: Steps involved in reproduction.S0, S1, S2, S3 are the parental candidates.Sc is the resulting child chromosome after initialcrossover.S′

c and S′′

c are the states of the child chromosome after conflicts resolution and mutation respectively.

conflicts, one of the redundant cores (selected at random) isreassigned a job fromJ ′ based onCost(j, c) (Figure 6(c)).Lastly, the newly formed child chromosome is mutated bytaking 2 randomly selected job assignments and swappingthem (no risk of creating new conflicts), reducing the proba-bility of converging at local optima (Figure 6(d)) .

4. Replace and Repeat:After a child solution is formed theweakest member, as defined by the cost function, of the ex-isting population is replaced by the new child. This con-cludes a single generation in the evolutionary cycle. Theprocess is repeated until a predetermined number of gener-ations,genmax, fails to produce an improved solution.2

4. EVALUATION AND ANALYSISThis section evaluates Maestro’s reliability-centric scheduling

policies using lifetime reliability simulations. A variety of systemparameters including CMP size and system utilization are varied toinvestigate their impact on Maestro’s performance. The effective-ness of each wearout-centric policy is measured in terms oflife-time throughput (LT), the number of cycles spent executing activejobs (real applications not idle threads), summed across all cores,throughout the entire lifetime of the CMP. LT improvement metricsare the result of comparisons with the naive, round-robin schedulerpresented in Section 3.2.1. Monte Carlo experiments are conductedusing a simulation setup similar to the framework in [12]. The stan-dard toolchain of SimAlpha, Wattch [5], and Hotspot [20] is usedto simulate the thermal characteristics of workloads and Varius [19]is used to model the impact of process variation. Results presentedin this section, unless otherwise indicated, are for a 16-core CMPwith processors modeled after the DEC Alpha 21264/21364 [1].

Given that CMPs have lifespans on the order of years (3-5 yearsin future computer systems [11]), detailed lifetime reliability sim-ulation is a computationally intensive task. This is especially true

2Given the size of the solution space, as many as16! possible schedulesfor a 16-core CMP, values ofgenmax from 0 to 100,000 were studied tounderstand the tradeoff between optimality and runtime. The actual valuesof genmax used in Section 4 were determined empirically based on theCMP size, with many runs producing good results withgenmax as low as1000.

when large numbers of Monte Carlo runs must be conducted to gen-erate statistically significant results. Since wearout damage takesyears to reach critical mass, results presented in this section weregathered using anadaptive simulation scheme. Short periods of de-tailed system-level reliability simulation, the darker phases in Fig-ure 7(a), are used to gather statistics on the progression ofCMPaging in light of dynamically changing workload streams andMae-stro’s reliability-centric scheduling. The simulation isthen rapidlyadvanced through a longer period of time (accelerated simulation)using the statistics generated during the most recent detailed phaseas a guide. To minimize error, the length of the accelerated simula-tion phase is limited by the amount of damage accumulated duringthe detailed interval according to Equation 2:

La = (Dfail

Dacc

) · AF · Ld (2)

where,La = length of the accelerated phase,Ld = length ofthe previous detailed phase,Dfail = amount of damage the weak-est core in the CMP can sustain before failing,Dacc = amount ofdamage accumulated by the weakest core during the previous de-tailed phase, andAF = parameter that trades off simulation timefor accuracy (0%-100%).

Dynamically adjusting the durations allows simulation to slowdown as cores near their failing point, where small changes in dam-age and scheduling decisions have larger implications. When acore fails in phasei, accelerated simulation resumes at a faster rate(Lai+1

> Lai), butLa soon contracts as the next core in the CMP

nears failure. Figure 7(a) illustrates (not to scale) how adjustingAF can influence the lengths of the accelerated phases. The valueof AF essentially dictates the number of detailed phases that aresimulated between core failures. At anAF of 100%, simulationsare accelerated from one core failure to the next. However, whenAF is dialed down to 50%, many more phases are required to coverthe same amount of simulated time, concentrating simulation effortaround times when cores are failing and improving simulation ac-curacy. Figure 7(b) shows both simulation time speedup and erroras a function ofAF , illustrating how simulation time can be tradedoff for fidelity. The experiments presented in this work use an AF

of 6%, resulting in simulation runtimes from 30 minutes to over 6hours for a single set of Monte Carlo runs.

(a) Interleaving of detailed and accelerated simulation phases.

(b) Simulation time/error v. acceleration factor (AF).

Figure 7: The adaptive simulation used to accelerate lifetimereliability simulations while incurring minimal experime ntalerror.

Figure 8: Performance of wearout-centric scheduling policiesverses CMP size and failure threshold.

4.1 Lifetime Throughput EnhancementFigure 8 shows the normalized LT improvement as a function

of the scheduling policy, CMP size, and failure threshold. In thecontext of this paper, failure threshold is defined as the numberof cores that must fail before a chip is considered unusable.Thisis the point at which the risks/costs associated with maintaininga system with only a fraction of its original computational capac-ity justifies replacing the chip. The CMP is considered dead eventhough functional cores still remain. The results shown in Figure 8are conducted for 2 to 16-core systems, and failure thresholds rang-ing from 1 core to all cores. The value of the failure threshold ispassed to the adaptive policy so that it can optimize for the appro-priate number of cores. Results are shown for CMP utilizationsof 100%, providing a lower-bound on the benefits of the adaptivepolicy (Section 4.3 examines the impact of CMP utilization).

As expected, both the greedy and adaptive policies perform wellacross all CMP sizes and the majority of failure thresholds.Asthe size of the CMP grows, Maestro has more cores to work with,increasing the chances of finding complementary job-to-core map-pings. This results in more effective schedules for both wearout-centric policies improving their performance. Yet even with the

(a) Failure distribution (Core)

(b) CMP failure distribution (CMP)

Figure 9: Failure distributions for individual cores and th e 16-core CMP with a failure threshold of 8 cores and 100% uti-lization. Trendlines are added (between markers) to improvereadability.

lack of scheduling alternatives in a 2-core system, both policiescan still achieve a respectable 30% improvement.

A strong dependence on failure threshold is also evident. Byag-gressively minimizing premature core failures, the greedysched-uler achieves large gains for small failure thresholds. However, asthe failure threshold nears the size of the CMP, the LT improve-ment attenuates. This is expected since under the greedy policy,stronger cores sacrifice a portion of their lifetime in orderto pre-serve their weaker counterparts. The cost of this sacrifice is mostapparent when the failure threshold allows all the cores to fail. Inthese systems, the increased contribution toward LT by the weakcores is offset by the loss in LT resulting from the strong coresfailing earlier. Notice also that the adaptive scheduler outperformsgreedy by the largest margins when the failure threshold is roughlyhalf the size of the CMP. In these situations, the adaptive sched-uler has the maximum freedom to sacrificeSecondaryCores topreservePrimaryCores (Section 3.2.3). At either extreme forfailure threshold, it performs similarly to greedy.

Lastly, it is important to note that, although the benefits ofwearout-centric scheduling are less impressive for these extreme values offailure threshold, the scenarios when a user could actuallyafford towait for all the cores within a system to fail are also quite remote.For the remainder of the paper, all the experiments shown arefor a16-core CMP with a failure threshold of 8 cores and 100% systemutilization unless otherwise indicated.

4.2 Failure DistributionsFigure 9 presents the failure distributions for the individual cores,

as well as the CMPs that correspond to the results in Figure 8.Figure 9(a) illustrates the effectiveness of the wearout-centric poli-cies at distributing the workload stress appropriately. The distri-

Figure 10: Impact of CMP utilization on reliability enhance-ment.

bution for the baseline naive policy reveals a bias towards earlypremature core failures. The greedy scheduler, exploitingeffec-tive wear-leveling, produced a tighter distribution, lacking in bothpremature failures as well as cores that significantly outlasted theirpeers. Lastly, the adaptive policy also delivers on its promises bypreserving a subset of cores for a longer period of time than eitherthe naive or greedy schedulers.

Figure 9(b) tells a similar story, but with chip-level failures. Aswith the individual core distributions, both wearout-centric policiesare able to increase the mean failure time of the CMP population.Note that because the failure time of a CMP is limited by the weak-est set of its constituent cores, the distributions in Figure 9(b) areconsiderably tighter than those in Figure 9(a). The correspondingtables of expected lifetimes embedded within the plots present thedata slightly differently. From a product yield/warranty perspec-tive, intelligent wearout-centric scheduling can be thought of as anadditional means of ensuring that cores meet their expectedreli-ability qualified lifetimes. For example, the table in Figure 9(b)shows that the adaptive scheduler enabled 99% of the chips tosur-vive beyond1.9 years, compared to just1.4 years with the naivebaseline, a 38% improvement. Granted, job assignment alonecan-not makeguarantees on lifetime, but it can complement existingmore aggressive techniques like thermal throttling.

4.3 Sensitivity to System UtilizationThe utilization of computer systems can be highly variable,both

within the same domain (e.g., variability inside data centers) andacross domains. One might expect computationally intensive sci-entific codes (e.g., physics simulations, oil exploration,etc.) to con-sistently utilize the hardware. On the other hand, since designersbuild web servers to accommodate peak loads (periodic by season,day, and hour), they are often over-provisioned for the commoncase. Some reports claim average utilization as low as 20% ofpeak [2].

Figure 10 plots the performance of Maestro’s wearout-centricschedulers as a function of system utilization. The resultsare shownfor nominal utilizations ranging from 20% (light duty mail server orembedded system) to 100% (scientific cluster)3. Note that initiallyas average utilization drops, improvement in lifetime throughputactually increases. A system that is slightly underutilized can bemore aggressively load balanced since some cores are allowed toremain idle. However, as utilization continues to drop these gainsare eventually lost, until finally improvements are actually worse

3Although the mean utilization per simulation run is fixed, the instanta-neous utilization experienced by the CMP is allowed to vary over time,sometimes peaking at 100% even for a system nominally at 20% load. Fur-thermore, the averageeffective utilization is also changing as cores on theCMP begin to fail.

(a)

(b)

Figure 11: Sensitivity to sensor noise. Although random sensornoise can be removed with the appropriate filtering, systematicerror due to manufacturing tolerances is more problematic.

than at full utilization. In these highly over-provisionedsystems,the efforts of wearout-centric scheduling to prevent premature fail-ures arepartially wasted because so few cores are actually neces-sary to sustain demand. Nevertheless, in the long run, the periodicspikes in utilization do accumulate, and thanks to the longer overallcore lifetimes (lower utilization means less overall stress that trans-lates to longer lifetimes), the greedy and adaptive schedulers stillmanage to exhibit improvements.

4.4 Sensitivity to Sensor NoiseFigure 11 illustrates how error-prone sensors could impactlife-

time reliability gains. Although the introduction of systematic er-ror, which is studied in Figure 11(b), does reduce the potential ofwearout-centric scheduling, the presence of random noise (morecommon for circuit-level sensors) shown in Figure 11(a) canbe ac-counted for and mitigated by the statistical filtering and trend anal-ysis schemes referenced in Section 3.1. Yet, even at the extreme of+/-15% systematic error, Maestro still achieves over 10% LTim-provement. Figure 11(b) also suggests that the adaptive scheduleris more sensitive to noise than the greedy scheduler. By aggres-sively trying to preservePrimaryCores, the adaptive heuristicrelies strongly on sensor feedback to accurately identify the bound-ary between its two classes of processors, making it less robustagainst sensor inaccuracy.

4.5 Sensor SelectionLastly, Figure 12 presents a comparison between the low-level

damage sensors advocated in this work and more conventionalhard-ware like temperature sensors and performance counters. Giventhat Maestro is targeting an environment with significant amountsof process variation, it is not surprising that employing temperatureand activity readings as proxies for wearout/manufacturing induceddamage is inadequate. They are unable to account for the extent to

Figure 12: Performance of wearout-centric scheduling withdif-ferent sensors. Results are shown for a failure threshold of1core to favor the temperature sensor and access counter basedapproaches.

which non-uniform, pre-existing damage within the CMP respondsto the same thermal stimuli. In the absence of variation, a sched-uler relying on only temperature might effectively enhancelifetimereliability by evenly distributing the thermal stress across the CMP.However, without any knowledge of CMP damage profiles, as pro-cess variation is swept from one extreme (no variation) to the other(100% expected variation at 32nm), thermal load balancing aloneis insufficient and Figure 12 shows a dramatic plunge in the effec-tiveness of these temperature based schemes. Similarly, the perfor-mance counter approach performed poorly across the spectrum ofvariation.

5. CONCLUSIONAs large CMP systems grow in popularity and technology scal-

ing continues to exacerbate lifetime reliability challenges, the re-search community must develop innovative ways for systems todynamically adapt. Although issues like process variationare thesource of design and validation nightmares, this inherent hetero-geneity in future systems is also a source of potential opportu-nity. Maestro recognizes that although emerging reliability obsta-cles cannot be ignored, with the appropriate monitoring andintelli-gent management, they can be overcome. By exploiting low-levelsensor feedback, Maestro was able to demonstrate the effective-ness of wearout-centric scheduling at preventing premature corefailures, improving expected CMP lifetimes by as much as 38%.Formulating wearout-centric schedules that achieved bothlocal andglobal wear-leveling, Maestro enhanced the lifetime throughput ofa 16-core CMP by as much as 180%. Future work that leveragessensor feedback to improve upon other traditional reliability man-agement mechanisms (e.g., DVFS) could demonstrate still morepotential.

6. REFERENCES[1] Alpha. 21364 family, 2001.

http://www.alphaprocessors.com/21364.htm.[2] A. Andrzejak, M. Arlitt, and J. Rolia. Bounding the resource

savings of utility computing models, Dec. 2002. HPLaboratories, http://www.hpl.hp.com/techreports/2002/HPL-2002-339.html.

[3] J. Blome, S. Feng, S. Gupta, and S. Mahlke. Self-calibratingonline wearout detection. InProc. of the 40th AnnualInternational Symposium on Microarchitecture, pages109–120, 2007.

[4] S. Borkar. Designing reliable systems from unreliable

components: The challenges of transistor variability anddegradation.IEEE Micro, 25(6):10–16, 2005.

[5] D. Brooks, V. Tiwari, and M. Martonosi. A framework forarchitectural-level power analysis and optimizations. InProc.of the 27th Annual International Symposium on ComputerArchitecture, pages 83–94, June 2000.

[6] A. Cabe, Z. Qi, S. Wooters, T. Blalock, and M. Stan. Smallembeddable nbti sensors (sens) for tracking on-chipperformance decay. Washington, DC, USA, Mar. 2009. IEEEComputer Society.

[7] J. Choi, C. Cher, , H. Franke, H. Haman, A. Wedger, andP. Bose. Thermal-aware task scheduling at the systemsoftware level. InProc. of the 2007 International Symposiumon Low Power Electronics and Design, pages 213–218, Aug.2007.

[8] P. C. Chu and J. E. Beasley. A genetic algorithm for thegeneralised assignment problem. 24(1):17–23, 1997.

[9] J. Donald and M. Martonosi. Techniques for multicorethermal management: Classification and new exploration. InProc. of the 33rd Annual International Symposium onComputer Architecture, June 2006.

[10] D. Ernst, S. Das, S. Lee, D. Blaauw, T. Austin, T. Mudge,N. S. Kim, and K. Flautner. Razor: Circuit-level correctionof timing errors for low-power operation. InProc. of the 37thAnnual International Symposium on Microarchitecture,pages 10–20, 2004.

[11] C. Evangs-Pughe. Live fast, die young [nanometer-scale iclife expectancy].IEE Review, 50(7):34–37, 2004.

[12] S. Feng, S. Gupta, and S. Mahlke. Olay: Combat the signs ofaging with intropsective reliability management. InProc. ofthe Workshop on Architectural Reliability, June 2008.

[13] J. Friedrich et al. Desing of the power6 microprocessor, Feb.2007. InProc. of ISSCC.

[14] S. Gupta, S. Feng, A. Ansari, J. Blome, and S. Mahlke. Thestagenet fabric for constructing resilient multicore systems.In Proc. of the 41st Annual International Symposium onMicroarchitecture, pages 141–151, 2008.

[15] X. Li, B. Huang, J. Qin, X. Zhang, M. Talmor, Z. Gur, andJ. B. Bernstein. Deep submicron cmos integrated circuitreliability simulation with spice. InProc. of the 2005International Symposium on Quality of Electronic Design,pages 382–389, Mar. 2005.

[16] Z. Lu, J. Lach, M. R. Stan, and K. Skadron. Improvedthermal management with reliability banking.IEEE Micro,25(6):40–49, Nov. 2005.

[17] M. Powell, M. Gomaa, and T. Vijaykumar. Heat-and-run:Leveraging smt and cmp to manage power density throughthe operating system. In12th International Conference onArchitectural Support for Programming Languages andOperating Systems, pages 260–270, Oct. 2004.

[18] D. Roberts, R. Dreslinski, E. Karl, T. Mudge, D. Sylvester,and D. Blaauw. When homogeneous becomesheterogeneous: Wearout aware task scheduling for streamingapplications. InProc. of the Workshop on Operationg SystemSupport for Heterogeneous Multicore Architectures, Sept.2007.

[19] S. Sarangi, B. Greskamp, R. Teodorescu, J. Nakano,A. Tiwari, and J. Torrellas. Varius: A model of processvariation and resulting timing errors for microarchitects. InIEEE Transactions on Semiconductor Manufacturing, pages3–13, Feb. 2008.

[20] K. Skadron, M. R. Stan, K. Sankaranarayanan, W. Huang,S. Velusamy, and D. Tarjan. Temperature-awaremicroarchitecture: Modeling and implementation.ACMTransactions on Architecture and Code Optimization,1(1):94–125, 2004.

[21] J. Srinivasan, S. V. Adve, P. Bose, and J. A. Rivers. The casefor lifetime reliability-aware microprocessors. InProc. of the31st Annual International Symposium on ComputerArchitecture, pages 276–287, June 2004.

[22] D. Sylvester, D. Blaauw, and E. Karl. Elastic: An adaptiveself-healing architecture for unpredictable silicon.IEEEJournal of Design and Test, 23(6):484–490, 2006.

[23] R. Teodorescu and J. Torrellas. Variation-aware applicationscheduling and power management for chip multiprocessors.In Proc. of the 35th Annual International Symposium onComputer Architecture, pages 363–374, June 2008.

[24] A. Tiwari, S. Sarangi, and J. Torrellas. Recycle: Pipelineadaptation to tolerate process variation. InProc. of the 34thAnnual International Symposium on Computer Architecture,pages 323–334, June 2007.

[25] A. Tiwari and J. Torrellas. Facelift: Hiding and slowingdown aging in multicores. InProc. of the 41st AnnualInternational Symposium on Microarchitecture, pages129–140, Dec. 2008.

[26] J. Winter and D. Albonesi. Scheduling algorithms forunpredictably heterogeneous cmp architectures. InProc. ofthe 2008 International Conference on Dependable Systemsand Networks, page To appear, June 2008.