metareasoning for planning under uncertaintymetareasoning for planning under uncertainty christopher...

Metareasoning for Planning Under Uncertainty

Christopher H. Lin ∗ Andrey Kolobov Ece Kamar Eric HorvitzUniversity of Washington Microsoft Research

Seattle, WA Redmond, [email protected] {akolobov,eckamar,horvitz}@microsoft.com

AbstractThe conventional model for online planning underuncertainty assumes that an agent can stop and planwithout incurring costs for the time spent planning.However, planning time is not free in most real-world settings. For example, an autonomous droneis subject to nature’s forces, like gravity, even whileit thinks, and must either pay a price for counteract-ing these forces to stay in place, or grapple with thestate change caused by acquiescing to them. Pol-icy optimization in these settings requires metarea-soning—a process that trades off the cost of plan-ning and the potential policy improvement that canbe achieved. We formalize and analyze the metar-easoning problem for Markov Decision Processes(MDPs). Our work subsumes previously studiedspecial cases of metareasoning and shows that inthe general case, metareasoning is at most polyno-mially harder than solving MDPs with any givenalgorithm that disregards the cost of thinking. Forreasons we discuss, optimal general metareason-ing turns out to be impractical, motivating approx-imations. We present approximate metareasoningprocedures which rely on special properties of theBRTDP planning algorithm and explore the effec-tiveness of our methods on a variety of problems.

1 IntroductionOffline probabilistic planning approaches, such as policy iter-ation [Howard, 1960], aim to construct a policy for every pos-sible state before acting. In contrast, online planners, such asRTDP [Barto et al., 1995] and UCT [Kocsis and Szepesvari,2006], interleave planning with execution. After an agenttakes an action and moves to a new state, these planners sus-pend execution to plan for the next step. The more planningtime they have, the better their action choices. Unfortunately,planning time in online settings is usually not free.

Consider an autonomous Mars rover trying to decide whatto do while a sandstorm is nearing. The size and uncertaintyof the domain precludes a-priori computation of a completepolicy, and demands the use of online planning algorithms.

∗Research was performed while the author was an intern at Mi-crosoft Research.

Normally, the longer the rover runs its planning algorithm,the better decision it can make. However, computation costspower; moreover, if it reasons for too long without takingpreventive action, it risks being damaged by the oncomingsandstorm. Or consider a space probe on final approach to aspeeding comet, when the probe must plan to ensure a safelanding based on new information it gets about the comet’ssurface. More deliberation time means a safer landing. Atthe same time, if the probe deliberates for too long, the cometmay zoom out of range — a similarly undesirable outcome.

Scenarios like these give rise to a general metareasoningdecision problem: how should an agent trade off the costof planning and the quality of the resulting policy for thebase planning task every time it needs to make a move, soas to optimize its long-term utility? Metareasoning aboutbase-level problem solving has been explored for probabilis-tic inference and decision making [Horvitz, 1987; Horvitz etal., 1989], theorem proving [Horvitz and Klein, 1995; Kautzet al., 2002], handling streams of problems [Horvitz, 2001;Shahaf and Horvitz, 2009], and search [Russell and Wefald,1991; Burns et al., 2013]. There has been little work explor-ing generalized approaches to metareasoning for planning.

We explore the general metareasoning problem for Markovdecision processes (MDPs). We begin by formalizing theproblem with a general but precise definition that subsumesseveral previously considered metareasoning models. Then,we show with a rigorous theoretical analysis that optimal gen-eral metareasoning for planning under uncertainty is at mostpolynomially harder than solving the original planning prob-lem with any given MDP solver. However, this increase incomputational complexity, among other reasons we discuss,renders such optimal general metareasoning impractical. Theanalysis raises the issue of allocating time for metareason-ing itself, and leads to an infinite regress of meta∗reasoning(metareasoning, metametareasoning, etc.) problems.

We next turn to the development and testing of fast ap-proximate metareasoning algorithms. Our procedures usethe Bounded RTDP (BRTDP [McMahan et al., 2005]) algo-rithm to tackle the base MDP problem, and leverage BRTDP-computed bounds on the quality of MDP policies to reasonabout the value of computation. In contrast to prior workon this topic, our methods do not require any training data,precomputation, or prior information about target domains.We perform a set of experiments showing the performanceof these algorithms versus baselines in several synthetic do-

arX

iv:1

505.

0039

9v1

[cs

.AI]

3 M

ay 2

015

mains with different properties, and characterize their perfor-mance with a measure that we call the metareasoning gap— a measure of the potential for improvement from metarea-soning. The experiments demonstrate that the proposed tech-niques excel when the metareasoning gap is large.

2 Related WorkMetareasoning efforts to date have employed strategies thatavoid the complexity of the general metareasoning prob-lem for planning via relying on different kinds of simpli-fications and approximations. Such prior studies includemetareasoning for time-critical decisions where expectedvalue of computation is used to guide probabilistic inference[Horvitz, 1987; Horvitz et al., 1989], and work on the guid-ing of sequences of single actions in search [Russell andWefald, 1991; Burns et al., 2013]. Several lines of workhave leveraged offline learning [Breese and Horvitz, 1990;Horvitz et al., 2001; Kautz et al., 2002]. Other studies haverelied on optimizations and inferences that leverage the struc-ture of problems, such as the functional relationships betweenmetareasoning and reasoning [Horvitz and Breese, 1990;Zilberstein and Russell, 1996], the structure of the prob-lem space [Horvitz and Klein, 1995], and the structure ofutility [Horvitz, 2001]. In other work, [Hansen and Zil-berstein, 2001] proposed a non-myopic dynamic program-ming solution for single-shot problems. Finally, several plan-ners rely on a heuristic form of online metareasoning whenmaximizing policy reward under computational constraints inreal-world time with no “conversion rate” between the two[Kolobov et al., 2012; Keller and Geißer, 2015]. In con-trast, our metareasoning model is unconstrained, with com-putational and base-MDP costs in the same “currency.”

Our investigation also has connections to research on al-locating time in a system composed of multiple sensingand planning components [Zilberstein and Russell, 1996;Zilberstein and Russell, 1993], on optimizing portfolios ofplanning strategies in scheduling applications [Dean et al.,1995], and on choosing actions to explore in Monte Carloplanning [Hay et al., 2012]. In other related work, [Chanelet al., 2014] consider how best to plan on one thread, while aseparate thread processes execution.

3 PreliminariesA key contribution of our work is formalizing the metarea-soning problem for planning under uncertainty. We buildon the framework of stochastic shortest path (SSP) MDPswith a known start state. This general MDP class includesfinite-horizon and discounted-reward MDPs as special cases[Bertsekas and Tsitsiklis, 1996], and can also be used to ap-proximate partially observable MDPs with a fixed initial be-lief state. An SSP MDP M is a tuple 〈S,A, T,C, s0, sg〉,where S is a finite set of states, A is a set of actions that theagent can take, T : (S,A, S) → [0, 1] is a transition func-tion, C : (S,A) → R is a cost function, s0 ∈ S is the startstate, and sg is the goal state. An SSP MDP must have acomplete proper policy, a policy that leads to the goal fromany state with probability 1, and all improper policies mustaccumulate infinite cost from every state from which they failto reach the goal with a positive probability. The objective is

to find a Markovian policy πs0 : S → A with the minimumexpected cost of reaching the goal from the start state s0 —in SSP MDPs, at least one policy of this form is globally op-timal.

Without loss of generality, we assume an SSP MDP tohave a specially designated NOP (“no-operation”) action.NOP is an action the agent chooses when it wants to “idle”and “think/plan”, and its semantic meaning is problem-dependent. For example, in some MDPs, choosing NOPmeans staying in the current state for one time step, while inothers it may mean allowing a tidal wave to carry the agent toanother state. Designating an action as NOP does not changeSSP MDPs’ mathematical properties, but plays a crucial rolein our metareasoning formalization.

4 Formalization and Theoretical Analysis ofMetareasoning for MDPs

The online planning problem of an agent, which involveschoosing an action to execute in any given state, is repre-sented as an SSP MDP that encapsulates the dynamics of theenvironment and costs of acting and thinking. We call thisproblem the base problem. The agent starts off in this en-vironment with some default policy, which can be as simpleas random or guided by an unsophisticated heuristic. Theagent’s metareasoning problem, then, amounts to deciding,at every step during its interaction with the environment, be-tween improving its existing policy or using this policy’s rec-ommended action while paying a cost for executing either ofthese options, so as to minimize its expected cost of gettingto the goal.

Besides the agent’s state in the base MDP, which we callthe world state, the agent’s metareasoning decisions are con-ditioned on the algorithm the agent uses for solving the baseproblem, i.e., intuitively, on the agent’s thinking process.To abstract away the specifics of this planning algorithmfor the purposes of metareasoning formalization, we view itas a black-box MDP solver and represent it, following theChurch-Turing thesis, with a Turing machine B that takes abase SSP MDP M as input. In our analysis, we assume thefollowing about Turing machine B’s operation:

• B is deterministic and halts on every valid base MDPM . This assumption does not affect the expressivenessof our model, since randomized Turing machines can betrivially simulated on deterministic ones, e.g., via seedenumeration (although potentially at an exponential in-crease in time complexity). At the same time, it greatlysimplifies our theorems.• An agent’s thinking cycle corresponds to B executing a

single instruction.• A configuration of B is a combination of B’s tape con-

tents, state register contents, head position, and next in-put symbol. It represents the state of the online plannerin solving the base problem M . We denote the set of allconfigurations B ever enters on a given input MDP Mas XB(M). We assume that B can be paused after ex-ecuting y instructions, and that its configuration at thatpoint can be mapped to an action for any world state sof M using a special function f : S ×XB(M) → A in

time polynomial in M ’s flat representation. The numberof instructions needed to compute f is not counted intoy. That is, an agent can stop thinking at any point andobtain a policy for its current world state.• An agent is allowed to “think” (i.e., execute B’s instruc-

tions) only by choosing the NOP action. If an agent de-cides to resume thinking after pausingB and executing afew actions, B re-starts from the configuration in whichit was last paused.

We can now define metareasoning precisely:Definition 1. Metareasoning Problem. Consider an MDPM = 〈S,A, T,C, s0, sg〉 and an SSP MDP solver repre-sented by a deterministic Turing machine B. Let XB(M) bethe set of all configurations B enters on input M , and letTB(M) : XB(M) × XB(M) → {0, 1} be the (deterministic)transition function of B on XB(M). A metareasoning prob-lem for M with respect to B, denoted MetaB(M) is an MDP〈Sm, Am, Tm, Cm, sm0 , smg 〉 s.t.

• Sm = S ×XB(M)

• Am = A

• Tm((s, χ), a, (s′, χ′))

=

T (s, a, s′) if a 6= NOP, χ = χ′, and a = f(s, χ)

T (s, a, s′) · TB(M)(χ, χ′) if a = NOP0 otherwise

• Cm((s, χ), a, (s′, χ′)) = C(s, a, s′) if T (s, a, s′) 6= 0,and 0 otherwise• sm0 = (s0, χ0), where χ0 is the first configuration B

enters on input M

• smg = (sg, χ), where χ is any configuration in XB(M)

Solving the metareasoning problem means finding a policyfor MetaB(M) with the lowest expected cost of reaching smg .

This definition casts a metareasoning problem for abase MDP as another MDP (a meta-MDP). Note that inMetaB(M), an agent must choose either NOP or an actioncurrently recommended by B(M); in other cases, the tran-sition probability is 0. Thus, MetaB(M)’s definition essen-tially forces an agent to switch between two “meta-actions”:thinking or acting in accordance with the current policy.

Modeling an agent’s reasoning process with a Turing ma-chine allows us to see that at every time step the metarea-soning decision depends on the combination of the currentworld state and the agent’s “state of mind,” as captured by theTuring machine’s current configuration. In principle, this de-cision could depend on the entire history of the two, but thefollowing theorem implies that, as forM , at least one optimalpolicy for MetaB(M) is always Markovian.Theorem 1. If the base MDP M is an SSP MDP, thenMetaB(M) is an SSP MDP as well, provided that B haltson M with a proper policy. If the base MDP M is an infinite-horizon discounted-reward MDP, then so is MetaB(M).If the base MDP M is a finite-horizon MDP, then so isMetaB(M).

Proof. Verifying the result for finite-horizon and infinite-horizon discounted-reward MDPs M is trivial, since the onlyrequirement MetaB(M) must satisfy in these cases is to havea finite horizon or a discount factor, respectively.

If M is an SSP MDP, then, per the SSP MDP definition[Bertsekas and Tsitsiklis, 1996], to ascertain the theorem’sclaim we need to verify that (1) MetaB(M) has at least oneproper policy and (2) every improper policy in MetaB(M)accumulates an infinite cost from some state.

To see why (1) is true, recall that MetaB(M)’s state spaceis formed by all configurations Turing machine B enters onM . Consider any state (s′0, χ

′0) of MetaB(M). Since B is

deterministic, as stated in Section 3, the configuration χ′0 liesin the linear sequence of configurations between the “des-ignated” initial configuration χ0 and the final proper-policyconfiguration that B enters according to the theorem. Thus,B can reach a proper-policy configuration from χ′0. There-fore, let the agent starting in the state (s′0, χ

′0) of MetaB(M)

choose NOP until B halts, and then follow the proper policycorresponding toB’s final configuration until it reaches a goalstate sg of M . This state corresponds to a goal state (sg, χ)of MetaB(M). Since this construct works for any (s′0, χ

′0),

it gives a complete proper policy for MetaB(M).To verify (2), consider any policy πm for MetaB(M) that

with a positive probability fails to reach the goal. Any infinitetrajectory of πm that fails to reach the goal can be mappedonto a trajectory in M that repeats the action choices of πm’strajectory in M ’s state space S. Since M is an SSP MDP,this projected trajectory must accumulate an infinite cost, andtherefore the original trajectory in MetaB(M) must do so aswell, implying the desired result.

We now present two results to address the difficulty ofmetareasoning.

Theorem 2. For an SSP MDP M and a deterministic Turingmachine B representing a solver for M , the time complexityof MetaB(M) is at most polynomial in the time complexityof executing B on M .

Proof. The main idea is to construct the MDP representingMetaB(M) by simulating B on M . Namely, we can run Bon M until it halts and record every configuration B enters toobtain the setX . GivenX , we can construct Sm = S×X andall other components of MetaB(M) in time polynomial in|X| and |M |. ConstructingX itself takes time proportional torunning time of B on M . Since, by Theorem 1, MetaB(M)is an SSP MDP and hence can be solved in time polynomialin the size of its components, e.g., by linear programming, theresult follows.

Theorem 3. Metareasoning for SSP MDPs is P -completeunder NC-reduction. (Please see the appendix for proof.)

At first glance, the results above look encouraging. How-ever, upon closer inspection they reveal several subtletiesmaking optimal metareasoning utterly impractical. First, al-though both SSP MDPs and their metareasoning counterpartswith respect to an optimal polynomial-time solver are in P ,doing metareasoning for a given MDPM is appreciably more

expensive than solving that MDP itself. Given that the addi-tional complexity due to metareasoning cannot be ignored,the agent now faces the new challenge of allocating computa-tional time between metareasoning and planning for the baseproblem. This challenge is a meta-metareasoning problem,and ultimately causes infinite regress, an unbounded nestedsequence of ever-costlier reasoning problems.

Second, constructing MetaB(M) by running B on M ,as the proof of Theorem 2 proceeds, may entail solving Min the process of metareasoning. While the proof doesn’tshow that this is the only way of constructing MetaB(M),without making additional assumptions about B’s operationone cannot exclude the possibility of having to run B untilconvergence and thereby completely solving M even beforeMetaB(M) is fully formulated. Such a construction woulddefeat the purpose of metareasoning.

Third, the validity of Theorems 2 and 3 relies on an im-plicit crucial assumption that the transitions of solver B onthe base MDP M are known in advance. Without this knowl-edge, MetaB(M) turns into a reinforcement learning prob-lem [Sutton and Barto, 1998], which further increases thecomplexity of metareasoning and the need for simulating Bon M . Neither of these is viable in reality.

The difficulties with optimal metareasoning motivate thedevelopment of approximation procedures. In this regard, thepreceding analysis provides two important insights. It sug-gests that, since running B on M until halting is infeasible, itmay be worth trying to predict B’s progress on M . Many ex-isting MDP algorithms have clear operational patterns, e.g.,evaluating policies in the decreasing order of their cost, aspolicy iteration does [Howard, 1960]. Regularities like thesecan be of value in forecasting the benefit of running B on Mfor additional cycles of thinking. We now focus on exploringapproximation schemes that can leverage these patterns.

5 Algorithms for Approximate MetareasoningOur approach to metareasoning is guided by value of com-putation (VOC) analysis. In contrast to previous work thatformulates V OC for single actions or decision-making prob-lems [Horvitz, 1987; Horvitz et al., 1989; Russell and Wefald,1991], we aim to formulate V OC for online planning. For agiven metareasoning problem MetaB(M), V OC at any en-countered state sm = (s, χ) is exactly the difference betweenthe Q-value of the agent following f(s, χ) (the action recom-mended by the current policy of the base MDP M ) and theQ-value of the agent taking NOP and thinking:

V OC(sm) = Q∗(sm, f(s, χ))−Q∗(sm,NOP). (1)

V OC captures the difference in long-term utility betweenthinking and acting as determined by these Q-values. Anagent should take the NOP action and think when the V OC ispositive. Our technique aims to evaluate V OC by estimatingQ∗(sm, f(s, χ)) and Q∗(sm,NOP). However, attempting toestimate these terms in a near-optimal manner ultimately runsinto the same difficulties as solving MetaB(M), such as sim-ulating the agent’s thinking process many steps into the fu-ture, and is likely infeasible. Therefore, fast approximationsfor the Q-values will generally have to rely on simplifyingassumptions. We rely on performing greedy metareasoning

analysis as has been done in past studies of metareasoning[Horvitz et al., 1989; Russell and Wefald, 1991]:Meta-Myopic Assumption. In any state sm of the meta-MDP,we assume that after the current step, the agent will neveragain choose NOP, and hence will never change its policy.

This meta-myopic assumption is important in allowing usto reduce V OC estimation to predicting the improvement inthe value of the base MDP policy following a single thinkingstep. The weakness of this assumption is that opportunitiesfor subsequent policy improvements are overlooked. In otherwords, the V OC computation only reasons about the currentthinking opportunity. Nonetheless, in practice, we computeV OC at every timestep, so the agent can still think later. Ourexperiments show that our algorithms perform well in spiteof their meta-myopicity.

5.1 Implementing Metareasoning with BRTDPWe begin the presentation of our approximation scheme withthe selection of B, the agent’s thinking algorithm. Since ap-proximating Q∗(sm, f(s, χ)) and Q∗(sm,NOP) essentiallyamounts to assessing policy values, we would like an on-line planning algorithm that provides efficient policy valueapproximations, preferably with some guarantees. Havingaccess to these policy value approximations enables us to de-sign approximate metareasoning algorithms that can evaluateV OC efficiently in a domain-independent fashion.

One algorithm with this property is Bounded RTDP(BRTDP) [McMahan et al., 2005]. It is an anytime plan-ning algorithm based on RTDP [Barto et al., 1995]. LikeRTDP, BRTDP maintains a lower bound on an MDP’s op-timal value function V ∗, which is repeatedly updated viaBellman backups as BRTDP simulates trials/rollouts to thegoal, making BRTDP’s configuration-to-configuration transi-tion function TB(M)(χ, χ′) stochastic. A key difference isthat in addition to maintaining a lower bound, it also main-tains an upper bound, updated in the same conceptual wayas the lower one. If BRTDP is initialized with a mono-tone upper-bound heuristic, then the upper-bound decreasesmonotonically as BRTDP runs. The construction of domain-independent monotone bounds is beyond the scope of thispaper, but is easy for the domains we study in our experi-ments. Another key difference between BRTDP and RTDPis that if BRTDP is stopped before convergence, it returnsan action greedy with respect to the upper, not lower bound.This behavior guarantees that the expected cost of a policy re-turned at any time by a monotonically-initialized BRTDP isno worse than BRTDP’s current upper bound. Our metarea-soning algorithms utilize these properties to estimate V OC.In the rest of the discussion, we assume that BRTDP is ini-tialized with a monotone upper-bound heuristic.

5.2 Approximating VOCWe now show how BRTDP’s properties help us with estimat-ing the two terms in the definition of V OC, Q∗(sm, f(s, χ))and Q∗(sm,NOP). We first assume that one “thinking cycle”of BRTDP (i.e., executing NOP once and running BRTDP inthe meantime, resulting in a transition from BRTDP’s cur-rent configuration χ to another configuration χ′) correspondsto completing some fixed number of BRTDP trials from theagent’s current world state s.

Estimating Q∗(sm,NOP)We first describe how to estimate the value of taking the NOPaction (thinking). At the highest level, this estimation firstinvolves writing down an expression for Q∗(sm,NOP), mak-ing a series of approximations for different terms, and thenmodeling the behavior of how BRTDP’s upper bounds on theQ-value function drop in order to compute the needed quan-tities.

When opting to think by choosing NOP, the agent may tran-sition to a different world state while simultaneously updatingits policy for the base problem. Therefore, we can expressQ∗(sm,NOP) =∑

s′

T (s,NOP, s′)∑χ′

TB(M)(χ, χ′)V ∗((s′, χ′)). (2)

Because of meta-myopicity, we have V ∗((s′, χ′)) = V χ′(s′)

where V χ′ is the value function of the policy correspond-ing to χ′ in the base MDP. However, this expression can-not be efficiently evaluated in practice, since we do not knowBRTDP’s transition distribution TB(M)(χ, χ′) nor the statevalues V χ

′(s′), forcing us to make further approximations.

To do so, we assume V χ′

and Qχ′

are random variables, andrewrite

∑χ′ TB(M)(χ, χ′)V χ

′(s′) =∑

a

P (Aχ′

s′ = a)E[Qχ′(s′, a)|Aχ

′

s′ = a]. (3)

where the random variableAχ′

s′ takes value a iff f(s′, χ′) = a

after one thinking cycle in state (s, χ). Intuitively, P (Aχ′

s′ =a) denotes the probability that BRTDP will recommend ac-tion a in state s′ after one thinking cycle. Now, let us denotethe Q-value upper bound corresponding to BRTDP’s currentconfiguration χ as Q

χ. This value is known. Then, let the up-

per bound corresponding to BRTDP’s next configuration χ′,

be Qχ′

. Because we do not know χ′, this value is unknown,and is a random variable. Because BRTDP selects actionsgreedily w.r.t. the upper bound, we follow this behavior anduse the upper bound to estimate Q-value by assuming that

Qχ′

= Qχ′

. Since the value of Qχ′

is unknown at the time of

the V OC computation, P (Aχ′

s′ = a) andE[Qχ′

(s′, a)|Aχ′

s′ =a] are computed by integrating over the possible values of

Qχ′

. We have that E[Qχ′

(s′, a)|Aχ′

s = a] =

∫Qχ′

(s′,a)

Qχ′

(s′, a)P (Aχ

′

s′ = a|Qχ′

(s′, a))P (Qχ′

(s′, a))

P (Aχ′

s′ = a),

and P (Aχ′

s′ = a) =∫Qχ′

(s′,a)

P (Qχ′

(s′, a))∏ai 6=a

P (Qχ′

(s′, ai) > P (Qχ′

(s′, a)).

Therefore, we must model the distribution that Qχ′

is drawn

from. We do so by modeling the change ∆Q = Qχ − Qχ

′

,due to a single BRTDP thinking cycle that corresponds to atransition from configuration χ to χ′. Since Q

χis known and

fixed, estimating a distribution over possible ∆Q gives us a

distribution over Qχ′

.Let ∆Qs,a be the change in Qs,a resulting from the most

recent thinking cycle for some state s and action a. We firstassume that the change resulting from an additional cycleof thinking, ∆Qs,a, will be no larger than the last change,∆Qs,a ≤ ∆Qs,a. This assumption is reasonable, becausewe can expect the change in bounds to decrease as BRTDPconverges to the true value function. Given this assumption,we must choose a distribution D over the interval [0, ∆Qs,a]

such that for the next thinking cycle, ∆Qs,a ∼ D. Figure 1aillustrates these modeling assumptions for two hypotheticalactions, a1 and a2.

One option is to make D uniform, so as to represent ourpoor knowledge about the next bound change. Then, com-puting P (Aχ

′

s′ = a) involves evaluating an integral of a poly-nomial of degree O(|A|) (the product of |A| − 1 CDF’s and

1 PDF), and computing E[Qχ′

(s′, a)|Aχ′

s = a] also entailsevaluating an integral of degree O(|A|), and thus computingthese quantities for all actions in a state can be computed intime O(|A|2). Since the overall goal of this subsection, ap-proximating Q∗(sm,NOP), requires computing P (Aχ

′

s′ = a)for all actions in all states where NOP may lead, assumingthere are no more than K << |A| such states, the complexitybecomes O(K|A|2) for each state visited by the agent on itsway to the goal.

A weakness of this approach is that the changes in the up-per bounds for different actions are modeled independently.For example, if the upper bounds for two actions in a givenstate decrease by a large amount in the previous thinking step,then it is unlikely that in the next thinking step one of themwill drop dramatically while the other drops very little. Thisindependence can cause the amount of uncertainty in the up-per bound at the next thinking step to be overestimated, lead-ing to V OC being overestimated as well.

Therefore, we create another version of the algorithm as-suming that the speed of decrease in Q-value upper boundsfor all actions are perfectly correlated; all ratios between fu-ture drops in the next thinking cycle are equal to the ratiosbetween the observed drops in the last thinking cycle. For-mally, for a given state s, we let ρ ∼ Uniform[0, 1]. Then, let∆Qs,a = ρ · ∆Qs,a for all actions a.

Now, to compute P (Aχ′

s′ = a), for each action a, we repre-

sent the range of its possible future Q-values Qχ′

s,a with a linesegment la on the unit interval [0,1] where la(0) = Q

χ

s,a and

la(1) = Qχ

s,a−∆Qs,a. Then, P (Aχ′

s′ = a) is simply the pro-portion of la which lies below all the other lines representingall other actions. We can naıvely compute these probabilitiesin time O(|A|2) by enumerating all intersections. Similarly,

computing E[Qχ′

(s′, a)|Aχ′

s = a] is also easy. This value isthe mean of the portion of la that is beneath all other lines.Figure 1b illustrates these computations.

Whether or not we make the assumption of action inde-pendence, we further speed up the computations by only cal-

culating E[Qχ′

(s′, a)|Aχ′

s = a] and P (Aχ′

s′ = a) for the two

Figure 1: a) Hypothetical drops in upper bounds on the Q-values of two actions, a1 and a2. We assume the next Q-value drop resulting from another cycle of thinking, ∆Q, isdrawn from a range equal to the last drop from thinking, ∆Qb) Assuming perfect correlation in the speed of decrease inthe Q-value upper bounds, as the upper bounds of the twoactions drop from an additional cycle of thinking, initially a2has a better upper bound, but eventually a1 overtakes a2.

“most promising” actions a, those with the lowest expectationof potential upper bounds. This limits the computation timeto the time required to determine these actions (linear in |A|),and makes the time complexity of estimating Q∗(sm,NOP)for one state s be O(K|A|) instead of O(K|A|2).

Estimating Q∗(sm, f(s, χ))Now that we have described how to estimate the value of tak-ing the NOP action, we describe how to estimate the valueof taking the currently recommended action, f(s, χ). We

estimate Q∗(sm, f(s, χ)) by computing E[Qχ′

(s, f(s, χ))],which takes constant time, keeping the overall time complex-ity linear. The reason we estimate Q∗(sm, f(s, χ)) using fu-ture Q-value upper bound estimates based on a probabilis-tic projection of χ′, as opposed to our current Q-value up-per bounds based on the current configuration χ, is to makeuse of the more informed bounds derived at the future utilityestimation. As the BRTDP algorithm is given more compu-tation time, it can more accurately estimate the upper boundof a policy. This type of approximation has been justifiedbefore [Russell and Wefald, 1991]. In addition, using fu-ture utility estimates in both estimating Q∗(sm, f(s, χ)) andQ∗(sm,NOP) provides a consistency guarantee: if thinkingleads to no policy change, then our method estimates V OCto be zero.

5.3 Putting It All TogetherThe core of our algorithms involves the computations wehave described, in every state s the agent visits on theway to the goal. In the experiments, we denote UnCorrMetareasoner as the metareasoner that assumes the ac-tions are uncorrelated, and Metareasoner as the metarea-soner that does not make this assumption. To complete thealgorithms, we ensure that they decide the agent should thinkfor another cycle if ∆Qs,a isn’t yet available for the agent’scurrent world state s (e.g., because BRTDP has never updated

bounds for this state’s Q-value so far), since V OC compu-tation is not possible without prior observations on ∆Qs,a.Crucially, all our estimates make metareasoning take timeonly linear in the number of actions, O(K|A|), per visitedstate.

6 ExperimentsWe evaluate our metareasoning algorithms in several syn-thetic domains designed to reflect a wide variety of factorsthat could influence the value of metareasoning. Our goal isto demonstrate the ability of our algorithms to estimate thevalue of computation and adapt to a plethora of world condi-tions. The experiments are performed on four domains, all ofwhich are built on a 100×100 grid world, where the agent canmove between cells at each time step to get to the goal locatedin the upper right corner. To initialize the lower and upperbounds of BRTDP, we use the zero heuristic and an appropri-ately scaled (multiplied by a constant) Manhattan distance tothe goal, respectively.

6.1 DomainsThe four world domains are as follows:• Stochastic. This domain adds winds to the grid world

to be analogous to worlds with stochastic state transi-tions. Moving against the wind causes slower movementacross the grid, whereas moving with the wind results infaster movement. The agent’s initial state is the south-east corner and the goal is located in the northeast cor-ner. We set the parameters of the domain as follows sothat there is a policy that can get the agent to the goalwith a small number of steps (in tens instead of hun-dreds) and where the winds significantly influence thenumber of steps needed to get to the goal: The agentcan move 11 cells at a time and the wind has a push-ing power of 10 cells. The next location of the agent isdetermined by adding the agent’s vector and the wind’svector except when the agent decides to think (executesNOP), in which case it stays in the same position. Thus,the winds can never push the agent in the opposite direc-tion of its intention. The prevailing wind direction overmost of the grid is northerly, except for the column ofcells containing the goal and starting position, where it issoutherly. Note that this southerly wind direction makesthe initial heuristic extremely suboptimal. To simulatestochastic state transitions, the winds have their prevail-ing direction in a given cell with 60% probability; with40% probability they have a direction orthogonal to theprevailing one (20% easterly and 20% westerly).We perform a set of experiments on this simplest do-main of the set, to observe the effect of different costsfor thinking and acting on the behaviors of algorithms.We vary the cost of thinking and acting between 1 and15. When we vary the cost of thinking, we fix the costof acting at 11, and when we vary the cost of acting, wefix the cost of thinking at 1.• Traps. This domain modifies the Stochastic domain to

resemble the setting where costs for thinking and act-ing are not constant among states. To simplify the pa-rameter choices, we fix the cost of thinking and acting

to be equal, respectively, to the agent’s moving distanceand wind strength. Thus, the cost of thinking is 10 andthe cost of acting is 11. To vary the costs of thinkingand acting between states, we make thinking and act-ing at the initial state extremely expensive at a cost of100, about 10 times the cost of acting and thinking inthe other states. Thus, the agent is forced to think out-side its initial state in order to perform optimally.

• DynamicNOP-1. In the previous domains, executing aNOP does not change the agent’s state. In this domain,thinking causes the agent to move in the direction of thewind, causing the agent to stochastically transition as aresult of thinking. In this domain, the cost of thinkingis composed of both explicit and implicit components;a static value of 1 unit and a dynamic component deter-mined by stochastic state transitions as a result of think-ing. The static value is set to 1 so that the dynamic com-ponent can dominate the decisions about thinking. Theagent starts in cell (98, 1). We change the wind direc-tions so that there are easterly winds in the most southernrow and northerly winds in the most eastern row that canpush the agent very quickly to the goal. Westerly windsexist everywhere else, pushing the agent away from thegoal. We change the stochasticity of the winds so thatthe westerly winds change to northerly winds with 20%probability, and all other wind directions are no longerstochastic. We lower the amount of stochasticity to bet-ter see if our agents can reason about the implicit costsof thinking. The wind directions are arranged so thatthere is potential for the agent to improve upon its initialpolicy but thinking is risky as it can move the agent tothe left region, which is hard to recover from since allthe winds push the agent away from the goal.

• DynamicNOP-2. This domain is just like the previousdomain, but we change the direction of the winds in thenorthern-most row to be easterly. These winds also donot change directions. In this domain, as compared tothe previous one, it is less risky to take a thinking ac-tion; even when the agent is pushed to the left region ofthe board, the agent can find strategies to get to the goalquickly by utilizing the easterly wind at the top regionof the board.

6.2 The Metareasoning GapWe introduce the concept of the metareasoning gap as away to quantify the potential improvement over the initialheuristic-implied policy, denoted as Heuristic, that ispossible due to optimal metareasoning. The metareasoninggap is the ratio of the expected cost of Heuristic for thebase MDP to the expected cost of the optimal metareasoningpolicy, computed at the initial state. Exact computation of themetareasoning gap requires evaluating the optimal metarea-soning policy and is infeasible. Instead, we compute an upperbound on the metareasoning gap by substituting the cost ofthe optimal metareasoning policy with the cost of the optimalpolicy for the base MDP (denoted OptimalBase). Themetareasoning gap can be no larger than this upper bound,because metareasoning can only add cost to OptimalBase.We quantify each domain with this upper bound (MGUB)

in Table 1 and show that our algorithms for metareasoningprovide significant benefits when MGUB is high. We notethat none of the algorithms use the metareasoning gap in itsreasoning.

Heuristic OptimalBase MGUB

Stochastic (Thinking) 1089 103.9 10.5Stochastic (Acting) 767.3 68.1 11.3Traps 979 113.5 8.6DynamicNOP-1 251.4 66 3.8DynamicNOP-2 119.4 66 1.8

Table 1: Upper bounds of metareasoning gaps (MGUB) forall test domains, defined as the ratio of the expected cost ofthe initial heuristic policy (Heuristic) to that of an optimalone (OptimalBase) at the initial state.

6.3 Experimental SetupWe compare our metareasoning algorithms against a numberof baselines. The Think∗Act baseline simply plans for ncycles at the initial state and then executes the resulting pol-icy, without planning again. We also consider the Prob base-line, which chooses to plan with probability p at each state,and executes its current policy with probability 1− p. An im-portant drawback of these baselines is that their performanceis sensitive to their parameters n and p, and the optimal pa-rameter settings vary across domains. The NoInfoThinkbaseline plans for another cycle if it does not have informa-tion about how the BRTDP upper bounds will change. Thisbaseline is a simplified version of our algorithms that doesnot try to estimate the V OC.

For each experimental condition, we run each metareason-ing algorithm until it reaches the goal 1000 times and averagethe results to account for stochasticity. Each BRTDP trajec-tory is 50 actions long.

6.4 ResultsIn Stochastic, we perform several experiments by varying thecosts of thinking (NOP) and acting. We observe (figures canbe found in the appendix) that when the cost of thinkingis low or when the cost of acting is high, the baselines dowell with high values of n and p, and when the costs are re-versed, smaller values do better. This trend is expected, sincelower thinking cost affords more thinking, but these baselinesdon’t allow for predicting the specific “successful” n and pvalues in advance. Metareasoner does not require pa-rameter tuning and beats even the best performing baselinefor all settings. Figure 2a compares the metareasoning al-gorithms against the baselines when the results are averagedover the various settings of the cost of acting, and Figure 2bshows results averaged over the various settings of the cost ofthinking. Metareasoner does extremely well in this do-main because the metareasoning gap is large, suggesting thatmetareasoning can improve the initial policy significantly.Importantly, we see that Metareasoner performs betterthan NoInfoThink, which shows the benefit from reason-ing about how the bounds on Q-values will change. UnCorrMetareasoner does not do as well as Metareasoner,

Figure 2: Comparison of Metareasoner and Uncorr Metareasoner with baselines on experimental domains. Somefigures do not include Heuristic when it performs especially poorly for readability.

probably because the assumption that actions’ Q-values areuncorrelated does not hold well.

We now turn to Traps, where thinking and acting in the ini-tial state incurs significant cost. Figure 2c again summarizesthe results. Think∗Act performs very poorly, because it islimited to thinking only at the initial state. Metareasonerdoes well, because it figures out that it should not thinkin the initial state (beyond the initial thinking step), andshould instead quickly move to safer locations. UncorrMetareasoner also closes the metareasoning gap signif-icantly, but again not as much as Metareasoner.

We now consider DynamicNOP-1, a domain adversarialto approximate metareasoning, because winds almost every-where push the agent away from the goal. There are onlya few locations from which winds can carry the agent tothe destination. Figure 2d shows that our algorithms do notachieve large gains. However, this result is not surprising.The best policy involves little thinking, because wheneverthe agent chooses to think, it is pushed away from the goal,and thinking for just a few consecutive time steps can takethe agent to states where reaching the goal is extremely dif-ficult. Consequently, Think∗Act with 1-3 thinking stepsturns out to be near-optimal, since it is pushed away from thegoal only slightly and can use a slightly improved heuristicto head back. Metareasoner actually does well in manyindividual runs, but occasionally thinks longer due to V OCcomputation stochasticity and can get stuck, yielding higheraverage policy cost. In particular, it may frequently be pushedinto a state that it has never encountered before, where it mustthink again because it does not have any history about howBRTDP’s bounds have changed in that state, and then subse-

quently get pushed into an unencountered state again. In thisdomain, our approximate algorithms can diverge away froman optimal policy, which would plan very little to minimizethe risk of being pushed away from the goal.

DynamicNOP-2 provides the agent more opportunitiesto recover even if it makes a poor decision. Figure 2edemonstrates that our algorithms perform much better inDynamicNOP-2 than in DynamicNOP-1. In DynamicNOP-2,even if our algorithms do not discover the jetstreams that canpush it towards the goal from initial thinking, they are pro-vided more chances to recover when they get stuck. Whenthinking can move the agent on the board, having more op-portunities to recover reduces the risk associated with mak-ing suboptimal thinking decisions. Interestingly, the metar-easoning gap is decreased at the initial state by the additionof the extra jetstream. However, the metareasoning gap atmany other states in the domain is increased, showing thatthe metareasoning gap at the initial state is not the most idealway to characterize the potential for improvement via metar-easoning in all domains.

7 Conclusion and Future WorkWe formalize and analyze the general metareasoning problemfor MDPs, demonstrating that metareasoning is only polyno-mially harder than solving the base MDP. Given the determi-nation that optimal general metareasoning is impractical, weturn to approximate metareasoning algorithms, which esti-mate the value of computation by relying on bounds given byBRTDP. Finally, we empirically compare our metareasoningalgorithms to several baselines on problems designed to re-flect challenges posed across a spectrum of worlds, and show

that the proposed algorithms are much better at closing largemetareasoning gaps.

We have assumed that the agent can plan only when it takesthe NOP action. A generalization of our work would allowvarying amounts of thinking as part of any action. Some ac-tions may consume more CPU resources than others, and ac-tions which do not consume all resources during executioncan allocate the remainder to planning. We also can relax themeta-myopic assumption, so as to consider the consequencesof thinking for more than one cycle. In many cases, assumingthat the agent will only think for one more step can lead to un-derestimation of the value of thinking, since many cycles ofthinking may be necessary to see significant value. This abil-ity can be obtained with our current framework by projectingchanges in bounds for multiple steps. However, in experi-ments to date, we have found that pushing out the horizonof analysis was associated with large accumulations of errorsand poor performance due to approximation upon approxi-mation from predictions about multiple thinking cycles. Fi-nally, we may be able to improve our metareasoners by learn-ing about and harnessing more details about the base-levelplanner. In our Metareasoner approximation scheme, wemake strong assumptions about how the upper bounds pro-vided by BRTDP will change, but learning distributions overthese changes may improve performance. More informedmodels may lead to accurate estimation of non-myopic valueof computation. However, learning distributions in a domain-independent manner is difficult, since the planner’s behavioris heavily dependent on the domain and heuristic at hand.

References[Barto et al., 1995] Andrew G. Barto, Steven J. Bradtke, and

Satinder P. Singh. Learning to act using real-time dynamicprogramming. Artificial Intelligence, 72:81–138, 1995.

[Bertsekas and Tsitsiklis, 1996] Dimitri P. Bertsekas andJohn Tsitsiklis. Neuro-dynamic Programming. AthenaScientific, 1996.

[Breese and Horvitz, 1990] John S. Breese and Eric Horvitz.Ideal reformulation of belief networks. In UAI, 1990.

[Burns et al., 2013] Ethan Burns, Wheeler Ruml, andMinh B. Do. Heuristic search when time matters. Jour-nal of Artificial Intelligence Research, 47:697–740, 2013.

[Chanel et al., 2014] Caroline P. Carvalho Chanel, CharlesLesire, and Florent Teichteil-Konigsbuch. A robotic exe-cution framework for online probabilistic (re)planning. InICAPS, 2014.

[Dean et al., 1995] Thomas Dean, Leslie Pack Kaelbling,Jak Kirman, and Ann Nicholson. Planning under timeconstraints in stochastic domains. Artificial Intelligence,76:35–74, 1995.

[Hansen and Zilberstein, 2001] Eric A Hansen and ShlomoZilberstein. Monitoring and control of anytime algorithms:A dynamic programming approach. Artificial Intelligence,126(1):139–157, 2001.

[Hay et al., 2012] Nick Hay, Stuart Russell, David Toplin,and Solomon Eyal Shimony. Selecting computations: The-ory and applications. In UAI, 2012.

[Horvitz and Breese, 1990] Eric J. Horvitz and John S.Breese. Ideal partition of resources for metareasoning.Technical Report KSL-90-26, Stanford University, 1990.

[Horvitz and Klein, 1995] Eric Horvitz and Adrian Klein.Reasoning, metareasoning, and mathematical truth: Stud-ies of theorem proving under limited resources. In UAI,1995.

[Horvitz et al., 1989] Eric J. Horvitz, Gregory F. Cooper, andDavid E.Heckerman. Reflection and action under scarceresources: Theoretical principles and empirical study. InIJCAI, 1989.

[Horvitz et al., 2001] Eric Horvitz, Yongshao Ruan, Carla P.Gomes, Henry Kautz, Bart Selman, and David M. Chick-ering. A bayesian approach to tackling hard computationalproblems. In UAI, 2001.

[Horvitz, 1987] Eric Horvitz. Reasoning about beliefs andactions under computational resource constraints. In UAI,1987.

[Horvitz, 2001] Eric Horvitz. Principles and applications ofcontinual computation. Artificial Intelligence, 126:159–196, 2001.

[Howard, 1960] R.A. Howard. Dynamic Programming andMarkov Processes. MIT Press, 1960.

[Kautz et al., 2002] Henry Kautz, Eric Horvitz, YongshaoRuan, Carla Gomes, and Bart Selman. Dynamic restartpolicies. In AAAI, 2002.

[Keller and Geißer, 2015] Thomas Keller and FlorianGeißer. Better be lucky than good: Exceeding expecta-tions in mdp evaluation. In AAAI, 2015.

[Kocsis and Szepesvari, 2006] Levente Kocsis and CsabaSzepesvari. Bandit based monte-carlo planning. In ECML,2006.

[Kolobov et al., 2012] Andrey Kolobov, Mausam, andDaniel S. Weld. Lrtdp vs uct for online probabilisticplanning. In AAAI, 2012.

[McMahan et al., 2005] H. Brendan McMahan, MaximLikhachev, and Geoffrey J. Gordon. Bounded real-time dynamic programming: Rtdp with monotone upperbounds and performance guarantees. In ICML, 2005.

[Russell and Wefald, 1991] Stuart Russell and Eric Wefald.Principles of metareasoning. Artificial intelligence,49(1):361–395, 1991.

[Shahaf and Horvitz, 2009] Dafna Shahaf and Eric Horvitz.Ivestigations of continual computation. In IJCAI, 2009.

[Sutton and Barto, 1998] Richard S. Sutton and Andrew G.Barto. Introduction to Reinforcement Learning. MITPress, 1998.

[Zilberstein and Russell, 1993] Shlomo Zilberstein and Stu-art J. Russell. Anytime sensing, planning and action: Apractical model for robot control. In IJCAI, 1993.

[Zilberstein and Russell, 1996] Shlomo Zilberstein and Stu-art Russell. Optimal composition of real-time systems. Ar-tificial Intelligence, 82:181–213, 1996.

Figure 3: The construction of a base MDP M ′ for a metarea-soning problem from an input SSP MDP M .

A AppendixA.1 Proof of Theorem 3Proof. By calling metareasoning P -complete we mean thatthere exists a Turing machine B s.t. (1) for any input SSPMDP M ′, MetaB(M ′) can be decided in time polynomialin |M ′|, i.e., MetaB(M) is in P , and (2) there is a class ofP -complete problems that can be converted to MetaB(M ′)via an NC-reduction, i.e., by constructing M ′ appropriatelyusing a polynomial number of parallel Turing machines, eachoperating in polylogarithmic time.

The first part of the above claim follows from Theorem 2:since SSP MDPs are solvable optimally by linear program-ming in polynomial time, MetaB(M) is in P if B encodes apolynomial solver for linear programs.

For the second part, we perform an NC-reduction from theclass of SSP MDPs to the class of SSP MDPs-based metarea-soning problems with respect to a fixed optimal polynomial-time solver B. Specifically, given an SSP MDP M with aninitial state, we show how to construct another SSP MDPM ′s.t., for the optimal polynomial-time solver B we describeshortly, deciding MetaB(M ′) is equivalent to deciding M .

The intuition behind converting a given SSP MDP M intoM ′, the SSP MDP that will serve as the base in our metarea-soning problem, is to augment M with new states where theagent can “think” by using a zero-cost NOP action until theagent arrives at an optimal policy for the original states ofM .Afterwards, the agent can transition from any of these newlyadded “thinking states” to M ’s original start state s0 and ex-ecute the optimal policy from there. Unfortunately, the proofis not as straightforward as it seems, because we cannot sim-ply build M ’ by equipping M with a new start state s′0 witha self-loop zero-cost NOP action — M ′ with such an actionwould violate the SSP MDP definition. Below, we show howto overcome this difficulty. Since thinking in the newly addedstates of M ′ costs nothing, the cost of an optimal policy forMetaB(M ′) is the same as for M , so deciding the formerproblem decides the latter.

The construction of M ′ from a given SSP MDP M is illus-trated in Figure 3. Consider the number of instruction-stepsit takes to solve M by linear programming. This number ispolynomial; namely, there exists a polynomial pLP (|M |) thatbounds M ’s solution time from above. To transform M intoM ′, we add a set of pLP (|M |) states, s′0, s

′1, . . . , s

′pLP (|M |) to

M . These new states connect into a chain via zero-cost NOP

actions: the start state s′0 of M ′ links to s′1, s′1 links to s′2, andso on until s′pLP (|M |) links to s0, the start state of M . In ad-dition, for all original states of M , we create a self-loop NOPaction with a positive cost. The entire transformation can beeasily implemented as an NC-reduction on pLP (|M |) + |S|computers, each recording the cost and transition function ofNOP for a separate state. Since for each state, NOP’s cost andtransition functions together can be encoded by just two num-bers (NOP transition function assigns probability 1 to a singletransition that is implicitly but unambiguously determined forevery state), each computer operates in polylogarithmic time.Moreover, initializing each of the parallel machines with theMDP state for which it is supposed to write out the transitionand cost function values is as simple as appropriately setting apointer to the input tape, and can be done in log-space. Thus,the above procedure is a valid NC-reduction. Note also thatM ′ is an SSP MDP: although it has zero-cost actions, they donot form loops/strongly connected components.

Our motivation for constructing M ′ as above was to pro-vide an agent with enough states where it can “think” to guar-antee that if the agent starts at s′0, it arrives atM ’s initial states0 with a computed optimal policy from s0 onwards. Thiswould imply that the expected cost of an optimal policy forMetaB(M ′) from s0 would be the same as for M . How-ever, for this guarantee to hold, we need a general SSP MDPsolverB that can solve/decideM ′ in timeO(poly(|M |)), notO(poly(|M ′|). The difference between O(poly(|M ′|)) andO(poly(|M |)) is very important, because M ′ is larger thanM , so the newly added chain of states may not be enough fora O(poly(|M ′|)) policy computation to have zero cost.

To circumvent this issue, we define B that recognizes“lollypop-shaped” MDPs M ′ as in Figure 3, which have anarbitrarily connected subset Sc of the state space representinga sub-MDP M c preceded by a chain of NOP-connected statesof size pLP (|M c|) leading to M c’s start state s0, and ignoresthe linear chain part. (Note that the policy for the linear chainpart is determined uniquely and there is no need to write it outexplicitly). For that, we assume the metareasoning problem’sinput SSP MDP to be in the form of a string

M c description###chain descriptionIn this string,M c description stands for the arbitrarily con-

nected part of the input MDP, and chain description stands forthe description of the linear NOP-connected chain. For MDPsviolating conditions in Figure 3 (i.e., having a different con-nectivity structure or having the linear part of the wrong size),chain description must be empty, with the entire MDP de-scription placed before “###”. B is defined to read that inputstring only up to “###” and solve that part using LP.

Constructing M ′ from M and recording M ′ in the afore-mentioned way ensures that the optimal policy for the metar-easoning problem MetaB(M ′) chooses NOP until the agentreaches M ’s start state s0, by which point the agent will havecomputed an optimal policy for M . Coupled with the factthat MetaB(M ′) is in P , this implies the theorem’s claim.

A.2 More FiguresFigures 4 through 11 show results for the Stochastic domainwhere we vary the cost of thinking and the cost of acting.

2 4 6 8 10 12 14 16 18Number of Thinking Cycles n for Think*Act Baseline

120

140

160

180

200

220

240

260

280

300

Avera

ge C

ost

to t

he G

oal

0.2 0.3 0.4 0.5 0.6 0.7 0.8Probability p for Prob Baseline

Think*Act

Metareasoner

UnCorr Metareasoner

NoInfoThink

Prob

Figure 4: Comparison of algorithms in Stochastic, with thecost of thinking = 1 and cost of acting = 11


180

200

220

240

260

280

300

320

340

Avera

ge C

ost

to t

he G

oal


Think*Act

Metareasoner

UnCorr Metareasoner

NoInfoThink

Prob



200

250

300

350

400

450

500

550

Avera

ge C

ost

to t

he G

oal


Think*Act

Metareasoner

UnCorr Metareasoner

NoInfoThink

Prob

Figure 6: Comparison of algorithms in Stochastic, with thecost of thinking =10 and cost of acting = 11


200

300

400

500

600

700

800

Avera

ge C

ost

to t

he G

oal


Think*Act

Metareasoner

UnCorr Metareasoner

NoInfoThink

Prob



20

40

60

80

100

120

140

Avera

ge C

ost

to t

he G

oal


Think*Act

Metareasoner

UnCorr Metareasoner

NoInfoThink

Prob

Figure 8: Comparison of algorithms in Stochastic, with thecost of acting = 1 and cost of thinking = 1


60

80

100

120

140

160

180

200

Avera

ge C

ost

to t

he G

oal


Think*Act

Metareasoner

UnCorr Metareasoner

NoInfoThink

Prob



100

120

140

160

180

200

220

240

Avera

ge C

ost

to t

he G

oal


Think*Act

Metareasoner

UnCorr Metareasoner

NoInfoThink

Prob



160

180

200

220

240

260

280

300

320

340

Avera

ge C

ost

to t

he G

oal


Think*Act

Metareasoner

UnCorr Metareasoner

NoInfoThink

Prob


metareasoning for planning under uncertaintymetareasoning for planning under uncertainty christopher...

Documents