Planning Under Continuous Time and Resource Uncertainty:A Challenge for AI

John Bresina, Richard Dearden�, Nicolas Meuleau�,Sailesh Ramakrishnan�, David Smith and Rich Washington�

NASA Ames Research CenterMail stop 269–2

Moffet Field, CA 94035–1000, USA

Abstract

We outline a class of problems, typical of Marsrover operations, that are problematic for cur-rent methods of planning under uncertainty. Theexisting methods fail because they suffer fromone or more of the following limitations: 1) theyrely on very simple models of actions and time,2) they assume that uncertainty is manifested indiscrete action outcomes, 3) they are only prac-tical for very small problems. For many realworld problems, these assumptions fail to hold.In particular, when planning the activities for aMars rover, none of the above assumptions isvalid: 1) actions can be concurrent and have dif-fering durations, 2) there is uncertainty concern-ing action durations and consumption of contin-uous resources like power, and 3) typical dailyplans involve on the order of a hundred actions.This class of problems may be of particular in-terest to the UAI community because both clas-sical and decision-theoretic planning techniquesmay be useful in solving it. We describe the roverproblem, discuss previous work on planning un-der uncertainty, and present a detailed, but verysmall, example illustrating some of the difficul-ties of finding good plans.

1 THE PROBLEM

Consider a rover operating on the surface of Mars. On agiven day, there are a number of different scientific obser-vations or experiments that the rover could perform, andthese are prioritized in some fashion (each observation or

� Research Institute for Advanced Computer Science (RI-ACS)

�QSS Group Inc.�QSS Group Inc.�RIACS

experiment is assigned a scientific value). Different ob-servations and experiments take differing amounts of timeand consume differing amounts of power and data storage.There are, in general, a number of constraints that governthe rovers activities:

� There are time, power, data storage, and posi-tioning constraints for performing different activi-ties. Time constraints often result from illuminationrequirement—that is, experiments may require that atarget rock or sample be illuminated with a certain in-tensity, or from a certain angle.

� Experiments have setup conditions (preconditions)that must hold before they can be performed. For ex-ample, the rover will usually need to be at a particularlocation and orientation for each experiment and willneed instruments turned on, initialized, and calibrated.In general, there may be multiple ways of achievingsome of these setup conditions (e.g. different travelroutes, different choice of cameras).

� The amount of power available varies according to thetime of day, since solar flux is a function of the angleof the sun.

Given these constraints, the objective is to maximize scien-tific return for the rover—that is, find the plan with maxi-mal utility. Unfortunately, for many rover activities, there isinherent uncertainty about the duration of tasks, the powerrequired, the data storage necessary, the position and orien-tation of the rover, and environmental factors that influenceoperations, e.g., soil characteristics, dust on the solar pan-els, ambient temperature, etc.

For example, in driving from one location to another, theamount of time required depends on wheel slippage andsinkage, which varies depending on slope, terrain rough-ness, and soil characteristics. All of these factors also in-fluence the amount of power that is consumed. The amountof energy collected by the solar panels during this traversedepends on the length of the traverse, but also on the an-

gle of the solar panels. This is dictated by the slope androughness of the terrain.

Similarly, for certain types of instruments, temperature af-fects the signal to noise ratio and, hence, affects the amountof time required to collect useful data. Since the tempera-ture varies depending on the time of day and the weatherconditions, this duration is uncertain. The amount of powerused depends upon the duration of the data collection. Theamount of data storage required depends on the effective-ness of the data compression techniques, which ultimatelydepends on the nature of the data collected.

In short, this domain is rife with uncertainty. Plans that donot take this uncertainty into account usually fail miserably.In fact, it has been estimated that the 1997 Mars Pathfinderrover spent between 40% and 75% of its time doing nothingbecause of plan failure.

One way to attack this problem is to rely on real-time or re-active replanning when failures occur. While this capabil-ity is certainly desirable, there are several difficulties withexclusive reliance on this approach:

� Spacecraft and rovers have severely limited computa-tional resources due to power limitations and radiationhardening requirements. As a result, it is not alwaysfeasible to do timely onboard replanning.

� Many actions are potentially risky and require pre-approval by mission operations personnel. Because ofthe cost and difficulty of communication, the rover re-ceives infrequent command uplinks (typically one perday). As a result, each daily plan must be constructedand checked for safety well in advance.

� Some contingencies require anticipation; e.g., switch-ing to a backup system may require that the backupsystem be warmed up in advance. For time critical op-erations such as orbit insertions or landing operationsthere is insufficient time to perform these setup oper-ations once the contingency has occurred, no matterhow fast the planning can be done.

For these reasons, it is sometimes necessary to plan in ad-vance for potential contingencies—that is, anticipate unex-pected outcomes and events and plan for them in advance.

The problem that we have just described is essentially adecision-theoretic planning problem. More precisely, theproblem is to produce a (concurrent) plan with maximalexpected utility, given the following domain information:

� A set of possible goals that may be achievable, eachof which has a value or reward associated with it.

� A set of initial conditions, which may involve uncer-tainty about continuous quantities like temperature,

energy available, solar flux, and position. This un-certainty is characterized by probability distributionsover the possible values.

� A set of possible actions, each of which is character-ized by:

– a set of conditions that must be true before the ac-tion can be performed. (These may include met-ric temporal constraints as well as constraints onresource availability.)

– an uncertain duration characterized by a proba-bility distribution.

– a set of certain and uncertain effects that describethe world following the action. Uncertain ef-fects on continuous variables are characterizedby probability distributions.

Decision-theoretic planning is already known to be quitehard both in theory [20] and in practice. However, thereare some characteristics of this domain, which, when takentogether, make this planning problem both difficult and dif-ferent from the kinds of problems that have been studied inthe past:

Time: actions take differing amounts of time and concur-rency is often necessary.

Continuous outcomes: most of the uncertainty is associ-ated with continuous quantities like time and power.In other words, actions do not have a small number ofdiscrete outcomes.

Problem size: a typical daily plan for a rover will involveon the order of a hundred actions.

While we have described this scenario for a rover, this kindof problem is not limited to robotics or even space applica-tions. For example, in a logistics problem, travel durationsare influenced by both traffic and weather considerations.Fuel use is likewise influenced by these “environmental”factors. There are temporal constraints on the availabilityand delivery of cargo, as well as on the availability of bothfacilities and crew. There are also constraints on fuel load-ing and availability, and on maintenance operations.

2 PREVIOUS WORK

There has been considerable work in AI on planning underuncertainty. Table 1 classifies much of this work along thefollowing two dimensions:

Representation of uncertainty: whether uncertainty ismodeled strictly logically, using disjunctions, or ismodeled numerically, with probabilities.

Observability assumptions: whether the uncertain out-comes of actions are not observable, partially observ-able, or fully observable.

Table 1: A classification of planners that deal with uncer-tainty. Planners in the top row are often referred to as con-formant planners, while those in the other two rows are of-ten referred to as contingency planners

Disjunction ProbabilityCGP [34]

Non CMBP [11, 1] Buridan [19]Observable C-PLAN [10, 15] UDTPOP [26]

Fragplan [18]SENSp [14] C-Buridan [12]

Cassandra [28] DTPOP [26]Partially PUCCINI [16] C-MAXPLAN [21]

Observable SGP [37] ZANDER [21]QBF-Plan [30] Mahinur [25]

GPT [7] POMDP [8]MBP [2]

JIC [13]Fully WARPLAN-C [36] Plinth [17]

Observable CNLP [27] Weaver [5]PGP [4]MDP [8]

We do not discuss this work in detail here. A survey ofsome of this work can be found in Blythe [6]. A moredetailed survey of work on MDPs and POMDPs can be foundin Boutilier, Dean and Hanks [8]. Instead we will focuson why this work is generally not applicable to the roverproblem and what can be done about this.

There are a number of difficulties in attempting to applyexisting work on planning under uncertainty to spacecraftor rovers. First of all, the work listed in Table 1 assumesa very simple model of action in which concurrent ac-tions are not permitted, explicit time constraints are notallowed, and actions are considered to be instantaneous.As we said above, none of these assumptions hold for typ-ical spacecraft or rover operations. These characteristicsare not as much of an obstacle for Partial-Order Planningframeworks such as SENSp [14], PUCCINI [16], WARPLAN-C [36], CNLP [27], Buridan [19], UDTPOP [26], C-Buridan[12], DTPOP [26], Mahinur [25] and Weaver [5]. In theory,these systems could represent plans with concurrent actionsand complex temporal constraints. The requirements fora rich model of time and action are more problematic forplanning techniques that are based on the MDP or POMDP

representations, satisfiability encodings, the graphplan rep-resentation, or state- space encodings. These techniquesrely heavily on a discrete model of time and action. (See[33] for a more detailed discussion of this issue.) Althoughsemi-Markov decision processes (SMDPs) [29] and tempo-

ral MDPs (TMDP) [9] can be used to represent actions withuncertain durations, they cannot model concurrent actionswith complex temporal dependencies. The factorial MDP

model has recently been developed to allow concurrent ac-tions in an MDP framework. However, this model is limitedto discrete time and state representations. Moreover, exist-ing solution techniques are either too general to be efficienton real-world problems (e.g. Singh and Cohn [31]), or toodomain-specific to be applicable to the rover problem (e.g.Meuleau et al. [22]).

A second, and equally serious, problem with existing con-tingency planning techniques is that they all assume thatuncertain actions have a small number of discrete out-comes. For example, in the representation popularized byBuridan and C-Buridan, a rover movement action mightbe characterized as shown in Figure 1. In this represen-tation, each arrow to a propositions on the right indicatesa possible outcome of the action, along with the associ-ated probability of that transition. To characterize wherea rover could end up after a move operation, we have tolist all the different possible discrete locations. We wouldneed to do something similar to characterize power usage.For most spacecraft and rover activities this kind of dis-crete representation is impractical most of the uncertaintyinvolves continuous quantities, such as the amount of timeand power an activity requires. Action outcomes are dis-tributions over these continuous quantities. There is somerecent work using models with continuous states and/or ac-tion outcomes in both the MDP [3, 23, 24, 32] and POMDP

[35] literature, but this has not yet been applied to SMDPsand has primarily been applied to reinforcement learningrather than planning problems.

Move([1,1],[4,4])

At([3,3])

At([3,4])

At([4,3])

At([4,4])

�

.5.1

.1

.05

Figure 1: A C-Buridan action for movement.

Ultimately, the state that results from performing an ac-tion determines the future actions that will be taken, sosome dimensions of an action’s outcomes are discretized.However, this discretization is not a static property of theactions—instead, it depends on what goals or subgoals theplanner is trying to accomplish. For example, suppose thatthe rover is trying to move to a certain location. If the ob-jective is to place an instrument on a particular rock feature,then the tolerance in position is quite small. In contrast, ifthe objective is to take a picture from a different vantagepoint, then the tolerance can be significantly larger.

A third problem with conventional contingency planningtechnology is that it does not scale to larger problems. Partof the problem is that most of the algorithms attempt to ac-count for all possible contingencies. In effect, they try toproduce policies. For spacecraft and rover operations, thisis not realistic or tractable—a daily plan can involve on theorder of a hundred operations, many of which have uncer-tain outcomes that can impact downstream actions. Theresulting plans must also be simple enough that they can beunderstood by mission operators, and it must be feasible todo detailed simulation and validation on them in a limitedtime period. This means that a planner can only afford toplan in advance for the “important” contingencies and mustleave the rest to run-time replanning. Of the planning sys-tems discussed above, only Just-In-Case (JIC) contingencyscheduling [13] and Mahinur [25] exhibit a principled ap-proach to choosing what contingencies to focus on. We willdiscuss this approach in more detail later.

3 A DETAILED EXAMPLE

In order to illustrate the problem further, in this section wegive a detailed example of a very small rover problem. Fig-ure 2 shows a “primary” plan and two potential branches.The primary plan consists of approaching a target point (Vi-sualServo), digging the soil (Dig), backing up (Drive), andtaking spectral images of the area (NIR). One potential al-ternate branch consists of replacing the spectral image witha high-resolution camera image of the target (Hi res). Asecond potential branch consists of taking a low-resolutionpanorama of the area (Lo res), performing on-board imageanalysis to find rocks in the panorama (Rock finder), andthen taking spectral images of the rocks found (NIR). Forthis example, we assume that energy is only being depleted.(More generally, a rover would also be receiving energy in-put from charging.

Precedence constraints are denoted by arrows in the fig-ure; for example, since HiRes can only be performed afterDrive, there is an arrow from Drive to HiRes. For eachaction, there may be preconditions, expectations, and a lo-cal utility; in the figure, these appear above the plan ac-tions. The preconditions specify under what conditions ex-ecution of the action may start. The expectations describethe expected resource consumption of the actions (in termsof mean and standard deviation); the relative width of dis-tributions is illustrated graphically as well. The local utilityis the reward received when the action terminates success-fully: in this example, this will be when the preconditionsare met and when the energy resource is non-negative at theend of execution.

In the example, consider the HiRes action. It has an energyprecondition � � ���� Ah and a time precondition of 9:00� � � 16:00. The expected energy usage is 0.01 Amp-hours (Ah) with a standard deviation of 0 Ah (so in this

Dig(60) Drive(-2) NIRVisualServo(27 13)

Lo res NIRRock finder

Hi res

µ = 1000 sσ = 500 s

µ = 60 sσ = 1 s

µ = 40 sσ = 20 s

µ = 5 Ahσ = 2.5 Ah

E > 10 Ahµ = .05 Ahσ = .02 Ah

E > .1 Ahµ = .2 Ahσ = .2 Ah

E > .6 Ahµ = 2 Ahσ = .5 Ah

E > 3 Ah

µ = 600 sσ = 60 s

t ∈∈∈∈ [10:00, 14:00]

µ = 120 sσ = 20 s

µ = .01 Ahσ = 0 Ah

E > .02 Ahµ = .1 Ahσ = .01 Ah

E > .12 Ahµ = 2 Ahσ = .5 Ah

E > 3 Ah

µ = 600 sσ = 60 s

t ∈∈∈∈ [10:00, 13:50]

µ = 5 sσ = 1 s

t ∈∈∈∈ [9:00, 16:00]

µ = .01 Ahσ = 0 Ah

E > .02 Ah

µ = 5 sσ = 1 s

t ∈∈∈∈ [9:00, 14:30]

v = 100

v = 10

v = 50v = 5

time

energy

energy

time

energy

time

Figure 2: A detailed rover problem. A “main” plan, andtwo possible alternative branch plans are shown. Probabil-ity distributions for time and energy usage are shown foreach action. Time and energy constraints for actions areshown in bold.

case there is no uncertainty in the model). The expectedduration is 5 seconds with a standard deviation of 1 second.The local utility of the action is � � ��.

4 APPROACHES

There are several possible ways of attacking this problemof planning with continuous uncertain variables. In thissection, we briefly discuss some of these, and the issuesthat arise.

4.1 COMPUTING THE OPTIMAL VALUEFUNCTION

Figure 3 shows the optimal value function for the problemin Figure 2. It represents the expected utility obtained byfollowing an optimal (Markov) policy of the MDP repre-

senting the problem. The figure was computed by work-ing backwards from all possible activities that have posi-tive reward and using dynamic programming to constructthe optimal plan, after a fine discretization of both time andenergy.1 The curved hump where there is lots of powerand time available corresponds to the primary plan, whilethe rectangular block corresponds to branching to the Rockfinder plan and completing the NIR. The tail of the curvedhump is a branch after the drive action to the HiRes plan.The flat surface with value 5 that covers nearly all the restof the space is again an immediate branch to the Rock-Finder plan, but in this area there is not enough power ortime to complete the plan, and only the LoRes reward is re-ceived. Figure 4 shows a cross-section through this surfacefor power equal to 11, showing how the various branchescontribute to the overall plan. The utility curve of eachbranch, identified by its goal, represents the expected re-ward if we commit to the branch before knowing the initialconditions (start time). The maximum (upper envelope) ofthese curves is the expected utility of the best plan that firstselects a branch depending on initial conditions, and thencommits to this branch. The utility of the optimal policy(labeled as “all”) is higher in some places than the util-ity of the best branch. This is because the optimal policynever commits prematurely to a branch. This increase inexpected reward is due to the benefits of waiting to see howmuch time is available after part of the best plan has beenperformed, and branching to an alternative plan if the bestone is unlikely to succeed in the remaining time, rather thancomitting to a particular plan immediately.

50

100

150

200

100

200

300

400

0

25

50

75

100

50

100

150

0

0

2

5

Figure 3: Optimal value function for the example in Figure2. The left axis is increasing energy from 0 to 20. The rightaxis is start time from 14:30 down to 13:20. Vertical axis isexpected utility.

1With a grid of 420 steps for time and 200 steps for energy,the size of the state space is about 2.7 ��

�. Moreover, it growsexcponentially with the number of actions in the problem, so thatthis approach is unfeasible for any real size problem.

0

10

20

30

40

50

60

70

80

90

100

48000 48500 49000 49500 50000 50500 51000 51500 52000

Exp

ecte

d ut

ility

Start time

main NIRsecondary NIR

High Resall

Figure 4: Slice of the optimal value function for energy =11 Ah, along with the component curves that contribute tothe overall utility.

Given a detailed contingent plan and the distributions fortime and resource usage, it is relatively straightforward toevaluate the expected utility of the plan. If the distribu-tions are very simple, it may be possible to compute thisquantity exactly; more generally, this will have to be donewith stochastic simulation. Thus, if we could generate allpossible contingent plans for a problem, we could evaluateeach of them and choose the one with highest utility. Ofcourse this is completely impractical for problems of anysize, partly because it is impossible to enumerate the con-ditions for conditional branches. The dynamic program-ming approach we took above is an alternative, but it toois computationally expensive, and it fails completely whenresource availability is not monotonically decreasing (be-cause optimization can no longer be performed through asingle backward pass).

4.2 HEURISTIC APPROACHES

One possibility is to try to plan for the worst case scenario.Thus, in the example from the last section, we could as-sume that the drive operation requires time and power thatis one or perhaps even two standard deviations above themean. The trouble is, this approach is overly conservativeand leads to plans with less science gain than is typicallypossible. In the example from the previous section, if planexecution was expected to begin at 13:45, this approachwould lead us to build a “safe” primary plan that replacesNIR with the HiRes action, with expected utility of 10 in allcases, instead of the more ambitious current primary plan,with expected utility of 0 in the worst case, but 32 in theaverage case and 100 in the best case.

A more ambitious approach to the problem would be tobuild an initial plan based on the expected behavior of vari-ous activities and then attempt to improve that plan by aug-menting it with contingent branches. This is the approach

taken by Drummond, Bresina and Swanson with their Just-in-Case (JIC) telescope scheduling [13]. This approach isintuitively simple and appealing, but extending it to prob-lems like the one we have outlined is non-trivial. Theprimary difficulty is to decide where contingent branchesshould be added to a plan. In JIC scheduling, brancheswere added at the points with the greatest probability ofplan failure. Given the distributions for time and resourceusage this is relatively easy to calculate by statistical simu-lation of the plan. Unfortunately, the points most likely tofail are not necessarily the points where useful alternativesare available. The points of maximal failure probabilitymay be too late in the plan to have enough time or powerleft for any useful alternative. A more efficient approachcould be to identify the earliest point in time where we canpredict with a given confidence that a failure is going tooccur.

Unfortunately, the problem of finding “high utility” branchpoints is non-trivial. Figure 5 shows the expected utilityover time of the possible plans with a single branch, for afixed starting energy of 11. Note that at earlier start times,the plans with the highest expected utility are those thatpostpone the decision to later in the primary plan, wherethe possibility of receiving the 100 reward for the NIR ac-tion can be more accurately assessed. Between 49200 and49700 seconds, the expected utility of 55 gained by im-mediately taking the RockFinder branch dominates as thatplan is likely to succeed when started later than the pri-mary plan. The value function for this branch drops offvery sharply because there is relatively little uncertaintyabout the duration of this plan. As time advances, the valueof branching later is apparent. Late branches look betterwhen time is short because of the chance that an earlieraction will happen unusually quickly, allowing the primaryplan to be completed. Late branches to the RockFinder per-form worse than to HiRes because there is rarely enoughtime remaining after the VisualServo action to complete theRockFinder plan. These plans finally dominate when thereis very little time available because even the HiRes branchis unlikely to be completed.

4.3 FINDING THE BRANCH CONDITIONS

Once we have decided to add a branch to a plan, there isstill a problem of deciding under what conditions to takethe branch. Once again, we could use dynamic program-ming to compute the optimal conditions, but this suffersfrom the problems we described above. In addition, as Fig-ure 3 illustrates, the optimal conditions can be extremelycomplex and hard to represent. The flat surfaces of utility5 and 55 correspond to branching to the RockFinder planbefore the first step of the primary plan. The primary plan(along with the later possible branch to the HiRes plan) isof higher expected utility where the surface is curved. Theconditions for the branch point at the beginning of the pri-

0

10

20

30

40

50

60

70

80

90

48000 48500 49000 49500 50000 50500 51000 51500

Expected Utility

Time

No branchesBranch at start to Rock Finder

Branch after Dig to Rock FinderBranch after Drive to Rock Finder

Branch after Drive to Hi-Res

Figure 5: Utility for a single branch at different possiblebranch points with energy = 11.

mary plan are thus the boundaries between the curved sur-faces and the flat surfaces. The boundaries are in this casediscontinuous, corresponding to a disjunctive condition.

It is important to bear in mind that the boundaries are gen-erally places where the values of two different branches areequal, which means that approximate solutions will usuallybe acceptable here. One possibility is to treat the continu-ous dimensions of the problem as independent, which re-sults in rectangular regions. This works well in most cases,but the boundaries must be chosen with care where thereare abrupt edges in the value function. This approximationmay also fail if there are dependencies between the dimen-sions, for example when the energy used for driving is de-pendent on the actual time spent, rather than being treatedindependently as in our example.

5 CONCLUSIONS

For a Mars rover, uncertainty is absolutely pervasive in thedomain. There is uncertainty in the duration of many activi-ties, in the amount of power that will be used, in the amountof data storage that will be required, and in the locationand orientation of the rover. Unfortunately, current tech-niques for planning under uncertainty are limited to simplemodels of time, and actions with discrete outcomes. In therover domain there is concurrent action, actions of differ-ing duration, and most of the uncertainty is associated withcontinuous quantities like time, power, position and orien-tation.

For any non-trivial problem, it seems unlikely that exactor optimal solutions will be possible. Nor do we havegood heuristic techniques for generating effective contin-gent plans. It seems that new and dramatically differentapproaches are needed to deal with this kind of problem.

Acknowledgements

Thanks to Tania Bedrax-Weiss, Jeremy Frank and KeithGolden for discussions on this subject and comments ondrafts of the paper. This research was supported by NASAAmes Research Center and the NASA Intelligent Systemsprogram.

References

[1] P. Bertoli, A. Cimatti, and M. Roveri. Heuristic search+ symbolic model checking = efficient conformantplanning. In Proceedings of the Seventeenth Inter-national Joint Conference on Artificial Intelligence,2001.

[2] P. Bertoli, A. Cimatti, M. Roveri, and P. Traverso.Planning in mondeterministic domains under partialobservability via symbolic model checking. In Pro-ceedings of the Seventeenth International Joint Con-ference on Artificial Intelligence, 2001.

[3] D. Bertsekas and J. Tsitsiklis. Neuro-dynamic Pro-gramming. Athena, Belmont, MA, 1996.

[4] A. Blum and J. Langford. Probabilistic planning inthe Graphplan framework. In Proceedings of the FifthEuropean Conference on Planning, 1999.

[5] J. Blythe. Planning under uncertainty in dynamic do-mains. PhD thesis, Carnegie Mellon University, 1998.

[6] J. Blythe. Decision-theoretic planning. AI Magazine,20(2):37–54, 1999.

[7] B. Bonet and H. Geffner. Planning with incompleteinformation as heuristic search in belief space. In Pro-ceedings of the Fifth International Conference on Ar-tificial Intelligence Planning and Scheduling, pages52–61, 2000.

[8] C. Boutillier, T. Dean, and S. Hanks. Decision the-oretic planning: structural assumptions and compu-tational leverage. Journal of AI Research, 11:1–94,1999.

[9] J. Boyan and M. Littman. Exact solutions to time-dependent MDPs. In Advances in Neural InformationProcessing Systems 13, pages 1–7. MIT Press, Cam-bridge, 2000.

[10] C. Castellini, E. Giunchiglia, and A. Tacchella. Im-provements to sat-based conformant planning. In Pro-ceedings of the Sixth European Conference on Plan-ning, 2001.

[11] A. Cimatti and M. Roveri. Conformant planning viasymbolic model checking. Journal of AI Research,11:305–338, 2000.

[12] D. Draper, S. Hanks, and D. Weld. Probabilistic plan-ning with information gathering and contingent exe-cution. In Proceedings of the Second InternationalConference on Artificial Intelligence Planning andScheduling, pages 31–36, 1994.

[13] M. Drummond, J. Bresina, and K. Swanson. Just-In-Case scheduling. In Proceedings of the TwelfthNational Conference on Artificial Intelligence, pages1098–1104, 1994.

[14] O. Etzioni, S. Hanks, D. Weld, D. Draper, N. Lesh,and M. Williamson. An approach to planning withincomplete information. In Proceedings of the ThirdInternational Conference on Principles of KnowledgeRepresentation and Reasoning, pages 115–125, 1992.

[15] E. Ferraris and E. Giunchiglia. Planning as satisfi-ability in nondeterministic domains. In Proceedingsof the Seventeenth National Conference on ArtificialIntelligence, 2000.

[16] K. Golden. Leap before you look: information gath-ering in the PUCCINI planner. In Proceedings of theFourth International Conference on Artificial Intelli-gence Planning and Scheduling, pages 70–77, 1998.

[17] R. Goldman and M. Boddy. Conditional linear plan-ning. In Proceedings of the Second InternationalConference on Artificial Intelligence Planning andScheduling, pages 80–85, 1994.

[18] J. Kurien, P. Nayak, and D. Smith. Fragment-basedconformant planning. In Proceedings of the SixthInternational Conference on Artificial IntelligencePlanning and Scheduling, 2002.

[19] N. Kushmerick, S. Hanks, and D. Weld. An algo-rithm for probabilistic planning. Artificial Intelli-gence, 76(1-2):239–286, 1995.

[20] M. Littman, J. Goldsmith, and M. Mundhenk. Thecomputational complexity of probabilistic planning.Journal of AI Research, 9:1–36, 1998.

[21] S. Majercik and M. Littman. Contingent planning un-der uncertainty via stochastic satisfiability. In Pro-ceedings of the Sixteenth National Conference on Ar-tificial Intelligence, 1999.

[22] N. Meuleau, M. Hauskrecht, K. Kim, L. Peshkin,L. Kaelbling, T. Dean, and C. Boutilier. Solving verylarge weakly coupled Markov decision processes. InProceedings of the Fifteenth National Conference onArtificial Intelligence, pages 165–172, 1998.

[23] R. Munos. Variable resolution discretization for high-accuracy solutions of optimal control problems. InProceedings of the Sixteenth International Joint Con-ference on Artificial Intelligence, 1999.

[24] R. Munos. A study of reinforcement learning in thecontinuous case by the means of viscosity solutions.Machine Learning, 40:265–299, 2000.

[25] N. Onder and M. Pollack. Conditional, probabilisticplanning: A unifying algorithm and effective searchcontrol mechanisms. In Proceedings of the SixteenthNational Conference on Artificial Intelligence, pages577–584, 1999.

[26] M. Peot. Decision-Theoretic Planning. PhD the-sis, Dept. of Engineering-Economic Systems, Stan-ford University, 1998.

[27] M. Peot and D. Smith. Conditional nonlinear plan-ning. In Proceedings of the First InternationalConference on Artificial Intelligence Planning andScheduling, pages 189–197, 1992.

[28] L. Pryor and G. Collins. Planning for contingencies:a decision-based approach. Journal of AI Research,4:287–339, 1996.

[29] M. Puterman. Markov Decision Processes: DiscreteStochastic Dynamic Programming. Wiley, New York,NY, 1994.

[30] J. Rintanen. Constructing conditional plans by a the-orem prover. Journal of AI Research, 10:323–352,1999.

[31] S. Singh and D. Cohn. How to dynamically mergeMarkov decision processes. In Advances in NeuralInformation Processing Systems 11. MIT Press, Cam-bridge, 1998.

[32] W. Smart and L. Kaelbling. Practical reinforcementlearning in continuous spaces. In Proceedings ofthe Seventeenth International Conference on MachineLearning, 2000.

[33] D. Smith, J. Frank, and A. Jonsson. Bridging the gapbetween planning and scheduling. The KnowledgeEngineering Review, 15(1), 2000.

[34] D. Smith and D. Weld. Conformant graphplan. InProceedings of the Fifteenth National Conference onArtificial Intelligence, pages 889–896, 1998.

[35] S. Thrun. Monte Carlo POMDPs. In Advancesin Neural Information Processing Systems 12, pages1064–1070. MIT Press, Cambridge, 1999.

[36] D. Warren. Generating conditional plans and pro-grams. In Proceedings of the Summer Conference onAI and Simulation of Behavior., 1976.

[37] D. Weld, C. Anderson, and D. Smith. ExtendingGraphplan to handle uncertainty and sensing actions.In Proceedings of the Fifteenth National Conferenceon Artificial Intelligence, pages 897–904, 1998.