decision making under uncertainty
DESCRIPTION
Decision Making Under Uncertainty. CMSC 471 – Spring 2014 Class #12– Thursday, March 6 R&N, Chapters 15.1-15.2.1, 16.1-16.3. material from Lise Getoor, Jean-Claude Latombe, and Daphne Koller. MODELING UNCERTAINTY OVER TIME. sensors. environment. ?. agent. actuators. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/1.jpg)
Decision Making Under Decision Making Under UncertaintyUncertainty
CMSC 471 – Spring 2014Class #12– Thursday, March 6R&N, Chapters 15.1-15.2.1, 16.1-16.3
material from Lise Getoor, Jean-Claude Latombe, and Daphne Koller 1
![Page 2: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/2.jpg)
MODELING UNCERTAINTY OVER TIME
2
![Page 3: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/3.jpg)
Temporal Probabilistic Temporal Probabilistic AgentAgent
environmentagent
?
sensors
actuators
t1, t2, t3, …
3
![Page 4: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/4.jpg)
Time and Uncertainty
The world changes; we need to track and predict itExamples: diabetes management, traffic monitoringBasic idea: Copy state and evidence variables for each time step Model uncertainty in change over time, incorporating new
observations as they arrive Reminiscent of situation calculus, but for stochastic domains
Xt – set of unobservable state variables at time te.g., BloodSugart, StomachContentst
Et – set of evidence variables at time te.g., MeasuredBloodSugart, PulseRatet, FoodEatent
Assumes discrete time steps
4
![Page 5: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/5.jpg)
States and Observations
Process of change is viewed as series of snapshots, each describing the state of the world at a particular timeEach time slice is represented by a set of random variables indexed by t:
1. the set of unobservable state variables Xt
2. the set of observable evidence variables Et
1. The observation at time t is Et = et for some set of values et
2. The notation Xa:b denotes the set of variables from Xa to Xb
5
![Page 6: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/6.jpg)
Stationary Process/Markov Assumption
Markov Assumption: Xt depends on some previous Xis
First-order Markov process: P(Xt|X0:t-1) = P(Xt|Xt-1)kth order: depends on previous k time stepsSensor Markov assumption:P(Et|X0:t, E0:t-1) = P(Et|Xt)
That is: The agent’s observations depend only on the actual current state of the world
Assume stationary process: Transition model P(Xt|Xt-1) and sensor model P(Et|Xt) are
time-invariant (i.e., they are the same for all t) That is, the changes in the world state are governed by
laws that do not themselves change over time
6
![Page 7: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/7.jpg)
Complete Joint Distribution
Given: Transition model: P(Xt|Xt-1) Sensor model: P(Et|Xt) Prior probability: P(X0)
Then we can specify the complete joint distribution of a sequence of states:
t
1iii1ii0t1t10 )X|E(P)X|X(P)X(P)E,...,E,X,...,X,X(P
7
![Page 8: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/8.jpg)
Example
Raint-1
Umbrellat-1
Raint
Umbrellat
Raint+1
Umbrellat+1
Rt-1 P(Rt|Rt-1)
TF
0.70.3
Rt P(Ut|Rt)
TF
0.90.2
This should look a lot like a finite state automaton (since it is one...) 8
![Page 9: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/9.jpg)
Inference Tasks
Filtering or monitoring: P(Xt|e1,…,et)Compute the current belief state, given all evidence to date
Prediction: P(Xt+k|e1,…,et) Compute the probability of a future state
Smoothing: P(Xk|e1,…,et) Compute the probability of a past state (hindsight)
Most likely explanation: arg maxx1,..xtP(x1,…,xt|e1,…,et)Given a sequence of observations, find the sequence of states that is most likely to have generated those observations
9
![Page 10: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/10.jpg)
Examples
Filtering: What is the probability that it is raining today, given all of the umbrella observations up through today?Prediction: What is the probability that it will rain the day after tomorrow, given all of the umbrella observations up through today?Smoothing: What is the probability that it rained yesterday, given all of the umbrella observations through today?Most likely explanation: If the umbrella appeared the first three days but not on the fourth, what is the most likely weather sequence to produce these umbrella sightings?
10
![Page 11: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/11.jpg)
Filtering
We use recursive estimation to compute P(Xt+1 | e1:t+1) as a function of et+1 and P(Xt | e1:t)We can write this as follows:
This leads to a recursive definition:f1:t+1 = FORWARD (f1:t, et+1)
€
P(X t+1 | e1:t+1) = P(X t+1 | e1:t ,et+1)
= α P(et+1 | X t+1,e1:t ) P(X t+1 | e1:t )
= α P(et+1 | X t+1) P(X t+1 | e1:t )
= α P(et+1 | X t+1) P(X t+1 | x t ) P(x t | e1:t )xt
∑
QUIZLET: What is α?11
![Page 12: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/12.jpg)
Filtering Example
Raint-1
Umbrellat-1
Raint
Umbrellat
Raint+1
Umbrellat+1
Rt-1 P(Rt|Rt-1)
TF
0.70.3
Rt P(Ut|Rt)
TF
0.90.2
What is the probability of rain on Day 2, given a uniform prior of rain on Day 0, U1 = true, and U2 = true? 12
€
P(X t+1 | e1:t+1) = α P(et+1 | X t+1) P(X t+1 | X t ) P(X t | e1:t )X t
∑
![Page 13: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/13.jpg)
DECISION MAKING UNDER UNCERTAINTY
![Page 14: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/14.jpg)
Decision Making Under Uncertainty
Many environments have multiple possible outcomesSome of these outcomes may be good; others may be badSome may be very likely; others unlikely
What’s a poor agent to do??
14
![Page 15: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/15.jpg)
Non-Deterministic vs. Non-Deterministic vs. Probabilistic UncertaintyProbabilistic Uncertainty
?
ba c
{a,b,c}
decision that is best for worst case
?
ba c
{a(pa), b(pb), c(pc)}
decision that maximizes expected utility valueNon-deterministic
modelProbabilistic model~ Adversarial search
15
![Page 16: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/16.jpg)
Expected UtilityExpected Utility
Random variable X with n values x1,…,xn and distribution (p1,…,pn)E.g.: X is the state reached after doing an action A under uncertaintyFunction U of XE.g., U is the utility of a stateThe expected utility of A is EU[A] = i=1,…,n p(xi|A)U(xi)
16
![Page 17: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/17.jpg)
s0
s3s2s1
A1
0.2 0.7 0.1100 50 70
U(A1, S0) = 100 x 0.2 + 50 x 0.7 + 70 x 0.1 = 20 + 35 + 7 = 62
One State/One Action One State/One Action ExampleExample
17
![Page 18: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/18.jpg)
s0
s3s2s1
A1
0.2 0.7 0.1100 50 70
A2
s40.2 0.8
80
• U (A1, S0) = 62• U (A2, S0) = 74• U (S0) = maxa{U(a,S0)} = 74
One State/Two Actions One State/Two Actions ExampleExample
18
![Page 19: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/19.jpg)
s0
s3s2s1
A1
0.2 0.7 0.1100 50 70
A2
s40.2 0.8
80
• U (A1, S0) = 62 – 5 = 57• U (A2, S0) = 74 – 25 = 49• U (S0) = maxa{U(a, S0)} = 57
-5 -25
Introducing Action CostsIntroducing Action Costs
19
![Page 20: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/20.jpg)
MEU Principle
A rational agent should choose the action that maximizes agent’s expected utilityThis is the basis of the field of decision theoryThe MEU principle provides a normative criterion for rational choice of action
20
![Page 21: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/21.jpg)
Not quite…
Must have a complete model of: Actions Utilities States
Even if you have a complete model, decision making is computationally intractableIn fact, a truly rational agent takes into account the utility of reasoning as well (bounded rationality)Nevertheless, great progress has been made in this area recently, and we are able to solve much more complex decision-theoretic problems than ever before
21
![Page 22: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/22.jpg)
Axioms of Utility Theory
Orderability (A>B) (A<B) (A~B)
Transitivity (A>B) (B>C) (A>C)
Continuity A>B>C p [p,A; 1-p,C] ~ B
Substitutability A~B [p,A; 1-p,C]~[p,B; 1-p,C]
Monotonicity A>B (p≥q [p,A; 1-p,B] >~ [q,A; 1-q,B])
Decomposability[p,A; 1-p, [q,B; 1-q, C]] ~ [p,A; (1-p)q, B; (1-p)(1-q), C]
22
![Page 23: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/23.jpg)
Money Versus Utility
Money <> Utility More money is better, but not always in
a linear relationship to the amount of money
Expected Monetary ValueRisk-averse: U(L) < U(SEMV(L))
Risk-seeking: U(L) > U(SEMV(L))
Risk-neutral: U(L) = U(SEMV(L))
23
![Page 24: Decision Making Under Uncertainty](https://reader035.vdocument.in/reader035/viewer/2022062517/56813acd550346895da2ee02/html5/thumbnails/24.jpg)
Value Function
Provides a ranking of alternatives, but not a meaningful metric scaleAlso known as an “ordinal utility function”
Sometimes, only relative judgments (value functions) are necessaryAt other times, absolute judgments (utility functions) are required
24