1
Search and Planning for Inference and Learning
in Computer Vision
Iasonas Kokkinos, Sinisa Todorovic and Matt (Tianfu) Wu
Markov Decision Processes & Reinforcement Learning
Sinisa Todorovic and Iasonas KokkinosJune 7, 2015
Multi-Armed Bandit Problem• A gambler faces K slot-machines ("armed bandits")• Each machine provides a random reward from an
unknown distribution specific to that machine• Problem:
In which order to play each machine to maximize the sum of rewards of a sequence of lever pulls
s
a1 a2 ak
R(s,a1) R(s,a2) R(s,ak)
…
… Robbins 1952
Outline
• Stochastic Process• Markov Property• Markov Chain• Markov Decision Process• Reinforcement Learning
Discrete Stochastic Process
• A collection of indexed random variables with well-defined ordering
• Characterized by probabilities that the variables take given values, called states
Andrey Makrov
Stochastic Process Example• Classic: Random Walk
– Start at state X0 at time t0
– At time ti, move a step Zi whereP(Zi = -1) = p and P(Zi = 1) = 1 - p
– At time ti, state Xi = X0 + Z1 +…+ Zi
http://en.wikipedia.org/wiki/Image:Random_Walk_example.png
Markov Property
• Also thought of as the “memoryless” property
• If the probability that Xn+1 has any given value depends only on Xn
Markov Chain
• Discrete-time stochastic process with the Markov property
• Example: Google’s PageRankLikelihood of random linking ending up on a page
http://en.wikipedia.org/wiki/PageRank
Markov Decision Process (MDP)
• Discrete time stochastic control process• Extension of Markov chains• Differences:
– Addition of actions (choice)– Addition of rewards (motivation)
• If the actions are fixed, an MDP reduces to a Markov chain
Description of MDPs
• Tuple (S, A, P(.,.), R(.)))– S -> state space– A -> action space– Pa(s, s’) = Pr(st+1 = s’ | st = s, at = a)– R(s) = immediate reward at state s
• Goal: maximize a cumulative function of the rewards = utility function
Example MDP
state node
action node
Solution to an MDP = Policy π
• Given a state, selects the optimal action regardless of history
Value function
Learning Policy
• Value Iteration
• Policy Iteration
• Modified Policy Iteration
• Prioritized Sweeping
Value Iteration
k Vk(PU) Vk(PF) Vk(RU) Vk(RF)
1
2
3
4
5
6
0 0 10 10
0 4.5 14.5 19
2.03 8.55 18.55 24.18
4.76 11.79 19.26 29.23
7.45 15.30 20.81 31.82
10.23 17.67 22.72 33.68
Why So Interesting?
• Straightforward if the transition probabilities are known, but...
• If the transition probabilities are unknown, then this problem is reinforcement learning.
A Typical Agent
• In reinforcement learning (RL), an agent observes a state and takes an action.
• Afterwards, the agent receives a reward.
Mission: Optimize Reward
• Rewards are calculated in the environment• Used to teach the agent how to reach a goal
state• Must signal what we ultimately want
achieved, not necessarily subgoals• May be discounted over time• In general, seek to maximize the expected
return
Monte Carlo Methods
• Instead of
• Compute:
• Qπ(s, a): Expected reward when starting in state s, taking action a, and thereafter following policy π
Monte-Carlo Tree Search
• Builds a tree rooted at current state by repeated Monte-Carlo simulation of a “rollout policy”
• Key Idea: Use statistics of previous trajectories to expand the tree in most promising direction
• No heuristic functions, unlike A*, and branch-and-bound methods
Kocsis & Szepesvari, 2006 Browne et. al., 2012
Monte-Carlo Tree Search
select the best state
so far
take an actionand move
to a new state
simulation backpropagationof the total reward
of simulation
Repeated until the maximum tree depth is reached
Monte-Carlo Tree Search• During construction each tree node s stores:
– state-visitation count n(s) – action counts n(s,a) – action values Q(s,a)
• Repeat until time is up1. Select action a
2. Update statistics of each node s on trajectory• Increment n(s) and n(s,a) for selected action a• Update Q(s,a) by total reward of simulation
Monte-Carlo Tree Search• During construction each tree node s stores:
– state-visitation count n(s) – action counts n(s,a) – action values Q(s,a)
• Repeat until time is up1. Select action a
2. Update statistics of each node s on trajectory• Increment n(s) and n(s,a) for selected action a• Update Q(s,a) by total reward of simulation
Monte-Carlo Tree Search
exploitationexploration
Theoretically, guaranteed to converge to optimal solutions if run long enough.
Practically, it often shows good anytime behavior.
Kocsis & Szepesvari, 2006 Browne et. al., 2012
24
Acknowledgements
NSF IIS 1302700
DARPA MSEE FA 8650-11-1-7149
