policy gradients - robot learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...policy...
TRANSCRIPT
![Page 1: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/1.jpg)
Policy Gradients
CS 294-112: Deep Reinforcement Learning
Sergey Levine
![Page 2: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/2.jpg)
Class Notes
1. Homework 1 milestone due today (11:59 pm)!• Don’t be late!
2. Remember to start forming final project groups
![Page 3: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/3.jpg)
Today’s Lecture
1. The policy gradient algorithm
2. What does the policy gradient do?
3. Basic variance reduction: causality
4. Basic variance reduction: baselines
5. Policy gradient examples
• Goals:• Understand policy gradient reinforcement learning
• Understand practical considerations for policy gradients
![Page 4: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/4.jpg)
The goal of reinforcement learningwe’ll come back to partially observed later
![Page 5: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/5.jpg)
The goal of reinforcement learning
infinite horizon case finite horizon case
![Page 6: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/6.jpg)
Evaluating the objective
![Page 7: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/7.jpg)
Direct policy differentiation
a convenient identity
![Page 8: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/8.jpg)
Direct policy differentiation
![Page 9: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/9.jpg)
Evaluating the policy gradient
generate samples (i.e. run the policy)
fit a model to estimate return
improve the policy
![Page 10: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/10.jpg)
Evaluating the policy gradient
![Page 11: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/11.jpg)
Comparison to maximum likelihood
trainingdata
supervisedlearning
![Page 12: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/12.jpg)
Example: Gaussian policies
![Page 13: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/13.jpg)
What did we just do?
good stuff is made more likely
bad stuff is made less likely
simply formalizes the notion of “trial and error”!
![Page 14: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/14.jpg)
Partial observability
![Page 15: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/15.jpg)
What is wrong with the policy gradient?
high variance
(image from Peters & Schaal 2008)
slow convergence
hard to choose learning rate
![Page 16: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/16.jpg)
generate samples (i.e.
run the policy)
fit a model to estimate return
improve the policy
Review
• Evaluating the RL objective• Generate samples
• Evaluating the policy gradient• Log-gradient trick
• Generate samples
• Understanding the policy gradient• Formalization of trial-and-error
• Partial observability• Works just fine
• What is wrong with policy gradient?
![Page 17: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/17.jpg)
Break
![Page 18: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/18.jpg)
Reducing variance
“reward to go”
![Page 19: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/19.jpg)
Baselines
but… are we allowed to do that??
subtracting a baseline is unbiased in expectation!
average reward is not the best baseline, but it’s pretty good!
a convenient identity
![Page 20: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/20.jpg)
Analyzing variance
This is just expected reward, but weightedby gradient magnitudes!
![Page 21: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/21.jpg)
generate samples (i.e.
run the policy)
fit a model to estimate return
improve the policy
Review
• The high variance of policy gradient
• Exploiting causality• Future doesn’t affect the past
• Baselines• Unbiased!
• Analyzing variance• Can derive optimal baselines
![Page 22: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/22.jpg)
Policy gradient is on-policy
• Neural networks change only a little bit with each gradient step
• On-policy learning can be extremely inefficient!
![Page 23: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/23.jpg)
Off-policy learning & importance sampling
importance sampling
![Page 24: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/24.jpg)
Deriving the policy gradient with IS
a convenient identity
![Page 25: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/25.jpg)
The off-policy policy gradient
![Page 26: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/26.jpg)
A first-order approximation for IS (preview)
We’ll see why this is reasonablelater in the course!
![Page 27: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/27.jpg)
Policy gradient with automatic differentiation
![Page 28: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/28.jpg)
Policy gradient with automatic differentiation
Pseudocode example (with discrete actions):
Maximum likelihood:# Given:# actions - (N*T) x Da tensor of actions# states - (N*T) x Ds tensor of states# Build the graph:logits = policy.predictions(states) # This should return (N*T) x Da tensor of action logitsnegative_likelihoods = tf.nn.softmax_cross_entropy_with_logits(labels=actions, logits=logits)loss = tf.reduce_mean(negative_likelihoods)gradients = loss.gradients(loss, variables)
![Page 29: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/29.jpg)
Policy gradient with automatic differentiation
Pseudocode example (with discrete actions):
Policy gradient:# Given:# actions - (N*T) x Da tensor of actions# states - (N*T) x Ds tensor of states# q_values – (N*T) x 1 tensor of estimated state-action values# Build the graph:logits = policy.predictions(states) # This should return (N*T) x Da tensor of action logitsnegative_likelihoods = tf.nn.softmax_cross_entropy_with_logits(labels=actions, logits=logits)weighted_negative_likelihoods = tf.multiply(negative_likelihoods, q_values)loss = tf.reduce_mean(weighted_negative_likelihoods)gradients = loss.gradients(loss, variables)
q_values
![Page 30: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/30.jpg)
Policy gradient in practice
• Remember that the gradient has high variance• This isn’t the same as supervised learning!
• Gradients will be really noisy!
• Consider using much larger batches
• Tweaking learning rates is very hard• Adaptive step size rules like ADAM can be OK-ish
• We’ll learn about policy gradient-specific learning rate adjustment methods later!
![Page 31: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/31.jpg)
generate samples (i.e.
run the policy)
fit a model to estimate return
improve the policy
Review
• Policy gradient is on-policy
• Can derive off-policy variant• Use importance sampling• Exponential scaling in T• Can ignore state portion
(approximation)
• Can implement with automatic differentiation – need to know what to backpropagate
• Practical considerations: batch size, learning rates, optimizers
![Page 32: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/32.jpg)
Advanced policy gradient topics
• What more is there?
• Next time: introduce value functions and Q-functions
• Later in the class: natural gradient and automatic step size adjustment
![Page 33: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/33.jpg)
Example: policy gradient with importance sampling
Levine, Koltun ‘13
• Incorporate example demonstrations using importance sampling
• Neural network policies
![Page 34: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/34.jpg)
Example: trust region policy optimization
Schulman, Levine, Moritz, Jordan, Abbeel. ‘15
• Natural gradient with automatic step adjustment (we’ll learn about this later)
• Discrete and continuous actions
• Code available (see Duan et al. ‘16)
![Page 35: Policy Gradients - Robot Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy...Policy gradients suggested readings •Classic papers •Williams (1992). Simple statistical](https://reader034.vdocument.in/reader034/viewer/2022042405/5f1c63a9334f99635c5a553a/html5/thumbnails/35.jpg)
Policy gradients suggested readings
• Classic papers• Williams (1992). Simple statistical gradient-following algorithms for connectionist
reinforcement learning: introduces REINFORCE algorithm• Baxter & Bartlett (2001). Infinite-horizon policy-gradient estimation: temporally
decomposed policy gradient (not the first paper on this! see actor-critic section later)• Peters & Schaal (2008). Reinforcement learning of motor skills with policy gradients:
very accessible overview of optimal baselines and natural gradient
• Deep reinforcement learning policy gradient papers• Levine & Koltun (2013). Guided policy search: deep RL with importance sampled policy
gradient (unrelated to later discussion of guided policy search)• Schulman, L., Moritz, Jordan, Abbeel (2015). Trust region policy optimization: deep RL
with natural policy gradient and adaptive step size• Schulman, Wolski, Dhariwal, Radford, Klimov (2017). Proximal policy optimization
algorithms: deep RL with importance sampled policy gradient