Upload File

deep reinforcement learning cs 294 -...

  • Home
  • Documents
  • Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement
18
Deep Reinforcement Learning CS 294 - 112

Upload: others

Post on 10-Aug-2020

2 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

Deep Reinforcement LearningCS 294 - 112

Page 2: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

Today

1. Course logistics (the boring stuff)

2. 20-minute introductions from each instructor

Page 3: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

Course Staff

Sergey LevineAssistant Professor

UC Berkeley

Chelsea FinnPhD StudentUC Berkeley

John SchulmanResearch Scientist

OpenAI

Page 4: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

Class Information & Resources

• Course website: rll.berkeley.edu/deeprlcourse/

• Piazza: UC Berkeley, CS294-112

• Subreddit (for non-enrolled students): www.reddit.com/r/berkeleydeeprlcourse/

• Office hours: after class each day (but not today), sign up in advance for a 10-minute slot on the course website

http://rll.berkeley.edu/deeprlcourse/
http://www.reddit.com/r/berkeleydeeprlcourse/
Page 5: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

Prerequisites & Enrollment

• All enrolled students must have taken CS189, CS289, or CS281A• Please contact Sergey Levine if you haven’t

• Please enroll for 3 units

• Wait list is (very) full, everyone near the top has been notified

• Lectures will be recorded• Since the class is full, please watch the lectures online if you are not enrolled

Page 6: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

What you should know

• Assignments will require training neural networks with standard automatic differentiation packages (TensorFlow or Theano)

• Review Section• Chelsea Finn will teach a review section in week 2

• Please fill out the poll here to help us choose a time: tinyurl.com/tfsection

• You should be able to at least do the TensorFlow MNIST tutorial (if not, come to the review section and ask questions!)

tinyurl.com/tfsection
Page 7: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

What we’ll cover

• Full syllabus on course website

1. From supervised learning to decision making

2. Basic reinforcement learning: Q-learning and policy gradients

3. Advanced model learning and prediction, distillation, reward learning

4. Advanced deep RL: trust region policy gradients, actor-critic methods, exploration

5. Open problems, research talks, invited lectures

Page 8: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

Assignments

1. Homework 1: Imitation learning (control via supervised learning)

2. Homework 2: Basic (shallow) RL

3. Homework 3: Deep Q learning

4. Homework 4: Deep policy gradients

5. Final project: Research-level project of your choice (form a group of up to 2-3 students, you’re welcome to start early!)

Grading: 40% homework (10% each), 50% project, 10% participation

Page 9: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

How do we building intelligent machines?

• Imagine you have to build an intelligent machine, where do you start?

Page 10: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

Learning as the basis of intelligence

• Some things we can all do (e.g. walking)

• Some things we can only learn (e.g. driving a car)

• We can learn a huge variety of things, including very difficult things

• Therefore our learning mechanism(s) are likely powerful enough to do everything we associate with intelligence• Though it may still be very convenient to “hard-code” a few really important

things

Page 11: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

A single algorithm?

[BrainPort; Martinez et al; Roe et al.]

Seeing with your tongue

Human echolocation (sonar)

Auditory

Cortex

adapted from A. Ng

• An algorithm for each “module”?

• Or a single flexible algorithm?

Page 12: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

What must that single algorithm do?

• Interpret rich sensory inputs

• Choose complex actions

Page 13: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

Why deep reinforcement learning?

• Deep = can process complex sensory input …and also compute really complex functions

• Reinforcement learning = can choose complex actions

Page 14: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

Some evidence in favor of deep learning

Page 15: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

Some evidence for reinforcement learning

• Percepts that anticipate reward become associated with similar firing patterns as the reward itself

• Basal ganglia appears to be related to reward system

• Model-free RL-like adaptation is often a good fit for experimental data of animal adaptation• But not always…

Page 16: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

What can deep learning & RL do well now?

• Acquire high degree of proficiency in domains governed by simple, known rules

• Learn simple skills with raw sensory inputs, given enough experience

• Learn from imitating enough human-provided expert behavior

Page 17: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

What has proven challenging so far?

• Humans can learn incredibly quickly• Deep RL methods are usually slow

• Humans can reuse past knowledge• Transfer learning in deep RL is an open problem

• Not clear what the reward function should be

• Not clear what the role of prediction should be

Page 18: Deep Reinforcement Learning CS 294 - 112rail.eecs.berkeley.edu/deeprlcoursesp17/docs/week_1_lecture_1_introduction.pdf1. From supervised learning to decision making 2. Basic reinforcement

Instead of trying to produce aprogramme to simulate the adultmind, why not rather try toproduce one which simulates thechild's? If this were then subjectedto an appropriate course ofeducation one would obtain theadult brain.

- Alan Turing

general learning algorithm

environment

ob

serv

atio

ns

acti

on

s


Reinforcement Learning & Apprenticeship Learning

Reinforcement Learning - Multi-Agent Reinforcement

Universal Reinforcement Learning Algorithms: … Reinforcement Learning Algorithms: Survey and ... class of models hypothesesenvironments M . ... Universal Reinforcement Learning Algorithms:

Reinforcement Learning

Deep Learning for Reinforcement Learning in · PDF fileDeep Learning for Reinforcement Learning in ... Deep Learning for Reinforcement Learning in Pacman Deep Learning für ... Während

Deep Reinforcement Learning with Forward Prediction, Memory, …rll.berkeley.edu/deeprlcoursesp17/docs/honglak_lecture.pdf · 1 Deep Reinforcement Learning with Forward Prediction,

Bayesian Reinforcement Learning - mlg.eng.cam.ac.ukmlg.eng.cam.ac.uk/rowan/files/BayesianReinforcementLearning.pdf · Introduction Bayesian Reinforcement Learning Bayesian Reinforcement

Reinforcement Learning and Deep Reinforcement Learningcse.ucdenver.edu/.../Class-22-Reinforcement-learning-DL.pdf · 2018. 11. 28. · Outlines 1 Principles of Reinforcement Learning

From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

Reinforcement Learning: Learning algorithms

Hierarchical Deep Reinforcement Learning: Integrating ...papers.nips.cc/paper/6233-hierarchical-deep-reinforcement-learning... · work that integrates deep reinforcement learning

Deep Learning for Reinforcement Learning in Pacman · Deep Learning for Reinforcement Learning in Pacman Deep Learning für Reinforcement Learning in Pacman Vorgelegte Bachelor-Thesis

Cooperative Inverse Reinforcement Learning...Cooperative Inverse Reinforcement Learning Dylan Hadfield-Menell CS237: Reinforcement Learning May 31, 2017

Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning Applications Summary

Eick: Reinforcement Learning. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning

Inverse Reinforcement Learning - Peoplecbfinn/_files/bootcamp_inverserl.pdf · Apprenticeship Learning via Inverse Reinforcement Learning. Good introduction to inverse reinforcement

Multi-Objective Reinforcement Learning using Sets of Pareto … · 2020. 10. 19. · learning and multi-objective reinforcement learning. 2.1 Reinforcement Learning A reinforcement

Reinforcement Learning - uni-freiburg.degki.informatik.uni-freiburg.de/.../recordings/reinforcement.pdf · Reinforcement Learning 3 What is Reinforcement Learning? Learning from interaction

Inverse Reinforcement Learning CS885 Reinforcement

Reinforcement Learning Chapter 13 What is Reinforcement Learning? Q-Learning Examples 1

Reinforcement Learning Lecture Inverse Reinforcement Learningipvs.informatik.uni-stuttgart.de/mlr/wp-content/uploads/2017/07/09... · Reinforcement Learning Inverse Reinforcement

The Reinforcement Learning Toolbox – Reinforcement Learning in Optimal Control Tasks

Tutorial: Deep Reinforcement Learning - Machine Learning ...hunch.net/~beygel/deep_rl_tutorial.pdfTutorial: Deep Reinforcement Learning - Machine Learning

Generalization in Reinforcement Learning: Successful ...papers.nips.cc/paper/1109-generalization-in-reinforcement-learning... · Generalization in Reinforcement Learning: Successful

What is Reinforcement Learning? - University of …rll.berkeley.edu/deeprlcoursesp17/docs/lec0.pdfLearning." AISTATS. 2011;M. Norouzi, S. Bengio, N. Jaitly, M. Schuster, Y. Wu, et

Reinforcement Learning or Active Inference?karl/Reinforcement Learning or Active... · Reinforcement Learning or Active Inference? ... From the point of view of reinforcement learning