1 decision making in basketball 2-point shot: easier, fewer points 3-point shot: more difficult,...
Post on 22-Dec-2015
218 views
TRANSCRIPT
1
Decision making in basketball• 2-point shot: easier, fewer points
• 3-point shot: more difficult, more points
Kobe BryantLA Lakers
31.6 PPG (2006-7)
3P attempts: 398 (23%)
2P attempts: 1,359 (77%)
Chris BoshToronto Raptors
26.3 PPG (2006-7)
35 (3%)
1,059 (97%)
3P success: 34%
2P success : 50%
34%
50%
2
NBA best 100 players (2006-2007)
2
3
3
N
N N
3
2 3
I
I I
N2,3 = # of 2,3 points shots
I2,3 = # 2,3 points earned0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
Bryant
Bosh
The matching law
3
The reward schedule
, 1 , 2 ,R t r A t A t A t
4
Herrnstein, JEAB, 1961
1
11
1 2
I
I I
1
1 2
N
N N
The matching law
5
Sugrue, Corrado & Newsome, Science, 2004
The matching law
6
The matching law
Gallistel et al., unpublished
7
Nj = # of attempts at alternative j investment in j
1 1
1 2 1 2
N I
N N I I
equal returns
Ij = # of points earned from alternative j income from j
The matching law
1 2
1 2
I I
N N
E 1 E 2R A R A
8
MATCHING MAXIMIZING
The matching law is very general. It is found in many animal types as well as humans, under very different experimental conditions.
9
freq [drugs]1–freq [work]
Example: addiction model
0 0.2 0.4 0.6 0.8 1
E[R|A=drugs]
after Herrnstein and Prelec, J Econ Perspect, 1991
10
Example: addiction model
0 0.2 0.4 0.6 0.8 1
freq [drugs]1–freq [work]
E[R|A=drugs]E[R|A=work]
matching
after Herrnstein and Prelec, J Econ Perspect, 1991
11
Example: addiction model
freq [drugs]1–freq [work]
0 0.2 0.4 0.6 0.8 1
E[R|A=drugs]E[R|A=work]E[R]
maximizing matching
after Herrnstein and Prelec, J Econ Perspect, 1991
12
Question:What is the neural basis of the matching law?
13
It is generally believed that learning is due to changes in the efficacy of synapses
Kennedy, Science, 2000
0.4 μm
14
Question:What is the neural basis of the matching law?
Question: What microscopic plasticity rules underlie adaptation to matching behavior?
15
Question:What is the neural basis of the matching law?
Hypothesis: the matching law results from synaptic plasticity that is driven by the covariance of reward and neural activity
16
Question:What is the neural basis of the matching law?
Hypothesis: the matching law results from synaptic plasticity that is driven by the covariance of reward and neural activity
17
• two random variables X, Y
Cov
Var Var
[ , ]
[ ] [ ]
X Yr
X Y
E[ ] E[ ]X X X Y Y Y
Covariance is a measure of dependence
X Y X Y
X Y
X Y
Cov , E[ ]
E[ ]
E[ ]
• correlation coefficient:
• covariance:
18
Covariance
Cov[X ,Y ] 0
Cov[X ,Y ]0
Cov[X ,Y ] 0
19
Hypothesis: the matching law results from synaptic plasticity that is driven by the covariance of reward and neural activity
20
Synaptic plasticity• Local signals affect synaptic
efficacies. Popular theory: Hebbain plasticity
pre postW DS S
• Global signals affect synaptic efficacies. Popular theory: dopamine gates Hebbian plasticity (Wickens)
pre postW S S
21Schultz, Dayan & Montague, Science, 1997
22
Synaptic plasticity• Local signals affect synaptic
efficacies. Popular theory: Hebbain plasticity
pre postW DS S
• Global signals affect synaptic efficacies. Popular theory: dopamine gates Hebbian plasticity (Wickens)
ED R R R
• Popular theory: dopamine codes the mismatch between reward and expected reward (Schultz)
pre postW S S
23
Synaptic plasticity
pre postW DS S
D R
pre postW R S S
E [ ] Cov[ , ]pre post pre postW R S S R S S E
Average trajectory approximation
24
Covariance-based plasticity rules
W R R N
W R N N
E
E
E Cov[ , ]W R N
Average trajectory approximation:
N=Spre , N=Spost , N=SpreSpost
25
The matching law
Stationary state of covariance-based plasticity
Hypothesis:
covariance-based synaptic plasticity The matching law
outline:
Cov , 0 R N
26
Assumptions
1. E[N|A=i] ≠ E[N|A≠i]
2. The dependence of the reward R on neural activity N is through the action A.
N1
N2
N3
N4N5
A R
hidden variables
neurons
action reward
27
Suppose that Assumptions 1 and 2 are satisfied
Cov , 0 E | 1 E | 2jR N R A R A The matching law
Theorem
28
Intuition
A Raction reward
Nneuron
Cov , 0 E | ER N R A i R
• In general R depends on A • If, as a result of the policy used by the subject,
R becomes independent of A then R also becomes independent of N
29
W R S S E
30
W R S S E
W RS
31
W R S S E
W RS
ij i jpre postW R R S M E
32
The matching law
Hypothesis:Covariance based synaptic plasticity underlies the matching law
Cov , 0 R N
Summary
Loewenstein & Seung, PNAS, 2006Loewenstein, PLoS Comp Biol, 2008
Theorem:
Disclaimer:There are learning rules that converge to Cov[R,N]=0 that are not driven by covariance