1 decision making in basketball 2-point shot: easier, fewer points 3-point shot: more difficult,...

1

Decision making in basketball• 2-point shot: easier, fewer points

• 3-point shot: more difficult, more points

Kobe BryantLA Lakers

31.6 PPG (2006-7)

3P attempts: 398 (23%)

2P attempts: 1,359 (77%)

Chris BoshToronto Raptors

26.3 PPG (2006-7)

35 (3%)

1,059 (97%)

3P success: 34%

2P success : 50%

34%

50%

2

NBA best 100 players (2006-2007)

2

3

3

N

N N

3

2 3

I

I I

N2,3 = # of 2,3 points shots

I2,3 = # 2,3 points earned0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

Bryant

Bosh

The matching law

3

The reward schedule

, 1 , 2 ,R t r A t A t A t

4

Herrnstein, JEAB, 1961

1

11

1 2

I

I I

1

1 2

N

N N

The matching law

5

Sugrue, Corrado & Newsome, Science, 2004

The matching law

6

The matching law

Gallistel et al., unpublished

7

Nj = # of attempts at alternative j investment in j

1 1

1 2 1 2

N I

N N I I

equal returns

Ij = # of points earned from alternative j income from j

The matching law

1 2

1 2

I I

N N

E 1 E 2R A R A

8

MATCHING MAXIMIZING

The matching law is very general. It is found in many animal types as well as humans, under very different experimental conditions.

9

freq [drugs]1–freq [work]

Example: addiction model

0 0.2 0.4 0.6 0.8 1

E[R|A=drugs]

after Herrnstein and Prelec, J Econ Perspect, 1991

10

Example: addiction model

0 0.2 0.4 0.6 0.8 1

freq [drugs]1–freq [work]

E[R|A=drugs]E[R|A=work]

matching

after Herrnstein and Prelec, J Econ Perspect, 1991

11

Example: addiction model

freq [drugs]1–freq [work]

0 0.2 0.4 0.6 0.8 1

E[R|A=drugs]E[R|A=work]E[R]

maximizing matching

after Herrnstein and Prelec, J Econ Perspect, 1991

12

Question:What is the neural basis of the matching law?

13

It is generally believed that learning is due to changes in the efficacy of synapses

Kennedy, Science, 2000

0.4 μm

14

Question:What is the neural basis of the matching law?

Question: What microscopic plasticity rules underlie adaptation to matching behavior?

15

Question:What is the neural basis of the matching law?

Hypothesis: the matching law results from synaptic plasticity that is driven by the covariance of reward and neural activity

16

Question:What is the neural basis of the matching law?

Hypothesis: the matching law results from synaptic plasticity that is driven by the covariance of reward and neural activity

17

• two random variables X, Y

Cov

Var Var

[ , ]

[ ] [ ]

X Yr

X Y

E[ ] E[ ]X X X Y Y Y

Covariance is a measure of dependence

X Y X Y

X Y

X Y

Cov , E[ ]

E[ ]

E[ ]

• correlation coefficient:

• covariance:

18

Covariance

Cov[X ,Y ] 0

Cov[X ,Y ]0

Cov[X ,Y ] 0

19

Hypothesis: the matching law results from synaptic plasticity that is driven by the covariance of reward and neural activity

20

Synaptic plasticity• Local signals affect synaptic

efficacies. Popular theory: Hebbain plasticity

pre postW DS S

• Global signals affect synaptic efficacies. Popular theory: dopamine gates Hebbian plasticity (Wickens)

pre postW S S

21Schultz, Dayan & Montague, Science, 1997

22

Synaptic plasticity• Local signals affect synaptic

efficacies. Popular theory: Hebbain plasticity

pre postW DS S

• Global signals affect synaptic efficacies. Popular theory: dopamine gates Hebbian plasticity (Wickens)

ED R R R

• Popular theory: dopamine codes the mismatch between reward and expected reward (Schultz)

pre postW S S

23

Synaptic plasticity

pre postW DS S

D R

pre postW R S S

E [ ] Cov[ , ]pre post pre postW R S S R S S E

Average trajectory approximation

24

Covariance-based plasticity rules

W R R N

W R N N

E

E

E Cov[ , ]W R N

Average trajectory approximation:

N=Spre , N=Spost , N=SpreSpost

25

The matching law

Stationary state of covariance-based plasticity

Hypothesis:

covariance-based synaptic plasticity The matching law

outline:

Cov , 0 R N

26

Assumptions

1. E[N|A=i] ≠ E[N|A≠i]

2. The dependence of the reward R on neural activity N is through the action A.

N1

N2

N3

N4N5

A R

hidden variables

neurons

action reward

27

Suppose that Assumptions 1 and 2 are satisfied

Cov , 0 E | 1 E | 2jR N R A R A The matching law

Theorem

28

Intuition

A Raction reward

Nneuron

Cov , 0 E | ER N R A i R

• In general R depends on A • If, as a result of the policy used by the subject,

R becomes independent of A then R also becomes independent of N

29

W R S S E

30

W R S S E

W RS

31

W R S S E

W RS

ij i jpre postW R R S M E

32

The matching law

Hypothesis:Covariance based synaptic plasticity underlies the matching law

Cov , 0 R N

Summary

Loewenstein & Seung, PNAS, 2006Loewenstein, PLoS Comp Biol, 2008

Theorem:

Disclaimer:There are learning rules that converge to Cov[R,N]=0 that are not driven by covariance