Progress in MURI 15 (2007- 09)

Mathematical modeling of decision behavior.

AFOSR, Alexandria, VA, Nov 17th, 2009

Phil Holmes1. Optimizing monkeys? Balancing noisy stimulus

information and reward expectations (Feng, Holmes, Rorie,

Newsome, PLoS Comp. Biol. 5 (2), 2009). 2. Time-dependent perturbations can distinguish

integrator dynamics (Zhou, Wong-

Lin, Holmes, Neural Comput., 21, 2336-2662, 2009). [3. Mean field reduction of a spiking neuron integrator

model with neuromodulation (Eckhoff, Wong-Lin, Holmes, SIAM J.

Appl. Dyn. Sys., in review, 2009).Not to be presented today. ]

-----------------------

1. Optimizing Monkeys&

Optimistic Math

Leaky competing accumulators reduce to OU

(actually, nonlinear DD/OU processes) via strong attraction to a 1-d subspace:

QuickTime™ and aVideo decompressor

are needed to see this picture.(Usher & McClelland, 1995,2001)

Subtracting the accumulated evidence yields an OU process (linearized DD case shown here). This generalizes to nonlinear systems: stochastic center manifolds.

y1 y2

s1s2

noise noise

leak leak

inhibn.

inhibn.

Fixed viewing time (cued response) tasks

can be modeled by DD/OU processes. We consider the PDF of sample paths of the SDE, which is governed by the forward Fokker-Planck or Kolmogorov PDE:

Interrogate solution at cue time T: is ? [Neyman-Pearson]

General solutions for time-varying drift(SNR) are available, so we can predict accuracy (psychometric functions):

QuickTime™ and aVideo decompressor

are needed to see this picture.

Integrating bottom-up (stimulus) and top-down (prior expectation) inputs: a motion discrimination task with four reward conditions. Introduces reward priors (expectations).

A. Rorie & W.T. Newsome, SfN 2006; in prep., 2008).

DD can incorporate priors: reward expectations

Reward cue Motion

We only model this part

HL

HH

LL

LH

Why monkeys?Electrophysiology!(Rorie, Reppas, Newsome)

DD/OU models with reward bias priors, 1

Choose a model for reward expectation. Two examples are:1. Initial condition set at start of motion period:

motion period only.2. Bias applied to drift rate throughout reward cue and motion periods:

reward cue period motion period.

for predicts shifted PMFs:

Feng, H, Rorie, Newsome, PLoS Comp. Biol. 5 (2), 2009.

Leak-dominated Balanced Inhibition-dominated(stable OU, recency) (DD, optimal) (unstable OU, primacy)

HL

LH

HH

LL

DD/OU models with reward bias priors, 2

These reward expectation models both lead to PMFs that can be written in the simple form

where b1 acuity = ability to

use signal (SNR)

b2 shift = estimate of

reward bias

depending on the model chosen. But unless we have a range of “priming” and viewing times* , the models can’t be distinguished on behavioral evidence alone: the parameters

cannot be separated. [*But see McClelland presentation.]

We can nonetheless compute shifts that maximize expected rewards,e.g., in simple case of fixed coherence : (More complex expressions hold for the mix of coherences actually used.)

~1/SNRreward ratio

DD/OU models with reward bias priors, 3

Fit the model to the Rorie-Newsome behavioral data from two adult male rhesus monkeys, averaged over all sessions:

monkey A monkey T

slope shift

For fixed the PMFs all reduceto a 2 parameter family:

DD/OU models with reward bias priors, 4

Maximize expected rewards: compute optimal shifts given animals’ slopes and compare with their actual shifts:

Both animals overshift, and T prefers alternative 2 when rewards are equal. But does this cost them much? How steep is the reward hill?

reward ratio sums of Gaussians

DD/OU models with reward bias priors, 5

Expected reward functions are rather flat, and overshifting costs less than undershifting. So they don’t lose much!

monkey A monkey T

99.5%performance bands

DD/OU models with reward bias priors, 6

Overall performance is good, although acuity (= slope b1) and shifts (b2) vary significantly from session to session. In spite of “visual correlation,” the animals do not exhibit significant b1 vs b2 correlations (performance tuning). Monkey A:

Blue is optimal; magenta curves are 99% & 97% of optimal. With few exceptions, A stays inside 97% in every session. Monkey T is not as good, shows bias to T2.

• DD processes extend to include top-down cognitive control: can be used to predict optimal PMFs. Two subjects tested both overshift, but garner 98 − 99% of maximum possible rewards, in spite of significant session-to-session variability. App of rigorous math methods.

• Brief impulsive perturbations during stimulus presentation can reveal the dynamics of integrators, distinguish pure DD processes from leak- and inhibition-dominated integrators*, and even from nonlinear model. (*stable, unstable OU processes)

Conclusions 1

2. Brief perturbations can distinguishing integrators

Drift-diffusion and OU processes with thresholds can model RT

tasks.

Zhou, Wong-Lin, H, Neural Comput. 21 (8), 2009.

Time-dependent perturbations can distinguish integrator types.

Deci

sion

vari

ab

le x

time

threshold for choice 1

threshold for choice 2

Mean RT

Variance

Perturbation modifies RT distributions

means standard deviations

Means and standard deviations of RT distributions are shifted in distinct manners for constant and rising drift rates (CD, TD) and for stable and unstable OU processes (SOU, UOU); positive pulse +, negative pulse _.

Zhou, Wong-Lin, H, Neural Comput. 21 (8), 2009.

Perturbn must be early to have strong effect!

Zero-effect perturbations

CD, TD, SOU and UOU are most easily distinguished by tuning a neg-pos pulse pair so that the mean RT is not changed.

These employ the recency/primacy effects previously noted by Usher-McClelland. The “simple” patterns seen here occur only for low noise and early perturbations (pert time T << mean RT; effects decay as T increases). For high noise the mean and SD patterns are less clear. Zhou, Wong-Lin, H, Neural Comput. 21

(8), 2009.

Further results and related work

The ideas extend to nonlinear systems (subcritical pitchfork case), although for effects are harder to interpret. Zhou, Wong-Lin, H, Neural Comput. 21 (8), 2009.

Gao-McClelland have replicated a slight decrease in the effect of perturbations delivered later, under the cued response (interrogation) protocol. Also, subjects appear more responsive to coherent motion early in the trial than later. See Gao presentation, and cf. fits of OU and variable drift DD models to data from J. Gold lab (Penn). (Eckhoff, H, Law, Conolly, Gold, New J. Phys.10, 2008.)

Perturbation experiments are planned (Usher, McClelland, Simen, …)

• DD processes extend to include top-down cognitive control: it can predict optimal PMFs. Two subjects tested both overshift, but garner 98 − 99% of their maximum possible rewards, in spite of significant session-to-session variability. App of rigorous math methods.

• Brief impulsive perturbations during stimulus presentation can reveal the dynamics of integrators, distinguish pure DD processes from leak- and inhibition-dominated integrators*, and even from nonlinear model. (*stable, unstable OU processes)

Conclusions 2

Preamble: optimal choicesThe drift-diffusion (DD) process, a cornerstone of 20th century physics, and perhaps the simplest stochastic ODE, is an optimal decision maker for two-alternative forced choice perception tasks with noisy data:

drift rate noise strength

Here is the accumulated evidence (the log likelihood ratio): when it reaches the threshold , declare R or L the winner. DD is a continuum limit of SPRT (Wald, 1947). The model has only 2 parameters:

Behavioral data, neural recordings in primates, fMRI and EEG in humans (MURI 16) and spiking neuron models support the contention that evidence is accumulated in this manner by cortical networks.

Behavioral evidence: RT distributions

Human reaction time data in free response mode can be fitted to the first passage threshold crossing times of a DD process.

Prior or bias toward one alternative can be implemented by setting starting point . Extended DD: variable drift rates & starting points.Simple expressions for mean RT, accuracy. Ratcliff et al., Psych Rev. 1978, 1999, 2004

Simen et al., J. Exp. Psych.:HPP, in press, 2009.

thresh. +Z

thresh. -Z

drift A

QuickTime™ and aVideo decompressor

are needed to see this picture.

Spiking neurons reduce to leaky accumulators Working hypothesis: motion sensitive cells in visual

cortex MT pass noisy signals on to LIP, FEF, … where integration occurs.

threshold

Experimental observations:K.H. Britten, M.N. Shadlen, W.T. Newsome, J.D. Schall & A. Movshon (various papers, 1992-2009).

Simulation, analysis and reduction of spiking neuron models:X.-J. Wang, Neuron, 2002; K-F. Wong & X.-J. Wang, J. Neurosci., 2006. Extension to model NE modulation of LIP: P. Eckhoff, K-F. Wong, PH. J. Neurosci, 29 (13), 2009;P. Eckhoff, K-F. Wong, PH. (in review, 2009).

MT

LIP

Neural evidence: firing ratesSpike rates of neurons in cortical areas rise during stimulus presentation, monkeys signal their choice after a threshold is crossed (like DD process).

J. Schall, V. Stuphorn, J. Brown, Neuron, 2002. Frontal eye field (FEF) recordings.

J.I Gold, M.N. Shadlen, Neuron, 2002. Lateral interparietal area (LIP) recordings.

thresholds

We can predict Psychometric Functions (PMFs):

Accuracy for fixed viewing time T, sorted by coherences.

DD model with variable drift:

scale coher. expt. asmpt. decay rate

Model fits to data from monkeys during training on the moving dots task. Animals learn to extract signal from noise.

Scale factor a: steady increase in SNR. (Eckhoff, H, Law, Conolly, Gold, New J. Phys., 2008)

Acc

ura

cy

Perturbing a nonlinear integrator

A nonlinear model can also be distinguished from CD, TD, SOU and UOU: both mean and SD effects decrease monotonically as T increases.

The “simple” patterns seen here occur only for low noise and early perturbations (T << mean RT).

Experiments are in planning stage (Usher, McClelland, Simen).

Zhou, Wong-Lin, H, Neural Comput. 21 (8), 2009.