a reinforcement learning model of song acquisition in the bird9.54/fall14/slides/class19_fee.pdfthat...

A reinforcement learning model of song

acquisition in the bird

Michale Fee

McGovern Institute

Department of Brain and Cognitive Sciences

Massachusetts Institute of Technology

9.54

November 12, 2014

0 kHz

10 kHz

1s

Fre

qu

en

cy

Motif Motif

Syllable

(~100ms)

Note (~10ms)

Structure of zebra finch song

Songbirds learn to sing by imitating their parents

De

cre

ase

d V

aria

bili

ty

Incre

ase

d S

imila

rity to

Tu

tor

Tutor Song

Subsong

Plastic Song

Crystallized

Overview

• The songbird as a model system for understanding how the brain

generates and learns complex sequential behaviors

• Review some current understanding of the mechanisms of song

production

• Describe progress in elucidating the role of cortical and basal ganglia

circuits in song learning.

• Some speculations on how insights from the songbird may inform our

understanding of mammalian BG function

A circuit for vocal production

RA

nXII

Motor Pathway

HVC

Cortex

Uva

Thalamus

Nottebohm et al, 1976, 1982

Hahnloser, Kozhevnikov and Fee, 2002

Antidromic activation allows identification

of RA-projecting neurons in HVC

HVC

RA

X

Stimulation

electrode

Extracellular

recording

electrode

HVC neurons burst throughout the song

Lynch, Okubo and Fee, in

preparation

Leonardo and Fee (J Neurosci, 2004)

Experimental evidence in favor of a clock model and

against a gesture trajectory extrema (GTE) model of HVC encoding Galen F. Lynch1, Tatsuo Okubo1*, Michael B. Lynn1, Alexander Hanuschkin2, Richard H.R. Hahnloser2, and Michale S. Fee1

1: McGovern Institute for Brain Research, Department of Brain and Cognitive Sciences, MIT, Cambridge, MA, USA

2: Institute of Neuroinformatics, University of Zurich and ETH Zurich, Zurich, Switzerland

LLL6

675.18

Introduction

HVC is a premotor area necessary for stereotyped adult song.

[4] Vu et al. (J. Neurosci. 1998)

[5] Long & Fee (Nature, 2008)

Songbirds learn and produce a complex sequence of vocal

gestures. •Zebra finches learn to sing by imitation

•Adults birds sing a precise, highly stereotyped song

•This learned requires precise motor control with10ms temporal resolution

• Projection neurons in HVC burst

sparsely during singing in adults [1].

• Different neurons are active at different

times

• Lesions of HVC eliminate stereotyped

elements of song [2,3].

• Disrupting HVC activity disrupts song

production [4].

• Cooling of HVC stretches stereotyped

components of the song [5]. Hahnloser et al. (Nature, 2002)

[1] Hahnloser et al. (Nature, 2002) [2] Nottebohm et al. (J Comp. Neurol., 1976) [3] Aronov et al. (Science, 2008)

Two different models of HVC coding have been proposed.

•It has been previously hypothesized that HVC neurons form a continuous

chain of activity, or ‘c

l

ock’ , that drives motor activity [6 ] .

•This model states HVC is active throughout the song, and that HVC activity

controls the timing of song.

*The Nakajima Foundation

and NIH Grant #DC009183

The clock model of HVC coding

The GTE model of HVC coding

Discriminating between models

Here we test the GTE hypotheses with a large

dataset of HVC neurons.

[6] Leonardo and Fee (J Neurosci., 2004) [7] Amador et al (Nature, 2013)

The GTE model makes testable predictions about the

statistics of HVC bursts. •The GTE model states that HVC bursts are clustered around GTE times, and

absent elsewhere [8].

•We compare the predictions of this model to a uniform null hypothesis in which

bursts are randomly placed with uniform probability.

•Clustering of bursts at GTEs makes specific predictions for

•The clustering of bursts around GTEs would be reflected in burst

statistics by an excess of short interburst intervals (IBIs). Long inter-

GTE intervals would produce gaps between bursts, and thus very long

IBIs. Combined, this would increase the variance and skew of the IBI

distriubtions compared to the variance and skew expected in the

uniform null hypothesis.

•The alignment of burst times to GTE times would be reflected by

significant cross correlation between burst times and GTE times.

HVC burst activity covers the song

motif, supporting the clock model.

Variability in burst density is

inconsistent with the GTE model

100ms

Burst #

1

66

100ms

Burst #

1

56

100ms

Burst #

1

91

Interburst intervals (IBIs) are


Bird A: 66 bursts, 40 neurons

Bird B: 56 bursts, 44 neurons

Bird C: 91 bursts, 64 neurons

Bursts are not cross correlated with

GTEs in the song

Conclusions


interneuron minima, inconsistent with

the GTE model

20 40 60 80 100 120 0

0.002

0.004

0.006

0.008

0.01

0.012

0.014

0.016 P = .51

P = 4x10-4

Longest IBI [ms]

P(L

on

gest

IBI)

Distribution of longest IBIs

Density variance [bursts2/ms2]

P(D

ensi

ty v

aria

nce

)

Bird C

0.01 0.02 0.03 0.04 0.05 0.06 0.07 0

20

40

60

80

100

120

140 P = .13

P < 1x10-6

• Amador et al. modeled the biophysics of song production as a low

dimensional system with two control parameters.

• With a relatively small dataset of projection neurons, Amador et al. observed

that burst times appeared to be aligned to extrema and discontinuities of

these control parameters, called gesture trajectory extrema (GTE).

Weak GTE hypothesis: HVC projection neurons preferentially

burst around GTEs

• Amador et al. further concluded that HVC projection neurons burst only at

GTEs, and nowhere else [7].

Strong GTE hypothesis: HVC projection neurons burst around

GTEs, and nowhere else.

• The weak GTE hypothesis is not inconsistent with the clock model.

• The strong GTE hypothesis is incompatible with the clock model, because it

predicts gaps in HVC activity.

•The GTE model predicts that HVC burst density

(#bursts per ms) has large variations

• We simulated burst density variance under the

GTE model and uniform null hypotheses

•The observed burst densities are highly

inconsistent with the GTE model and are

consistent with the null hypothesis

0

0.5

Bu

rst

De

nsi

ty

Distributions of burst density variance

0.005 0.01 0.015 0.02 0

50

100

150

200

250

300

350


P(D

ensi

ty v

aria

nce

)

P = .09

P =3x10-6

GTE Null

Bird A

Distribution of the # of IBIs < 5ms

10 20 30 40 50 60 70 80 0

0.005

0.01

0.015

0.02

0.025

# IBIs

P (

#IB

Is)

P = .81

P = 0.002

GTE

Null

20 40 60 80 100 120 0

0.002

0.004

0.006

0.008

0.01

0.012

P = .68

P = 0.028

Longest IBI [ms] P

(Lo

nge

st IB

I)


0 5 10 15 20 25 30 35 40 45

0.005

0.01

0.015

0.02

0.025

0.03


# IBIs

P (

#IB

Is)

P = 0.011

P = .71 GTE

Null

Bird C

Bird B

Bird A

50 100 150 0

1

2

3

4

5

6

7

8

9 x 10

-3 Distribution of longest IBIs

P = 9x10-4

P = .09

Longest IBI [ms]

P(L

on

ge

st IB

I)

10 15 20 25 30 35 40 45 0

0.005

0.01

0.015

0.02

0.025

0.03


P = 0.003

P = .27

GTE

Null

# IBIs

P (

#IB

Is)

GTEs for bird C:

• The GTE model states that interneuron minima should preferentially occur at

GTEs.

• Thus interneuron minima should be correlated with projection neuron bursts.

• We tested this prediction by analyzing 237 minima times from 17 interneurons

and 66 bursts from 40 projection neurons recorded in a single bird

• We observe no significant correlation between minima times and burst times.

Amador et al. (Nature, 2013)

•We test the GTE hypothesis by looking for these signatures in a large dataset

of HVC projection neurons

•We determine the probability of the data under each hypothesis by generating

surrogate datasets under the uniform null hypothesis and under the GTE

hypothesis.

Long IBIs Many short IBIs

GTEs

Bursts placed under GTE model

Bursts placed under null model

50ms

50ms

Bursts:

-0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0

1

2

3

4

5

6

7

8

9 Distribution of cross correlations at 0ms delay

Cross Correlation at 0ms

P(C

orr

ela

tio

n)

GTE Null

P = .19

P < 3x10-6

-100 -50 0 50 100 -0.4

-0.2

0

0.2

0.4

0.6

0.8

1

offset (burst-GTE ) [ms]

Cro

ss c

orr

ela

tio

n

Cross correlation between burst and GTE times

GTE

5-95% Null 5%

FWER Bird C

-100 -50 0 50 100 -0.2

0

0.2

0.4

0.6

0.8

offset (ms)

Cro

ss c

orr

ela

tio

n

Average cross correlation between

HVC burst and GTE times for each model

GTE

Null

Transitions for bird B: 50ms

Bursts:

-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0

0.5

1

1.5

2

2.5

3

3.5

Distriubtion of cross correlations at 0ms delay


P(C

orr

ela

tio

n)

Null

P = .20

-100 -50 0 50 100 -0.4

-0.2

0

0.2

0.4

0.6

offset (burst-GTE) [ms]

Cro

ss c

orr

ela

tio

n

Cross correlation between HVC bursts and song features

Null 5%

FWER Bird A

-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0

0.5

1

1.5

2

2.5

3

3.5

4 Distriubtion of cross correlations at 0ms delay


P(C

orr

ela

tio

n)

P = .90

Null

-100 -50 0 50 100 -0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

offset bursts-minima [ms]

Cro

ss c

orr

ela

tio

n

Cross correlation between projection neuron bursts

and interneuron minima

Null 5%

FWER

Bird A

PN bursts

Int. minima

25 100

1

2

4

Variance [ms2]

Ske

w [

ms

3]

Distribution of variance vs. skew

0 10 20 30 40 50 0

10

20

30

40

IBI (ms)

0 10 20 30 40 50

IBI (ms)

Co

un

t

Interburst interval distributions

Uniform

5-95%

Bird B

GTE

5-95%

25 100 400

0.5

1

2

4

Variance [ms2]

Ske

w [

ms

3]


25 100 400

0.5

1

2

4


Variance [ms2]

Ske

w [

ms

3]

0 10 20 30 40 50 0

20

40

60

IBI (ms)

0 10 20 30 40 50

IBI (ms)

Co

un

t


Uniform

5-95%

Bird C

GTE

5-95%

0 10 20 30 40 50

IBI (ms) 0 10 20 30 40 50

0

10

20

30

40

IBI (ms)

Null

5-95%

GTE

5-95%

Co

un

t Bird A


•We examined various measures of the IBI distribution under the GTE and

uniform null hypotheses

•The observed IBI distributions are highly inconsistent with the GTE hypothesis

and are consistent with the null hypothesis

•The GTE model states that projection neurons should burst near GTEs.

•This predicts that GTE times would be cross correlated with bursts times.

•We are testing this by deriving GTEs from the with published techniques[7,8]. In

one bird so far, we found no correlation between burst and GTE times.

• In another bird, we found that acoustic transitions in song are not correlated with

burst times.

• We have tested the GTE model with a dataset of 213 bursts from 148

identified projection neurons in HVC, recorded from three birds.

• We find that interburst intervals are highly inconsistent with the GTE model,

and are consistent with a uniform null model.

• We additionally find that HVC burst times are not correlated with GTE times or

acoustic transitions in the song.

• We find that interneuron minima are not significantly correlated with these

projection neuron bursts.

• Our findings are highly inconsistent with the strong form of the GTE

hypothesis. In addition, our data do not reject the uniform null hypothesis, and

therefore do not support even the weak GTE hypothesis.

[8] Perl et al (Phys. Rev. E, 2011)

10 20 30 40 0

20

40

60

80

Interburst interval (IBI) [ms]

Cou

nt

Average IBI distribution for each model

GTE

Null

0.005 0.01 0.015 0.02 0.025 0.03 0.035 0

50

100

150

200

250

P = .73

P =.009


P(D

ensi

ty

vari

ance

)

Bird B






LLL6

675.18

Introduction











times




production [4].







l






























100ms

Burst #

1

66

100ms

Burst #

1

56

100ms

Burst #

1

91







GTEs in the song

Conclusions



the GTE model

20 40 60 80 100 120 0

0.002

0.004

0.006

0.008

0.01

0.012

0.014

0.016 P = .51

P = 4x10-4

Longest IBI [ms]

P(L

on

gest

IBI)



P(D

ensi

ty v

aria

nce

)

Bird C

0.01 0.02 0.03 0.04 0.05 0.06 0.07 0

20

40

60

80

100

120

140 P = .13

P < 1x10-6







burst around GTEs















0

0.5

Bu

rst

Den

sity


0.005 0.01 0.015 0.02 0

50

100

150

200

250

300

350


P(D

ensi

ty v

aria

nce

)

P = .09

P =3x10-6

GTE Null

Bird A


10 20 30 40 50 60 70 80 0

0.005

0.01

0.015

0.02

0.025

# IBIs

P (

#IB

Is)

P = .81

P = 0.002

GTE

Null

20 40 60 80 100 120 0

0.002

0.004

0.006

0.008

0.01

0.012

P = .68

P = 0.028

Longest IBI [ms]

P(L

on

gest

IBI)


0 5 10 15 20 25 30 35 40 45

0.005

0.01

0.015

0.02

0.025

0.03


# IBIs

P (

#IB

Is)

P = 0.011

P = .71 GTE

Null

Bird C

Bird B

Bird A

50 100 150 0

1

2

3

4

5

6

7

8

9 x 10


P = 9x10-4

P = .09

Longest IBI [ms]

P(L

on

ge

st IB

I)

10 15 20 25 30 35 40 45 0

0.005

0.01

0.015

0.02

0.025

0.03


P = 0.003

P = .27

GTE

Null

# IBIs

P (

#IB

Is)

GTEs for bird C:


GTEs.










hypothesis.


GTEs



50ms

50ms

Bursts:

-0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0

1

2

3

4

5

6

7

8



P(C

orr

ela

tio

n)

GTE Null

P = .19

P < 3x10-6

-100 -50 0 50 100 -0.4

-0.2

0

0.2

0.4

0.6

0.8

1


Cro

ss c

orr

ela

tio

n


GTE

5-95% Null 5%

FWER Bird C

-100 -50 0 50 100 -0.2

0

0.2

0.4

0.6

0.8

offset (ms)

Cro

ss c

orr

ela

tio

n



GTE

Null


Bursts:

-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0

0.5

1

1.5

2

2.5

3

3.5



P(C

orr

ela

tio

n)

Null

P = .20

-100 -50 0 50 100 -0.4

-0.2

0

0.2

0.4

0.6


Cro

ss c

orr

ela

tio

n


Null 5%

FWER Bird A

-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0

0.5

1

1.5

2

2.5

3

3.5



P(C

orr

ela

tio

n)

P = .90

Null

-100 -50 0 50 100 -0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4


Cro

ss c

orr

ela

tio

n



Null 5%

FWER

Bird A

PN bursts

Int. minima

25 100

1

2

4

Variance [ms2]

Ske

w [

ms

3]


0 10 20 30 40 50 0

10

20

30

40

IBI (ms)

0 10 20 30 40 50

IBI (ms)

Co

un

t


Uniform

5-95%

Bird B

GTE

5-95%

25 100 400

0.5

1

2

4

Variance [ms2]

Ske

w [

ms

3]


25 100 400

0.5

1

2

4


Variance [ms2]

Ske

w [

ms

3]

0 10 20 30 40 50 0

20

40

60

IBI (ms)

0 10 20 30 40 50

IBI (ms)

Co

un

t


Uniform

5-95%

Bird C

GTE

5-95%

0 10 20 30 40 50

IBI (ms) 0 10 20 30 40 50

0

10

20

30

40

IBI (ms)

Null

5-95%

GTE

5-95%

Co

un

t Bird A











burst times.













10 20 30 40 0

20

40

60

80


Co

un

t


GTE

Null

0.005 0.01 0.015 0.02 0.025 0.03 0.035 0

50

100

150

200

250

P = .73

P =.009


P(D

ensi

ty

vari

ance

)

Bird B






LLL6

675.18

Introduction











times




production [4].







l






























100ms

Burst #

1

66

100ms

Burst #

1

56

100ms

Burst #

1

91







GTEs in the song

Conclusions



the GTE model

20 40 60 80 100 120 0

0.002

0.004

0.006

0.008

0.01

0.012

0.014

0.016 P = .51

P = 4x10-4

Longest IBI [ms]

P(L

on

gest

IBI)



P(D

ensi

ty v

aria

nce

)

Bird C

0.01 0.02 0.03 0.04 0.05 0.06 0.07 0

20

40

60

80

100

120

140 P = .13

P < 1x10-6







burst around GTEs















0

0.5

Bu

rst

Den

sity


0.005 0.01 0.015 0.02 0

50

100

150

200

250

300

350


P(D

ensi

ty v

aria

nce

)

P = .09

P =3x10-6

GTE Null

Bird A


10 20 30 40 50 60 70 80 0

0.005

0.01

0.015

0.02

0.025

# IBIs

P (

#IB

Is)

P = .81

P = 0.002

GTE

Null

20 40 60 80 100 120 0

0.002

0.004

0.006

0.008

0.01

0.012

P = .68

P = 0.028

Longest IBI [ms]

P(L

on

gest

IBI)


0 5 10 15 20 25 30 35 40 45

0.005

0.01

0.015

0.02

0.025

0.03


# IBIs

P (

#IB

Is)

P = 0.011

P = .71 GTE

Null

Bird C

Bird B

Bird A

50 100 150 0

1

2

3

4

5

6

7

8

9 x 10


P = 9x10-4

P = .09

Longest IBI [ms]

P(L

on

ge

st IB

I)

10 15 20 25 30 35 40 45 0

0.005

0.01

0.015

0.02

0.025

0.03


P = 0.003

P = .27

GTE

Null

# IBIs

P (

#IB

Is)

GTEs for bird C:


GTEs.










hypothesis.


GTEs



50ms

50ms

Bursts:

-0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0

1

2

3

4

5

6

7

8



P(C

orr

ela

tio

n)

GTE Null

P = .19

P < 3x10-6

-100 -50 0 50 100 -0.4

-0.2

0

0.2

0.4

0.6

0.8

1


Cro

ss c

orr

ela

tio

n


GTE

5-95% Null 5%

FWER Bird C

-100 -50 0 50 100 -0.2

0

0.2

0.4

0.6

0.8

offset (ms)

Cro

ss c

orr

ela

tio

n



GTE

Null


Bursts:

-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0

0.5

1

1.5

2

2.5

3

3.5



P(C

orr

ela

tio

n)

Null

P = .20

-100 -50 0 50 100 -0.4

-0.2

0

0.2

0.4

0.6


Cro

ss c

orr

ela

tio

n


Null 5%

FWER Bird A

-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0

0.5

1

1.5

2

2.5

3

3.5



P(C

orr

ela

tio

n)

P = .90

Null

-100 -50 0 50 100 -0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4


Cro

ss c

orr

ela

tio

n



Null 5%

FWER

Bird A

PN bursts

Int. minima

25 100

1

2

4

Variance [ms2]

Ske

w [

ms

3]


0 10 20 30 40 50 0

10

20

30

40

IBI (ms)

0 10 20 30 40 50

IBI (ms)

Co

un

t


Uniform

5-95%

Bird B

GTE

5-95%

25 100 400

0.5

1

2

4

Variance [ms2]

Ske

w [

ms

3]


25 100 400

0.5

1

2

4


Variance [ms2]

Ske

w [

ms

3]

0 10 20 30 40 50 0

20

40

60

IBI (ms)

0 10 20 30 40 50

IBI (ms)

Co

un

t


Uniform

5-95%

Bird C

GTE

5-95%

0 10 20 30 40 50

IBI (ms) 0 10 20 30 40 50

0

10

20

30

40

IBI (ms)

Null

5-95%

GTE

5-95%

Co

un

t Bird A











burst times.













10 20 30 40 0

20

40

60

80


Cou

nt


GTE

Null

0.005 0.01 0.015 0.02 0.025 0.03 0.035 0

50

100

150

200

250

P = .73

P =.009


P(D

ensi

ty

vari

ance

)

Bird B

Activity of RA neurons during

singing

Yu and Margoliash, 1996

Motif

Leonardo and Fee, 2005

HVC

RA

Extracellular

recording

electrode

Simple sequence generation circuit

Sparse representation of time

Output


Sparse representation of time

Output

Simple sequence generation circuit


HVC is the ‘clock’ of the song motor

pathway

Brain cooling to localize dynamics

np

RA

nXII

HVC0.0A

0.25A

-0.25A

-0.5A

-0.75A

...

Bilateral cooling of HVC causes

uniform slowing of the song

5 mm

Long and Fee, Nature 2008

A simple reinforcement model of song

learning

Song motor

systemSong Exploratory

variability

Song evaluation

Auditory

feedback

Auditory Memory

Doya and Sejnowski, 1989

--

Error/Reinforcement signal

RA

nXII

Motor Pathway

HVC

A separate circuit for song learning

Cortex

LMAN

DLM

Thalamus

Anterior Forebrain Pathway

(AFP)

•The learning pathway is not necessary for adult song production , but is required

for learning (Bottjer, 1984, Scharff and Nottebohm, 1991)

Instructive signal

•Bottjer proposed that the AFP transmits an instructive signal that guides plasticity

in the motor pathway

Basal Ganglia

Area X

RA

nXII

LMAN

HVC

Separate premotor pathways for

stereotyped song and variability

Sequence generator

Variability generator

Kao et al, 2005

Ölveczky et al, 2005

Aronov et al, 2008

Stepanek and Doupe, 2010

RA

nXII

LMAN

HVC

TTX or Muscimol



Sequence generator


Kao et al, 2005


Aronov et al, 2008


Transient inactivation of the learning pathway

55 day old bird

RA

nXII

LMAN

HVC

Olveczky, Andalman, and Fee, 2005

LMAN drives exploratory variability in song

LMAN intact

LMAN inactivated

0 20 40 60

-20

0

20

Resid

ua

l P

itch (

Hz)

Time (ms)

LMAN intact

250 ms

LMAN inactivated

30 dB

LMAN also drives early song ‘babbling’

Goldberg and Fee, 2011

RA

nXII

LMAN

Motor Pathway

Learning Pathway (AFP)

HVC

HVC lesions abolish all stereotyped song structure

HVC lesions abolish all stereotyped song structure

Pre HVC lesion Post HVC lesion

Transient pharmacological inactivation of HVC produces the same effect

Aronov, Andalman and Fee, Science 2008,

Subsong bird

Plastic song bird

Adult bird

The basal ganglia are not necessary for

subsong or vocal variability in juvenile birds

30

dB

SubsongPre-lesion

250 ms

Post-lesion

HVC

RA

nXIIts

LMAN

XDLM X

Goldberg and Fee, 2011

• Lesions of the BG have little or no acute effect on juvenile song variability.

• Local cooling in LMAN slow timescales of babbling exploratory vocal variability is generated by local circuit dynamics within LMAN.

RA

nXII



HVC

Sequence generator

LMAN


Kao et al, 2005


Aronov et al, 2008


RA

nXII

LMAN

HVC



Sequence generator


Kao et al, 2005


Aronov et al, 2008


Instructive signal

RA

nXII

LMAN

HVC



Sequence generator


Instructive signal

Area XDLM

Tchernichovski, Mitra, Lints, Nottebohm, 2001

TutorPupil

Days of Training

5 8 12 20 30

606 Hz

Harmonic

Stack

Pitch (Hz) 568 554 551 596 607

Song learning is slow

Experimental control of song learning

0.5

0.6

Targeted region

of syllable

Vocaliz

ed

Pitch

(kH

z)

Heard

Feedback Noise

Pitch threshold

Andalman and Fee 2009; Tumer and Brainard 2007

Speaker

DSP

Brain

Cranial airsac

Mic

Conditional auditory feedback drives pitch

learning

Tumer and Brainard 2007

Pitch (

Hz)

550

650

25 ms

0 h 2 h 4 h

Pitch (

Hz)

550

650

Andalman and Fee 2009

-50 0 500

5

10

15

ΔPitch, Day (Hz)

Ob

se

rva

tio

ns

Many days of sequential learning

120 125 130 135 140 145 150 155 160 165

500

600

700

141 142 143 144470

600

161 162 163 164520

650

Pitch

(H

z)

Days Post Hatch Days Post Hatch

Days Post Hatch

Pitch

(H

z)

-50 0 500

5

10

15

ΔPitch, Overnight (Hz)

Ob

se

rva

tio

ns Up Days

Down Days

Up Days

Down Days

Motor parameter space

AFP-driven

variability

Where does this learning occur in the song

control circuit?

Motor pathway


AFP-driven

variability

AFP-driven

bias

Error gradient

(reduced error)

Motor pathway


control circuit?


AFP-driven

variability

AFP-driven

bias

Error gradient

(reduced error)

Motor pathway

Plasticity

in motor

pathway


control circuit?

AFP-driven

bias

Plasticity

in motor

pathway

HVC

RA

nXIIts

LMAN

XDLM

Motor PathwayAnterior Forebrain Pathway (AFP)

HVC

RA

nXIIts

LMAN

XDLM


control circuit?

during LMAN inactivation

25 Hz

Pitch

(H

z) TTX

2 h ∆Pitch (Hz)

-50 0 500

2

4

6

8 TTX

Ob

se

rva

tio

ns

Up Days

Down Days

Δ

Vehicle

Pitch

(H

z)

Vehicle

∆Pitch (Hz)-50 0 50

0

2

4

6

8

10

Up Days

Down Days

Ob

se

rva

tio

ns

Andalman and Fee, PNAS 2009

470

600

TTX TTX TTX TTXPBS PBS PBS PBS

?

Pitch

(H

z)

drug

reservoir

cap

inflow tube

outlet tube

dialysis membrane

skull

dental acrylic

LMAN

Does AFP-driven variability become biased to

reduce vocal errors?


AFP-driven

variability

AFP-driven

error-reducing

bias

Error gradient

(reduced error)

Plasticity

in motor

pathway

Yes!!

Motor pathway

Is all song learning mediated by AFP bias?

120 125 130 135 140 145 150 155 160 165

500

600

700

Days Post Hatch

Pitch

(H

z)

Many days of sequential learning

120 125 130 135 140 145 150 155 160 165

500

600

Days post-hatch

Pitch (

Hz)

baseline

LMAN(+)

LMAN(-)

Is all song learning mediated by AFP bias?

120 125 130 135 140 145 150 155 160 165

500

600

Days post-hatch

Pitch (

Hz)

baseline

LMAN(+)

LMAN(-)

AFP bias is highly predictive of motor pathway plasticity

within the next 24 hoursP

itch

β

Day 1

Δm

Nig

ht

Day 2 Day 3

Andalman and Fee, 2009

Warren et al, 2011

-4 -2 0 2 40

0.2

0.4

0.6

0.8

1

Lag (days)

Corr

ela

tion C

oeffic

ient (r

2)

-100 0 1000

50

100

Lag = -2 d Lag = -1 d Lag = 0 d

Δm

(H

z)

(dow

n d

ays in

v.)

Estimated AFP bias (Hz, down days inverted)

Days


AFP-driven

variability

AFP-driven

bias

motor

pathway

plasticityMotor pathway

motor

pathway

plasticity

Error gradient

(reduced error)

Day 1 Day 2 Day 3

Motor pathway plasticity appears to ‘integrate’AFP bias

X

How is AFP bias generated?

RA

nXII

HVC

LMAN

• Area X receives an efference copy of variability signals sent to RA.

• If Area X also receives an evaluation signal, then X could figure out which variations

lead to better song performance.

• Dopaminergic midbrain (VTA) has been shown to signal reward prediction error

• Do X-projecting VTA neurons carry error-related signals?

VTA

Schultz, 2000

A descending pathway from higher-order auditory areas

to VTA/SNc

AIV

RA

VTA

X

CM

Aud

Keller and Hahnloser, 2008

Gale, Perkel 2008

Mandelblat-Cerf et al, 2014

Retrograde label from VTA

AIV

Ventral Intermediate Arcopallium (AIV)

X

Is AIV necessary for song learning?

nXII

HVC

LMAN

VTA

NeuN

Las, Denisenko, Mandelblat-Cerf, eLife, 2014

Is AIV necessary for song learning?

Bird tutored in

home cage

AIV lesion

40 90

Check

imitationBird isolated

Age (days post hatch)

AIV lesioned pupil #2 – Adult song

Example 1

Example 2

Tutor

AIV lesion produces profound song

learning deficits

AIV lesions produce profound song

learning deficits

AIV lesioned –

adult song

Tutor

Lesioned controlUnlesioned controlAIV lesion

Similarity of

unrelated birds

Do AIV neurons transmit an ‘error’ signal to

VTA during singing?

XnXII

HVC

LMAN

VTA-2 0 2 4 6 8 10 12 14

-0.4

-0.2

0.0

0.2

0.4

Vo

lta

ge

(m

V)

Time from stim (ms)

200 ms

Noise burst

Do AIV neurons transmit an ‘error’ signal to

VTA during singing?

AIV neurons show error-related signals

Mandelblat-Cerf, Las, Denisenko, under review

A descending pathway from higher-order auditory areas

to VTA/SNc

AIV

RA

VTA

X

CM

Aud

Keller and Hahnloser, 2008

Gale, Perkel 2008

Mandelblat-Cerf et al, 2014

Retrograde label from VTA

AIV

Ventral Intermediate Arcopallium (AIV)

RA

nXII

HVC

LMAN

How it all works: a hypothesis

VTA

X


CM

Aud

X

VTA

LMAN

HVC


DLM

X

LMAN

HVC


RA

HVC

LMAN


321

VTA

HVC (Time Sequence)

LMAN

Pallidal

Thalamus

MSN

To RA

Area X

HVC(X) firing patterns

4 k

Hz

1

2

3

4

5

6

7

100 ms

A model of basal ganglia function with

functionally distinct inputs for context, motor

efference copy, and reward

The AFP forms a classic

cortical-BG-thalamo-cortical loop

321

VTA

HVC (Time Sequence)

LMAN

Pallidal

Thalamus

MSN

To RA

Area X

Learning rule:

Strengthen HVC synapse

after coincidence

of LMAN, HVC and DA inputs VTA




321

VTA

HVC (Time Sequence)

LMAN

Pallidal

Thalamus

MSN

To RA

Area X




Time-dependent

bias of one LMAN

neuron Goldberg and Fee 2010

LMAN

HVC

1

2

3

MSN

To RA

321

VTA

HVC (Time Sequence)

LMAN

Pallidal

Thalamus

MSN

To RA

Area X

HVC synapses

Timing

Drive MSNs

Plastic

Selective for single synapses

LMAN synapses

Action

Do NOT drive MSNs

Not plastic

Global signal




VTALMAN

MSN

HVC

LMAN

MSN

HVC

A learning rule with an eligibility trace allows

delayed reward

LMAN

HVC

VTA

ΔWHVC-X

Eligibility

traceEHVC-X

EHVC-X = LH

DWHVC-X = bEHVC-XR

HVC 1 HVC 2

LMAN

MSN

Hypothesis for HVC-LMAN synaptic interaction

on striatal MSNs

HVC on spines

LMAN on

dendritic shafts

t = 1

HVC 1 HVC 2

LMAN

MSN


on striatal MSNs

t = 2

HVC 1 HVC 2

LMAN

MSN


on striatal MSNs

t = 2

HVC 1 HVC 2

LMAN

MSN


on striatal MSNs

Dopamine

HVC 1 HVC 2

LMAN

MSN

Synapse

Strengthened


on striatal MSNs

Serial Block Face Scanning EM

Collaboration with Winfried Denk and Jörgen Kornfeld

LMAN

200 µm

200 µm

HVC

200 µm

200 µm

Distinct morphology of HVC and LMAN axons

Michael Stetner

Axonal arbor of LMAN neuron in Area X Axonal arbor of HVC neuron in Area X

MSN HVC-like LMAN-like

~94% of synapses onto spines are

from HVC-like axons

Inputs onto MSN spines originate primarily from HVC

Putative LMAN axons

Putative HVC axons

The role of the basal ganglia in songbird

vocal learning

• LMAN directly drives ‘exploratory variability’ in the song motor pathway.

• LMAN-driven variability becomes biased during learning, in the direction of

improved song performance.

• We have found evidence that a dopaminergic pathway to the songbird BG may

carry ‘performance’ error-related information.

• We hypothesize that the basal ganglia determine which song variations lead to

better performance and bias the variability in the direction of improved

performance.

• We have proposed a testable model of basal ganglia function that explicitly

incorporates an efference copy of cortically-generated motor actions.

The Fee Lab

Current Lab Members

• Anusha Narayan

• Natalia Denissenko

• Tatsuo Okubo

• Michael Stetner

• Emily Mackevicius

• Galen Lynch

web.mit.edu/feelab

Former Lab Members

• Richard Hahnloser

• Alexay Kozhevnikov

• Anthony Leonardo

• Michael Long

• Bence Ölveczky

• Dmitriy Aronov

• Aaron Andalman

• Lena Veit

• Jakob Förster

• Liora Las

• Jesse Goldberg

• Yael MandelblatFunding: National Institutes of Health - NIMH, NIDCD



RA

Sequence

Stereotypy

Precision

HVC

UvaRandomness

Variability

Exploration

LMAN

DLM

Motor Output

SubsongAdult song X

LMAN drives subsong

Stimulating electrode

nXII

LMAN

Area XDLM

HVC

RA

Insta

nta

ne

ous

firing r

ate

(Hz)

700

0

20 dB

Sound amplitude

250 ms

LMAN(RA) neurons exhibit premotor correlation

with subsong syllables

Insta

nta

ne

ous

firing r

ate

(Hz)

700

0

20 dB

Sound amplitude

200 ms


with subsong syllables

20 dB

700

0

Insta

nta

neous

firing r

ate

(Hz)

Neuron 14

Sound amplitude

250 ms

*

*

-100 0 100 2000

20

40

60

Time relative to offset (ms)

Mean f

irin

g r

ate

(H

z)

Sound amplitude


with subsong acoustic structure

Summary

• The AFP can generate a direct premotor bias that reduces vocal errors.

• The learning accumulated across many days of training is encoded primarily in plasticity in the motor pathway.

• The contribution of the AFP is limited to the learning that occurred most recently (during the same day).

• AFP bias is predictive of subsequent plasticity in the motor pathway within the next 24 hours

a reinforcement learning model of song acquisition in the bird9.54/fall14/slides/class19_fee.pdfthat...

Documents