lecture 13: associative memory references: d amit, n brunel, cerebral cortex 7, 237-252 (1997) n...
TRANSCRIPT
Lecture 13: Associative Memory
References:
D Amit, N Brunel, Cerebral Cortex 7, 237-252 (1997)
N Brunel, Network 11, 261-280 (2000)
N Brunel, Cerebral Cortex 13, 1151-1161 (2003)
J Hertz, in Models of Neural Networks IV (L van Hemmen, J Cowan and E Domany, eds) Springer Verlag, 2002; sect 1.4
What is associative memory?
• “Patterns”: firing activity of specific sets of neurons (Hebb: “assemblies”)
What is associative memory?
• “Patterns”: firing activity of specific sets of neurons (Hebb: “assemblies”)
• “Store” patterns in synaptic strengths
What is associative memory?
• “Patterns”: firing activity of specific sets of neurons (Hebb: “assemblies”)
• “Store” patterns in synaptic strengths
• Recall: Given input (initial activity pattern) not equal to any stored pattern, network dynamics should take it to “nearest” (most similar) stored pattern
What is associative memory?
• “Patterns”: firing activity of specific sets of neurons (Hebb: “assemblies”)
• “Store” patterns in synaptic strengths
• Recall: Given input (initial activity pattern) not equal to any stored pattern, network dynamics should take it to “nearest” (most similar) stored pattern
(categorization, error correction, …)
Implementation in balanced excitatory-inhibitory network
Model (Amit & Brunel): p non-overlapping excitatory subpopulations
Implementation in balanced excitatory-inhibitory network
Model (Amit & Brunel): p non-overlapping excitatory subpopulationseach of size n = fN (fp < 1)
Implementation in balanced excitatory-inhibitory network
Model (Amit & Brunel): p non-overlapping excitatory subpopulationseach of size n = fN (fp < 1)
stronger connections within subpopulations (“assemblies”)
Implementation in balanced excitatory-inhibitory network
Model (Amit & Brunel): p non-overlapping excitatory subpopulationseach of size n = fN (fp < 1)
stronger connections within subpopulations (“assemblies”)weakened connections between subpopulations
Implementation in balanced excitatory-inhibitory network
Model (Amit & Brunel): p non-overlapping excitatory subpopulationseach of size n = fN (fp < 1)
stronger connections within subpopulations (“assemblies”)weakened connections between subpopulations
Looking for selective states: higher rates in a single assembly
Model
Like Amit-Brunel model (Lecture 9) except for exc-exc synapses:
From within the same assembly:
Model
Like Amit-Brunel model (Lecture 9) except for exc-exc synapses:
From within the same assembly: 1111 JgJ
Model
Like Amit-Brunel model (Lecture 9) except for exc-exc synapses:
From within the same assembly: 1111 JgJ (strengthened, “Hebb” rule)
Model
Like Amit-Brunel model (Lecture 9) except for exc-exc synapses:
From within the same assembly:
From outside the assembly:
1111 JgJ
1111 JgJ
(strengthened, “Hebb” rule)
(weakened, “anti-Hebb”)
Model
Like Amit-Brunel model (Lecture 9) except for exc-exc synapses:
From within the same assembly:
From outside the assembly:
Otherwise: no change
1111 JgJ
1111 JgJ
(strengthened, “Hebb” rule)
(weakened, “anti-Hebb”)
Model
Like Amit-Brunel model (Lecture 9) except for exc-exc synapses:
From within the same assembly:
From outside the assembly:
Otherwise: no change
1111 JgJ
1111 JgJ
(strengthened, “Hebb” rule)
(weakened, “anti-Hebb”)
To conserve average strength: 1)1( gffg
Model
Like Amit-Brunel model (Lecture 9) except for exc-exc synapses:
From within the same assembly:
From outside the assembly:
Otherwise: no change
1111 JgJ
1111 JgJ
(strengthened, “Hebb” rule)
(weakened, “anti-Hebb”)
To conserve average strength:
=>
1)1( gffg
ffg
g
11
Mean field theoryRates: active assembly
inactive assembliesrest of excitatory neuronsinhibitory neuronsext input neurons
actr
r1r
2r0r
Mean field theory
Input current to neurons in the active assembly:
212111010 ])1()1([ rJrgpfrgpfrfgJrJI actact
Rates: active assemblyinactive assembliesrest of excitatory neuronsinhibitory neuronsext input neurons
actr
r1r
2r0r
Mean field theory
Input current to neurons in the active assembly:
212111010 ])1()1([ rJrgpfrgpfrfgJrJI actact
Rates: active assemblyinactive assembliesrest of excitatory neuronsinhibitory neuronsext input neurons
actr
r1r
2r0r
to rest of assemblies:
212111010 ])1(])2([[ rJrgpfrgpgfrfgJrJI act
Mean field theory
Input current to neurons in the active assembly:
212111010 ])1()1([ rJrgpfrgpfrfgJrJI actact
Rates: active assemblyinactive assembliesrest of excitatory neuronsinhibitory neuronsext input neurons
actr
r1r
2r0r
to rest of assemblies:
212111010 ])1(])2([[ rJrgpfrgpgfrfgJrJI act
to other excitatory neurons:
2121110101 ])1()1([ rJrpfrpffrJrJI act
Mean field theory
Input current to neurons in the active assembly:
212111010 ])1()1([ rJrgpfrgpfrfgJrJI actact
Rates: active assemblyinactive assembliesrest of excitatory neuronsinhibitory neuronsext input neurons
actr
r1r
2r0r
to rest of assemblies:
212111010 ])1(])2([[ rJrgpfrgpgfrfgJrJI act
to other excitatory neurons:
2121110101 ])1()1([ rJrpfrpffrJrJI act
to inhibitory neurons:
2221210202 ])1()1([ rJrpfrpffrJrJI act
Mean field theory (2)Noise variances (white noise approximation):
2
22
121
222
1
211
0
02
102 ])1()1([K
rJrgpfrgpfrfg
KJ
KrJ
actact
Mean field theory (2)Noise variances (white noise approximation):
2
22
121
222
1
211
0
02
102 ])1()1([K
rJrgpfrgpfrfg
KJ
KrJ
actact
2
22
121
2222
1
211
0
02
102 ])1(])2([[K
rJrgpfrgpgfrfg
KJ
KrJ
act
Mean field theory (2)Noise variances (white noise approximation):
2
22
121
222
1
211
0
02
102 ])1()1([K
rJrgpfrgpfrfg
KJ
KrJ
actact
2
22
121
2222
1
211
0
02
102 ])1(])2([[K
rJrgpfrgpgfrfg
KJ
KrJ
act
2
22
121
1
211
0
02
1021 ])1()1([
KrJ
rpfrpffrKJ
KrJ
act
Mean field theory (2)Noise variances (white noise approximation):
2
22
121
222
1
211
0
02
102 ])1()1([K
rJrgpfrgpfrfg
KJ
KrJ
actact
2
22
121
2222
1
211
0
02
102 ])1(])2([[K
rJrgpfrgpgfrfg
KJ
KrJ
act
2
22
121
1
211
0
02
1021 ])1()1([
KrJ
rpfrpffrKJ
KrJ
act
2
2222
11
221
0
02202
2 ])1()1([K
rJrpfrpffr
KJ
KrJ
act
Mean field theory (2)Noise variances (white noise approximation):
2
22
121
222
1
211
0
02
102 ])1()1([K
rJrgpfrgpfrfg
KJ
KrJ
actact
2
22
121
2222
1
211
0
02
102 ])1(])2([[K
rJrgpfrgpgfrfg
KJ
KrJ
act
2
22
121
1
211
0
02
1021 ])1()1([
KrJ
rpfrpffrKJ
KrJ
act
2
2222
11
221
0
02202
2 ])1()1([K
rJrpfrpffr
KJ
KrJ
act
Rate of an I&F neuron driven by white noise:
Mean field theory (2)Noise variances (white noise approximation):
2
22
121
222
1
211
0
02
102 ])1()1([K
rJrgpfrgpfrfg
KJ
KrJ
actact
2
22
121
2222
1
211
0
02
102 ])1(])2([[K
rJrgpfrgpgfrfg
KJ
KrJ
act
2
22
121
1
211
0
02
1021 ])1()1([
KrJ
rpfrpffrKJ
KrJ
act
2
2222
11
221
0
02202
2 ])1()1([K
rJrpfrpffr
KJ
KrJ
act
),()erf1)(exp(
12/)(
/)(
aaI
IVr
a IFxxdx
raa
aareset
Rate of an I&F neuron driven by white noise:
Spontaneous activity:
212111010 })1(])1([{ rJrgpfrgpgfJrJI
212111010 ])1(])2([[ rJrgpfrgpgfrfgJrJI act
All assemblies inactive:
becomes
Spontaneous activity:
212111010 })1(])1([{ rJrgpfrgpgfJrJI
2121110101 ])1([ rJrpfpfrJrJI
212111010 ])1(])2([[ rJrgpfrgpgfrfgJrJI act
All assemblies inactive:
becomes
Similarly,
Spontaneous activity:
212111010 })1(])1([{ rJrgpfrgpgfJrJI
2121110101 ])1([ rJrpfpfrJrJI 2221210202 ])1([ rJrpfpfrJrJI
212111010 ])1(])2([[ rJrgpfrgpgfrfgJrJI act
All assemblies inactive:
becomes
Similarly,
Spontaneous activity:
212111010 })1(])1([{ rJrgpfrgpgfJrJI
2121110101 ])1([ rJrpfpfrJrJI
2
22
121
222
1
211
0
02
102 })1(])1([{K
rJrgpfrgpgf
KJ
KrJ
2221210202 ])1([ rJrpfpfrJrJI
212111010 ])1(])2([[ rJrgpfrgpgfrfgJrJI act
All assemblies inactive:
becomes
Similarly,
and
Spontaneous activity:
212111010 })1(])1([{ rJrgpfrgpgfJrJI
2121110101 ])1([ rJrpfpfrJrJI
2
22
121
222
1
211
0
02
102 })1(])1([{K
rJrgpfrgpgf
KJ
KrJ
2221210202 ])1([ rJrpfpfrJrJI
2
22
121
1
211
0
02
1021 ])1([
KrJ
rpfpfrKJ
K
rJ 2
2222
11
221
0
02202
2 ])1([K
rJrpfpfr
KJ
KrJ
212111010 ])1(])2([[ rJrgpfrgpgfrfgJrJI act
All assemblies inactive:
becomes
Similarly,
and
Spontaneous activity:
212111010 })1(])1([{ rJrgpfrgpgfJrJI
2121110101 ])1([ rJrpfpfrJrJI
2
22
121
222
1
211
0
02
102 })1(])1([{K
rJrgpfrgpgf
KJ
KrJ
2221210202 ])1([ rJrpfpfrJrJI
2
22
121
1
211
0
02
1021 ])1([
KrJ
rpfpfrKJ
K
rJ 2
2222
11
221
0
02202
2 ])1([K
rJrpfpfr
KJ
KrJ
212111010 ])1(])2([[ rJrgpfrgpgfrfgJrJI act
All assemblies inactive:
becomes
Similarly,
and
Solve for 2121 ,,;,, rrr
Simplified model (Brunel 2000)
• pf << 1
• g+ ~1/f >> 1
• variances + =act, 1 as in spontaneous-activity state
Simplified model (Brunel 2000)
• pf << 1
• g+ ~1/f >> 1
• variances + =act, 1 as in spontaneous-activity state
Define L = fJ11g+
Simplified model (Brunel 2000)
• pf << 1
• g+ ~1/f >> 1
• variances + =act, 1 as in spontaneous-activity state
Define L = fJ11g+
Then
(1) spontaneous activity state has r+= r1,
Simplified model (Brunel 2000)
• pf << 1
• g+ ~1/f >> 1
• variances + =act, 1 as in spontaneous-activity state
Define L = fJ11g+
Then
(1) spontaneous activity state has r+= r1,
(2) In recall state with ract > r+, r1 and r2 are same as in spontaneous activity state
Simplified model (Brunel 2000)
• pf << 1
• g+ ~1/f >> 1
• variances + =act, 1 as in spontaneous-activity state
Define L = fJ11g+
Then
(1) spontaneous activity state has r+= r1,
(2) In recall state with ract > r+, r1 and r2 are same as in spontaneous activity state
(3) ract is determined by]),([ ,11,1 spontactspontact rrLIFr
Graphical solution
)( mactact I (This L = (our L) x m)
(r -> )
1-assembly memory/recall state stable for big enough L (or g+ )~ describes “working memory” in prefrontal cortex
Capacity problem
In this model, memory assemblies were non-overlapping.This is unrealistic.
Alternative model: neurons in each assembly independently chosenA single neuron can be in many assemblies
Capacity problem
In this model, memory assemblies were non-overlapping.This is unrealistic.
Alternative model: neurons in each assembly independently chosenA single neuron can be in many assemblies
How many patterns can be stored using N neurons before interference between patterns destroys the recall ability?
Capacity problem
In this model, memory assemblies were non-overlapping.This is unrealistic.
Alternative model: neurons in each assembly independently chosenA single neuron can be in many assemblies
How many patterns can be stored using N neurons before interference between patterns destroys the recall ability?
Here: solve this for a simplified model (binary neurons, can beeither excitatory on inhibitory, “Hebbian” synapse formula)
Model N Binary neurons: )(0),(1 firing"not"firing""iS
Assemblies/patterns : p sets of n = fN neurons with Si = 1i
Model N Binary neurons: )(0),(1 firing"not"firing""iS
Assemblies/patterns : p sets of n = fN neurons with Si = 1i
ff ii 1)0(P;)1(P
Model N Binary neurons: )(0),(1 firing"not"firing""iS
Assemblies/patterns : p sets of n = fN neurons with Si = 1i
ff ii 1)0(P;)1(P
)1(,1
OnfNOfNnN
Of
Model N Binary neurons: )(0),(1 firing"not"firing""iS
Assemblies/patterns : p sets of n = fN neurons with Si = 1i
ff ii 1)0(P;)1(P
(Synchronous) dynamics:
jjiji tSJtS )()1(
)1(,1
OnfNOfNnN
Of
Model N Binary neurons: )(0),(1 firing"not"firing""iS
Assemblies/patterns : p sets of n = fN neurons with Si = 1i
ff ii 1)0(P;)1(P
(Synchronous) dynamics:
jjiji tSJtS )()1(
Synapses:Npf
nJ j
p
iij
1
1
)1(,1
OnfNOfNnN
Of
Model N Binary neurons: )(0),(1 firing"not"firing""iS
Assemblies/patterns : p sets of n = fN neurons with Si = 1i
ff ii 1)0(P;)1(P
(Synchronous) dynamics:
jjiji tSJtS )()1(
Synapses:Npf
nJ j
p
iij
1
1 (global inhibitory term makesaverage Jij = 0)
)1(,1
OnfNOfNnN
Of
Net input to neuron i:
j
jiji SJh
j
jjj
ji SNpf
Sn
,
1
j
jjj
jijj
ji SNpf
Sn
Sn 1,
11 11
01 21 fQpffQfNp
Nfhhm iii
Fluctuations QpfS
nS
Npf
Sn
h jj
jij
jjj
jii2
1,1,
11
kjjk
kjikjjk
kijii SSn
SSn
h
222 11
kjkj
kjijj
ji SSn
Sn
,
22
11
Fluctuations QpfS
nS
Npf
Sn
h jj
jij
jjj
jii2
1,1,
11
kjjk
kjikjjk
kijii SSn
SSn
h
222 11
kjkj
kjijj
ji SSn
Sn
,
22
11
2322
22 )(
)(1
)(1
fQfpNNf
fQfNpNf
Fluctuations QpfS
nS
Npf
Sn
h jj
jij
jjj
jii2
1,1,
11
kjjk
kjikjjk
kijii SSn
SSn
h
222 11
kjkj
kjijj
ji SSn
Sn
,
22
11
2322
22 )(
)(1
)(1
fQfpNNf
fQfNpNf
)1( 2NQffQN
p
Fluctuations QpfS
nS
Npf
Sn
h jj
jij
jjj
jii2
1,1,
11
kjjk
kjikjjk
kijii SSn
SSn
h
222 11
with = p/N
kjkj
kjijj
ji SSn
Sn
,
22
11
2322
22 )(
)(1
)(1
fQfpNNf
fQfNpNf
)1( 2NQffQN
p )1( nfQfQ
Fluctuations QpfS
nS
Npf
Sn
h jj
jij
jjj
jii2
1,1,
11
kjjk
kjikjjk
kijii SSn
SSn
h
222 11
with = p/N (recall nf = O(1))
kjkj
kjijj
ji SSn
Sn
,
22
11
2322
22 )(
)(1
)(1
fQfpNNf
fQfNpNf
)1( 2NQffQN
p )1( nfQfQ
Mean field equations
For neurons in pattern 1, h = m + Gaussian noise
=>
2
2
21
2exp
2 h
h
h
dhm
m
2
Hh
m
with )exp(2
)(H 221 x
dxx
x
Mean field equations
For neurons in pattern 1, h = m + Gaussian noise
=>
2
2
21
2exp
2 h
h
h
dhm
m
2
Hh
m
with )exp(2
)(H 221 x
dxx
x
Mean field equations
For neurons in pattern 1, h = m + Gaussian noise
=>
2
2
21
2exp
2 h
h
h
dhm
m
2
Hh
m
with )exp(2
)(H 221 x
dxx
x
For other neurons, h = Gaussian noise
Mean field equations
For neurons in pattern 1, h = m + Gaussian noise
=>
2
2
21
2exp
2 h
h
h
dhm
m
2
Hh
m
with )exp(2
)(H 221 x
dxx
x
For other neurons, h = Gaussian noise
=>
2H)1(
hffmfQ
Mean field equations
For neurons in pattern 1, h = m + Gaussian noise
=>
2
2
21
2exp
2 h
h
h
dhm
m
2
Hh
m
with )exp(2
)(H 221 x
dxx
x
For other neurons, h = Gaussian noise
=>
2H)1(
hffmfQ
Solve for m and Q
Capacity estimateWeight in tail (h > m) of big gaussian centered at 0 must be < weight in small one centered at m
Capacity estimateWeight in tail (h > m) of big gaussian centered at 0 must be < weight in small one centered at m
Can take
1,1 mQ
Capacity estimateWeight in tail (h > m) of big gaussian centered at 0 must be < weight in small one centered at m
Can take
=>
1,1 mQ
1)1(
exp)1(2
2
21
1
nffh
nff
dh
Capacity estimateWeight in tail (h > m) of big gaussian centered at 0 must be < weight in small one centered at m
Can take
=>
1,1 mQ
1)1(
exp)1(2
2
21
1
nffh
nff
dh
i.e., 1)1(
1H
nff
Capacity estimateWeight in tail (h > m) of big gaussian centered at 0 must be < weight in small one centered at m
Can take
=>
1,1 mQ
1)1(
exp)1(2
2
21
1
nffh
nff
dh
i.e., 1)1(
1H
nff
Use asymptotic form of H: )exp(2
1)(H 2
21 x
xx