frank_neuroinformatics11.pdf
DESCRIPTION
Computational psychiatry symposium at NeuroInformatics 2011 meeting in Boston, MA, USA.TRANSCRIPT
![Page 1: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/1.jpg)
Modelin
gdecisio
nmak
ingdefi
citsin
frontostriatal
diso
rders
Mich
aelFran
kLab
orato
ryforNeu
ralComputatio
nan
dCognitio
nBrownUniversity
![Page 2: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/2.jpg)
Computatio
nal
Psych
iatryan
d...
Neu
rogen
ocomputomics
•Man
ydiso
rders
broad
lych
aracterizedbych
anges
inmotiv
ation
•Sev
eralfro
nto-striatal
diso
rders
hav
esu
bstan
tialgen
eticheritab
ility
•Individual
differen
cesin
reinforcem
entlearn
ing?
![Page 3: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/3.jpg)
Computatio
nal
Psych
iatryan
d...
Neu
rogen
ocomputomics
•Man
ydiso
rders
broad
lych
aracterizedbych
anges
inmotiv
ation
•Sev
eralfro
nto-striatal
diso
rders
hav
esu
bstan
tialgen
eticheritab
ility
•Individual
differen
cesin
reinforcem
entlearn
ing?
•But...
Can
didate
gen
eeffects
aregen
erallysm
all
•Which
gen
es?Which
task?Which
measu
re?
![Page 4: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/4.jpg)
Computatio
nal
Psych
iatryan
d...
Neu
rogen
ocomputomics
•Man
ydiso
rders
broad
lych
aracterizedbych
anges
inmotiv
ation
•Sev
eralfro
nto-striatal
diso
rders
hav
esu
bstan
tialgen
eticheritab
ility
•Individual
differen
cesin
reinforcem
entlearn
ing?
•But...
Can
didate
gen
eeffects
aregen
erallysm
all
•Which
gen
es?Which
task?Which
measu
re?
•N
eedth
eoreticalm
od
el!(an
dco
nvergin
gp
harm
acolo
gy/im
agin
g)
Fran
k&
Fossella,2011;
Maia
&Fran
k,2011;
Huyset
al,2011
![Page 5: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/5.jpg)
Rein
forcem
entlearn
ingan
ddopam
ine:
pred
ictionerro
rs
Positiv
ePE:
Neg
ativePE:
dopam
ine:
Montag
ue,D
ayan
&Sejn
owksi96;
Doya,2002;
O’Reilly,
Fran
k,H
azy&
Watz
06...
δ(t)
=(
r(t)
+γV̂(t+
1))
−V̂(t)
![Page 6: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/6.jpg)
Rein
forcem
entlearn
ingan
ddopam
ine:
pred
ictionerro
rs
Positiv
ePE:
Neg
ativePE:
dopam
ine:
Montag
ue,D
ayan
&Sejn
owksi96;
Doya,2002;
O’Reilly,
Fran
k,H
azy&
Watz
06...
δ(t)
=(
r(t)
+γV̂(t+
1))
−V̂(t)
![Page 7: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/7.jpg)
D1effects
onstriatal
learning:Positiv
ePE
![Page 8: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/8.jpg)
D1effects
onstriatal
learning:Positiv
ePE
Three
factorlearn
ing:presy
nap
tic,postsy
nap
tican
dDA
![Page 9: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/9.jpg)
D2effects
onstriatal
learning:Neg
ativePE
Fran
k2005
![Page 10: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/10.jpg)
Neu
ralmodel
ofbasal
gan
glia
anddopam
ine
Integ
ratesawideran
geofdata
into
asin
gle
coheren
tfram
ework
Sep
arateGoan
dNoGopopulatio
nsinteg
ratestatistics
ofrein
forcem
ent
Input
Stria
tum
Thalamus
GPi/S
Nr
preSMASTN
SNc
Go
NoGo
GPe
ge ≈i w
ij >+
i wij
≈V
m−Θ
[]+
Vm
−Θ[
]++
1jnet
<x
x
βN=
yγ
γge ge
Vm
=
c
gg
g iigll
+ ++...
Ee
m V[
]
EV
[i
]mE
V[
l]m
∆w
ij≈
(xpi y
pj )−(xti y
tj )
Fran
k,2005,
2006JCogNeu
rosci,N
eural
Netw
orks
![Page 11: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/11.jpg)
Max
imizin
gRew
ardvia
RTAdap
tation:
Tem
poral
Utility
Integ
rationTask
01000
20003000
40005000
Tim
e (ms)
0.00.10.20.30.40.50.60.70.80.91.0
ProbabilityC
EV
DE
VIE
VC
EV
R
Rew
ard F
requ
ency
01000
20003000
40005000
Tim
e (ms)
0 50
100
150
200
250
300
350
# Points Gained
CE
VD
EV
IEV
CE
VR
Rew
ard M
agn
itud
e
01000
20003000
40005000
Tim
e (ms)
0 5 10 15 20 25 30 35 40 45 50 55 60
Expected Value (freq*mag)
CE
VD
EV
IEV
CE
VR
Exp
ected V
alue
![Page 12: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/12.jpg)
RLmodel:
Fitto
data
across
allsu
bjects
RLmodel
:ad
just
RTsas
afunctio
nofrew
ardpred
ictionerro
rs
Fran
k,D
oll,O
as-Terp
stra&
Moren
o(2009,
Natu
reNeu
roscien
ce)
![Page 13: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/13.jpg)
Neu
rogen
etican
dpharm
acological
modulatio
nof
reinforcem
entlearn
ingparam
eters
Fran
k&
Fossella,2011
![Page 14: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/14.jpg)
Single
subject
Data...
010
2030
4050
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000S
ingle Subject C
EV
RT (ms)
Trial
010
2030
4050
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000S
ingle Subject D
EV
RT (ms)
Trial
010
2030
4050
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000S
ingle Subject IE
V
RT (ms)
Trial
010
2030
4050
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000S
ingle Subject C
EV
R
RT (ms)
Trial
![Page 15: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/15.jpg)
Exploratio
nvsExploitatio
n
•Byexploitin
glearn
edstrateg
ies,weknow
wecan
get
acertain
amount
ofrew
ard
•Butdon’tknow
how
gooditcan
get.
⇒Need
toE
xplo
re
•Theo
ry:Explore
based
onrelativ
euncertain
tyab
outwheth
erother
actionsmightyield
better
outco
mes
than
statusquo
(Day
an&
Sejn
owksi96)
![Page 16: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/16.jpg)
Exploratio
nvsExploitatio
n
•Byexploitin
glearn
edstrateg
ies,weknow
wecan
get
acertain
amount
ofrew
ard
•Butdon’tknow
how
gooditcan
get.
⇒Need
toE
xplo
re
•Theo
ry:Explore
based
onrelativ
euncertain
tyab
outwheth
erother
actionsmightyield
better
outco
mes
than
statusquo
(Day
an&
Sejn
owksi96)
![Page 17: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/17.jpg)
Uncertain
ty-Based
Exploratio
n
510
1520
2530
3540
4550
−4000
−3000
−2000
−1000 0
1000
2000
3000
4000E
xploration
Trial
RT Diff (ms)
Single S
ubject, CE
V Model E
xp termR
T diff
![Page 18: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/18.jpg)
PFCGen
e-Dose
Effect
onUncertain
ty-Based
Exploratio
n
0.000.050.100.150.200.250.300.350.400.450.50
(x 1e4)val/valval/m
etm
et/met
CO
MT
gene-dose effectsU
ncertainty-exploration parameter
ε
Fran
k,Doll,O
as-Terp
stra&
Moren
o(2009,
Natu
reNeu
roscien
ce)
![Page 19: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/19.jpg)
Does
thebrain
trackrelativ
euncertain
tyforexploratio
n?
![Page 20: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/20.jpg)
Does
thebrain
trackrelativ
euncertain
tyforexploratio
n?
ǫ>
0(’ex
plorers’)
explorers
>non-ex
plorers
Bad
re,Doll,L
ong&
Fran
k,under
review
![Page 21: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/21.jpg)
EEG
reveals
temporal
dynam
ics
![Page 22: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/22.jpg)
EEG
reveals
temporal
dynam
ics
Relativ
euncertain
tyrep
resented
prio
rto
choice,
andmore
soin
explorato
rytrials
Cav
anag
h,Cohen
,Figuero
a&
Fran
k,under
review
![Page 23: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/23.jpg)
Neg
ativesymptomsin
schizo
phren
ia:Uncertain
ty-Based
Exploratio
n
(uncert)0.00
0.050.100.15
0.200.250.30
0.350.40
(x 1e4)
SZ
CN
Un
certainty-d
riven exp
loratio
n
ε
**
ε0
12
34
Glo
bal A
nh
edo
nia
-1.2-1.0-0.8-0.6-0.4-0.2 00.20.40.60.8
(x1e4)
An
hed
on
ia & E
xplo
ration
r = -.44, p =
.002
ε•
Anhed
onia
=beh
avioral
componen
tofrew
ardseek
ing(e.g
.,initiatin
gsocial/
recreational
activities)
notcap
acityto
experien
cepleasu
re
•Anhed
onia
relatedto
exploratio
nan
dnotlearn
ingfro
mrew
ardpred
ictionerro
rs
Strau
sset
al,2011,
Biological
Psych
iatry
![Page 24: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/24.jpg)
Obsessiv
eCompulsiv
eDiso
rder:
Aversio
nto
Uncertain
ty
gainslosses
-0.4
-0.2
0.0
0.2
0.4
0.6
(x 1e4)
CN
OC
D
Un
certainty-d
riven exp
loratio
n
ε
prelim
inary
data,N
=17
per
group
with
Masch
avan
’tWout,B
enGreen
berg
,Stev
eRasm
ussen
![Page 25: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/25.jpg)
Summary
•Dopam
inemodulates
reinforcem
entlearn
ingan
dch
oice
based
on
positiv
ean
dneg
ativeoutco
mes:
patien
ts,pharm
acology,g
enetics,
imag
ing
•Prefro
ntal
cortex
tracksoutco
meuncertain
tyso
asto
reduce
it
•Disru
ptio
nofthese
mech
anism
sisasso
ciatedwith
fronto-striatal
diso
rders,
Park
inson’s,sch
izophren
ia,OCD
•Models
integ
ratebetw
eenmultip
lelev
elsofan
alysis:
neu
ralmech
anism
toab
stractcomputatio
n(see
Thomas
Wieck
i
dem
onstratio
ntomorro
w!).
![Page 26: Frank_NeuroInformatics11.pdf](https://reader033.vdocument.in/reader033/viewer/2022060118/558a6532d8b42a5f4a8b45b9/html5/thumbnails/26.jpg)
Than
ksTo...
Brad
leyDoll
Christin
aFiguero
aJim
Cav
anag
hDav
idBad
reJeff
Cock
burn
AnneCollin
sThomas
Wieck
iJim
Gold
Ken
tHutch
ison
Masch
avan
’tWout
Nico
leLong
MikeCohen
Ahmed
Moustafa
ScottSherm
an
Thepatien
ts
Lab
forNeu
ralComputatio
nan
dCognitio
n