toward the neural causes of human visual perception and ... · pdf filetoward the neural...
TRANSCRIPT
![Page 1: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/1.jpg)
Toward the neural causes of human visual perception and
behavior
Kaoru AmanoCenter for Information and Neural Networks (CiNet)
National Institute of Information and Communications Technology (NICT)
Graduate School of Frontier Biosciences, Osaka University
PREST, Japan Science and Technology Agency (JST)
![Page 2: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/2.jpg)
Outline
• Neural correlates of perception– Exp. 1: Alpha oscillation as a clock signal for visual processing
• From neural correlates to neural causes of perception:decoded neurofeedback as a tool to study causality
– Exp. 2: Creation of color perception by manipulation of neural activity in the primary and secondary visual areas
– Exp. 3: Change in perceptual confidence by manipulation of neural activity in the fronto-parietal areas
![Page 3: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/3.jpg)
Seeing (visual perception) is the interpretation of the visual input by brain
• Humans interpret the external world, rather than copying the visual input (as camera does)
• Illusion (mismatch between visual input and percept) is a typical example indicating the interpretation process different from camera
• Illusion can be a crucial clue for understanding the neural mechanisms underlying visual perception
VS.
![Page 4: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/4.jpg)
“A” looks much darker than “B”, but …Checkerboard shadow illusion
(Adelson)
![Page 5: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/5.jpg)
Stationary image looks like movingRotating snake illusion(Akiyoshi Kitaoka)
![Page 6: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/6.jpg)
Yellow dots disappear and re-appear…Motion induced blindness
(Bonneh et al.)
![Page 7: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/7.jpg)
Adaptation induced blindnessMotoyoshi et al. 2010
![Page 8: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/8.jpg)
How to study the neural correlates of perception
Dots are invisible
Neural correlates of perception
Dots are visible
Motion induced blindness
Compare the neural activities corresponding to different percepts with keeping the input constant
![Page 9: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/9.jpg)
Non-invasive human brain measurement
• Electrical activities of neurons• High temporal resolution (ms)• Difficult to accurately estimate the
locus of activity
• Changes in blood flow associated with electrical activities of neurons
• Low temporal resolution (s)• High spatial resolution (mm)
Magnetoencephalography: MEG Functional magnetic resonance imaging: fMRI
Need to utilize different techniques for each purpose
![Page 10: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/10.jpg)
Outline
• Neural correlates of perception– Exp. 1: Alpha oscillation as a clock signal for visual processing
• From neural correlates to neural causes of perception:decoded neurofeedback as a tool to study causality
– Exp. 2: Creation of color perception by manipulation of neural activity in the primary and secondary visual areas
– Exp. 3: Change in perceptual confidence by manipulation of neural activity in the fronto-parietal areas
![Page 11: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/11.jpg)
Outline
• Neural correlates of perception– Exp. 1: Alpha oscillation as a clock signal for visual processing
• From neural correlates to neural causes of perception:decoded neurofeedback as a tool to study causality
– Exp. 2: Creation of color perception by manipulation of neural activity in the primary and secondary visual areas
– Exp. 3: Change in perceptual confidence by manipulation of neural activity in the fronto-parietal areas
![Page 12: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/12.jpg)
Alpha oscillation
• Neural oscillations at 8-13 Hz• Increases when you relax• Originally thought to reflect
brain “idling”• Suppression of irrelevant
information?
Jensen et al., 2012
Spatial attention suppress contralateral alpha and enhances ipsilateral alpha
What is the functional role of alpha oscillation?
~100 ms~100 ms~100 ms
![Page 13: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/13.jpg)
Alpha oscillation as a clock signal for visual processing?
Alpha oscillation might work as a clock signal that determines the timing of interaction
between visual areas
Ventral pathwaye.g. Color, Object
Dorsal pathwaye.g. Motion, Depth
Information transmission between visual areas
at alpha rhythm
![Page 14: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/14.jpg)
Motion-induced spatial conflictArnold and Johnston, 2003
![Page 15: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/15.jpg)
Motion-induced spatial conflict
Green bars were either (1) darker than, (2) isoluminant with, or (3) brighter than red squares
Slow(isoluminant
motion)
Fast(luminance
motion)
• Red-green borders were set to be isoluminant
• Perceived speed of isoluminantred-green boarders is slower than that of a luminance defined red-black borders
• Isoluminant border is perceived to be jittering
Arnold & Johnston, 2005
Illusory jitter MEG responses
Amano et al., 2008
![Page 16: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/16.jpg)
Correlation between jitter perception and alpha oscillation
Amano et al., 2008
Perceived jitter frequency
Dark Iso Bright
~10 Hz DarkIsoluminantBright
MEG spectrum(Left occipital)
Increase in alpha power during jitter perception
• Frequency of illusory jitter, perceived only in the isoluminant condition, was at around 10 Hz
• Frequency of modulated neural oscillation matches the perceived jitter frequency
![Page 17: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/17.jpg)
Does perceived jitter frequency reflect individual alpha frequency?
Illusory jitter perception at ~10Hz
Neural oscillation at ~10Hz
Highjitter frequency
(( ))
Lowjitter frequency
( )
Alpha frequency is measured during restBEFORE jitter frequency measurement
Low alpha frequency
High alpha frequency
Subject A
Subject B
Neural correlates
![Page 18: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/18.jpg)
― Open― Close
Frequency (Hz)
10
8
4
2
0
6
Pow
er (m
V2 /Hz
)Measurement of alpha oscillation
……
30s 30s 30s 30s
f = 8.3Hzσ = 3.1Hz
FFT analysis on MEG data
• Whole-head MEG system (Elekta)
• Sampling frequency: 500 Hz
• 306 channels
• Open/close eyes – One beep: open eyes– Two beeps: close eyes
• 5 repetitions of open/close session
• Subjects: 10
Peak alpha frequency
![Page 19: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/19.jpg)
Measurement of perceived jitter frequency
• Frequency of real jitter was randomly chosen from 4-13 Hz (20 times each)
• Subjects judged whether the real jitter is faster than the illusory jitter
Real jitter
Illusory jitter
Real
jitt
er fa
st (%
)
Real jitter frequency (Hz)
Perceived jitter frequency
![Page 20: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/20.jpg)
Individual resting alpha and perceived jitter frequency
Perceived jitter frequency mirrors individual peak alpha frequency
Sub1― Open: 9.8 Hz― Close: 10.3 Hz
10
8
6
4
2
0
Pow
er (m
V2 /Hz
)
― Open: 10.3 Hz― Close: 9.8 Hz Sub2
Frequency (Hz) Real jitter frequency (Hz)
f: 9.7 Hz
Real
jitt
er fa
st (%
)100
80
60
40
200
13119751311975
Sub3― Open: 8.3 Hz― Close: 8.3 Hz
f: 8.3 Hz
Real jitter frequency (Hz)1311975
Real
jitt
er fa
st (%
)100
80
60
40
20
0Frequency (Hz)
1311975
10
8
6
4
2
0
Pow
er (m
V2 /Hz
)
Real jitter frequency (Hz)1311975
Real
jitt
er fa
st (%
)
100
80
60
40
20
0
Frequency (Hz)1311975
10
8
6
4
2
0
Pow
er (m
V2 /Hz
)
f: 10.2Hz
![Page 21: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/21.jpg)
Individual alpha frequency, not amplitude, correlates with jitter frequency
• Illusory jitter is perceived at the frequency of intrinsic alpha oscillation
• Alpha power is not correlated with jitter frequency
r = 0.87 (P < 0.01)
N=10
r = 0.10 (P = 0.77)
N=10
![Page 22: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/22.jpg)
Resting state alpha frequency is kept constant during jitter perception
Peak alpha frequency during the perception of illusory jitter was highly correlated with that during the resting state
r = 0.95 (P < 0.01)
N=6
![Page 23: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/23.jpg)
Possible functional role of alpha oscillation
Alpha oscillation may function as a clock signal for the interaction between visual areas
Object-based position representation
Low temporal resolutionHigh spatial resolution
Motion-based position representation
High temporal resolutionLow spatial resolution
delay delay
compensation
Update of position representation
at alpha rhythm
Dorsal pathway
Ventral pathway
![Page 24: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/24.jpg)
Summary• Alpha oscillation is enhanced when illusory
jitter is perceived• Perceived jitter frequency was highly
correlated with the peak alpha frequency during the resting state
• Alpha oscillation is the neural correlates of illusory jitter perception
Alpha oscillation might work as a clock signal that determines the timing of interaction between ventral and dorsal visual areas
![Page 25: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/25.jpg)
Outline
• Neural correlates of perception– Exp. 1: Alpha oscillation as a clock signal for visual processing
• From neural correlates to neural causes of perception:decoded neurofeedback as a tool to study causality
– Exp. 2: Creation of color perception by manipulation of neural activity in the primary and secondary visual areas
– Exp. 3: Change in perceptual confidence by manipulation of neural activity in the fronto-parietal areas
![Page 26: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/26.jpg)
How to study the neural correlates of perception
Dots are invisible
Neural correlates of perception
Dots are visible
Motion induced blindness
Compare the neural activities corresponding to different percepts with keeping the input constant
![Page 27: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/27.jpg)
Neural correlates are not necessarily neural causes of visual perception
Dots are invisible
Neural causes of perception Epiphenomena
Dots are visible
Neural correlates of perception? ?
![Page 28: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/28.jpg)
FMRI decoded neurofeedback (DecNef)
Correlation with the target
pattern
Subjects manipulate their brain activities
Feedback
Target pattern
Target area
Shibata et al., 2011
Current pattern
A specific brain activity pattern in a specific area can be induced with DecNef
![Page 29: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/29.jpg)
Perception
Behavior
Cognition
Perception
Behavior
Cognition
Variable Observation
Correlation analysis
DecNef
Decoded neurofeedback (DecNef)as a tool for cognitive neuroscience
Shibata et al., 2011
![Page 30: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/30.jpg)
PurposeCreate color perception associated with an orientation by manipulating neural activity in the primary and secondary visual areas (V1/V2) using decoded fMRI neurofeedback
Is an achromatic grating perceived to be reddish?
Achromaticvertical grating
V1/V2 activity patterncorresponding to red
Association
![Page 31: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/31.jpg)
Experimental procedure31
• Day 1: Retinotopic mapping• Day 2: Color decoder construction
• Day 3: Decoded neurofeedback training• Day 4: Decoded neurofeedback training • Day 5: Decoded neurofeedback training
Subjective color testChromatic psychometric function measurement
• 12 subjects participated in the 5-day session• 6 control subjects participated in only a subjective color test
and chromatic psychometric function measurement
None of the subjects were colorblind
1–2 weeks
![Page 32: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/32.jpg)
Day 1: Retinotopic mapping
Wandell et al., 2007
![Page 33: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/33.jpg)
Day 2: Color (red vs. green) decoder construction
・・・
Color decoding accuracy in V1/V2(irrespective of orientation)
• Each combination of color (red/green/gray) and orientation (vertical/horizontal) was randomly presented 24 times in 12 runs
• Green and gray are set to be perceptually equiluminant to red
• SF: 1 cycle/deg, TF: 0.5 Hz
12 s
12 s
12 s
12 s
12 s
Red likelihood of V1/V2 activity pattern was calculated by sparse logistic regression (SLR) (>0.5, red; <0.5, green)
![Page 34: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/34.jpg)
Days 3-5: Decoded neurofeedback training
Time
Rest (6 s)Feedback (1 s)Rest (7 s)
Task: “Somehow regulate your brain activity to make the feedback disk as large as possible”
Task (6 s)
High red likelihood in V1/V2
Low red likelihood in V1/V2
Large feedback disk
Small feedback disk
≈
≈
![Page 35: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/35.jpg)
V1/V2 activity pattern during the presentation of achromatic vertical grating became “reddish”
The strategy of induction• “I did mental multiplication.”• “I tried to remember scenes of TV
programs or video games. I also imaginedwatching music videos.”
• “I counted numbers.”• “I imagined singing or running. I also
calculated or remembered sentences.”• “I tried to remember what I did yesterday.”• … and so on.
None of these reported strategies was related in any way to the target color (red)
Red
likel
ihoo
d in
crea
se[%
]Day 1 Day 2 Day 3 All
0
10
20
30
40
50
relative to the likelihood of an achromaticvertical grating being classified as the targetcolor (red) in the decoder construction stage
Induction performance
![Page 36: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/36.jpg)
Day 5: Subjective color test
• Each achromatic grating was presented 10 times• Duration, 1.5 s; ISI, 4.5 s• Judge the color of the inner grating (red, green or achromatic)
Vertical Horizontal
Control subjects
Vertical Horizontal
Chro
mat
ic re
spon
se [%
] 604020
0-20-40-60
*
*
Subjects with DecNef
Vertical Horizontal
Chro
mat
ic re
spon
se [%
] 604020
0-20-40
Red
Green
Red
Green-60
Vertical grating used for DecNef was perceived to be reddish
![Page 37: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/37.jpg)
Day 5: Chromatic psychometricfunction measurement
• 32 stimuli (4 orientations × 8 colors) were randomly presented– Orientation
– Color• Inner: 8 levels from greenish to reddish (middle is gray)• Outer: gray
• Judge the color of the inner grating (red vs. green)
Vertical Horizontal +45 deg -45 deg
![Page 38: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/38.jpg)
Shift in psychometric function by DecNef training
Subjects with DecNef Control subjects
Orientation-specific change in color perception by DecNef
![Page 39: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/39.jpg)
Long-lasting effect of DecNef training
Color perception associated with an orientation is long-lasting, as has been reported for other types of associative learning
Day3 3 – 5 months later
![Page 40: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/40.jpg)
Summary
Using decoded fMRI neurofeedback, we created long-lasting color perception associated with a specific orientation
Achromaticvertical grating
V1/V2 activitycorresponding to red
Association
The early visual cortex has the capability of creating an orientation-specific color perception
![Page 41: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/41.jpg)
Outline
• Neural correlates of perception– Exp. 1: Alpha oscillation as a clock signal for visual processing
• From neural correlates to neural causes of perception:decoded neurofeedback as a tool to study causality
– Exp. 2: Creation of color perception by manipulation of neural activity in the primary and secondary visual areas
– Exp. 3: Change in perceptual confidence by manipulation of neural activity in the fronto-parietal areas
![Page 42: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/42.jpg)
Outline
• Neural correlates of perception– Exp. 1: Alpha oscillation as a clock signal for visual processing
• From neural correlates to neural causes of perception:decoded neurofeedback as a tool to study causality
– Exp. 2: Creation of color perception by manipulation of neural activity in the primary and secondary visual areas
– Exp. 3: Change in perceptual confidence by manipulation of neural activity in the fronto-parietal areas
![Page 43: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/43.jpg)
Confidence in perceptual judgment
• Confidence is a meta-cognitive function for decisions• Confidence generally reflects perceptual accuracy but
can dissociate from accuracy (e.g. blidsight)• Confidence affects a variety of cognitive functions• LIP, DLPFC, and Pulvinar are associated with
confidence (e.g. Kiani et al. 2009, Komura et al. 2013)Are these areas causally associated with the confidence of perceptual judgment?
Left
Not sure…
Right
Pretty sure Confidence
Perceptual judgment
![Page 44: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/44.jpg)
Experimental procedure
Day 1
ConfidenceDecoder
construction
Psyc
hoph
ysic
al te
st
Psyc
hoph
ysic
al te
st
High- orLow-
confidence DecNef
Day 2
The order of High- and Low-confidence DecNefwas counterbalanced across subjects (N=10)
High- orLow-
confidence DecNef
Day 3
Psyc
hoph
ysic
al te
st
Psyc
hoph
ysic
al te
st
Low- orHigh-
confidence DecNef
Day 4
Low- orHigh-
confidence DecNef
Day 5
![Page 45: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/45.jpg)
Confidence decoder construction
• Motion direction judgment• Confidence rating of the motion direction
judgment (1-4)1 s
+
+
1 s
2 s
4 s
6 s
2 s
2 s
Time
+
Fixation
+
Incoherent motion
Coherent motion at threshold coherence
+
Delay RightLeft
Direction judgment
3 2 4
1
Confidencerating
ITI
Confidence decoder was built from the activity • in visual and frontoparietal areas • during the coherent motion and delay periods
![Page 46: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/46.jpg)
Confidence decoding from parietal and frontal areas
• Confidence decoded from the activity of parietal (IPL) and frontal (IFS, MFS, MFG) areas
• Confidence decoding accuracy was lower in visual areas
Visual areas Frontoparietalareas
![Page 47: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/47.jpg)
High/low confidence DecNef
• Feedback disk size represents the similarity of frontoparietalactivities between the induction period and the high or low confidence trials in the decoder construction
• Incoherent (random) motion was presented for 2 s at a random time during the induction period
Rest Feedback ITI
6 s 6 s6 s 2 s
Induction
Before and after each DecNef session (high/low), direction discrimination accuracy and confidence were measured for the threshold level coherent motion
2 s
Incoherent motion
Somehow regulate the brain activity during the induction period to make the feedback disk as large as possible
Induction of activity patterns in the frontoparietalareas corresponding to high/low confidence
Online confidence decoding
![Page 48: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/48.jpg)
DecNef induced confidence changes (without change in discrimination accuracy)
• No change in direction discrimination accuracy by DecNef (decoded neurofeedbck)
• Significant increase in confidence by high confidence DecNef
High-confidenceDecNef
Low-confidenceDecNef
High-confidenceDecNef
Low-confidenceDecNef
![Page 49: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/49.jpg)
Conclusions• Confidence was successfully decoded from
fronto-parietal areas • Induction of brain activities corresponding to
high and low confidence resulted in the change in confidence rating without changing motion direction judgment accuracy
• Confidence is a rather separated entity from the other measures of perceptual accuracy
Fronto-parietal areas are causally related to confidence during the perceptual judgment
![Page 50: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/50.jpg)
Overall summary• Neural correlates of perception
– Perceived jitter frequency mirrors intrinsic alpha frequency
• Decoded neurofeedback to study neural causes of perception– Long-lasting color perception associated with an
orientation created by manipulation of V1/V2 activities– Change in confidence during direction judgment task
achieved by manipulation of fronto-parietal activities
![Page 51: Toward the neural causes of human visual perception and ... · PDF fileToward the neural causes of human visual perception and behavior ... • Frequency of modulated neural oscillation](https://reader030.vdocument.in/reader030/viewer/2022021510/5ab7f7c57f8b9a684c8c3669/html5/thumbnails/51.jpg)
CollaboratorsSorato Minami (CiNet)
Aurelio Cortese(ATR)
Hakwan Lau(UCLA)
Takeo Watanabe(Brown University)
Yuka Sasaki(Brown University)
Kazuhisa Shibata(Brown University)
Ai Koizumi(CINet)
Mitsuo Kawato(ATR)
Color DecNef
Confidence DecNefAlpha