pixels to voxels: modelling visual representations in the human
TRANSCRIPT
Pixels to Voxels: Modelling Visual Representations in the Human Brain
Pulkit Agrawal, Dustin Stansbury, Jitendra Malik, Jack Gallant
(Under Review, arXiv preprint available online)
Thanks to Brian Cheung, Mark Lescroart and Alex Huth for the helpful discussions
Goal
“ Obtain a functional mapping of the human visual cortex ”
or
“ What visual features does the brain code for? ”
Why solve this?
- We can’t fix it, unless we know what it does → Neural Prosthesis.
- Brain Reading.
- A scientific pursuit. “What are the underlying computational principles used by biological vision?”
- Understanding limitations of state of art computer vision systems.
functional Magnetic Resonance Imaging (fMRI)
What does fMRI measure?
Neural Activity → Oxygen Consumed → Oxygen level in blood drops
fMRI measures “local” changes in Oxygen level
Indirect measurement of neural activity!
[Slide taken from Tom Mitchell]
Traditional Neuroscience approach
- Hypothesis Driven. - Experimenter’s bias.
- Measures binary contrast only!
- Cannot provide holistic description of stimulus tuning of voxels.
Basic Idea
[Gallant et al.]
Basic Idea
[Gallant et al.]
Gabor Model → Early Visual Cortex
[Kay et al.]
Semantic Model → Higher Visual Cortex
[Stansburry et al.]
i.e. until now,
- Different models for different brain areas
- Human Annotation for semantic Model :(
- No model explains mid-level visual areas :(
[Stansburry et al.]
This work,
- Modelling the entire visual hierarchy
- No need for human annotation
- Enable exploratory analysis of neural representations!!
Overview
ConvNet v/s Semantic Model
Semantic Model
Con
vNet
Mod
el
Functional Selectivity
Increased Activity Decreased Activity
Clustering in EBA
Increased Activity Decreased Activity
In a nutshell
- A single annotation free for modelling brain responses throughout the visual cortex.
- Replicate results of multiple localizer studies in a single experiment.
- Novel insights into representations such as clustering in EBA.
Questions?
Fisher Vector v/s Semantic Model
Semantic Model
Fish
er V
ecto
r M
odel
ConvNet v/s Fisher Vector
Fisher Vector Model
Con
vNet
Mod
el
Brief Description of Fisher Vectors