RDM mixtures for predicting visual cortices responses
Agustin Lage Castellanos1,2 and Federico De Martino2
1-Cuban Neuroscience Center, 2-Maastricht University
Algonauts Challenge 2019
Intuition behind our method
Perceptual
Cat
ego
rica
l
fMRI-EVC
MEG-early
fMRI-ITC
MEG-Late
DNN-L1
DNN-L3DNN-L2
DNN-L5DNN-L4
Combining RDMs to improve predictions
Predicted RDM Perceptual Categorical DNN
= ๐ค1 + ๐ค2 + ๐ค3
Perceptual RDMs
Perceptual RDMs
Only uses image information
Extract Edges and Smooth
Perceptual-RDMPixel Overlap
Categorical RDMs
Categorical Structure of the 92 image set
Objects-Scenes
animals
Human
Fruits-vegetables
Faces
Hands
Monkey faces
Animal Faces
Within category RDM based on fMRI/MEG data similarity
92 x 92 8 x 8
mean
Between image fMRI/MEG similarity Between category fMRI/MEG similarity
fMRI-ITC
Training a GNB classifier as predicting category
GNB
Class Labels
Last fully connected layer (defines category membership)
Leave one out CV on the 92 image training set
Classification of the 78 test set images
๐๐บ๐๐ต
Predicted Labels
x
Predicted as Human Faces
Predicted as Animal Faces in the 78 set
Objects-Scenes
Animal Faces
animalsHumanFruits-vegetablesFacesHands
Assigning distances between new test images based on categorical RDM and predicted labels
Test Set Image 1
Test Set Image 2
human face
animal face
Assigned distance0.37
Predicted categorical RDM for the 78 images test data
Same distance for all the images within the same category
Mixing perceptual and categorical components
Large impact on fMRI-ITC and MEG-Late.
๐ = 1 โ ๐ค2 ๐ ๐๐๐ + ๐ค2๐ ๐๐๐ก
Training data: 92 image set
Results Test set: Perceptual + Categorical RDMs
DNN based RDMs
RDM based on DNN features at one layer
117 ๐ฅ 117
mean 0.12
corrDNN
1
64
1
64
2
63
2
63
Vgg L-1
Model Improvement including DNN Based RDMs
๐ = 1 โ ๐ค3 ๐ (๐๐๐+๐๐๐ก) +๐ค3๐ ๐๐๐
Improvement of ๐ 2 (explained variance) in EVCfor the 92 image set
๐ค3 ๐ค3๐ค3
Results Test set including DNNs
Conclusions
โข A mixture of perceptual and categorical RDMs made the largest contribution to the prediction accuracy in fMRI-ITC/MEG-Late.
โข VGG was the DNN that produced the largest improvement on the model performance.
โข However, it is necessary to evaluate the perceptual-categorical vs DNN contribution in the inverse order.
