rdm mixtures for predicting visual cortices...
TRANSCRIPT
RDM mixtures for predicting visual cortices responses
Agustin Lage Castellanos1,2 and Federico De Martino2
1-Cuban Neuroscience Center, 2-Maastricht University
Algonauts Challenge 2019
Intuition behind our method
Perceptual
Cat
ego
rica
l
fMRI-EVC
MEG-early
fMRI-ITC
MEG-Late
DNN-L1
DNN-L3DNN-L2
DNN-L5DNN-L4
Combining RDMs to improve predictions
Predicted RDM Perceptual Categorical DNN
= 𝑤1 + 𝑤2 + 𝑤3
Perceptual RDMs
Perceptual RDMs
Only uses image information
Extract Edges and Smooth
Perceptual-RDMPixel Overlap
Categorical RDMs
Categorical Structure of the 92 image set
Objects-Scenes
animals
Human
Fruits-vegetables
Faces
Hands
Monkey faces
Animal Faces
Within category RDM based on fMRI/MEG data similarity
92 x 92 8 x 8
mean
Between image fMRI/MEG similarity Between category fMRI/MEG similarity
fMRI-ITC
Training a GNB classifier as predicting category
GNB
Class Labels
Last fully connected layer (defines category membership)
Leave one out CV on the 92 image training set
Classification of the 78 test set images
𝑊𝐺𝑁𝐵
Predicted Labels
x
Predicted as Human Faces
Predicted as Animal Faces in the 78 set
Objects-Scenes
Animal Faces
animalsHumanFruits-vegetablesFacesHands
Assigning distances between new test images based on categorical RDM and predicted labels
Test Set Image 1
Test Set Image 2
human face
animal face
Assigned distance0.37
Predicted categorical RDM for the 78 images test data
Same distance for all the images within the same category
Mixing perceptual and categorical components
Large impact on fMRI-ITC and MEG-Late.
𝑅 = 1 − 𝑤2 𝑅𝑝𝑒𝑟 + 𝑤2𝑅𝑐𝑎𝑡
Training data: 92 image set
Results Test set: Perceptual + Categorical RDMs
DNN based RDMs
RDM based on DNN features at one layer
117 𝑥 117
mean 0.12
corrDNN
1
64
1
64
2
63
2
63
Vgg L-1
Model Improvement including DNN Based RDMs
𝑅 = 1 − 𝑤3 𝑅(𝑝𝑒𝑟+𝑐𝑎𝑡) +𝑤3𝑅𝑑𝑛𝑛
Improvement of 𝑅2 (explained variance) in EVCfor the 92 image set
𝑤3 𝑤3𝑤3
Results Test set including DNNs
Conclusions
• A mixture of perceptual and categorical RDMs made the largest contribution to the prediction accuracy in fMRI-ITC/MEG-Late.
• VGG was the DNN that produced the largest improvement on the model performance.
• However, it is necessary to evaluate the perceptual-categorical vs DNN contribution in the inverse order.