dissimilarity learning via siamese network predicts...
TRANSCRIPT
![Page 1: Dissimilarity learning via Siamese network predicts …algonauts.csail.mit.edu/slides/Algonauts2019_Aakash...Aakash Agrawal Why Algonauts? Letter representation in the brain Visual](https://reader034.vdocument.in/reader034/viewer/2022050510/5f9a87146a309e6b520104de/html5/thumbnails/1.jpg)
Dissimilarity learning via Siamese network predicts brain imaging data
Aakash Agrawal
![Page 2: Dissimilarity learning via Siamese network predicts …algonauts.csail.mit.edu/slides/Algonauts2019_Aakash...Aakash Agrawal Why Algonauts? Letter representation in the brain Visual](https://reader034.vdocument.in/reader034/viewer/2022050510/5f9a87146a309e6b520104de/html5/thumbnails/2.jpg)
Why Algonauts?
![Page 3: Dissimilarity learning via Siamese network predicts …algonauts.csail.mit.edu/slides/Algonauts2019_Aakash...Aakash Agrawal Why Algonauts? Letter representation in the brain Visual](https://reader034.vdocument.in/reader034/viewer/2022050510/5f9a87146a309e6b520104de/html5/thumbnails/3.jpg)
![Page 4: Dissimilarity learning via Siamese network predicts …algonauts.csail.mit.edu/slides/Algonauts2019_Aakash...Aakash Agrawal Why Algonauts? Letter representation in the brain Visual](https://reader034.vdocument.in/reader034/viewer/2022050510/5f9a87146a309e6b520104de/html5/thumbnails/4.jpg)
Letter representation in the brainVisual Search
Dissimilarity = 1/response time
Arun 2012
![Page 5: Dissimilarity learning via Siamese network predicts …algonauts.csail.mit.edu/slides/Algonauts2019_Aakash...Aakash Agrawal Why Algonauts? Letter representation in the brain Visual](https://reader034.vdocument.in/reader034/viewer/2022050510/5f9a87146a309e6b520104de/html5/thumbnails/5.jpg)
Behaviour
Behaviour + fMRI
![Page 6: Dissimilarity learning via Siamese network predicts …algonauts.csail.mit.edu/slides/Algonauts2019_Aakash...Aakash Agrawal Why Algonauts? Letter representation in the brain Visual](https://reader034.vdocument.in/reader034/viewer/2022050510/5f9a87146a309e6b520104de/html5/thumbnails/6.jpg)
fMRIBehaviour
Pramod & Arun, 2016Khaligh‐Razavi & Kriegeskorte, 2014
![Page 7: Dissimilarity learning via Siamese network predicts …algonauts.csail.mit.edu/slides/Algonauts2019_Aakash...Aakash Agrawal Why Algonauts? Letter representation in the brain Visual](https://reader034.vdocument.in/reader034/viewer/2022050510/5f9a87146a309e6b520104de/html5/thumbnails/7.jpg)
Siamese networks
where L is the loss function,y the class label (0 or 1) and p is the prediction.
https://medium.com
Contrastive Loss
L - y log p 1-y log 1-p
![Page 8: Dissimilarity learning via Siamese network predicts …algonauts.csail.mit.edu/slides/Algonauts2019_Aakash...Aakash Agrawal Why Algonauts? Letter representation in the brain Visual](https://reader034.vdocument.in/reader034/viewer/2022050510/5f9a87146a309e6b520104de/html5/thumbnails/8.jpg)
Modified siamese networks
EVC/ MEG early – AlexNetIT/ MEG late – VGG16
Exploration list
1) Architecture2) Layers to fine‐tune3) Feature extraction layer4) Epoch5) Loss function6) Training dataset 7) Other hyperparameter –
learning rate, batch size, etc.
![Page 9: Dissimilarity learning via Siamese network predicts …algonauts.csail.mit.edu/slides/Algonauts2019_Aakash...Aakash Agrawal Why Algonauts? Letter representation in the brain Visual](https://reader034.vdocument.in/reader034/viewer/2022050510/5f9a87146a309e6b520104de/html5/thumbnails/9.jpg)
Pre‐processing
![Page 10: Dissimilarity learning via Siamese network predicts …algonauts.csail.mit.edu/slides/Algonauts2019_Aakash...Aakash Agrawal Why Algonauts? Letter representation in the brain Visual](https://reader034.vdocument.in/reader034/viewer/2022050510/5f9a87146a309e6b520104de/html5/thumbnails/10.jpg)
MEG test RDMs
layer 12 from epoch 164 (MEG_early_RDMs)
layer 34 from epoch 13 of (MEG_late_RDMs)
![Page 11: Dissimilarity learning via Siamese network predicts …algonauts.csail.mit.edu/slides/Algonauts2019_Aakash...Aakash Agrawal Why Algonauts? Letter representation in the brain Visual](https://reader034.vdocument.in/reader034/viewer/2022050510/5f9a87146a309e6b520104de/html5/thumbnails/11.jpg)
Which layers were affected by fine‐tuning?
![Page 12: Dissimilarity learning via Siamese network predicts …algonauts.csail.mit.edu/slides/Algonauts2019_Aakash...Aakash Agrawal Why Algonauts? Letter representation in the brain Visual](https://reader034.vdocument.in/reader034/viewer/2022050510/5f9a87146a309e6b520104de/html5/thumbnails/12.jpg)
Which layers were affected by fine‐tuning?
![Page 13: Dissimilarity learning via Siamese network predicts …algonauts.csail.mit.edu/slides/Algonauts2019_Aakash...Aakash Agrawal Why Algonauts? Letter representation in the brain Visual](https://reader034.vdocument.in/reader034/viewer/2022050510/5f9a87146a309e6b520104de/html5/thumbnails/13.jpg)
Summary
Exploration list
1) Architecture2) Layers to fine‐tune3) Feature extraction layer4) Epoch5) Loss function6) Training dataset 7) Other hyperparameter –
learning rate, batch size, etc.
Trained from scratch