classifying instantaneous cognitive states from fmri data
DESCRIPTION
Classifying Instantaneous Cognitive States from fMRI Data. Tom Mitchell, Rebecca Hutchinson, Marcel Just, Stefan Niculescu, Francisco Pereira, Xuerui Wang Carnegie Mellon University November, 2003. …. Does fMRI contain enough information? - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/1.jpg)
1
Classifying Instantaneous Cognitive States from fMRI Data
Tom Mitchell, Rebecca Hutchinson, Marcel Just, Stefan Niculescu, Francisco Pereira, Xuerui Wang
Carnegie Mellon University
November, 2003
![Page 2: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/2.jpg)
2
…
![Page 3: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/3.jpg)
3
![Page 4: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/4.jpg)
4
Cognitive state sequence
COGNITIVE TASK
“Virtual sensors” of cognitive state
1. Does fMRI contain enough information?
2. Can we devise learning algorithms to construct such “virtual sensors”?
![Page 5: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/5.jpg)
5
Learning Virtual Sensors
• Learn fMRI(t,t+k) CognitiveState
• Classifiers:– Gaussian Naïve Bayes, SVM, kNN
• Trained per subject, per experiment
• Feature selection/abstraction– Select subset of voxels (by signal, by anatomy)– Select subinterval of time– Average activities over space, time– Normalize voxel activities
![Page 6: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/6.jpg)
6
Study 1: Pictures and Sentences
• Trial: read sentence, view picture, answer whether
sentence describes picture
• Picture presented first in half of trials, sentence first
in other half
• Three possible objects: star, dollar, plus
• Collected by Just et al.
![Page 7: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/7.jpg)
7
It is true that the star is above the plus?
![Page 8: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/8.jpg)
8
![Page 9: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/9.jpg)
9
+
---
*
![Page 10: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/10.jpg)
10
![Page 11: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/11.jpg)
11
Is Subject Viewing Picture or Sentence?
• Learn fMRI(t,t+8) {Picture, Sentence}
• Leave two out cross-validation was used to assess the performance of the classifiers
• SVMs and GNB worked better than kNN
• Some Details: – 12 subjects, 40 pictures, 40 sentences– 1397 - 2864 voxels per subject, 7 ROIs – fMRI snapshot taken every half second
![Page 12: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/12.jpg)
12
Error for Single-Subject Classifiers
•Error computed by averaging over all subjects
•95% confidence intervals per subject are ~ 10% large
• Error of default classifier is 50%
Dataset \ Classifier GNB SVM 1NN 2NN 5NN
Picture vs Sentence 0.16 0.09 0.20 0.18 0.19
![Page 13: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/13.jpg)
13
• Approach: define supervoxels based on anatomically defined regions of interest– Normalize per voxel activity for each subject
• Each value scaled now in [0,1]
– Abstract to seven brain region supervoxels– 16 snapshots for each supervoxel
• Train on n-1 subjects, test on nth– Leave one subject out cross validation
Can We Train Subject-Indep Classifiers?
![Page 14: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/14.jpg)
14
• NO Feature Selection used in this experiment
•95% confidence intervals approximately 5% large
•Error of default classifier is 50%
Error for Cross Subject Classifiers
Dataset \ Classifier GNB SVM 1NN 2NN 5NN
Cross-Subject Pict vs Sent
0.30 0.25 0.36 0.33 0.32
![Page 15: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/15.jpg)
15
Study 2: Word Categories
• Family members
• Occupations
• Tools
• Kitchen items
• Dwellings
• Building parts
• 4 legged animals
• Fish
• Trees
• Flowers
• Fruits
• Vegetables
![Page 16: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/16.jpg)
16
Word Categories Study
• Stimulus:
– 12 blocks of words:
• Category name (2 sec)
• Word (400 msec), Blank screen (1200 msec); answer
• Word (400 msec), Blank screen (1200 msec); answer
• …
– Subject answers whether each word in category
– 20 words per block, nearly all in category
![Page 17: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/17.jpg)
17
Training Classifier for Word Categories
• Learn fMRI(t) Word Category
• Training methods: kNN, GNB
• Leave one example out from each class used to assess performance
• Some Details: – 10 subjects, 20 examples per class– 8470 - 11,136 voxels per subject, 30 ROIs– fMRI snapshot taken every second
![Page 18: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/18.jpg)
18
Study 2: Results
Classifier outputs ranked list of classesEvaluate by the fraction of classes ranked ahead of true
class– 0=perfect, 0.5=random, 1.0 unbelievably poor
Dataset \ Classifier
GNB 1NN 3NN 5NN
Words 0.08 0.30 0.20 0.16
![Page 19: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/19.jpg)
19
Study 3: Syntactic Ambiguity
• Is subject reading ambiguous or unambiguous sentence?– “The experienced soldiers warned about the dangers
conducted the midnight raid.”
– “The experienced soldiers spoke about the dangers before the midnight raid.”
• Almost random results if no feature selection used • With feature selection:
– SVM - 77% accuracy
– GNB - 75% accuracy
– 5NN – 72% accuracy
![Page 20: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/20.jpg)
20
• Four feature selection methods:
• Active (n most active available voxels compared to baseline fixation activity, according to a t-test)
• RoiActive (n most active voxels in each ROI)
• RoiActiveAvg (average of the n most active voxels in each ROI)
• Disc (n most discriminating voxels according to a trained classifier)
• Active works best
Feature Selection
![Page 21: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/21.jpg)
21
Dataset
Feature
Selection
GNB SVM 1NN 3NN 5NN
PictureSent
No 0.29 0.32 0.43 0.41 0.37
Active 0.16 0.09 0.20 0.18 0.19
Words
No 0.10 N/A 0.40 0.40 0.40
Active 0.08 N/A 0.30 0.20 0.16
SyntAmb
No 0.43 0.38 0.50 0.46 0.47
Active 0.25 0.23 0.29 0.29 0.28
Feature Selection
![Page 22: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/22.jpg)
22
Summary
• Proved that there is enough information in the fMRI signal to allow decoding of Cognitive States
• Successful training of classifiers for instantaneous cognitive state in three studies
• Cross subject classifiers trained by abstracting to anatomically defined ROIs
• Feature selection and abstraction are essential
![Page 23: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.vdocument.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/23.jpg)
23
Research Opportunities
• Learning temporal models– HMM’s, Temporal Bayes Nets
• Learn to discriminate whether a subject has certain mental disease
• Discovering useful data abstractions– ICA, PCA, hidden layers in Neural Nets
• Merging data from multiple sources– fMRI, ERP, reaction times