Learning to Identify Overlapping and Hidden

Cognitive Processes from fMRI Data

Rebecca Hutchinson, Tom Mitchell, Indra Rustandi

Carnegie Mellon University

Decide whether consistent

How can we track hidden cognitive processes?

Read sentence

View picture

Cognitive processes:

?

Observed fMRI:

cortical region 1:

cortical region 2:

Observed button press:

Typical BOLD response

Sig

nal

Am

plitu

de

Time (seconds)

• At left is a typical averaged BOLD response

• Here, subject reads a word, decides whether it is a noun or verb, and pushes a button in less than 1 second.

Related Work

• General linear model (GLM) applied to fMRI– E.g., [Dale 1999]; SPM;– Accommodates multiple, overlapping processes,– But not unknown process timing

• Dynamic Bayesian Networks – Family of probabilistic models for time series– E.g., Factorial HMMs [Ghahramani & Jordan 1998]– Accommodate hidden timings/states– But do not capture convolution of overlapping states– Require learning detailed next-state function

Approach: Hidden Process Models

• Probabilistic model– Can evaluate P(model | data), P(data | model)

• Describe hidden processes by their– Type, duration, start time, fMRI signature

• Algorithms for learning model, interpreting data– Learn maximum likelihood models and data

interpretations

Hidden Process Models

Process ID = 3

Process ID = 2Process Instances:

Observed fMRI:

Processes:

ID: 1 Timing: P(start=+O)Response:

ID: 2Timing: P(start=+O)Response:

ID: 3 Timing: P(start=+O)Response:

Process ID = 1 Process ID = 1

Time landmarks: ¢ 1¢ 2¢ 1 ¢ 3

Process: ViewPicture

Duration d: 11 sec. P(Offset times): , Response signature W:

Configuration C of Process Instances h 1, 2, … i

Observed data Y:

Input Stimulus :

14

Timing landmarks : ¢ 2¢ 1 ¢ 3

2

Process instance: 2

Process h: ViewPicture Timing landmark : 2

Offset time O: 1 sec Start time ´ + O

sentencepicture

sentence

3

Hidden Process Models

HPMs More Formally…

Process h =

h d, , W i

Process Instance =

h h, , O i

Configuration C = set of Process Instances

Hidden Process Model HPM = h H, , C, i

• H: set of processes• : prior probs over H• C: set of candidate

configurations

• : h 1 … v i voxel noise model

HPM Generative ModelProbabilistically generate data using a configuration

of N process instances with known landmarks:

1. Generate a configuration C of process instances: For i=1 to N, generate process instance i

• Choose a process hi according to P(h| i , )• Choose an offset Oi according to P(O| (h) )

2. Generate all observed fMRI data ytv given C:

HPM Inference

• Given:– An HPM,

• including a set of candidate configurations• we typically assume processes known, but not timing

– Observed data Y

• Determine:– The most probable process instance

configuration c– P(C=c|Y, HPM) P(Y|C=c, HPM) P(C=c | HPM)

Inference: Example

Configuration 1:

Observed data

ProcessID=1, S=1

ProcessID=2, S=17

ProcessID=3, S=21

Configuration 2:

ProcessID=2, S=1

ProcessID=1, S=17

ProcessID=3, S=23

Prediction 1

Prediction 2

Learning HPMs with unknown timing O(), known processes h()

EM (Expectation-Maximization) algorithm

• E-step– Estimate the conditional distribution over start

times of the process instances given observed data, P(O(1)…O(N) | Y, h(1)… h(N), HPM).

• M-step– Use the distribution from the E step to get

maximum-likelihood estimates of the HPM parameters.

* In real problems, some timings are often known

HPMs are learnable from realistic amounts of data

Figure 1. The learner was given 80 training examples with known start times for only the first two processes. It chooses the correct start time (26) for the third process, in addition to learning the HDRs for all three processes.

true signal

Observed noisy signal

true response W

learned W

Process 1 Process 2 Process 3

fMRI Study: Pictures and Sentences

• Each trial: determine whether sentence correctly describes picture

• 40 trials per subject.• Picture first in 20 trials, Sentence first in other 20• Images acquired every 0.5 seconds.

Read Sentence

View Picture Read Sentence

View PictureFixation

Press Button

4 sec. 8 sec.t=0

Rest

Decide whether consistent

HPM model for Picture-Sentence Comparison

Read sentence

View picture

Cognitive processes:

?

Observed fMRI:

cortical region 1:

cortical region 2:

Observed button press:

Learned HPM with 3 processes (S,P,D), and R=13sec (TR=500msec).

P PS S

D?

observed

Learned models:

S

P

D

D start time chosen by program as t+18

reconstructed

P PS S

D D

D?

HPMs provide more accurate classification of unknown

processes than earlier methods

(e.g., Gaussian Naïve Bayes (GNB) classifier)

Standard classifier formulation

View PictureOr

Read Sentence

Read SentenceOr

View PictureFixation

Press Button

4 sec. 8 sec.t=0

Rest

picture or sentence? picture or sentence?

16 sec.

GNB:

Standard formulation of classification problem (e.g., Gaussian Naïve Bayes (GNB)):

Train on labeled data: known Processes, known StartTimes

Test on unlabeled data: unknown Processes, known StartTimes

HPM classifier accounts for overlap

View PictureOr

Read Sentence

Read SentenceOr

View PictureFixation

Press Button

4 sec. 8 sec.t=0

Rest

picture or sentence? picture or sentence?

16 sec.

GNB:

picture or sentence?

picture or sentence?

HPM:

View PictureOr

Read Sentence

Read SentenceOr

View PictureFixation

Press Button

4 sec. 8 sec.t=0

Rest

picture or sentence? picture or sentence?

16 sec.

GNB:

picture or sentence?

picture or sentence?

HPM:

Results

HPM with overlapping processes improves accuracy by 15% on average.

HPMs allow detecting and examining hidden processes

with unknown timing

Decide whether consistent

Two cognitive processes, or three?

Read sentence

View picture

Cognitive processes:

?

Observed fMRI:

cortical region 1:

cortical region 2:

Observed button press:

Choosing Between Alternative HPM Models

• Train 2-process HPM2 on training data

• Train 3-process HPM3 on training data

• Test HPM2 and HPM3 on separate test data

– Which predicts process identities better?– Which has higher probability given the test data?– (use n-fold cross-validation for test)

2-process HPM, 3-process HPM, GNB

Summary

• Hidden Process Model formalism

• Superiority over earlier classification methods

• Basis for studying hidden cognitive processes

Future Directions

• Add temporal and/or spatial smoothness constraints to process fMRI signatures

• Allow variable duration processes

• Give processes input arguments, output results

• Feature selection for HPMs

• Process libraries, hierarchies