Inference in Dynamic Environments

Mark Steyvers

Scott Brown

UC Irvine

This work is supported by a grant from the US Air Force Office of Scientific Research (AFOSR grant number FA9550-04-1-0317)

Overview

• Experiments with dynamically changing environments

– How do observers adapt their decision-making strategies?

– How quickly can observers detect changes?

• Theory development

– Bayesian ideal observers

– Process models

– Measure individual differences in adaptation

Part I Decision Criteria Adaption

( a quick overview from last year’s presentation)

DifficultBlock

EasyBlock

Decision Environment

parameter

Subject’sdecision criterion

Decision lag

EasyBlock

DifficultBlock

Time

Environments alternating between easy and hard blocks

Time

Realistic 3D gaming tasks

• Decide which of two types of missiles is approaching

• How quickly can a participant adapt to changes in the similarity between the two missiles?

A=“not puffy”

B=“very puffy”

Modeling Results

• Developed a lagged signal detection model

• Model estimates that participants take an average of 9 trials to switch to a new decision criterion

• Individual differences

lag

0 10 20 30 40

#Par

tici

pan

ts

Part IIPrediction and Change Detection

Change detection and prediction

• Prediction of future observations in time-series data

• Changepoint models– E.g. changing to different coins at random times

• Accurate prediction requires detection of change– stock market– “hot hand” players

Basic Task

• Given a sequence of random numbers, predict the next one

= observed data

= prediction

Two-dimensional prediction task

• Touch screen monitor

• 1500 trials • Self-paced • Same sequence

for all subjects

0

5

10

0

5

10

Subject 4

40 50 60 70 80 90 100 110 120 1300

5

10

Time

Subject 12

Sequence Generation

• (x,y) locations are drawn from a binomial distribution of size 10, and parameter θ

• At every time step, probability 0.1 of changing θ to a new random value in [0,1]

• Example sequence:

Time

θ=.12 θ=.95 θ=.46 θ=.42 θ=.92 θ=.36

0

5

10

Optimal Bayesian Solution

0

5

10

Subject 4

40 50 60 70 80 90 100 110 120 1300

5

10

Time

Subject 12

= observed sequence

Optimal Bayesian Solution= prediction

0

5

10

Optimal Bayesian Solution

0

5

10

Subject 4

40 50 60 70 80 90 100 110 120 1300

5

10

Time

Subject 12

Subject 4 – change detection too slow0

5

10

Optimal Bayesian Solution

0

5

10

Subject 4

40 50 60 70 80 90 100 110 120 1300

5

10

Time

Subject 12

Subject 12 – change detection too fast

(sequence from block 5)

Tradeoffs

• Detecting the change too slowly will result in lower accuracy and less variability in predictions than an optimal observer.

• Detecting the change too quickly will result in false detections, leading to lower accuracy and higher variability in predictions.

0.5 1 1.5 2 2.5 3 3.52.7

2.8

2.9

3

3.1

3.2

3.3

3.4

3.5

Mean Absolute Movement

Me

an

Abs

olu

te T

ask

Err

or

12

34

5

6

7

8 9

OPTIMAL SOLUTION

Average Error vs. Movement

= subject

Relatively many changes

Relatively few changes

A simple model

1. Make new prediction some fraction α of the way between old prediction and recent outcome. α = change proportion

2. Fraction α is a linear function of the error made on last trial

3. Two free parameters: A, B

A<B bigger jumps with higher error

A=B constant smoothing

1 (1 )t t tp y p

t t tError y p

α

0

1

A

B

BA

Average Error vs. Movement

0.5 1 1.5 2 2.5 3 3.52.7

2.8

2.9

3

3.1

3.2

3.3

3.4

3.5

Mean Absolute Movement

Me

an

Abs

olu

te T

ask

Err

or

12

34

5

6

7

8 9

OPTIMAL SOLUTION

= subject

= model

One-dimensional Prediction Task

1

2

3

4

5

6

7

8

9

10

11

12

12 PossibleLocations

• Where will next blue square arrive on right side?

Average Error vs. Movement

0 0.5 1 1.5 2

1.2

1.4

1.6

1.8

2

2.2

2.4

Mean Absolute Movement

Me

an

Abs

olu

te T

ask

Err

or

12

3

4

56

7

8

910

11

12

13

14

15

16

17

1819

2021

OPTIMAL SOLUTION

= subject

= model

Inference and Prediction Judgments

• Many errors in prediction

– too much movement

• What is causing this?

– Faulty change detection or faulty prediction judgments?

– Gambler’s fallacy: people predict too many alternations

• New experiments:

– ask directly about the generating process

– inference judgment: what currently is the state of the system?

Tomato Cans Experiment

• Cans roll out of pipes A, B, C, or D

• Machine perturbs position of cans (normal noise)

• At every trial, with probability 0.1, change to a new pipe (uniformly chosen)

(real experiment has response buttons and is subject paced)

A B C D

Tomato Cans Experiment

(real experiment has response buttons and is subject paced)

A B C D • Cans roll out of pipes A, B, C, or D

• Machine perturbs position of cans (normal noise)

• At every trial, with probability 0.1, change to a new pipe (uniformly chosen)

• Curtain obscures sequence of pipes

Tasks

A B C D• Inference:

what pipe produced the last can?

A, B, C, or D?

• Prediction: in what region will the next can arrive?

1, 2, 3, or 4?

1 2 3 4

Experiment 1

• 63 subjects

• 12 blocks

– 6 blocks of 50 trials for inference task

– 6 blocks of 50 trials for prediction task

– Identical trials for inference and prediction

INFERENCE PREDICTION

Sequence

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

1

2

3

4

5

6

1 2 3 4 5 6A B C D

Trial

INFERENCE PREDICTION

Sequence

Ideal Observer

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

1

2

3

4

5

6

1 2 3 4 5 6A B C D

Trial

INFERENCE PREDICTION

Sequence

Ideal Observer

Individualsubjects

Trial0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

1

2

3

4

5

6

1 2 3 4 5 6A B C D

INFERENCE PREDICTION

Sequence

Ideal Observer

Trial0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

1

2

3

4

5

6

1 2 3 4 5 6A B C D

Individualsubjects

INFERENCE PREDICTION

Sequence

Ideal Observer

Trial0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

1

2

3

4

5

6

1 2 3 4 5 6A B C D

Individualsubjects

0 20 40 6030

40

50

60

70

80

90

100

Changes (%)

Acc

ura

cy

0 20 40 6030

40

50

60

70

80

90

100

Changes (%)

Acc

ura

cy

= Subjectideal

ideal

INFERENCE PREDICTION

0 20 40 6030

40

50

60

70

80

90

100

Changes (%)

Acc

ura

cy

ideal

0 20 40 6030

40

50

60

70

80

90

100

Changes (%)

Acc

ura

cy

ideal

INFERENCE PREDICTION

= Process model

= Subject

50

60

70

80

90

100low alpha

50

60

70

80

90

100

% A

ccu

racy

(ag

ain

st T

rue

)

med alpha

0 10 20 30 40 50 6050

60

70

80

90

100

% Changes

high alpha

Varying Change Probability

• 136 subjects

• Inference judgments only

Subjects track changes in alpha

ideal

ideal

ideal

Prob. = .08

Prob. = .16

Prob. = .32

Number of Perceived Changes per Subject

Low medium high

Change Probability

(Red line shows ideal number of changes)

Subject #1

Number of Perceived Changes per Subject

55% of subjects show increasing pattern

45% of subjects show non-increasing pattern

Low, medium, high change probability Red line shows ideal number of changes

Conclusion

• Adaptation in non-stationary decision

environments

• Subjects are able to track changes in dynamic decision environments

• Individual differences

– Over-reaction: perceiving too much change

– Under-reaction: perceiving too little change

Potential deliverable outcomes

• Simple measurement models of decision making in dynamic environments.

• Classify different observers’ decision making abilities under different kinds of environments

– E.g., does this observer respond too slowly to changes in their environment?

– Is another observer close to the theoretically optimal decision mechanism?

– Conversely, classify different observers as optimally suited to different tasks, depending on dynamic properties of decision making environments in those tasks.

Publications

• Brown, S.D., & Steyvers, M. (2005). The Dynamics of Experimentally Induced Criterion Shifts. Journal of Experimental Psychology: Learning, Memory & Cognition, 31(4), 587-599.

• Steyvers, M., & Brown, S. (2005). Prediction and Change Detection. In: Advances in Neural Information Processing Systems, 19

• Navarro, D.J., Griffiths, T.L., Steyvers, M., & Lee, M.D. (2006). Modeling individual differences using dirichlet processes. Journal of Mathematical Psychology, 50, 101-122.

• Wagenmakers, E.J., Grunwald, P., & Steyvers, M. (2006). Accumulative prediction error and the selection of time series models. Journal of Mathematical Psychology, 50, 149-166.