psychophysics 3 research methods fall 2010 tamás bőhm

Psychophysics 3

Research Methods

Fall 2010

Tamás Bőhm

Signal detection theory

• Aka. sensory decision theory (SDT)• A model & a data analysis method for decision

problems with uncertainty (noise)• Originates from World

War II: aircraft detection on radar signals

• Today: widely used in psychophysics, medicine, radiology and machine learning


• Experiment setup:– In some trials a stimulus (signal) is presented, in

others there is no stimulus;– Observer reports if she/he saw a signal or not– Calculate how many times the observer detected a

signal when she/he was presented one (hit rate)• Is the hit rate all we want to know?

Two observers achieved the same hit rate. Are they certainly behaving the same way?

• NO, we also need to know how many times the observer said “I see” when there was no signal (false alarm rate)


• Confusion matrix: contains all the information about the observer’s performance


• Confusion matrix: contains all the information about the observer’s performance

• As columns add up to 100%, it is enough to know one item from each column

40 trials

20 20

18

2

6

14

= 100% = 100%

= 90%

= 10% = 70%

= 30%


• Perfect detection:

100%

100%0%

0%


• No detection at all (1st example): always reporting “Seen”

100%

0%0%

100%


• No detection (2nd example): always reporting “Not seen”

0%

100%100%

0%


• No detection (3rd example): flipping a coin

50%

50%50%

50%


• No detection (4th example): reporting “Seen” in 30% of the trials (no matter what is presented)

30%

70%70%

30%=

=

Rows equal no detection


• Receiver operating characteristic (ROC):

false alarm rate

hit

rat

e

100%

100%



false alarm rate

hit

rat

e

100%

100%

90% 30%

10% 70%



false alarm rate

hit

rat

e

100%

100%

100% 0%

0% 100%

Perfect detection



false alarm rate

hit

rat

e

100%

100%

100% 100%

0% 0%

No detection: always “yes”



false alarm rate

hit

rat

e

100%

100%

0% 0%

100% 100%

No detection: always “no”



false alarm rate

hit

rat

e

100%

100%

50% 50%

50% 50%

No detection: reporting “yes” in 50% of the trials (flipping a coin)



false alarm rate

hit

rat

e

100%

100%

40% 40%

60% 60%

No detection: reporting “yes” in 40% of the trials



false alarm rate

hit

rat

e

100%

100%

30% 30%

70% 70%




false alarm rate

hit

rat

e

100%

100%

60% 60%

40% 40%




false alarm rate

hit

rat

e

100%

100%Diagonal: no detection


• SDT model:

• No way to remove the noise• But sensation can be separated from decision by

using ROCs

Sensation

Noise

DecisionSignal

present/absent

Sensation level (SL)

SL ≥ β

Criterion (β)

SL < β

YES

NO


Sensation

(Noise)

DecisionSignal

present/absent


SL ≥ β

Criterion (β)

SL < β

YES

NO

sensation level

pro

bab

ility

Without noise: perfect detection is possible

criterion signal present

signal absent


Sensation

(Noise)

DecisionSignal

present/absent


SL ≥ β

Criterion (β)

SL < β

YES

NO

sensation level

pro

bab

ility

criterion signal present

signal absent

100% 0%

0% 100%


Sensation

Noise

DecisionSignal

present/absent


SL ≥ β

Criterion (β)

SL < β

YES

NO

sensation level

pro

bab

ility

Noise: smears the distributions perfect detection is impossible (if the two distributions overlap)

signal absent(noise only)

signal present(signal+noise)criterion

online demo

http://psych.hanover.edu/JavaTest/Media/Chapter2/MedFig.SignalDetectionTheory.html


Sensation level

Sensation level

http://www-psych.stanford.edu/~lera/psych115s/notes/signal/


Sensation level

Sensation level

false alarm rateh

it r

ate


false alarm rateh

it r

ate

ROC curve

β = 8

β = 6

β = 10

β = 6

β = 8

β = 10


false alarm rateh

it r

ate

β

sensation level

pro

bab

ility

• Criterion (β): specifies where we are on the ROC curve

• The ROC curve is specified by sensory capacities only(discriminability)


• Discriminability: how well the observer can separate the presence of signal from its absence~ overlap between the two distributions~ bowing out of the ROC curve

• Measured by d’ (discriminability index,also called sensitivity)

http://www-psych.stanford.edu/~lera/psych115s/notes/signal/


d’: selects the ROC curve

β: specifies a point on the selected ROC curve

same information as hit rate & false alarm rate, but:

hit rate, false alarm rate:both reflect sensation & decision characteristics;cannot separate the two

d’: depends only on sensation

β: depends only on decision

β

The two processes are separated

http://psych.hanover.edu/JavaTest/Media/Chapter2/MedFig.ROC.html









Fechner’s methods:Is a stimulus detectable? Yes or no?

• Clear-cut threshold value (with some variability) that can be measured– Stimulus intensity >

threshold detectable– Stimulus intensity <

threshold not detectable

• Dichotic outcome, categorical model

Signal detection theory:How well is it detectable? How sensitive the observer is to the stimulus?

• Measured by d’– The higher d’ is, the more

the stimulus is detectable– d’ = 0

not detectable at all

• Scalar outcome, dimensional model


Sensation

(Noise)

Stimulus


Different task

Correct Incorrect

Forced-choice: eliminates the criterion

SDT: separates the criterion

Decision

SL ≥ β

Criterion (β)

SL < β

YES

NO

• Problem with Fechner’s methods: criterion


Psychophysical measurements with SDT:1. Create a stimulus set with a range of intensities (like

in the method of constant stimuli)2. Test each stimulus many times with each observer3. On each trial, either present a randomly selected

stimulus or do not present anything4. Ask the observer if he/she detected the stimulus5. Calculate the hit rate and false alarm rate for each

observer, for each stimulus intensity6. Use the formula/table to calculate d’ for each case7. Examine how d’ changes with intensity: the higher

d’ is for a stimulus intensity, the greater the observer’s ability to detect this intensity

http://psych.hanover.edu/JavaTest/Media/Chapter2/MedFig.SignalDetection.html









• Main results: changes in d’ values

Caudek–Rubin Vision Res. 2001


• There is also a β value for each d’ value• It can be informative about the decision

behavior:– Balanced: false alarm and

miss rates are equal– Liberal: the observer

says “yes” whenever there may be a signal

– Conservative: decision is yes only when it is almost certain that there is a signal

sensation level

pro

bab

ility

balanced

conservativeliberal

psychophysics 3 research methods fall 2010 tamás bőhm

Documents

signal detection theory

perfect detection slide

aircraft detection

signal false alarm rate

stimulus signal

hit rate

coin slide

characteristic roc