11/14/2011
1
G89.2223 Perception
Sensory Cue Combination
Laurence T. Maloney
Some examples of cues to depthor shape
Retinal projection depends on size and distance
Monocular depth cues Epstein (1965) familiar size experiment
How far away is the coin?
Cast Shadows Shading and contour
11/14/2011
2
Texture
1. Density2. Foreshortening
3. Size
Linear perspective
Binocular depth cues Vergence Angle As One Binocular Source
Vergence Angle As One Binocular Source
Vergence Angle As One Binocular Source
11/14/2011
3
Wheatstone stereoscope (c. 1838)
Sir Charles Wheatstone
Dual mirror stereoscope
Uncrossed disparity
Zero retinal disparity
Crossed disparity
Disparity
zero disparity
uncrossed disparity
crossed disparity
Fixation point
Cues in conflict
What cues?
11/14/2011
4
Ames room Ames room
Ames room
Cue Combination
Rock & Victor (1964)
Visually and haptically specified shapes differed.What shape is perceived?
View object through distorting lens while exploring object haptically
Irv Rock
Rock & Victor (1964)Experimental Design
11/14/2011
5
Rock & Victor (1964)Results
1.90 0.98 1.85
13.4 23.1 14.1 mm
14.1 20.5 14.5 mm
Rock & Victor (1964)Results
1.90 0.98 1.85
13.4 23.1 14.1 mm
14.1 20.5 14.5 mm
Visual Capture
How should we combine cues?
V
HS
S
haptic size estimate
visual size estimate
random variables
Modeling Cue Combination
,
,
H
V V
HS Gaussian s
S Gaussian s
true location
s
VS HS
s
,
,
H
V V
HS Gaussian s
S Gaussian s
standard deviation
s
VS HS
s
11/14/2011
6
location-scale families
s
VS
,V VS Gaussian s
V
VE S s
10 ,1
10 ,2
H
V
S Gaussian cm cm
S Gaussian cm cm
s
VS HS $10 if you are within1 cm of s
Which cue?
Chances of winning?
s
VS HS
Can we do better by combining cues?
1V HS wS w S
weighted linear combination
s
VS HS
1
(1 )V HE S wE S w E S
ws w s s
s
VS HS
1
(1 )V HE S wE S w E S
ws w s s
Gaussian ,S Gaussian s w
What is SD?
s
VS HS
22
2 2 2 2
1
(1 )
V H
V H
Var S w Var S w Var S
w w
a parabola in w
11/14/2011
7
2 2 2 2(1 )V HVar S w w
0 1w
2V
2H
2 2 2 2(1 )V HVar S w w
0 1w
2V
2H
could it be?
s
VS HS
2 2
2
2 2
2 2(1 ) ! 0V H
H
V H
Var Sw w
w
w
0 1w
2V
2H
s
VS HS
2
2 2H
V H
w
some examples ….
2 2 2 2(1 )V HVar S w w
0 1w
2V
2H
optimal cue combination
effective cue combination
Rock & Victor (1964)
Visually and haptically specified shapes differed.What shape is perceived?
View object through distorting lens while exploring object haptically
Irv Rock
Why visual capture?
11/14/2011
8
Visual/Haptic Setup
Visual Capture ?
Why should vision be the “gold standard”all other modalities are compared to?
SVH wV SV wH SH
Variance
Weights
wV H
2
V
2 H
2
1
VH
2
1
V
2
1
H
2
Visual Capture ?
Why should vision be the “gold standard”all other modalities are compared to?
SVH wV SV wH SH
Variance
Weights
wV H
2
V
2 H
2
1
VH
2
1
V
2
1
H
2
Visual Capture ?
Why should vision be the “gold standard”all other modalities are compared to?
SVH wV SV wH SH
Variance
Weights
wV H
2
V
2 H
2
1
VH
2
1
V
2
1
H
2
Visual Capture ?
Why should vision be the “gold standard”all other modalities are compared to?
SVH wV SV wH SH
Variance
Weights
wV H
2
V
2 H
2
1
VH
2
1
V
2
1
H
2
11/14/2011
9
Experimental Outline
1) determine (& manipulate) within-modality variances• discrimination thresholds (2-IFC, constant stimuli)
2) make predictions for combined performance • using MLE model to predict weights & combined variance.
3) measure combined performance & compare to prediction• similar to within-modality 2-IFC discrimination task (get PSE and thresholds)
Standard Comparison
no feedback!
2‐IFC Task
Visual-HapticVisual-alone Haptic-alone
Three Conditions
Predictions
Determining Within‐Modality Variance Determining Within‐Modality Variance
Threshold
Determining Within‐Modality Variance
Threshold
Determining Within‐Modality Variance
Threshold
11/14/2011
10
Determining Within‐Modality Variance
Threshold
Determining Within‐Modality Variance
Threshold
From Variance to Threshold
visual-haptic varianceestimators weights
Predicted weights for combined performance from within-modal data
Predicted combined threshold from within-modal data
wV H
2
V
2 H
2
visual-haptic thresholdestimators weights
wV JNDH
2
JNDV2 JNDH
2
1
JNDVH2
1
JNDV2
1
JNDH2
1
VH
2
1
V
2
1
H
2
JNDi 2 i JNDi 2 i
Visual‐Haptic Discrimination
Visual‐Haptic Discrimination Visual‐Haptic Discrimination
11/14/2011
11
Visual‐Haptic Discrimination Visual‐Haptic Discrimination
Empirical Thresholds and Weights
“visual capture”
“haptic capture”
Weights & PSEs
Individual Differences
LAS
LAS
RSB
1
RSB
MOEE
mp
iric
al V
isu
al W
eig
ht
MOE
HTE
HTE
JWW
JWWKML
KML
0% noise
133% noise
Predicted Visual Weight
0.80.60.40.20
1
0.8
0.6
0.4
0.2
Conclusions
Combination reduces variance.
Linear weighting scheme for visual-haptic perception.
Explains behavior like “visual capture” or visual dominance.i.e, vision is given a weight of ~ 1.0 if the variance of the visual estimate is less then the variances of the other modalities.