image statistics and the perception of 3d shape roland w. fleming max planck institute for...
TRANSCRIPT
Image Statistics and the Image Statistics and the Perception of 3D ShapePerception of 3D ShapeRoland W. FlemingMax Planck Institute
for Biological Cybernetics
Yuanzhen Li
Edward H. AdelsonMassachusetts Institute of Technology
Matte
Glossy
Mirrored
Hen
ry M
oore
Visual system estimates surface orientation from image intensity
Classical Classical Shape from ShadingShape from Shading
reflectance mapimage
Visual system estimates surface orientation from image intensity
Classical Classical Shape from ShadingShape from Shading
reflectance map
Problems: Intensities are ambiguous
Reflectance map is unknown
No principled way to predict successes vs. failures of shape perception
Surface reflectanceSurface reflectance
A parametric space of glossy plastic materials (using Ward model)
Diffuse Reflectance, dDiffuse Reflectance, d
Sp
ecu
lar
Reflect
an
ce,
sS
pecu
lar
Reflect
an
ce,
s
Don’t use image intensity ! Use the kinds of image measurements the visual
system employs at the front end
Alternative approachAlternative approach
reflectance mapimage
Don’t use image intensity ! Use the kinds of image measurements the visual
system employs at the front end
Alternative approachAlternative approach
image
What can these measurements tell us about 3D shape ?
Can filter responses predict human shape perception ?
highly curved
Curvatures determineCurvatures determinedistortionsdistortions
slightlycurved
Anisotropies in surface curvature lead to powerful distortions of the reflected world
Curvatures determineCurvatures determinedistortionsdistortions
Population codesPopulation codes
Population codesPopulation codes
Population codesPopulation codes
Population codesPopulation codes
StatisticsStatisticsIlluminations
Sh
ap
es
Render many images: 50 Shapes 12 Illuminations 5 Reflectances
Measure the distribution of orientations (i.e. filter population response) for every point in every image
Look for regularities
Orientation fieldsOrientation fields
Ground truth
Orientation fieldsOrientation fields
Error (estimate - ground truth)
Surface reflectanceSurface reflectance
Diffuse Reflectance, dDiffuse Reflectance, d
Sp
ecu
lar
Reflect
an
ce,
sS
pecu
lar
Reflect
an
ce,
s
If the visual system relies on these measurements then:
1: Shape perception should be stable across changes that do not affect these measurements
2: Perceived shape should vary systematically when scene or image modifications do affect these measurements
PredictionsPredictions
Perceived shape should be extremely stable across changes in surface glossiness.
Prediction 1Prediction 1
Nefs, Koenderink & Kappers, 2006
“We found no evidence that the perceived shapes of glossy objects are different from the perceived shapes of matte objects...”
ExperimentExperiment
Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection
ExperimentExperiment
Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection
Orientation fieldsOrientation fields
ground truth
Orientation fieldsOrientation fields
ground truth
For shaded surfaces, perceived shape should undergo (subtle) changes across variations in illumination
Prediction 2Prediction 2
For shaded surfaces, perceived shape should undergo (subtle) changes across variations in illumination
Prediction 2Prediction 2
For shaded surfaces, perceived shape should undergo (subtle) changes across variations in illumination
Prediction 2Prediction 2
Todd, Norman, Koenderink & Kappers (1997) report little effect of illumination. But that was with additional cues.
Koenderink, van Doorn, Christou & Lappin (1996)
Nefs, Koenderink & Kappers, 2006
For shaded surfaces, perceived shape should undergo (subtle) changes across variations in illumination
Prediction 2Prediction 2
Caniard & Fleming, 2007
If the visual system relies on these measurements then:
1: Shape perception should be stable across changes that do not affect these measurements. Even when these changes are not natural.
2: Perceived shape should vary systematically when scene or image modifications do affect these measurements
PredictionsPredictions
Test improbable combination of lighting and reflectance
Decouple intensity from image orientation
non-linear intensity transfer function
normal shadingnormal shading ‘‘weird’ shadingweird’ shading
““Weird” ShadingWeird” Shading
normal shadingnormal shading‘‘weird’ shadingweird’ shading
““Weird” shadingWeird” shading
normal shadingnormal shading‘‘weird’ shadingweird’ shading
““Weird” shadingWeird” shading
perceived tiltperceived tilt
perceived slantperceived slant
normal shadingnormal shading
““ weir
d”
shadin
gw
eir
d”
shad
ing
S1
S1
normal shadingnormal shading‘‘weird’ shadingweird’ shading
““Weird” shadingWeird” shading
perceived tiltperceived tilt
perceived slantperceived slant
normal shadingnormal shading
““ weir
d”
shadin
gw
eir
d”
shad
ing
S2
S2
normal shadingnormal shading
‘‘ weir
d’
shad
ing
weir
d’
shad
ing
““Weird” shadingWeird” shading
Pooled data across 6 shapes
tilttilt slantslant
normal shadingnormal shading
r2 = 0.93 r2 = 0.88
Affine TransformationAffine Transformation
Shear:- does affect first derivatives- does NOT affect second derivatives
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
Matching TaskMatching Task
Subject adjusts shear of match until it appears to be same shape as test
test match
Matching TaskMatching Task
Subject adjusts shear of match until it appears to be same shape as test
test match
Matching TaskMatching Task
Subject adjusts shear of match until it appears to be same shape as test
test match
Matching TaskMatching Task
Subject adjusts shear of match until it appears to be same shape as test
test match
PredictionsPredictions
test shear
matc
h s
hear ve
ridi
cal
image statisticsprediction
ResultsResults
test shear
matc
h s
hear
If the visual system relies on these measurements then:
1: Shape perception should be stable across changes that do not affect these measurements.
2: Perceived shape should vary systematically when scene or image modifications do affect these measurements. Even when these changes are not natural.
PredictionsPredictions
Illusory distortionsIllusory distortionsof shapeof shape
Inspired by Todd & Thaler VSS 05
Illusory distortionsIllusory distortionsof shapeof shape
Inspired by Todd & Thaler VSS 05
Illusory distortionsIllusory distortionsof shapeof shape
Inspired by Todd & Thaler VSS 05
Illusory distortionsIllusory distortionsof shapeof shape
Illusory distortionsIllusory distortionsof shapeof shape
Experiment
ResultsResultsveridicalstimulus
ResultsResultspredictedstimulus
ResultsResultsresultsstimulus
Dot product between subject’s data and predictions
ResultsResults
Dot product between subject’s data and predictions
ResultsResults
Dot product between subject’s data and predictions
ResultsResults
“veridical”prediction
“ori
en
tati
on
field
”p
red
icti
on
Dot product between subject’s data and predictions
ResultsResults
“veridical”prediction
“ori
en
tati
on
field
”p
red
icti
on
ConclusionsConclusions
Useful shape cues can be derived from relatively simple image measurements at the front end of vision
In some cases these measurements are surprisingly robust across variations in other scene properties (e.g. illumination, reflectance).
Scale and orientation measurements can predict certain successes and failures of human 3D shape perception across a range of natural and unnatural stimuli.
Thank youThank youFunding
RF supported byDFG FL 624/1-1
Generative space of all possible combinations of surface curvature and local orientation in the reflectance map
Expected errorsExpected errors
Reflectance as IlluminationReflectance as Illumination
a(f) = 1 / f
= 0 = 0.4 = 0.8 = 1.2
= 1.6 = 2.0 = 4.0 = 8.0
Cues to 3D ShapeCues to 3D Shape
specularities shading texture
Conventional wisdom: different cues have different physical causes must be processed differently by visual system (‘modules’)
specularities shading texture
Goal: Find commonalities between cues.
Cues to 3D ShapeCues to 3D Shape
Cues to 3D ShapeCues to 3D Shape
Cues to 3D ShapeCues to 3D Shape
Fleming, Torralba, Adelson
Todd and colleagues
Mingolla and Grossberg
Koenderink and van Doorn
Zucker and colleagues
Zaidi and Li
Malik and Rosenholtz
Rendering withRendering withReflectance mapsReflectance maps
Reflectance map is a lookup-table that specifies image intensity for all surface normals Surface normals are indices for accessing values from the reflectance map
Within a local patch of surface, the normal changes smoothly This maps a small patch of the reflectance map “texture” into the image The rate at which the indices sweep through the reflectance map determines the warping
transformation that is applied to the texture patch during the mapping
Hierarchy of shapeHierarchy of shapeattributesattributes
We often refer to “stereo” or “texture”, or “shading” as “cues” to shape.
Traditional definition of shape cue: a physical property that can inform us about shape, e.g. “stereo”, or “texture”, or “shading”
New definition of cue: a specific image measurement that provides statistically reliable information about a specific property of the scene.
Any given cue on its own may be highly ambiguous, specifying some abstract, high level scene property that does not uniquely specify the object
Hierarchy of shapeHierarchy of shapeattributesattributes
Easily measurable image statistics that can inform us about any property of shape
It is remarkable that we can recover 3D shape:
No motion No stereo No shading No texture
image consists of nothing more than a distorted reflection of the world surrounding the object
Ideal mirrored surface
Fleming et al. (2004). JOV
Shape from SpecularitiesShape from Specularities
As the object moves from scene to scene, the image changes dramatically.
Yet, somehow we are able to recover the 3D shape.
Shape from SpecularitiesShape from Specularities
Approach IApproach I::inverse opticsinverse optics
Estimate shape by inverting the physics of mirror reflections.
Image from Savarese and Perona
Make an explicit model of the environment
Make assumptions about specific environmental features (e.g. ‘lines are straight’)
Estimate shape directly from the image Collect image measurements that are reliable
across ‘typical’ environments
Approach IIApproach II::direct perceptiondirect perception
No need to estimate the environment
ust use the pattern of distortions in the image
Pattern of compressions and rarefactions across the image indicates something about the 3D shape.
Shape from TextureShape from Texture
Real-world illumination is highly structured Specular reflections of the real world are a bit like texture Can we solve the 3D shape of mirrors using shape-from-
texture ?
Shape from TextureShape from Texture
??
Slant distorts texture but not reflections
Image distortionsImage distortions
Image distortionsImage distortions
Image distortionsImage distortions
Curvature distorts reflections but not texture
Image distortionsImage distortions
Shape-from-textureShape-from-textureandand
shape-from-specularityshape-from-specularityfollow different rulesfollow different rules
For texture, image compression depends on surface slant
first derivative of surface
For reflections, image compression depends on surface curvature properties
second derivatives of surface
Local analysis: Local analysis: banding patternsbanding patterns
Gauge Figure TaskGauge Figure Task
Subject adjusts 3D orientation of “gauge figure” to match local orientation of surface
Slant and TiltSlant and Tilt
Image from Palmer, 1999
Results IResults I
objective tilt
subje
ctiv
e t
ilt
TiltTilt
objective slant
subje
ctiv
e s
lant
SlantSlant
objective tilt
subje
ctiv
e t
ilt
objective slant
subje
ctiv
e s
lant
TiltTilt SlantSlant
Results IIResults II
objective tilt
subje
ctiv
e t
ilt
TiltTilt
objective slant
subje
ctiv
e s
lant
SlantSlant
objective tilt
subje
ctiv
e t
ilt
objective slant
subje
ctiv
e s
lant
TiltTilt SlantSlant
Is it just the occluding contour?Is it just the occluding contour?
No, it is not
Interpreting distortedInterpreting distortedreflectionsreflections
Effects of Effects of compressioncompression
3D shape appears to be conveyed by the continuously varying patterns of orientation across the image of a surface
Beyond specularityBeyond specularity
Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection
Differences betweenDifferences betweendiffuse and specular reflectiondiffuse and specular reflection
Differences betweenDifferences betweendiffuse and specular reflectiondiffuse and specular reflection
Differences betweenDifferences betweendiffuse and specular reflectiondiffuse and specular reflection
ShinyShiny
Painted Painted
Beyond specularityBeyond specularity
Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection
Latent orientationLatent orientationstructurestructure
Orientation fieldsOrientation fieldsin shadingin shading
Orientation fieldsOrientation fieldsin shadingin shading
highly curved
slightlycurved
Anisotropies in surface curvature lead to anisotropies in the image.
TextureTexture
Anisotropic compression of texture depends on surface slant
TextureTexture
Anisotropic compression of texture depends on surface slant
Orientation fieldsOrientation fieldsin texturein texture
Orientation fieldsOrientation fieldsin texturein texture
Orientation fieldsOrientation fieldsin texturein texture
No need for visual system to estimate reflectance or illumination explicitly.
Classical shape from shading uses the reflectance map to estimate surface normals from image intensities
Reflectance map is usually unknown and ambiguous
Potential of Potential of Orientation FieldsOrientation Fields
Visual system estimates surface orientation from image intensity
Classical Classical Shape from ShadingShape from Shading
reflectance mapimage
Stable across albedo discontinuities.
Breton and Zucker (1996), Huggins and Zucker (2001)
Potential of Potential of Orientation FieldsOrientation Fields
Uses biologically plausible measurements
Orientation selectivity maps in primary visual cortex of tree shrew. After Bosking et al. (1997).
Potential of Potential of Orientation FieldsOrientation Fields
May explain how images with no obvious BRDF interpretation nevertheless yield 3D percepts
Potential of Potential of Orientation FieldsOrientation Fields
Ohad Ben-Shahar
Converting between cuesConverting between cues
input imageinput image
Todd & Oomes 2004
( )2
Latent shadingLatent shading
( )2
Converting between cuesConverting between cues
input imageinput image
Todd & Oomes 2004
Latent shadingLatent shading
Matte vs. ShinyMatte vs. Shiny Same generative statistics, different mappings
Mapped Mapped as as texturetexture
Mapped as Mapped as reflectionreflection
Mapped Mapped as as texturetexture
Mapped as Mapped as reflectionreflection
Texture vs. ReflectanceTexture vs. Reflectance
Texture vs. ReflectanceTexture vs. Reflectance
Texture vs. ReflectanceTexture vs. Reflectance
Texture vs. ReflectanceTexture vs. Reflectance
ConclusionsConclusions
Orientation fields are potentially a very powerful source of information about 3D shape
For the early stages of 3D shape processing, seemingly different cues may have more in common than previously thought
Todd’s BlobsTodd’s Blobs
Todd’s BlobsTodd’s Blobs
What still needs to be explained?What still needs to be explained?
For Lambertian materials (or blurry illuminations), the reflectance map is so smooth that it is significantly anisotropic.
Therefore shading orientation fields vary considerably with changes in illumination.
sidefront top
What still needs to be explained?What still needs to be explained?
Note analogy to textures of different orientations
Todd et al. (2004)
Two possibilitiesTwo possibilities
I. Change in orientation field predicts (subtle) changes in perceived 3D shape
II. There are higher-order invariants in the orientation fields
sidefront top
Eigenvectors of Hessian matrix
Intrinsic principal curvatures
Matte dark grey
Rough metal
Glossy light grey
PlasticsPlastics
(a) Mirror (b) Smooth plastic (c) Rough plastic
When the world is anisotropicWhen the world is anisotropic
Brushed horizontally Brushed vertically
Stripy worldStripy world
Matte vs. ShinyMatte vs. Shiny Same generative statistics, different mappings
Mapped as texture
Mapped as reflection
Mapped as texture
Mapped as reflection
Hypothesis: the way the reflections are distorted is systematically related to properties of the 3D shape
Shape from SpecularitiesShape from Specularities