combining photometric and geometric constraints
DESCRIPTION
Combining Photometric and Geometric Constraints. Yael Moses IDC, Herzliya. Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion. Problem 1:. Recover the 3D shape of a general smooth surface from a set of calibrated images. Problem 2:. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/1.jpg)
Y. Moses 1
Combining Photometric and Geometric
Constraints
Yael Moses
IDC, Herzliya
Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion
![Page 2: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/2.jpg)
Y. Moses 2
Recover the 3D shape of a general smooth surface from a set of calibrated images
Problem 1:
![Page 3: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/3.jpg)
Y. Moses 3
Problem 2:
Recover the 3D shape of a smooth bilaterally symmetric object from a single image.
![Page 4: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/4.jpg)
Y. Moses 4
Shape Recovery
Geometry: Stereo Photometry:
Shape from shading Photometric stereo
Main problems: Calibrations and Correspondence
![Page 5: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/5.jpg)
Y. Moses 5
3D Shape Recovery
Photometry: Shape from
shading Photometric
stereo
Geometry:Stereo
Structure from motion
![Page 6: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/6.jpg)
Y. Moses 6
Geometric Stereo
2 different images Known camera parameters Known correspondence
+ +
![Page 7: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/7.jpg)
Y. Moses 7
Photometric Stereo
3D shape recovery: surface normals from two or more
images taken from the same viewpoint
![Page 8: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/8.jpg)
Y. Moses 8
Photometric Stereo
Solution:
)ˆ(
3
2
1
3
2
1
n
l
l
l
I
I
I
IT
T
T
LnI
IL
ILn
1
1
nlyxI
nlyxI
nlyxI
ˆ),(
ˆ),(
ˆ),(
33
22
11
Three images Matrix notation
IL
ILn
1
![Page 9: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/9.jpg)
Y. Moses 9
Photometric Stereo
3D shape recovery (surface normals) Two or more images taken from the
same viewpoint
Main Limitation:
Correspondence is obtained by a fixed viewpoint
IL
ILn
1
![Page 10: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/10.jpg)
Y. Moses 10
Overview
Combining photometric and geometric stereo: Symmetric surface, single image Non symmetric: 3 images
Mono-Geometric stereo Mono-Photometric stereo Experimental results.
![Page 11: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/11.jpg)
Y. Moses 11
The input Smooth featureless surface Taken under different viewpoints Illuminated by different light sources
The Problem: Recover the 3D shape from a set of calibrated images
![Page 12: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/12.jpg)
Y. Moses 12
Assumptions
Given correspondence the normals can be computed (e.g., Lambertian, distant point light source …)
*
*
ln
nlI ˆ
n
Three or more images
Perspective projection
![Page 13: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/13.jpg)
Y. Moses 13
Our method
Combines photometric and geometric stereo
We make use of:
Given Correspondence:
Can compute a normal
Can compute the 3D point
![Page 14: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/14.jpg)
Y. Moses 14
IL
ILn
1
1
ˆ
Basic Method
GivenCorrespondence
![Page 15: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/15.jpg)
Y. Moses 15
First Order Surface Approximation
![Page 16: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/16.jpg)
Y. Moses 16
First Order Surface Approximation
![Page 17: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/17.jpg)
Y. Moses 17
First Order Surface Approximation
P() = (1 - )O1 + P,
N)O - (P
N)O - (PT
1
T1
N (P() - P) = 0
![Page 18: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/18.jpg)
Y. Moses 18
First Order Surface Approximation
![Page 19: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/19.jpg)
Y. Moses 19
PMp ii
New Correspondence
![Page 20: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/20.jpg)
Y. Moses 20
IL
ILn
1
1
ˆ
New Surface Approximation
![Page 21: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/21.jpg)
Y. Moses 21
Dense Correspondence
![Page 22: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/22.jpg)
Y. Moses 22
Basic Propagation
![Page 23: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/23.jpg)
Y. Moses 23
Basic Propagation
![Page 24: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/24.jpg)
Y. Moses 24
Basic method: First Order
Given correspondence pi and L
P and n Given P and n T Given P, T and Mi
a new correspondence qi
![Page 25: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/25.jpg)
Y. Moses 25
Extensions
Using more than three images Propagation:
Using multi-neighbours Smart propagation
Second error approximation Error correction:
Based on local continuity Other assumptions on the surface
IL
ILn
1
![Page 26: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/26.jpg)
Y. Moses 26
Multi-neighbors Propagation
![Page 27: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/27.jpg)
Y. Moses 27
Smart Propagation
![Page 28: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/28.jpg)
Y. Moses 28
Second Order: a Sphere
P()
N+N
N
N
P
(P-P())(N+N)=0
![Page 29: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/29.jpg)
Y. Moses 29
Second Order Approximation
![Page 30: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/30.jpg)
Y. Moses 30
Second Order Approximation
![Page 31: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/31.jpg)
Y. Moses 31
Using more than three images
Reduce noise of the photometric stereo
Avoid shadowed pixels Detect “bad pixels”
Noise Shadows Violation of assumptions on the surface
![Page 32: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/32.jpg)
Y. Moses 32
Smart Propagation
![Page 33: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/33.jpg)
Y. Moses 33
Error correction
The compatibility of the local 3D shape can be used to correct errors of:
Correspondence Camera parameters Illumination parameters
![Page 34: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/34.jpg)
Y. Moses 34
Score
Continuity: Shape Normals Albedo
The consistency of 3D points locations and the computed normals: General case: full triangulation Local constraints
![Page 35: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/35.jpg)
Y. Moses 35
Extensions
Using more than three images Propagation:
Using multi-neighbours Smart propagation
Second error approximation Error correction:
Based on local continuity Other assumptions on the surface
![Page 36: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/36.jpg)
Y. Moses 36
Real Images
Camera calibration Light calibration
Direction Intensity Ambient
![Page 37: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/37.jpg)
Y. Moses 37
Error correction + multi-neighbor5 Images
![Page 38: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/38.jpg)
Y. Moses 38
5pp
3pp 3nn
5nn5pn
![Page 39: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/39.jpg)
Y. Moses 39
![Page 40: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/40.jpg)
Y. Moses 40
![Page 41: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/41.jpg)
Y. Moses 41
![Page 42: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/42.jpg)
Y. Moses 42
![Page 43: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/43.jpg)
Y. Moses 43
![Page 44: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/44.jpg)
Y. Moses 44
Detected Correspondence
![Page 45: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/45.jpg)
Y. Moses 45
Error correction + multi-neighbordMulti-neighborsBasic scheme (3 images)Error correction no multi-neighbors
![Page 46: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/46.jpg)
Y. Moses 46
New ImagesSynthetic Images
![Page 47: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/47.jpg)
Y. Moses 47
Sec a
Ground truthBasic schemeMulti-neighbors Error correction
![Page 48: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/48.jpg)
Y. Moses 48
Sec b
Ground truthBasic schemeMulti-neighbors Error correction
![Page 49: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/49.jpg)
Y. Moses 49
Sec c
Ground truthBasic schemeMulti-neighbors Error correction
![Page 50: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/50.jpg)
Y. Moses 50
Ground truthBasic schemeMulti-neighbors
approx.Error correction
Sec d
Ground truthBasic schemeMulti-neighbors Error correction
![Page 51: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/51.jpg)
Y. Moses 51
Combining Photometry and
Geometry
Yields a dense correspondence and dense shape recovery of the object
in a single path
![Page 52: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/52.jpg)
Y. Moses 52
Assumptions
Bilaterally Symmetric object Lambertian surface with constant
albedo Orthographic projection Neither occlusions nor
shadows Known “epipolar geometry”
![Page 53: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/53.jpg)
Y. Moses 53
Geometric Stereo
2 different images Known camera parameters Known viewpoints Known correspondence
3D shape recovery
![Page 54: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/54.jpg)
Y. Moses 54
Computing the Depth from Disparity
pl
pr
P
ql
qrZ
Z
Orthographic
Projection
![Page 55: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/55.jpg)
Y. Moses 55
Symmetry and Geometric Stereo
Non frontal view of a symmetric object
Two different images of the same object
![Page 56: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/56.jpg)
Y. Moses 56
Symmetry and Geometric Stereo
Non frontal view of a symmetric object
Two different images of the same object
![Page 57: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/57.jpg)
Y. Moses 57
Geometry
Weak perspective projection:
)(~ tsRpprojp
Around X Around Z Around Y
zyx RRRR
xR zR yRI I
![Page 58: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/58.jpg)
Y. Moses 58
Geometry
Projection of Ry:
010
)sin(0)cos()(' yRprojR
Around Y
is the only pose parameter
PRp '~
Image point
Object point
![Page 59: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/59.jpg)
Y. Moses 59
objectx
z
image
),,( zyxP l Tr zyxP ),,(
)~,~(~ rrr yxp )~,~(~ lll yxp
Correspondence
Assume YxZ is the symmetry plane.
x~
![Page 60: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/60.jpg)
Y. Moses 60
Mono-Geometric Stereo
3D reconstruction: given correspondence and ,
unknown
known
PRp '~
x~
z
image
x object
)sin()cos(~)sin()cos(~
~~
zxx
zxx
yyy
l
r
lr
xx
rx~lx~
![Page 61: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/61.jpg)
Y. Moses 61
Viewpoint Invariant
Given the correspondence and unknown
2
1
22
11
~~
~~
x
x
pp
pplr
lr
*1
~p
*2~p
*3~p 3*
~p2*
~p1*
~p
lr pp 11~~
)sin()cos(~)sin()cos(~
~~
zxx
zxx
yyy
l
r
lr
Invariant
![Page 62: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/62.jpg)
Y. Moses 62
Photometric Stereo
2 images Lambertian reflectance Known illuminations Known correspondence
(same viewpoint)
3D shape recovery
![Page 63: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/63.jpg)
Y. Moses 63
Symmetry and Photometric Stereo
Non-frontal illumination of a symmetric object
Two different images of the same object
![Page 64: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/64.jpg)
Y. Moses 64
Notation: Photometry
Corresponding object points:
Illumination:
Tyx
r zzn )1,,( Tyx
l zzn )1,,( Tzyx
r nnnn ),,(ˆ Tzyx
l nnnn ),,(ˆ
Tzyx eeeE ),,(
Tzyx
r nnnn ),,(ˆ Tzyx
l nnnn ),,(ˆ
![Page 65: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/65.jpg)
Y. Moses 65
Mono-Photometric Stereo
3D reconstruction given correspondence and E (up to a twofold ambiguity):
unknown
known
zzyyxxr neneneI
zzyyxxl neneneI
)(2
2
zzyylr
xxlr
neneIII
neIII
![Page 66: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/66.jpg)
Y. Moses 66
Invariance to Illumination
Given correspondence and E unknown
Invariant:x
x
n
n
I
I
2
1
2
1
)(2
2
zzyylr
xxlr
neneIII
neIII
![Page 67: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/67.jpg)
Y. Moses 67
Mono-Photometric Stereo
3D reconstruction E unknown but correspondence is given
Frontal viewpoint with non-frontal illumination. Use image first derivatives.
![Page 68: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/68.jpg)
Y. Moses 68
Mono-Photometric Stereo Using image derivatives
3 global unknowns: E For each pair:
5 unknowns zx zy zxx zxy zyy
6 equations 3 pairs are sufficient
![Page 69: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/69.jpg)
Y. Moses 69
Mono-Photometric StereoUnknown Illumination
![Page 70: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/70.jpg)
Y. Moses 70
Correspondence
No correspondence => no stereo. Hard to define correspondence in
images of smooth surfaces. Almost any correspondence is legal
when: Only geometric constraints are
considered. Only photometric constraints are
considered.
![Page 71: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/71.jpg)
Y. Moses 71
Combining Photometry and
Geometry Yields a dense correspondence (dense shape recovery of the
object).
Enables recovering of the global parameters.
![Page 72: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/72.jpg)
Y. Moses 72
Self-Correspondence A self-correspondence function:
lr ppC ~)~(
*~ lp
rp~*lrr xyxC ~)~,~(
![Page 73: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/73.jpg)
Y. Moses 73
Dense Correspondence using Propagation
Assume correspondence between a pair of points, p0
l and p0r.
rr yxCydCxdpCpdpC ~~
~~)~()~~(
lp2~
rp0*~
rp1*~
rp2~
*1~ lp
*0~ lp
)~,~(~ ydxdpd
![Page 74: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/74.jpg)
Y. Moses 74
Dense Correspondence using Propagation
* ***
**
lp2~
rp0*~
rp1*~
rp2~
*1~ lp
*0~ lp
![Page 75: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/75.jpg)
Y. Moses 75
x
x~
z
image
objectn̂
dxdx
dz
)sin()cos(~ dzdxxd )cos(~ dxxd
xdxcxdxc ~)~()~~(
dx
dz
xCxdxCxdxC ~~)~()~~(
![Page 76: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/76.jpg)
Y. Moses 76
Tyx zzn )1,,(
First derivatives of the Correspondence
Assume known Assume known E
)sin()cos(
)2sin(
;)sin()cos(
)sin()cos(
~
~
x
y
y
x
xx
z
zC
z
zC
r
r
![Page 77: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/77.jpg)
Y. Moses 77
)sin()cos(~)sin()cos(~
~~
zxxzxx
yyy
l
r
lr
Computing and
ryy
ryxy
rxy
rxxx
yCxCC
yCxCC
rr
rr
~~
~~
~~
~~
rr yxCC ~~
Object coordinates:Given computing and is trivial
Moving from object to image coordinates depends on the viewing parameter
rr yxCCn̂
![Page 78: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/78.jpg)
Y. Moses 78
Derivatives with respect to the object coordinates:
Derivatives with respect to the image coordinates:
![Page 79: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/79.jpg)
Y. Moses 79
x
x~
z
image
object
xd~
yx cc ~~
E
)~,~(~ rrr yxp
)~,~(~ lll yxp
![Page 80: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/80.jpg)
Y. Moses 80
Given a corresponding pair and E n=(zx,zy,-1)T Given and n
cx and cy
Given cx and cy
a new corresponding pair
General Idea
![Page 81: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/81.jpg)
Y. Moses 81
Results on Real Images: Given global parameters
![Page 82: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/82.jpg)
Y. Moses 82
Finding Global Parameters
Assume E and are unknown. Assume a pair of corresponding
points is given. Two possibilities:
Search for E and directly.
Compute E and from the image second derivatives.
![Page 83: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/83.jpg)
Y. Moses 83
All roads lead to Rome …
Find and verify correct correspondence
Recover global parameters, E and
Integration Constraint:Circular Tour
![Page 84: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/84.jpg)
Y. Moses 84
Finding Global Parameters
Consider image second derivatives Due to foreshortening effect:
and
We can relate image and object derivatives by
xll
xxrr
xIIII rr ~~
ry
r
y
ry
r
x
ry
rx
r
y
rx
r
x
rx
yIxII
yIxII
rr
rr
~~
~~
~~
~~
![Page 85: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/85.jpg)
Y. Moses 85
For each corresponding pair:
and
Plus 4 linear equations in 3 unknown.
Where
Testing E and : Image second derivatives
),,,(34
EzzfA yxx
yy
xy
xx
l
y
r
y
l
x
r
x
zzz
A
I
III
r
r
r
r
~
~
~
~
rrll nEInEI ˆˆ
![Page 86: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/86.jpg)
Y. Moses 86
Counting
5 unknowns for each pair: zx zy,zxx zxy zyy
4 global unknowns: E, For each pair: 6 equations. For n pairs: 5n+4 unknowns
6n equations. 4 pairs are sufficient
![Page 87: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/87.jpg)
Y. Moses 87
Results on Simulated Data
Ground Truth Recovered Shape
![Page 88: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/88.jpg)
Y. Moses 88
Recovering the Global Parameters
![Page 89: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/89.jpg)
Y. Moses 89
Degenerate Case
Close to frontal view: problems with geometric-stereo.
reconstruction problem Close to frontal illumination:
problems with photometric-stereo. correspondence problem
![Page 90: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/90.jpg)
Y. Moses 90
Future work
Perspective photometric stereo Use as a first approximation to global
optimization methods Test on other reflection models Recovering of the global parameters:
Light Cameras
Detect the first pair of correspondence
![Page 91: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/91.jpg)
Y. Moses 91
Future Work
Extend to general 3 images under 3 viewpoints and 3 illuminations.
Extend to non-lambertian surfaces.
Y. Moses 4
Assumptions
Bilaterally Symmetric object Lambertian surface with constant
albedo Orthographic projection Neither occlusions nor shadows Known “epipolar geometry”
![Page 92: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/92.jpg)
Y. Moses 92
Thanks
![Page 93: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/93.jpg)
Y. Moses 93
x
x~
z
image
object
)cos(~
dx
xd
n̂
dx
dz
dx
dz
![Page 94: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/94.jpg)
Y. Moses 94
Integration Constraint
ry
r
y
ry
r
x
ry
rx
r
y
rx
r
x
rx
yIxII
yIxII
rr
rr
~~
~~
~~
~~
l
xylx
l
y
r
xyrx
r
y
x
lxl
x
x
rxr
x
lr
rr
r
r
IznEI
IznEI
z
nEI
z
nEI
~~
~~
~
~
)sin(ˆ
)sin(ˆ
)sin()cos(
ˆ
)sin()cos(
ˆ
![Page 95: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/95.jpg)
Y. Moses 95
Integration Constraint
)(2
2
zzyy
xx
neneI
neI
l
xylx
l
y
r
xyrx
r
y
x
lxl
x
x
rxr
x
lr
rr
r
r
IznEI
IznEI
z
nEI
z
nEI
~~
~~
~
~
)sin(ˆ
)sin(ˆ
)sin()cos(
ˆ
)sin()cos(
ˆ
![Page 96: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/96.jpg)
Y. Moses 96
Searching for E
Illumination must satisfy:
E is further constrained by the image second derivatives.
),max( lr IIE
)(2 zzyy neneI
xxneI 2
![Page 97: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/97.jpg)
Y. Moses 97
Image second derivatives:
xll
x
xrr
x
yrl
y
yrr
y
II
II
II
II
r
r
r
r
~
~
~
~
lx
l
x
rx
r
x
ly
l
y
ry
r
y
nEI
nEI
nEI
nEI
r
r
r
r
ˆ
ˆ
ˆ
ˆ
~
~
~
~
),,,,(ˆ yyxyxxyxrx zzzzzn
yy
xy
xx
l
y
r
y
l
x
r
x
zzz
A
I
III
r
r
r
r
~
~
~
~ll nEI ˆrr nEI ˆ
Where ),,(34
EzzfA yxx
4 linear equations in 3 unknown
![Page 98: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/98.jpg)
Y. Moses 98
For each corresponding pair and E: 4 linear equations in 3 unknown.
Where
Image second derivatives
),,(34
EzzfA yxx
yy
xy
xx
l
y
r
y
l
x
r
x
zzz
A
I
III
r
r
r
r
~
~
~
~
zzyyxxr neneneI
zzyyxxl neneneI
![Page 99: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/99.jpg)
Y. Moses 99
Counting
5 unknowns for each pair: zx,zy,zxx,zxy,zyy
3 global unknowns: E For each pair: 6 equations. For n pairs: 5n+3 unknowns
6n equations.3 pairs are sufficient
![Page 100: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/100.jpg)
Y. Moses 100
Correspondence
Tr zyxP ),,( Tl zyxP ),,(
)~,~(~ rrr yxp )~,~(~ lll yxp
![Page 101: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/101.jpg)
Y. Moses 101
Variations
Known/unknown distant light source Known/unknown viewpoint Symmetric/non-symmetric image
Frontal/non-frontal viewpoint Frontal/non-frontal illumination
![Page 102: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/102.jpg)
Y. Moses 102
Correspondence
Epipolar geometry is the only geometric constraint on the correspondence.
Weak photometric constraint on the correspondence.
)(2
2
zzyy
xx
neneI
neI
![Page 103: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/103.jpg)
Y. Moses 103
Lambertian Surface
Basic radiometric
)cos(ˆ),(Re EnEqpf I =
E
*
*P
EE
n2
n1
n̂
![Page 104: Combining Photometric and Geometric Constraints](https://reader035.vdocument.in/reader035/viewer/2022062221/56812b93550346895d8fb2a2/html5/thumbnails/104.jpg)
Y. Moses 104
E
Photometric Stereo
First proposed by Woodham, 1980. Assume that we have two images ..
nEInEI ˆˆ 2211