binocular stereo #1
DESCRIPTION
Binocular Stereo #1. Topics. 1. Principle 2. binocular stereo basic equation 3. epipolar line 4. features and strategies for matching. single image is ambiguous. A. a”. a’. another image taken from a different direction gives the unique 3D point. Binocular stereo. Base line. - PowerPoint PPT PresentationTRANSCRIPT
Binocular Stereo #1
Topics
1. Principle2. binocular stereo basic equation3. epipolar line4. features and strategies for matching
Binocular stereo
single image is ambiguous
A
another image taken from a different direction gives the unique 3D point
a’a”
Epipolar line
Epipolar plane
Epipolar line constraints
Corresponding points lie on the Epipolar lines
Epipolar line constratints
Base line
One image pointPossible line of sight
Epipolar geometry (multiple points)
C1
C2
e1e2
Epipoles:• intersections of baseline with image planes• projection of the optical center in another image• the vanishing points of camera motion direction
Examples of epipolar geometry
Examples of epipolar geometry
Examples of epipolar geometry
Characteristics of epipolar line
•rectification
Basic binocular stereo equation
A physical point
focal length
right image point
z
left image point
base line length
right image planeleft image plane
World coordinate systemleft image centerright image center
Camera Model
Pinhole camera
Camera Model
geometry
(X, Y, Z)
Image plane
X
Y
-Z
xy
(x, y)
f : focal length
Z
Yf
Z
Xfyx ,),(
Perspective projection
View point
(Optical center) (sX, sY, sZ)
Basic binocular stereo equation
z=-2df/(x”-x’)x”-x’: disparity2d : base line length
x” x’
-z
fd d
z
d + x
)("
"
dxz
fx
f
x
z
xd
d - x
)('
'
dxz
fx
f
x
z
xd
dz
fdxdx
z
fxx 2)('"
Classic algorithms for binocular Stereo
Marr-PoggioMarr-Poggio-GrimsonNishihara-Poggio
Lucas-KanadeOhta-KanadeMatthie-KanadeOkutomi-Kanade
BakerHannahMoravec
Barnard-Thompson
MIT group
CMU group
Stanford group
Features for matching
a. brightness
b. edges
c. edge intervals
d. interest points
10 11 1210 11 12
10 11 1210 11 1211 15 16
a. relaxation
b. coarse to fine
c. dynamic programming
local optimam local optimam
Strategies for matching
global optimam
),(),()(),,( 32321211321 xxfxxfxfxxxf
10 10 1010 5 1010 10 10
10 10 1010 5 1010 10 10
10 10 1010 10 1010 10 10
Main purpose of development
simulate human stereosimulate human stereo
map makingmap makingmap makingmap making
map makingnavigationnavigation
navigation
Marr-PoggioMarr-Poggio-GrimsonNishihara-Poggio
Lucas-KanadeOhta-KanadeMatthie-KanadeOkutomi-Kanade
BakerHannahMoravec
Barnard-Thompson
Features for matching
points(random dots)edgesintervals
brightness(gradient)intervalsbrightnessbrightness
edgesinterest pointsinterest points
interest points
Marr-PoggioMarr-Poggio-GrimsonNishihara-Poggio
Lucas-KanadeOhta-KanadeMatthie-KanadeOkutomi-Kanade
BakerHannahMoravec
Barnard-Thompson
Strategies for matching
relaxationcoarse to finecoarse to fine
relaxationdynamic programmingRelaxation (Kalman filter)relaxation
dynamic programmingcoarse to finecoarse to fine
relaxation
Marr-PoggioMarr-Poggio-GrimsonNishihara-Poggio
Lucas-KanadeOhta-KanadeMatthie-KanadeOkutomi-Kanade
BakerHannahMoravec
Barnard-Thompson
Summary
1.binocular stereo takes two images of 3D point from two different positions and determines its 3D coordinate system.2. Epipolar line
2D matching ↓1D matching
3. Features for matching---brightness,edges,edge interval,interest point
4. Strategies for matching---relaxation,coarse to fine,dynamic programming
5. ReadB&B pp.88-93Horn pp.299-303
Binocular Stereo #2
Topics
case studyarea-based stereoMarr-poggio stereosimulate human visual systemOhta-Kanade stereoaerial image analysisMoravec stereonavigation
Classification of stereo method
1. Features for matchinga. brightness valueb. pointc. edged. region2. Strategies for matchinga. brute-force (not a strategy ???)b. coarse-to-finec. relaxationd. dynamic programming3. Constraints for matchinga. epipolar linesb. disparity limitc. continuityd. uniqueness
Area-based stereo
1. method
b c
b
c
2. problema. trade-off of window size and resolutionb. dull peak
b c
1. Features for matchinga. brightness valueb. pointc. edged. region2. Strategies for matchinga. brute-force (not a strategy ???)b. coarse-to-finec. relaxationd. dynamic programming3. Constraints for matchinga. epipolar linesb. disparity limitc. continuityd. uniqueness
Area-based stereo
Marr-Poggio Stereo(`76)
Simulating human visual system(random dot stereo gram)
Marr,Poggio “Coopertive computation of stereo disparity” Science 194,283-287
Input : random dot stereo
left image
random dot
shift the catch pat
right image
we can see the height different between the central and peripheral area
Constraints– Epipolar line constraint
– Uniqueness constraint» each point in a image has only one depth value
O.K. No.
– Continuity constraint» each point is almost sure to have a depth value near the values o
f neighbors
O.K. No.
Uniqueness constraint prohibits two or more matching points on one horizontal or vertical lines
continuity constraint attracts more matching on a diagonal line
ABC
D E F
D E F
A
B
C
A
B
C
(E-A)
(E-B)
(E-C)
prohibit
attract
attract
(D-A)
(E-B)
(F-C)Same depth
n n+1
relaxation
10 10 1010 5 1010 10 10
10 10 1010 5 1010 10 10
10 10 1010 10 1010 10 10
),( jicn
)1,( jicn
),(1 jicn
)1,( jicn
),1( jicn
),1( jicn
Pr
''''1
''''
),(),(),(ji
nExji
nn jicjicjic
),(0 jic ),(1 jic ),(1 jicn
1. Features for matchinga. brightness valueb. pointc. edged. region2. Strategies for matchinga. brute-force (not a strategy ???)b. coarse-to-finec. relaxationd. dynamic programming3. Constraints for matchinga. epipolar linesb. disparity limitc. continuityd. uniquenesssimulate the human visual system (MIT)
Marr-Poggio Stereo(`76)
Ohta-Kanade Stereo(`85)
Map making
Ohta,Kanade “Stereo by intra- and inter-scanline search using dynamic programming” ,IEEE Trans.,Vol. PAMI-7,No.2,pp.139-14
now matching become 1D to 1D
yet, N line * ML * MR (512 * 100 * 100 * 10 m sec = 15 hours)
L1L2L3L4L5L6
R1R2R3R4R5R6
L
R
disparity
Path Search
Matching problem can be considered as a path search problem
define a cost at each candidate of path segment based some ad-hoc function
10 100 100
Dynamic programming
We can formalize the path finding problem as the following iterative formula
optimum cost to K
cost between M and K
)();(min)(}{
kDkMdMDk
)1()1;0(),2()2;0(),3()3;0(min)0( DdDdDdD
3 0
2 1
Optimum costs are known
stereo pair
edges
path disparity
depth
stereo pair
edges
depth
1. Features for matchinga. brightness valueb. pointc. edged. region2. Strategies for matchinga. brute-force (not a strategy ???)b. coarse-to-finec. relaxationd. dynamic programming3. Constraints for matchinga. epipolar linesb. disparity limitc. continuityd. uniquenessaerial image analysis (CMU)
Ohta-Kanade Stereo(`85)
Brightness of interval
Moravec Stereo(`79)
navigation
Moravec “Visual mapping by a robot rover” Proc 6th IJCAI,pp.598-600 (1979)
Moravec’s cart
Slide stereo
Motion stereo
Slider stereo (9 eyes stereo)
9C2 = 36 stereo pairs!!! each stereo has an uncertainty measure uncertainty = 1 / base-line
each stereo has a confidence measure
22
2
ba
ab
long base line
large uncertainty
Coarse to fine
expand
expand
matching
matching
matching
σ
estimated distance
σ:uncertainty measure
area:confidence measure
9C2 = 36 curves
Interest point
1. Features for matchinga. brightness valueb. pointc. edged. region2. Strategies for matchinga. brute-force (not a strategy ???)b. coarse-to-finec. relaxationd. dynamic programming3. Constraints for matchinga. epipolar linesb. disparity limitc. continuityd. uniquenessnavigation (Stanford)
Moravec Stereo(`81)
interest point
Summary
1. Two images from two different positions give depth information
2. Epipolar line and plane
3. Basic equationZ=-2df/(x”-x’)x”-x’: disparity 2d : base line length
4. case studyarea-based stereoMarr-poggio stereo simulate human visual systemOhta-Kanade stereo aerial image analysisMoravec stereo navigation
5. Read Horn pp.299-303
F matrix
Camera Model
Pinhole camera
Camera Model
geometry
(X, Y, Z)
Image plane
X
Y
-Z
xy
(x, y)
f : focal length
Z
Yf
Z
Xfyx ,),(
Perspective projection
View point
(Optical center) (sX, sY, sZ)
Camera Model
Z
Yf
Z
Xfyx ,),( Perspective projection
RsZ
Y
X
f
f
y
x
s
10100
000
000
1
formularization
Perspective projection
(Non-linear)
Affine projection
(Linear)
Projection matrix
Affine Camera Models
General formularization
11000
0010
0001
1Z
Y
X
y
x
s•Orthographic
10100
000
000
1Z
Y
X
f
f
y
x
s•Perspective
•Affine camera
10001 34
24232221
14131211
Z
Y
X
a
aaaa
aaaa
y
x
s
Affine Cameras
perspective orthographic
Focal length
Distance from camera
Intrinsic parameters
Image plane : an ideal image
CCD : an actual picture
Not equal !
CCD elements
Intrinsic parameters
yAn ideal image on the Image plane
x
u
v
θ An actual picture
u0
v0
(x, y)
(u, v)
1100
sin0
cot
10
0
y
x
vk
ukk
v
u
v
uu
Intrinsic parameters
1100
sin0
cot
10
0
y
x
vk
ukk
v
u
s v
uu
e.g. perspective projection
10100
000
000
100
sin0
cot
0
0
Z
Y
X
f
f
vk
ukk
v
uu
XPAZ
Y
X
vfk
ufkfk
v
uu
10100
0010
0001
100
sin0
cot
0
0
Intrinsic matrix
Projection matrix (normalized)
T
Extrinsic parameters
Y
X
Z
P
),,( zyxp
i
j
k
Extrinsic parameters
T
Y
X
Z
P
),,( zyxp
i
j
k
TkzjyixP
TixPi tt
ti
Extrinsic parameters
TkzjyixP
TkzPk
TjyPj
TixPi
tt
tt
tt
T
k
j
i
P
k
j
i
z
y
x
t
t
t
t
t
t
Extrinsic parameters
T
k
j
i
P
k
j
i
z
y
x
t
t
t
t
t
t
tPR
TRPRp
R : rotation matrix t : translation vector
Summary (intrinsic & extrinsic parameters)
Y
X
Z (X,Y,Z)
World coordinate
R, t
(u, v)
picture
)(
1
tPRApAv
u
s
Camera coordinate
World coordinate
Summary (intrinsic & extrinsic parameters)
Y
X
Z (X,Y,Z)
World coordinate
R, t
(u, v)
picture
11
Z
Y
X
tRAt
Z
Y
X
RAv
u
s
3 × 4 matrix MtRAms~~
Epipolar geometry
C1
C2
t
xx
xRt
R
p
tt
tt
tt
ptpt
0
0
0
12
13
23 xRt
0 xRtx t
Essential matrix : E
Essential & Fundamental matrix
x x
0 xEx t Image planes (ideal)
Pictures (actual)
m
m
xAm
0)(
)()(1
21
1
111
mAEAm
mAEmA
xEx
tt
t
t
Fundamental matrix : F
Image 1 Image 2
F matrix
m
m
(u, v, 1) (u’, v’, 1)
0 mFmt
F & (u, v) known
0 mFmt 0 cvbua
Calculate the epipolar line
picture 1 picture 2
0..0,for
eFeimFmm t
2 picture in the epipole theis e
1 picture in the epipole theis ,0 satisfies, similaly,
eFe t
Computing F matrix (Linear solution)
0 mFmt
0
1
1
333231
232221
131211
v
u
fff
fff
fff
vu
01
33
32
31
23
22
21
13
12
11
f
f
f
f
f
f
f
f
f
vuvvvvuuuvuu
819)(33
matrix F of freedom) of (degree D.O.F
scaleambiguity
Corner detector
Extract interest points in each images
x
y
2
22
2
2
2
y
I
yx
Iyx
I
x
I
C
04.0
)(det 2
k
traceCkCR
Harris corner detector
Matching
),( jiI ),( jiI
)()(
),(
II
IICovCorr
or
2),(),( jiIjiId
Computing F matrix (Linear solution)
0
0
0
0
0
0
0
0
1
1
33
32
31
23
22
21
13
12
11
888888888888
111111111111
f
f
f
f
f
f
f
f
f
vuvvvvuuuvuu
vuvvvvuuuvuu
Suppose we found 8 pairs of corresponding points ·····
12
33
2
12
2
11 fff
Computing F matrix (Singularity constraint)
Epipolar pencil by linear solution (due to noise and error)
Computing F matrix (Singularity constraint)
Singular value decomposition (SVD)
321
3
2
1
00
00
00
VUF
2rank F Without noise, σ3 must be 0
modification
VUF
000
00
00
2
1
Computing F matrix (Singularity constraint)
VUF
3
2
1
00
00
00
VUF
000
00
00
2
1
Summary Pinhole camera and Affine camera
Intrinsic and extrinsic camera parameter
Epipolar geometry
Fundamental matrix