class 61 multi-linear systems and invariant theory in the context of computer vision and graphics...
TRANSCRIPT
Class 6 1
Multi-linear Systems and Invariant Theory
in the Context of Computer Vision and Graphics
Class 6: Duality and Shape Tensors
CS329Stanford University
Amnon Shashua
Class 6 2
Material We Will Cover Today
• The Point-View Duality Principle
• Multi-point Constraints of a single view
• Single-View Tensors
• Properties of Single View Tensors
• Single View Tensors under a stabilized reference plane
Class 6 3
Point-View Duality Principle
i
i
i
i
i
W
Z
Y
X
P points in 3D projective space
ii MPp points in 2D projective space
Approach: with enough points, we can “eliminate” M
Class 6 4
Point-View Duality Principle
iii TPMTMPp 1
We are free to choose a projective basis for the 3D space, and we fix12 degrees of freedom (out of 16):
Let us choose a representation of 3D space such that the first 4 pointshave the “standard” coordinates:
0
0
0
1
1P
0
0
1
0
2P
0
1
0
0
3P
1
0
0
0
4P
each point fixes 3 degrees of freedom
Class 6 5
Point-View Duality Principle
We can also choose a basis for the 2D (image) projective space:
0
0
0
1
0
0
1
M
0
0
1
0
0
1
0
M
0
1
0
0
1
0
0
M
1
0
0
0
1
1
1
M
Class 6 6
Point-View Duality Principle
0
0
0
1
0
0
1
M
0
0
1
0
0
1
0
M
0
1
0
0
1
0
0
M
1
0
0
0
1
1
1
M
0
01
M
0
0
2 M
0
0
3M
4M
00
00
00
M
Class 6 7
Point-View Duality Principle
00
00
00
M
Note: in the new basis, M is represented by 4 numbers (!)
Note: in the new basis we cannot assume that
1
y
x
p
(third coordinate may vanish), thus
z
y
x
p
Class 6 8
Point-View Duality Principle
ii
ii
ii
i
i
i
i
i
WZ
WY
WX
W
Z
Y
X
p
00
00
00
00
00
00
....7,6,5ifor
MPMPp iiiˆˆ
M̂
/1
/1
/1
/1
)(Mnull
Class 6 9
Single-view Multilinear Constraints
Let ', ii ll be two lines coincident with ip
0iTi pl 0' i
Ti pl
MPp iiˆˆ
0ˆˆ MPl iTi
0ˆˆ' MPl iTi
Class 6 10
8,7,6,5ifor
Single-view Multilinear Constraints
0ˆ
ˆ.
ˆ
ˆ.
ˆ
488'8
5'5
88
55
M
Pl
Pl
Pl
Pl
T
T
T
T
Every 4x4 minor vanishes !
The choice of 4 rows can include 2,3,4 points (in addition to the 4 basis points)
Class 6 11
Single-view Bilinear Constraints
Choose the 4 rows such that they include only 2 points:
0
ˆ
ˆ
ˆ
ˆ
det
6'6
66
5'5
55
Pl
Pl
Pl
Pl
T
T
T
T
Class 6 12
Single-view Bilinear Constraints
0
ˆ
ˆ
ˆ
ˆ
det
6'6
66
5'5
55
Pl
Pl
Pl
Pl
T
T
T
T
We still have 4 d.o.f left, so we can setthe coordinates of
1
1
1
1
5P
Let
W
Z
Y
X
P6
x
z
l 0
z
y
x
p
y
zl
0
'
Class 6 13
Single-view Bilinear Constraints
0
ˆ
ˆ
ˆ
ˆ
det
6'6
66
5'5
55
Pl
Pl
Pl
Pl
T
T
T
T
x
z
l 0
y
zl
0
'
1100
1010
1001
5̂P
WZ
WY
WX
P
00
00
00
6̂
0
0
0
0
0
det
6666
6666
5555
5555
WzWyZyYz
WzWxZxXz
zyyz
zxxz
Class 6 14
Single-view Bilinear Constraints
0
0
0
0
0
det
6666
6666
5555
5555
WzWyZyYz
WzWxZxXz
zyyz
zxxz
056 GppT
0
0
0
WYXYWXXY
WZXZXWXZ
WZYZYWWY
G
for all viewing positions !
65 , pp change with viewing position
Class 6 15
Single-view Bilinear Constraints
056 GppT
0
0
0
WYXYWXXY
WZXZXWXZ
WZYZYWWY
G
0332211 GGG ij
ijG 1
0)det( G
The dual fundamental matrix has additional 4 linear constraints!(compared to the fundamental matrix).
Class 6 16
056 GppT
The Dual Fundamental Matrix
5Gp is a line coincident with 6p
choose the camera projection matrix such that 06 MP
that is, the center of projection is at 6P
06 MPlT l 0ˆ6̂ MPlT
05 GppT p 05 Gp
0)det( GWhy ?
0)det( G
Since 056 GppT holds for all camera matrices,
Class 6 17
5p
5P
6P
5Gp
056 Ge56e
C
)ˆ(ˆ6556 PnullPe
(center of proj)
A “sketch” – the sketch is somewhat misleading since there are two (dual) epipoles.
Note: while C changes so doesthe image plane!
when 6PC
is the projection of
5P
5
5
5
5
66
66
66
6
6
6
6
55
55
55
/1/100
/10/10
/100/1
/1
/1
/1
/1
00
00
00
W
Z
Y
X
WZ
WY
WX
W
Z
Y
X
WZ
WY
WX
)ˆ(ˆ65 PnullP when 6)( PMnull projection of 5P
)ˆ(ˆ5665 PnullPe is the projection of 6P when 5PC
065 eGT
Class 6 18
The Dual Fundamental Matrix
0332211 GGG ij
ijG 1Why ?
Unlike the multiview tensors, the single-view point tensors have internallinear constraints (“synthetic” constraints) which are related to the factthe dual projection matrices are sparse.
Since 056 GppT holds for all camera matrices,
consider the camera matrices jM
0000
0000
0001
1M
1e
0000
0010
0000
2M
2e
0100
0000
0000
3M
3e
1000
1000
1000
4M
4e
Class 6 19
The Dual Fundamental Matrix
0332211 GGG ij
ijG 1Why ?
0jj ePM P
0
ˆ
ˆ
ˆ
ˆ
det
6'6
66
5'5
55
Pl
Pl
Pl
Pl
T
T
T
T
'66
'55 ,,, llll coincident with je
holds for all choices of 65ˆ,ˆ PP !!
0jTjGee 4,...,1j
Class 6 20
6-point Single-view Indexing
056 GppT is a function of 6 points (4 basis, 2 additional)
Each view provides one constraint for G. How many views are necessary?
G is defined up to scale and has 4 linear constraints – thus there are4 parameters to solve for.
For a linear solution: 4 views are necessary.For non-linear: 3 views are necessary (3-fold ambiguity).
Once G is recovered, the function 056 GppT is view-independent
thus provides a connection between 6 image points and their corresponding3D points which does not depend on the point of view (indexing function).
Class 6 21
7-point Single-view Tensor
7'7
77
6'6
66
5'5
55
ˆ
ˆ
ˆˆ
ˆ
ˆ
det0
Pl
Pl
PlPl
Pl
Pl
T
T
T
T
T
T
12 choices to choose 4 rows which include 7 points (4 basis +3).there are 3 groups: choose 2 rows from a singlepoint and the remaining two rows one from each remaining point.
choose these two rows
choose 1 rows from here
choose 1 rows from here
4 trilinear constraints (view independent)(per group)
Class 6 22
7-point Single-view Tensor
7'7
77
6'6
66
5'5
55
ˆ
ˆ
ˆˆ
ˆ
ˆ
det0
Pl
Pl
PlPl
Pl
Pl
T
T
T
T
T
T
0765 jk
ikji Tllp
6l is some line incident with 6p7l is some line incident with 7p
jkiT is a trilinear function of 765 ,, PPP alone.
Class 6 23
Since holds for all camera matrices,
consider the camera matrices jM
0000
0000
0001
1M
1e
0000
0010
0000
2M
2e
0100
0000
0000
3M
3e
1000
1000
1000
4M
4e
7-point Single-view Tensor
The “synthetic” constraints for the dual trilinear tensor:
0765 jk
ikji Tllp
0jj ePM P
0'5 jk
ikji Tllp where ', ll incident with jee
0' jkikj
i Tlle
4 constraints (2 choices for each line ', ll )
16 synthetic constraints (4 for each jee )
Class 6 24
7-point Single-view Tensor
The 7-point tensor is defined by 11 parameters (up to scale).
A linear solution will requires 3 views (each view 4 constraints)
How many Non-linear constraints?
The number of parameters needed to describe a 7-point configurationIs 3+3=6 (because the first 5 points can be assigned the standard coordinates).
There should be 5 non-linear constraints (1 for the overallscale, and 4 additional).
Class 6 25
7-point Single-view Tensor
The dual reprojection equation:
kjkij
i pTlp 76
5
Given point 5p and a line 6l coincident with 6p
then the contraction jkij
i Tlp 65 produces the point 7p
6 image points and the shape tensor determine the 7th point
Class 6 26
Items we Skip in Class
• 8-point dual tensor: 3x3x3x3 tensor, 58 synthetic constraints, 2 views are necessary to recover tensor (first view 12 constraint, second view 11 constraints).
• Homography duality (and slices of 7-point tensor), where do the non-linear constraints come from.
Class 6 27
Duality Under a Fixed Reference Plane
Assume some plane has been identified and “stabilized” throughout thesequence of images (i.e., points on the reference plane project to fixedPoints throughout the sequence).
The house façade is stabilized The road sign is stabilized
(movie clips due to Michal Irani)
Class 6 28
Duality Under a Fixed Reference Plane
In this case, the first 4 basis points are coplanar:
0
0
0
1
1P
0
0
1
0
2P
0
1
0
0
3P
0
1
1
1
4P
Class 6 29
0
0
0
1
0
0
1
M
0
0
1
0
0
1
0
M
0
1
0
0
1
0
0
M
0
1
1
1
1
1
1
M
0
0
1
1M
0
1
0
2M
1
0
0
3M
4M
00
00
00
M
Duality Under a Fixed Reference Plane
1
1
1
321 MMM
(unconstrained)
Class 6 30
ii
ii
ii
i
i
i
i
i
ZW
YW
XW
W
Z
Y
X
p
00
00
00
00
00
00
....7,6,5ifor
MPMPp iiiˆˆ
M̂
)(Mnull
Duality Under a Fixed Reference Plane
Class 6 31
M̂
)(Mnull
Duality Under a Fixed Reference Plane
/1
/1
/1
/1
)(Mnull
(general) (ref plane stabilized)
When M̂ varies
along a linear subspace
(say a line)
)(Mnull varies
along an algebraic surface
)(Mnull varies
along a linear subspaceof the same dimension
(easy geometric interp)
Class 6 32
0
ˆ
ˆ
ˆ
ˆ
det
6'6
66
5'5
55
Pl
Pl
Pl
Pl
T
T
T
T
1100
1010
1001
5̂P
ZW
YW
XW
P
00
00
00
6̂
6-point Tensor Under a Fixed Reference Plane
What is different?
056 GppT
Class 6 33
6-point Tensor Under a Fixed Reference Plane
Since 4321 ,,, PPPP are coplanar
0nPTi where Tn )1,0,0,0(
Since the bilinear constraint holds for all choices ofM
consider: TunM 0uMP P
056 GppT
0
ˆ
ˆ
ˆ
ˆ
det
6'6
66
5'5
55
Pl
Pl
Pl
Pl
T
T
T
T
'66
'55 ,,, llll coincident with u
holds for all choices of 65ˆ,ˆ PP !!
0GuuT
Gis skew-symmetric (only 3 parameters)
Class 6 34
5p
5P
6P
5Gp
056 Ge
e
C (center of proj)
065 eGT
TGG
6556 ee
There is only one (dual) epipole!
Simple Geometric Interpretation
Class 6 35
The Dual Image Ray
Recall: image ray in multi-view geometry is defined as follows:
0MPlT
0' MPl TP has a 1-parameter degree of freedom
defined by the intersection of the two planes
MlMl TT ',
Let ', ll be 2 lines coincident with MPp
this is the image ray
(the lines passing throughp and )(Mnull )
Class 6 36
The Dual Image Ray
In Dual:
0ˆˆ MPlT
0ˆˆ' MPl TM̂ has a 1-parameter degree of freedom
The line passing through p and )ˆ(Pnull
In general case: the camera center varies along a non-linear curve!
Class 6 37
The Dual Image Ray
In ref-plane stabilized: the camera center varies along a line,
and the line is the one joining p and P
ZW
YW
XW
lMPl TT
00
00
00ˆˆ0
Pnull )(Mnull
)(Mnull varies along the linepP
Class 6 38
Items we Skip in Class
• 8-point dual tensor: 3x3x3x3 tensor, 72 synthetic constraints, no non-linear constraints. 2 views are required (first contributes 5 constraints, the second 4 constraints).
• Number of Synthetic Constraints for 7-point tensor is 21. There no non-linear constraints. Only 2 views are required to solve.