bundle adjustment - · bob alice x 1 p bob p alice u 'p 'x black: given variables red:...
TRANSCRIPT
Bundle Adjustment
v
Bob Alice
X
bobP aliceP
u
pp
X
1
2
geom
2
2
3
2
3
u
P X P X
P X X
u
P
E
x y
Black: given variables Red: unknowns
Camera & Point Jacobian
v
Bob Alice
X
bobP aliceP
u
pp
X
1
2
geom
2
2
3
2
3
u
P X P X
P X X
u
P
E
x y
Black: given variables Red: unknowns
( )
p,X
u
v
w
f w
Camera & Point Jacobian
v
Bob Alice
X
bobP aliceP
u
pp
X
1
2
geom
2
2
3
2
3
u
P X P X
P X X
u
P
E
x y
Black: given variables Red: unknowns
( )
p,X
u
v
w
f w 2
2
( )
p pp,X
p p
p p
u ww u
u
w w
v u wv v
w
w
f
2
2
( )
X Xp,X
X X
X X
u ww u
u
w w
v u wv v
ww
f
Camera & Point Jacobian
v
Bob Alice
X
bobP aliceP
u
pp
X
1
2
geom
2
2
3
2
3
u
P X P X
P X X
u
P
E
x y
Black: given variables Red: unknowns
( )
p,X
u
v
w
f w 2
2
( )
p pp,X
p p
p p
u ww u
u
w w
v u wv v
w
w
f
Camera & Point Jacobian
v
Bob Alice
X
bobP aliceP
u
pp
X
1
2
geom
2
2
3
2
3
u
P X P X
P X X
u
P
E
x y
Black: given variables Red: unknowns
( )
p,X
u
v
w
f w 2
2
( )
p pp,X
p p
p p
u ww u
u
w w
v u wv v
w
w
f
2
2
( )
X Xp,X
X X
X X
u ww u
u
w w
v u wv v
ww
f
Camera & Point Jacobian
v
Bob Alice
1X
bobP aliceP
u
pp
X
Black: given variables Red: unknowns
Camera & Point Jacobian ( ) ( )
p X
p ,X p ,XJ J J
p Xj i j i
ij ij ij
j i
f f
J1,bobpJ 2 70 1,bobXJ
1,aliceXJ1,alicepJ2 70
v
Bob Alice
1X
bobP aliceP
u
pp
X
Black: given variables Red: unknowns
Camera & Point Jacobian ( ) ( )
p X
p ,X p ,XJ J J
p Xj i j i
ij ij ij
j i
f f
J1,bobpJ 2 70 1,bobXJ
1,aliceXJ1,alicepJ2 70
2X
2,bobpJ 2 70 2,bobXJ
2,aliceXJ2,alicepJ2 70
2 30
2 30
2 30
2 30
J =
Cam 1 Pt 1 Pt 4
# of unknowns: 3x7+4x3 # of projections: 3x4
Cam 2 Cam 3 Pt 2 Pt 3
Pt 1
Pt 2
Pt 3
Pt 4
Camera 1
Camera 2 Camera 3
J =
Cam 1 Pt 1 Pt 4
# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)
Cam 2 Cam 3 Pt 2 Pt 3
Pt 1
Pt 2
Pt 3
Pt 4
Camera 1
Camera 2 Camera 3
J =
Cam 1 Pt 1 Pt 4
# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)
J JTsize of : 24x24
x J J J b f(x)-1T T
Cam 2 Cam 3 Pt 2 Pt 3
Pt 1
Pt 2
Pt 3
Pt 4
Camera 1
Camera 2 Camera 3
J =
Cam 1 Pt 1 Pt 4
# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)
J JTsize of : 24x24
x J J J b f(x)-1T T
Main computational bottle neck
Cam 2 Cam 3 Pt 2 Pt 3
Pt 1
Pt 2
Pt 3
Pt 4
Camera 1
Camera 2 Camera 3
Camera trajectory
3D point
# of cameras: 3009 # of 3D points: 1,298,317
# of unknowns: 7x3009+3x1,298,317
J JTsize of : 6,167,690x6,167,690
J =
J = pJXJ
Camera 3D point
p XJ J J
J J =T
p XJ J JT T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
p XJ J JT T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
p XJ J JT T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T T T
T T T- ( )
p p p X p p
X p X X X X
J J J J J epb X
XJ J J J J ef
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T T T
T T T- ( )
p p p X p p
X p X X X X
J J J J J epb X
XJ J J J J ef
T T
T T
p p p X p
X p X X X
J J I J J ep
XJ J J J I eor
T
p
X
eA B p
B D X eor
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
T T T
T T T- ( )
p p p X p p
X p X X X X
J J J J J epb X
XJ J J J J ef
Normal equation:
T T
T T
p p p X p
X p X X X
J J I J J ep
XJ J J J I eor
T
p
X
eA B p
B D X eor
-1 -1
T
- -
p
X
eA B pI BD I BD
B D X0 I 0 I e
1
d
D
dM
-1
1
-1
-1
d
D
dM
Inversion of block diagonal matrix can be efficiently computed.
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T
p
X
eA B p
B D X e1
d
D
dM
-1
1
-1
-1
d
D
dM
Inversion of block diagonal matrix can be efficiently computed.
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T
p
X
eA B p
B D X e1
d
D
dM
-1
1
-1
-1
d
D
dM
Inversion of block diagonal matrix can be efficiently computed.
-1 -1
T
- -
p
X
eA B pI BD I BD
B D X0 I 0 I e
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T
p
X
eA B p
B D X e1
d
D
dM
-1
1
-1
-1
d
D
dM
Inversion of block diagonal matrix can be efficiently computed.
-1 -1
T
- -
p
X
eA B pI BD I BD
B D X0 I 0 I e
-1-1 T
T
--
p X
X
e BD epA BD B 0
XB D e
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T
p
X
eA B p
B D X e
-1 -1
T
- -
p
X
eA B pI BD I BD
B D X0 I 0 I e
-1-1 T
T
--
p X
X
e BD epA BD B 0
XB D e
-1-1 T -1
-1 T
- -
-
p X
X
p A BD B e BD e
X D e B p
-1 T-A BD BNote: is Schur complement of D
1
d
D
dM
-1
1
-1
-1
d
D
dM
Inversion of block diagonal matrix can be efficiently computed.
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T
p
X
eA B p
B D X e
-1 -1
T
- -
p
X
eA B pI BD I BD
B D X0 I 0 I e
-1-1 T
T
--
p X
X
e BD epA BD B 0
XB D e
-1-1 T -1
-1 T
- -
-
p X
X
p A BD B e BD e
X D e B p
-1 T-A BD BNote: is Schur complement of D
1
d
D
dM
-1
1
-1
-1
d
D
dM
Inversion of block diagonal matrix can be efficiently computed.
Small size matrix