Algebraic Vision 2015 Slide 1

(radial) multi-focal tensors concept, internal constraints and geometric insight

Marc Pollefeys

Algebraic Vision 2015 Slide 2

Backprojection

Represent point as intersection of row and column

Useful presentation for deriving and understanding multiple view geometry (notice 3D planes are linear in 2D point coordinates)

Condition for solution?

Algebraic Vision 2015 Slide 3

Multi-view geometry

(intersection constraint)

(multi-linearity of determinants)

(= epipolar constraint!) (counting argument: 11x2-15=7)

Algebraic Vision 2015 Slide 4

Epipolar geometry

(courtesy Hartley and Zisserman)!

x’TFx=0 for all x!x’ !

Algebraic Vision 2015 Slide 5

Internal constraint for fundamental matrix

Geometric interpretation of rank-2 constraint !

(courtesy Hartley and Zisserman)!

e’TFx=0, !x " e’TF=0 similarly Fe=0!

Algebraic Vision 2015 Slide 6

Multi-view geometry

(multi-linearity of determinants)

(= trifocal constraint!)

(3x3x3=27 coefficients)

Algebraic Vision 2015 Slide 7

T has 27 coefficients Q has 18DOF (i.e. 3x(4x3-1)-(4x4-1)=33-15) Q has 8 internal consistency constraints (i.e. 27-1-18)

T internal consistency constraint

Notice T31k is a point in view 3 corresponding to the projection of the intersection of reprojected (x,y)=(0,0) from view 1 and x=0 from view 2. There are 9 such points in view 3.

T31k, T322k, T333k have to be collinear (as projections of points on same 3D line), i.e. det(T11kl)=0. There are 3 such cubic constraints in view 3. Also, the above 3 lines need to intersect in epipole e13 (degree 6) (four additional constraints more complicated, see Resl PhD 2003)

Algebraic Vision 2015 Slide 8

Multi-view geometry

(multi-linearity of determinants)

(= quadrifocal constraint!)

(3x3x3x3=81 coefficients)

Algebraic Vision 2015 Slide 9

Q has 81 coefficients Q has 29 DOF (i.e. 4x(4x3-1)-(4x4-1)=44-15) Q has 51 internal consistency constraints (i.e. 81-1-29)

Q internal consistency constraint

Notice Q111l is a point in image 4 corresponding to the projection of the intersection of reprojected x=0 from image 1; x=0 from image 2; and x=0 from image 3. There are 27 such points in image 4.

Obviously Q111l, Q112l, Q113l have to be collinear (as projections of points on same 3D line), i.e. det(Q11kl)=0, same for all 9 combinations of view 1 & 2 in this view Similarly all 6 view-pairs yield 9 constraints (i.e. 1-2, 1-3, 1-4, 2-3, 2-4, 3-4) We verified that in general these 54 constraints yield 51 independent ones

Algebraic Vision 2015 Slide 10

perspective camera (2 constraints / feature)

radial camera (uncalibrated) (1 constraints / feature)

3 constraints allow to reconstruct 3D point

more constraints also tell something about cameras

multilinear constraints known as epipolar, trifocal and quadrifocal constraints

(0,0)

l=(y,-x)

(x,y)

Multiple view geometry

Algebraic Vision 2015 Slide 11

ℓ1!

ℓ4!ℓ

3!ℓ2!

ℓ5!

"1!

"2!

"3!

"4!"5!

crad! Optical Axis!

Radial 1D camera

Algebraic Vision 2015 Slide 12 12

Quadrifocal constraint

with!

Algebraic Vision 2015 Slide 13

QRAD has 16 coefficients QRAD has 13 DOF (i.e. 4x(4x2-1)-(4x4-1)=28-15) QRAD has 2 internal consistency constraints (i.e. 16-1-13) 4 optical axes have 2 lines that they all intersect with

QRAD internal consistency constraint

Pick projection of one of these lines in 3 views, then 4th line is arbitrary => requires liljlkQijk1,2=0, same for other views (compare to F.e=0) Feasible QRAD needs existence of 2 special lines => 2 internal constraints

(Thirthala and Pollefeys IJCV 2012)

12 degree polynomials characterized in (Lin and Sturmfels, 2009)!

Algebraic Vision 2015 Slide 14

•" Linearly compute radial quadrifocal tensor Qijkl from 15 pts in 4 views

•" Reconstruct 3D scene and use it for calibration

(2x2x2x2 tensor)

(x,y)

Radial quadrifocal tensor (Thirthala and Pollefeys IJCV 2012)

Algebraic Vision 2015 Slide 15 15

Synthetic quadrifocal tensor example

•" Perspective •" Fish-eye •" Spherical mirror •" Hyperbolic mirror

Algebraic Vision 2015 Slide 16 16

Perspective! Fish-eye!

Algebraic Vision 2015 Slide 17 17

Spherical mirror! Hyperbolic mirror!

Algebraic Vision 2015 Slide 18

•" Radial trifocal tensor Tijk from 7 points in 3 views

•" Reconstruct 2D panorama and use it for calibration

•" Linearly compute radial quadrifocal tensor Qijkl from 15 pts in 4 views

•" Reconstruct 3D scene and use it for calibration

(2x2x2x2 tensor)

(2x2x2 tensor)

Not easy for real data, hard to avoid degenerate cases (e.g. 3 optical axes intersect in single point). However, degenerate case leads to simpler 3 view

algorithm for pure rotation

(x,y)

Radial quadrifocal tensor

Algebraic Vision 2015 Slide 19

•" Radial trifocal tensor Tijk from 7 points in 3 views

•" Reconstruct 2D panorama and use it for calibration (2x2x2 tensor)

(x,y)

Radial trifocal tensor

Internal consistency constraints? TRAD has 8 coefficients TRAD has 7 DOF (i.e. 3x(3x2-1)-(3x3-1)=15-8) TRAD has NO internal consistency constraints (i.e. 8-1-7) All set of 8 numbers represent valid radial trifocal tensors

Same as (Quan and Kanade‘97)

Algebraic Vision 2015 Slide 20 20

•" Two-step linear approach to compute radial distortion

•" Estimates distortion polynomial of arbitrary degree

undistorted image

estimated distortion (4-8 coefficients)

Dealing with Wide FOV cameras (Thirthala and Pollefeys, ICCV’05/IJCV‘12)

Algebraic Vision 2015 Slide 21 21

unfolded cubemap estimated distortion

(4-8 coefficients)

Dealing with Wide FOV cameras

•" Two-step linear approach to compute radial distortion

•" Estimates distortion polynomial of arbitrary degree

(Thirthala and Pollefeys, ICCV’05/IJCV‘12)

Algebraic Vision 2015 Slide 22 22

Non-parametric distortion calibration

•" Models fish-eye lenses, cata-dioptric systems, etc.

Algebraic Vision 2015

(Thirthala and Pollefeys, ICCV’05/IJCV‘12)

normalized radius

angl

e

Algebraic Vision 2015 Slide 23 23 Algebraic Vision 2015

normalized radius

angl

e

90o

Non-parametric distortion calibration

•" Models fish-eye lenses, cata-dioptric systems, etc.

(Thirthala and Pollefeys, ICCV’05/IJCV‘12)

Algebraic Vision 2015 Slide 24

Non-Parametric Structure-Based Calibration of Radially Symmetric Cameras

24!

Camposeco et al. ICCV 2015!

Algebraic Vision 2015 Slide 25

Minimal relative pose with know vertical

25

-g

Fraundorfer, Tanskanen and Pollefeys, ECCV2010!

5 linear unknowns # linear 5 point algorithm 3 unknowns # quartic 3 point algorithm

Vertical direction can often be estimated!•" inertial sensor!•" vanishing point!

Algebraic Vision 2015 Slide 26

Multi-camera systems

•" Light rays do not meet at a single center of projection. xj!

x2!26!

Algebraic Vision 2015 Slide 27

Multi-camera systems

•" 6-vector Plücker line to represent the light rays. xj!

uij!

tCi!V : Reference frame!

uij : Unit direction of ray!Tci : Translation from Ci to V!

27!

Algebraic Vision 2015 Slide 28

Generalized Epipolar Constraint

•" Generalized Epipolar constraint:

•" E = [R]xt : conventional Essential matrix.

•" [R, t] is the relative transformation.

•" Absolute scale can be obtained!

Using many cameras as one #Robert Pless!IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2003 !

28!

Algebraic Vision 2015 Slide 29

Generalized Epipolar Constraint

•"Problems:

–" Linear solution requires 17-point correspondences. ! ~600k RANSAC loops needed.

–" Minimal problem needs 6-point correspondences but gives 64 solutions.

29!

Algebraic Vision 2015 Slide 30

Motion Estimation: Ackermann Constraint

•" Enforce Ackermann motion constraint:

•" 2 degree-of-freedom

! 2-point minimal problem.

17 RANSAC Loops!!

Motion Estimation for a Self-Driving Car with a Generalized Camera##Gim Hee Lee, Friedrich Fraundorfer, and Marc Pollefeys$#IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2013$!

30!

Algebraic Vision 2015 Slide 31

Motion Estimation: Ackermann Constraint

•" Generalized Essential matrix with Ackermann constraint:

•" Solve for " and # from the Epipolar constraint.

! 2 polynomial equations with 2 unknowns.

02

cossincos

02

cossincos

2222

1111

=+++

=+++

ecba

ecba

!"!!

!"!!

31!

Algebraic Vision 2015 Slide 32

•" Can be solved in closed-form with .

•" Up to 6 real solutions.

Motion Estimation: Ackermann Constraint

32!

Algebraic Vision 2015 Slide 33

Motion estimation with generalized camera

33!

Algebraic Vision 2015 Slide 34

Structure from Sound

sound source location si !microphone location mj!dij!

measure time difference of arrival (TDOA) at microphones tij !

dij =v.(tij - ti )!

time of emission ti !

We want to solve for:!

similar problem formulation in sensor networks, range-only SLAM, ultra-wide band localization, … !

Algebraic Vision 2015 Slide 35

Sound and microphone factorization

with!

rank 5 matrix!

(Pollefeys and Nister, ICASSP’08)

rank 5 matrix (which contains [1 1… 1]T )!

Given time-difference-of-arrival , !compute position of microphones!and time of emission and position of sound sources ,!

Algebraic Vision 2015 Slide 36

Subspace intersection problem

36!

(from Teller and Hohmeier 99)!

Find subspaces that intersect a set of subspaces!e.g. given 4 lines in 3D, find lines that intersect them!

e.g. sound factorization: given 5 lines and 1 point in 10D, !! !find the 5D subspace that intersects all of them!

(more details in Roland Angst PhD)!

Algebraic Vision 2015 Slide 37

Conclusion

•" Polynomial constraints in computer vision often have intuitive geometric interpretation

•" Multi-view geometry of radial cameras allow to handle much more general camera models

•" Additional constraints can greatly simplify the problem

•" Also interesting geometric problems with sound (and vision)

•" Subspace intersection problem

Thank you!

37!