lecture 6 multiple-view reconstruction from scene knowledge
DESCRIPTION
Multiple-View Geometry for Image-Based Modeling. Lecture 6 Multiple-View Reconstruction from Scene Knowledge. INTRODUCTION: Scene knowledge and symmetry. Symmetry is ubiquitous in man-made or natural environments. INTRODUCTION: Scene knowledge and symmetry. Parallelism (vanishing point). - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/1.jpg)
MASKS © 2004 Invitation to 3D vision
Lecture 6 Lecture 6 Multiple-View Reconstruction Multiple-View Reconstruction
from Scene Knowledgefrom Scene Knowledge
Multiple-View Geometry for Image-Based Modeling
![Page 2: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/2.jpg)
MASKS © 2004 Invitation to 3D vision
Symmetry is ubiquitous in man-made or natural environments
INTRODUCTION: Scene knowledge and symmetry
![Page 3: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/3.jpg)
MASKS © 2004 Invitation to 3D vision
INTRODUCTION: Scene knowledge and symmetry
Parallelism (vanishing point) Orthogonality
Congruence Self-similarity
![Page 4: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/4.jpg)
MASKS © 2004 Invitation to 3D vision
INTRODUCTION: Wrong assumptions
Ames room illusion Necker’s cube illusion
![Page 5: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/5.jpg)
MASKS © 2004 Invitation to 3D vision
INTRODUCTION: Related literature
Cognitive Science: Figure “goodness”: Gestalt theorists (1950s), Garner’74, Chipman’77, Marr’82, Palmer’91’99…
Computer Vision (isotropic & homogeneous textures): Gibson’50, Witkin’81, Garding’92’93, Malik&Rosenholtz’97, Leung&Malik’97…
Mathematics: Hilbert 18th problem: Fedorov 1891, Hilbert 1901, Bieberbach 1910, George Polya 1924, Weyl 1952
Detection & Recognition (2D & 3D): Morola’89, Forsyth’91, Vetter’94, Mukherjee’95, Zabrodsky’95, Basri & Moses’96, Kanatani’97, Sun’97, Yang, Hong, Ma’02
Reconstruction (from single view): Kanade’81, Fawcett’93,Rothwell’93, Zabrodsky’95’97, van Gool et.al.’96, Carlsson’98, Svedberg and Carlsson’99, Francois and Medioni’02, Huang, Yang, Hong, Ma’02,03
![Page 6: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/6.jpg)
MASKS © 2004 Invitation to 3D vision
MUTIPLE-VIEW MULTIPLE-OBJECT ALIGNMENT• Scale alignment: adjacent objects in a single view• Scale alignment: same object in multiple views
SUMMARY: Problems and future work
ALGORITHMS & EXAMPLES• Building 3-D geometric models with symmetry • Symmetry extraction, detection, and matching • Camera calibration
SYMMETRY & MULTIPLE-VIEW GEOMETRY• Fundamental types of symmetry• Equivalent views• Symmetry based reconstruction
Multiple-View Reconstruction from Scene Knowledge
![Page 7: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/7.jpg)
MASKS © 2004 Invitation to 3D vision
SYMMETRY & MUTIPLE-VIEW GEOMETRY
• Why does an image of a symmetric object give away its structure?
• Why does an image of a symmetric object give away its pose?
• What else can we get from an image of a symmetric object?
![Page 8: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/8.jpg)
MASKS © 2004 Invitation to 3D vision
Equivalent views from rotational symmetry
90O
![Page 9: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/9.jpg)
MASKS © 2004 Invitation to 3D vision
Equivalent views from reflectional symmetry
![Page 10: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/10.jpg)
MASKS © 2004 Invitation to 3D vision
Equivalent views from translational symmetry
![Page 11: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/11.jpg)
MASKS © 2004 Invitation to 3D vision
GEOMETRY FOR SINGLE IMAGES – Symmetric Structure
Definition. A set of 3-D features S is called a symmetric structureif there exists a nontrivial subgroup G of E(3) that acts on it such that for every g in G, the map
is an (isometric) automorphism of S. We say the structure S has agroup symmetry G.
![Page 12: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/12.jpg)
MASKS © 2004 Invitation to 3D vision
GEOMETRY FOR SINGLE IMAGES – Multiple “Equivalent” Views
![Page 13: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/13.jpg)
MASKS © 2004 Invitation to 3D vision
GEOMETRY FOR SINGLE IMAGES – Symmetric Rank Condition
Solving g0 from Lyapunov equations:
with g’i and gi known.
![Page 14: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/14.jpg)
MASKS © 2004 Invitation to 3D vision
THREE TYPES OF SYMMETRY – Reflective Symmetry
Pr
![Page 15: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/15.jpg)
MASKS © 2004 Invitation to 3D vision
THREE TYPES OF SYMMETRY – Rotational Symmetry
![Page 16: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/16.jpg)
MASKS © 2004 Invitation to 3D vision
THREE TYPES OF SYMMETRY – Translatory Symmetry
![Page 17: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/17.jpg)
MASKS © 2004 Invitation to 3D vision
SINGLE-VIEW GEOMETRY WITH SYMMETRY – Ambiguities
“(a+b)-parameter” means there are an a-parameter family of ambiguity in R0 of g0 and a b-parameter family of ambiguity in T0 of g0.
P
Pr
N
Pr
![Page 18: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/18.jpg)
MASKS © 2004 Invitation to 3D vision
Symmetry-based reconstruction (reflection)
Reflectional symmetry
Virtual camera-camera
12
3 4(3)(4)
(2)(1)
![Page 19: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/19.jpg)
MASKS © 2004 Invitation to 3D vision
Epipolar constraint
Homography
12
3 4(3)(4)
(2)(1)
Symmetry-based reconstruction
![Page 20: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/20.jpg)
MASKS © 2004 Invitation to 3D vision
2 pairs of symmetric image points
Decompose or to obtain
Solve Lyapunov equation
to obtain and then .
Recover essential matrix or homography
Symmetry-based reconstruction (algorithm)
![Page 21: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/21.jpg)
MASKS © 2004 Invitation to 3D vision
Symmetry-based reconstruction (reflection)
![Page 22: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/22.jpg)
MASKS © 2004 Invitation to 3D vision
Symmetry-based reconstruction (rotation)
![Page 23: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/23.jpg)
MASKS © 2004 Invitation to 3D vision
Symmetry-based reconstruction (translation)
![Page 24: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/24.jpg)
MASKS © 2004 Invitation to 3D vision
ALIGNMENT OF MULTIPLE SYMMETRIC OBJECTS
?
![Page 25: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/25.jpg)
MASKS © 2004 Invitation to 3D vision
Pick the image of a point on the intersection line
Correct scales within a single image
![Page 26: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/26.jpg)
MASKS © 2004 Invitation to 3D vision
Correct scale within a single image
![Page 27: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/27.jpg)
MASKS © 2004 Invitation to 3D vision
Correct scales across multiple images
![Page 28: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/28.jpg)
MASKS © 2004 Invitation to 3D vision
Correct scales across multiple images
![Page 29: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/29.jpg)
MASKS © 2004 Invitation to 3D vision
Correct scales across multiple images
![Page 30: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/30.jpg)
MASKS © 2004 Invitation to 3D vision
Image alignment after scales corrected
![Page 31: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/31.jpg)
MASKS © 2004 Invitation to 3D vision
ALGORITHM: Building 3-D geometric models
1. Specify symmetric objects and correspondence
![Page 32: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/32.jpg)
MASKS © 2004 Invitation to 3D vision
Building 3-D geometric models
2. Recover camera poses and scene structure
![Page 33: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/33.jpg)
MASKS © 2004 Invitation to 3D vision
Building 3-D geometric models
3. Obtain 3-D model and rendering with images
![Page 34: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/34.jpg)
MASKS © 2004 Invitation to 3D vision
ALGORITHM: Symmetry detection and matching
Extract, detect, match symmetric objects acrossimages, and recover the camera poses.
![Page 35: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/35.jpg)
MASKS © 2004 Invitation to 3D vision
1.Color-based segmentation (mean shift)
2. Polygon fitting
Segmentation & polygon fitting
![Page 36: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/36.jpg)
MASKS © 2004 Invitation to 3D vision
Symmetry verification & recovery
3. Symmetry verification (rectangles,…)
4. Single-view recovery
![Page 37: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/37.jpg)
MASKS © 2004 Invitation to 3D vision
Symmetry-based matching
5. Find the only one set of camera poses that are consistent with all symmetry objects
![Page 38: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/38.jpg)
MASKS © 2004 Invitation to 3D vision
MATCHING OF SYMMETRY CELLS – Graph Representation
123
123
36 possible matchings
(1,2,1, g)
Cell in image 1
Cell in image 2
# of possible matching
Camera transformation
The problem of finding the largest set of matching cells is equivalent to the problem of finding the maximal complete subgraphs (cliques) in the matching graph.
![Page 39: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/39.jpg)
MASKS © 2004 Invitation to 3D vision
Camera poses and 3-D recovery
Side view Top view Generic view
Length ratio Reconstruction Ground truth
Whiteboard 1.506 1.51
Table 1.003 1.00
![Page 40: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/40.jpg)
MASKS © 2004 Invitation to 3D vision
Multiple-view matching and recovery (Ambiguities)
![Page 41: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/41.jpg)
MASKS © 2004 Invitation to 3D vision
Multiple-view matching and recovery (Ambiguities)
![Page 42: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/42.jpg)
MASKS © 2004 Invitation to 3D vision
ALGORITHM: Calibration from symmetry
(vanishing point)
Calibrated homography
Uncalibrated homography
![Page 43: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/43.jpg)
MASKS © 2004 Invitation to 3D vision
ALGORITHM: Calibration from symmetry
Calibration with a rig is also simplified: we only need to know that there are sufficient symmetries, not necessarily the 3-D coordinates of points on the rig.
![Page 44: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/44.jpg)
MASKS © 2004 Invitation to 3D vision
SUMMARY: Multiple-View Geometry + Symmetry
Multiple (perspective) images = multiple-view rank condition
Single image + symmetry = “multiple-view” rank condition
Multiple images + symmetry = rank condition + scale correction
Matching + symmetry = rank condition + scale
correction + clique identification
![Page 45: Lecture 6 Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocument.in/reader035/viewer/2022070404/56813b44550346895da42177/html5/thumbnails/45.jpg)
MASKS © 2004 Invitation to 3D vision
Multiple-view 3-D reconstruction in presence of symmetry
• Symmetry based algorithms are accurate, robust, and simple.
• Methods are baseline independent and object centered.
• Alignment and matching can and should take place in 3-D space.
• Camera self-calibration and calibration are simplified and linear.
Related applications
• Using symmetry to overcome occlusion.
• Reconstruction and rendering with non-symmetric area.
• Large scale 3-D map building of man-made environments.
SUMMARY