computer vision: algorithms and applications · introduction what is stereo vision? the word...
TRANSCRIPT
![Page 1: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/1.jpg)
Stereo Correspondence
Reference: R. Szeliski. Computer Vision: Algorithms and Applications. 2010. 1.
Jing Luo | Megvii Tech Talk | Apr 2018
Computer Vision:Algorithms and Applications
![Page 2: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/2.jpg)
1.Introduction to Stereo Vision
![Page 3: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/3.jpg)
Introduction
▣ What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid shape
▣ Stereo matching Take two or more images and estimate a 3D model of the scene by
finding matching pixels in the images and converting their 2D positions into 3D depths.
▣ Application Photogrammetric matching of aerial images Modeling of the human visual system Robotic navigation and manipulation View interpolation and image-based rendering 3D model building
![Page 4: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/4.jpg)
Introduction
![Page 5: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/5.jpg)
Introduction
![Page 6: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/6.jpg)
2.Epipolar Geometry
![Page 7: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/7.jpg)
Two-frame structure from motion
▣ 3D rotation Also known as 3D rigid body motion or the 3D Euclidean transformation,
it can be written as𝑥′ = 𝑅𝑥 + 𝑡 or 𝑥′ = 𝑅 𝑡 ҧ𝑥
𝑅 is a 3 × 3 orthonormal rotation matrix with 𝑅𝑅𝑇 = 𝐼 and 𝑅 = 1.
▣ Epipolar geometry𝑑1 ො𝑥1 = 𝑝1 = 𝑅𝑝0 + 𝑡 = 𝑅(𝑑0 ො𝑥0) + 𝑡
𝑑1 𝑡 × ො𝑥1= 𝑑0 𝑡 ×𝑅ො𝑥0
𝑑0 ො𝑥1T 𝑡 ×𝑅 ො𝑥0= 𝑑1 ො𝑥1
T 𝑡 × ො𝑥1= 0
▣ Epipolar constraintො𝑥1
TEො𝑥0 = 0, where E= 𝑡 ×𝑅 is the essential matrix.
![Page 8: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/8.jpg)
Two-frame structure from motion
▣ Another perspective: Epipolars: 𝑒0 𝑒1 Epipolar plane: 𝑐0 𝑐1 and 𝑝 define a plane Epipolar line: Intersections of epipolar
plane with the image planes Epipolar constraint: Corresponding
points on conjugate epipolar lines
ො𝑥0 in image 0 𝑙1 in image 1𝑙1 = 𝐸 ො𝑥0
![Page 9: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/9.jpg)
Two-frame structure from motion
▣ ො𝑥0 in image 0 𝑙1 in image 1𝑙1 = 𝐸 ො𝑥0
![Page 10: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/10.jpg)
Two-frame structure from motion
![Page 11: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/11.jpg)
Two-frame structure from motion
![Page 12: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/12.jpg)
Two-frame structure from motion
▣ How to calculate essential matrix?
Method 1: SVD with more than eight equations Method 2: make use of the condition that E is rank-deficient
![Page 13: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/13.jpg)
Rectification
▣ Rectifying (i.e, warping) the input images so that corresponding horizontal scanlines are epipolar lines
![Page 14: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/14.jpg)
Rectification
▣ After rectification:
![Page 15: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/15.jpg)
Plane sweep
▣ Sweeping a set of planes through a scene:
![Page 16: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/16.jpg)
3.Sparse Correspondence
![Page 17: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/17.jpg)
3D curves and profiles
▣ Surface reconstruction from occluding contours
![Page 18: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/18.jpg)
4.Dense Correspondence
![Page 19: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/19.jpg)
Dense correspondence algorithms
▣ 4 steps: 1. matching cost computation; 2. cost (support) aggregation; 3. disparity computation and optimization; 4. disparity refinement.
▣ Local algorithm use a matching cost that is based on a support region
▣ Global algorithm make explicit smoothness assumptions and then solve a global
optimization problem
![Page 20: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/20.jpg)
Similarity measures
▣ Sum-of-squared difference technique SSD is the template matching method done by finding the lowest
difference value between input and template. The differences are squared in order to remove the sign.
▣ Other methods Normalized correlation coefficients Mutual information Normalized gradient field
![Page 21: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/21.jpg)
Local methods
▣ Local and window-based methods aggregate the matching cost by summing or averaging over a support region. support region can be either two-dimensional at a fixed disparity
(favoring fronto-parallel surfaces), or three-dimensional in x-y-d space (supporting slanted surfaces).
▣ Aggregation with a fixed support region can be performed using 2D or 3D convolution.
![Page 22: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/22.jpg)
Local methods
![Page 23: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/23.jpg)
Local methods
![Page 24: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/24.jpg)
Global optimization
▣ Many global methods are formulated in an energy-minimization framework. the objective is to find a solution d that minimizes a global energy
![Page 25: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/25.jpg)
Global optimization
▣ Simulated annealing ▣ Max-flow / Graph cut
![Page 26: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/26.jpg)
Global optimization
▣ Dynamic programming
![Page 27: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/27.jpg)
Global optimization
▣ Segmentation-based techniques
![Page 28: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/28.jpg)
Global optimization
▣ Z-keying and background replacement
![Page 29: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/29.jpg)
5.Multi-view stereo
![Page 30: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/30.jpg)
Epipolar plane
▣ Epipolar plane image
![Page 31: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/31.jpg)
3D reconstruction
▣ Volumetric and 3D surface reconstruction
![Page 32: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/32.jpg)
3D reconstruction
▣ Shape from silhouettes
![Page 33: Computer Vision: Algorithms and Applications · Introduction What is stereo vision? The word “stereo” comes from the Greek for “solid” Stereo vision: how we perceive solid](https://reader034.vdocument.in/reader034/viewer/2022042303/5ece2e79ee11c142a623dff9/html5/thumbnails/33.jpg)
3D reconstruction
▣ Shape from silhouettes