midterm shape-from-x (i.e., reconstruction) · cse 152 lecture 13 cse152, spr 04 intro computer...
TRANSCRIPT
1
CSE152, Spr 04 Intro Computer Vision
Stereo Vision II
Introduction to Computer VisionCSE 152
Lecture 13
CSE152, Spr 04 Intro Computer Vision
Announcements
• Assignment 3: Due today.– Extended to 5:00PM, sharp. Turn in hardcopy to
my office 3101 AP&M• No Discussion section this week.• Guest lecturer, Dr. Jeff Ho, next week.• Today
– Midterm summary– Shape from X– Stereo Vision I
CSE152, Spr 04 Intro Computer Vision
Midterm• High: 87• Low: 31• Mean: 57• Median: 55
Problem regrades: Must be requested today in class and in writing.
CSE152, Spr 04 Intro Computer Vision
Shape-from-X(i.e., Reconstruction)
• Methods for estimating 3-D shape from image data. X can be one (or more) of many cues.– Stereo (two or more views, known viewpoints)– Motion (moving camera or object)– Shading– Changing lighting (Photometric Stereo)– Texture variation– Focus/blur– Prior knowledge/context– structured light/lasers
CSE152, Spr 04 Intro Computer Vision
Example: Helmholtz StereoDepth + Normals + BRDF
CSE152, Spr 04 Intro Computer Vision
Stereo
2
CSE152, Spr 04 Intro Computer Vision
Binocular Stereopsis: MarsGiven two images of a scene where relative locations of cameras are known, estimate depth of all common scene points.
Two images of Mars
CSE152, Spr 04 Intro Computer Vision
An Application: Mobile Robot Navigation
The Stanford Cart,H. Moravec, 1979.
The INRIA Mobile Robot, 1990.
Courtesy O. Faugeras and H. Moravec.
CSE152, Spr 04 Intro Computer Vision
Commercial Stereo Heads
Trinocular stereoTrinocular stereo Binocular stereoBinocular stereo
CSE152, Spr 04 Intro Computer Vision
Stereo can work well
CSE152, Spr 04 Intro Computer Vision
Need for correspondence
Truco Fig. 7.5
CSE152, Spr 04 Intro Computer Vision
Triangulation
Nalwa Fig. 7.2
3
CSE152, Spr 04 Intro Computer Vision
Stereo Vision Outline• Offline: Calibrate cameras & determine
“epipolar geometry”• Online
1. Acquire stereo images2. Rectify images to convenient epipolar geometry3. Establish correspondence 4. Estimate depthA
B
C
D
CSE152, Spr 04 Intro Computer Vision
BINOCULAR STEREO SYSTEMEstimating Depth
Z
X(0,0) (d,0)
Z=fXL XR
DISPARITY(XL - XR)
Z = (f/XL) XZ= (f/XR) (X-d)
(f/XL) X = (f/XR) (X-d)X = (XLd) / (XL - XR)
Z = d f(XL - XR)
X = d XL(XL - XR)
(Adapted from Hager)
CSE152, Spr 04 Intro Computer Vision
Reconstruction: General 3-D case
• Linear Method: find P such that
• Non-Linear Method: find Q minimizingCSE152, Spr 04 Intro Computer Vision
Two Approaches• A) From each image, process “monocular”
image to obtain cues.B) Establish correspondence between
cues.• Directly compare image regions between
the two images.
CSE152, Spr 04 Intro Computer Vision
Human Stereopsis: Binocular Fusion
How are the correspondences established?
Julesz (1971): Is the mechanism for binocular fusiona monocular process or a binocular one??• There is anecdotal evidence for the latter (camouflage).
• Random dot stereograms provide an objective answer BP!CSE152, Spr 04 Intro Computer Vision
Random Dot Stereograms
4
CSE152, Spr 04 Intro Computer Vision
Random Dot Stereograms
CSE152, Spr 04 Intro Computer Vision
A Cooperative Model (Marr and Poggio, 1976)
CSE152, Spr 04 Intro Computer Vision
Epipolar Constraint
• Potential matches for p have to lie on the corresponding epipolar line l’.
• Potential matches for p’ have to lie on the corresponding epipolar line l.
CSE152, Spr 04 Intro Computer Vision
Epipolar Geometry
• Epipolar Plane
• Epipoles
• Epipolar Lines
• Baseline
CSE152, Spr 04 Intro Computer Vision
Family of epipolar Planes(standard approach)
CSE152, Spr 04 Intro Computer Vision
JEFF, HERE’s WHERE I ENDED• Jeff, I covered up to this point.• I suggest that you
– review the pictorial part of the epipolar geometry, • discuss the essentnial matrix, showing how it can
be computed from R&T, and that this can come from calibration.
• Discuss rectification in qualitative way, showing that result is epipolar lines become parallel lines.
• Then discuss matching, mostly SSD & SAD metric.
• The mention issue with half occluded regions.• Perhaps mention some challenges, dynamic
programming, etc.
5
CSE152, Spr 04 Intro Computer Vision
Family of epipolar Planes(standard approach)
CSE152, Spr 04 Intro Computer Vision
Epipolar Constraint: Calibrated Case
Essential Matrix(Longuet-Higgins, 1981)
CSE152, Spr 04 Intro Computer Vision
Properties of the Essential Matrix
• E p’ is the epipolar line associated with p’.
• ETp is the epipolar line associated with p.
• E e’=0 and ETe=0.
• E is singular.
• E has two equal non-zero singular values(Huang and Faugeras, 1989).
T
T
CSE152, Spr 04 Intro Computer Vision
Calibration
Determine intrinsic parameters and extrinsic relation of two cameras
CSE152, Spr 04 Intro Computer Vision
The Eight-Point Algorithm (Longuet-Higgins, 1981)
|F | =1.
Minimize:
under the constraint2
Set F33 to 1
CSE152, Spr 04 Intro Computer Vision
Epipolar geometry example
6
CSE152, Spr 04 Intro Computer Vision
Example: converging cameras
courtesy of Andrew Zisserman CSE152, Spr 04 Intro Computer Vision
Example: motion parallel with image plane
(simple for stereo → rectification)courtesy of Andrew Zisserman
CSE152, Spr 04 Intro Computer Vision
Example: forward motion
e
e’
courtesy of Andrew Zisserman CSE152, Spr 04 Intro Computer Vision
RectificationGiven a pair of images, transform both images so that epipolar lines are scan lines.
CSE152, Spr 04 Intro Computer Vision
Rectification
All epipolar lines are parallel in the rectified image plane.CSE152, Spr 04 Intro Computer Vision
Image pair rectification
simplify stereo matching by warping the images
Apply projective transformation so that epipolar linescorrespond to horizontal scanlines
e
e
map epipole e to (1,0,0)try to minimize image distortion
He001
=
Note that rectified images usually not rectangular
7
CSE152, Spr 04 Intro Computer Vision
RectificationGiven a pair of images, transform both images so that epipolar lines are scan lines.
Input Images
CSE152, Spr 04 Intro Computer Vision
RectificationGiven a pair of images, transform both images so that epipolar lines are scan lines.
Rectified ImagesSee Section 7.3.7 for specific method
CSE152, Spr 04 Intro Computer Vision
Features on same epipolar line
Truco Fig. 7.5
CSE152, Spr 04 Intro Computer Vision
Mobi: Stereo-based navigation
CSE152, Spr 04 Intro Computer Vision
Epipolar correspondence
This version is feature-based: detect edges in 1-D signal, and use dynanic progrmaming toe find correspondences that minimize an error function.
CSE152, Spr 04 Intro Computer Vision
A challenge: Multiple Interpretations
Each feature on left epipolar line match oneand only one feature on right epipolar line.
8
CSE152, Spr 04 Intro Computer Vision
Multiple Interpretations
Each feature on left epipolar line match oneand only one feature on right epipolar line.
CSE152, Spr 04 Intro Computer Vision
Multiple Interpretations
Each feature on left epipolar line match oneand only one feature on right epipolar line.
CSE152, Spr 04 Intro Computer Vision
Multiple Interpretations
Each feature on left epipolar line match oneand only one feature on right epipolar line.
CSE152, Spr 04 Intro Computer Vision
Dense Correspondence: A Photometric constraint
• Same world point has same intensity in both images (Constant Brightness Constraint)– Lambertian fronto-parallel– Issues:
• Noise• Specularity• Foreshortening
CSE152, Spr 04 Intro Computer Vision
Using epipolar & constant Brightness constraints for stereo matching
For each epipolar lineFor each pixel in the left image
• compare with every pixel on same epipolar line in right image
• pick pixel with minimum match cost• This will never work, so:
Improvement: match windows
(Seitz)CSE152, Spr 04 Intro Computer Vision
Comparing Windows: ==??
ff gg
MostMostpopularpopular
(Camps)
For each window, match to closest window on epipolar line in other image.
9
CSE152, Spr 04 Intro Computer Vision
Match Metric Summary
( )( ) ( )( )
( )( ) ( )( )∑ ∑
∑
−+⋅−
−+⋅−
vu vu
vu
IvduIIvuI
IvduIIvuI
, ,
222
211
,2211
,,
,,
( ) ( )( )∑ +−vu
vduIvuI,
221 ,,
( )( )( )( )
( )( )( )( )
∑∑∑
−+
−+−
−
−
vu
vuvu
IvduI
IvduI
IvuI
IvuI,
2
,
222
22
,
211
11
,
,
,
,
( ) ( )∑ +−vu
vduIvuI,
21 ,,
( ) ( )( )∑ +−vu
vduIvuI,
'2
'1 ,,
( ) ( ) ( )∑ <=nm
kkk vuInmIvuI,
' ,,,
( ) ( )( )∑ +vu
vduIvuIHAMMING,
'2
'1 ,,,
( ) ( ) ( )( )vuInmIBITSTRINGvuI kknmk ,,, ,' <=
MATCH METRIC DEFINITION
Normalized Cross-Correlation (NCC)
Sum of Squared Differences (SSD)
Normalized SSD
Sum of Absolute Differences (SAD)
Zero Mean SAD
Rank
Census
These two are actually
the same
( ) ( )∑ −+−−vu
IvduIIvuI,
_
22
_
11 ),(),(
CSE152, Spr 04 Intro Computer Vision
Correspondence Search Algorithm (simple version for Cross Correlation)
For i = 1:nrowsfor j=1:ncols
best(i,j) = -1for k = mindisparity:maxdisparity
c = CC(I1(i,j),I2(i,j+k),winsize)if (c > best(i,j))
best(i,j) = cdisparities(i,j) = k
endend
endend
O(nrows * ncols * disparities * winx * winy)
I1 I2
uv
d
I1 I2
uv
d
CSE152, Spr 04 Intro Computer Vision
Stereo results
Ground truthScene
– Data from University of Tsukuba
(Seitz)CSE152, Spr 04 Intro Computer Vision
Results with window correlation
Window-based matching(best window size)
Ground truth
(Seitz)
CSE152, Spr 04 Intro Computer Vision
Results with better method
State of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.
Ground truth
(Seitz)CSE152, Spr 04 Intro Computer Vision
Window size
W = 3 W = 20
Better results with adaptive window• T. Kanade and M. Okutomi, A Stereo Matching
Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.
• D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998
• Effect of window size
(Seitz)
10
CSE152, Spr 04 Intro Computer Vision
Stereo ConstraintsCONSTRAINT BRIEF DESCRIPTION
1-D Epipolar Search Arbitrary images of the same scene may be rectified based on epipolar geometry such that stereo matches lie along one-dimensional scanlines. This reduces the computational complexity and also reduces the likelihood of false matches.
Monotonic Ordering Points along an epipolar scanline appear in the same order in both stereo images, assuming that all objects in the scene are approximately the same distance from the cameras.
Image Brightness Constancy
Assuming Lambertian surfaces, the brightness of corresponding points in stereo images are the same.
Match Uniqueness For every point in one stereo image, there is at most one corresponding point in the other image.
Disparity Continuity Disparities vary smoothly (i.e. disparity gradient is small) over most of the image. This assumption is violated at object boundaries.
Disparity Limit The search space may be reduced significantly by limiting the disparity range, reducing both computational complexity and the likelihood of false matches.
Fronto-Parallel Surfaces
The implicit assumption made by area-based matching is that objects have fronto-parallel surfaces (i.e. depth is constant within the region of local support). This assumption is violated by sloping and creased surfaces.
Feature Similarity Corresponding features must be similar (e.g. edges must have roughly the same length and orientation).
Structural Grouping Corresponding feature groupings and their connectivity must be consistent.
(From G. Hager) CSE152, Spr 04 Intro Computer Vision
Problem of Occlusion
CSE152, Spr 04 Intro Computer Vision
Stereo Matching using Dynamic Programming
Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and MachineIntelligence, 7(2):139-154 (1985). 1985 IEEE.
(Ohta and Kanade, 1985)
CSE152, Spr 04 Intro Computer Vision
Stereo matching
Optimal path(dynamic programming )
Similarity measure(SSD or NCC)
Constraints• epipolar• ordering• uniqueness• disparity limit• disparity gradient limit
Trade-off• Matching cost (data)• Discontinuities (prior)
(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)(From Pollefeys)
CSE152, Spr 04 Intro Computer Vision
Some Challenges & Problems• Photometric issues:
– specularities– strongly non-Lambertian BRDF’s
• Surface structure– lack of texture– repeating texture within horopter bracket
• Geometric ambiguities– as surfaces turn away, difficult to get accurate reconstruction
(affine approximate can help)– at the occluding contour, likelihood of good match but incorrect
reconstruction
CSE152, Spr 04 Intro Computer Vision
Variations on Binocular Stereo1. Trinocular Stereopsis2. Helmholtz Reciprocity Stereopsis
11
CSE152, Spr 04 Intro Computer Vision
Trinocular Epipolar Constraints
These constraints are not independent!
CSE152, Spr 04 Intro Computer Vision
More on stereo …
The Middleburry Stereo Vision Research Pagehttp://cat.middlebury.edu/stereo/
Recommended reading
D. Scharstein and R. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms. IJCV 47(1/2/3):7-42, April-June 2002. PDF file (1.15 MB) - includes current evaluation. Microsoft Research Technical Report MSR-TR-2001-81, November 2001.
Myron Z. Brown, Darius Burschka, and Gregory D. Hager. Advances in Computational Stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(8):993-1008, 2003.