Correspondence and Correspondence and StereopsisStereopsis

Original notes by W. Correa. Figures from [Forsyth & Ponce] and [Trucco & Verri]Original notes by W. Correa. Figures from [Forsyth & Ponce] and [Trucco & Verri]

IntroductionIntroduction

• Disparity:Disparity:– Informally: difference between two picturesInformally: difference between two pictures– Allows us to gain a strong sense of depthAllows us to gain a strong sense of depth

• Stereopsis:Stereopsis:– Ability to perceive depth from disparityAbility to perceive depth from disparity

• Goal:Goal:– Design algorithms that mimic stereopsisDesign algorithms that mimic stereopsis

Stereo VisionStereo Vision

• Two partsTwo parts– Binocular fusion of features observed by the Binocular fusion of features observed by the

eyeseyes– Reconstruction of their three-dimensional Reconstruction of their three-dimensional

preimagepreimage

Stereo Vision – Easy CaseStereo Vision – Easy Case

• 1 single point being observed1 single point being observed– The preimage can be found at the The preimage can be found at the

intersection of the rays from the focal points intersection of the rays from the focal points to the image pointsto the image points

Stereo Vision – Hard CaseStereo Vision – Hard Case

• Many points being observedMany points being observed– Need some method to establish Need some method to establish

correspondencescorrespondences

Components of Stereo Vision Components of Stereo Vision SystemsSystems

• Camera calibrationCamera calibration: previous lecture: previous lecture• Image rectificationImage rectification: simplifies the search : simplifies the search

for correspondencesfor correspondences• CorrespondenceCorrespondence: which item in the left : which item in the left

image corresponds to which item in the image corresponds to which item in the right imageright image

• ReconstructionReconstruction: recovers 3-D information : recovers 3-D information from the 2-D correspondencesfrom the 2-D correspondences

Epipolar GeometryEpipolar Geometry

• Epipolar constraintEpipolar constraint: corresponding : corresponding points must lie on conjugate epipolar points must lie on conjugate epipolar lineslines– Search for correspondences becomes a 1-D Search for correspondences becomes a 1-D

problemproblem

Image RectificationImage Rectification

• Warp images such Warp images such that conjugate that conjugate epipolar lines epipolar lines become collinear become collinear and parallel to and parallel to uu axisaxis

Image Rectification (cont.)Image Rectification (cont.)

• Perform by rotatingPerform by rotatingthe camerasthe cameras

• NotNot equivalent to equivalent to rotating the imagesrotating the images

• The lines through the The lines through the centers become centers become parallel to each other, parallel to each other, and the epipoles and the epipoles move to infinitymove to infinity

Image Rectification (cont.)Image Rectification (cont.)

• Given extrinsic parameters T and R Given extrinsic parameters T and R (relative position and orientation of the (relative position and orientation of the two cameras)two cameras)– Rotate the left camera about the projection Rotate the left camera about the projection

center so that the the epipolar lines become center so that the the epipolar lines become parallel to the horizontal axisparallel to the horizontal axis

– Apply the same rotation to the right cameraApply the same rotation to the right camera– Rotate the right camera by RRotate the right camera by R– Adjust the scale in both camera reference Adjust the scale in both camera reference

framesframes

DisparityDisparity

• With rectified images, disparity is just With rectified images, disparity is just (horizontal) displacement of (horizontal) displacement of corresponding features in the two corresponding features in the two imagesimages– Disparity = 0 for distant pointsDisparity = 0 for distant points– Larger disparity for closer pointsLarger disparity for closer points– Depth of point proportional to 1/disparityDepth of point proportional to 1/disparity

CorrespondenceCorrespondence

• Given an element in the left image, find Given an element in the left image, find the corresponding element in the right the corresponding element in the right imageimage

• Classes of methodsClasses of methods– Correlation-basedCorrelation-based– Feature-basedFeature-based

Correlation-Based Correlation-Based CorrespondenceCorrespondence

• Input: rectified stereo pair and a point (u,v)Input: rectified stereo pair and a point (u,v)in the first imagein the first image

• Method:Method:– Form window of size (2m+1)Form window of size (2m+1)(2n+1) centered (2n+1) centered

at (u,v) and assemble points into the vector wat (u,v) and assemble points into the vector w– For each potential match (u+d,v) in the second For each potential match (u+d,v) in the second

image, compute w' and the normalized image, compute w' and the normalized correlation between w and w‘correlation between w and w‘

Sum of Squared DifferencesSum of Squared Differences

• Recall: SSD for image similarityRecall: SSD for image similarity

• Negative sign so that higher values Negative sign so that higher values mean greater similaritymean greater similarity

2)(),( vuvu

Normalized Cross-CorrelationNormalized Cross-Correlation

• Normalize to eliminate brightness Normalize to eliminate brightness sensitivity:sensitivity:

wherewhere

• Helps for non-diffuse scenes, can hurt for Helps for non-diffuse scenes, can hurt for perfectly diffuse onesperfectly diffuse ones

vu

vvuuvu

))((),(

)deviation( standard)average(

uuu

u

Correlation-Based Correlation-Based Correspondence (cont.)Correspondence (cont.)

• Main problem:Main problem:– Assumes that the observed surface is locally parallel Assumes that the observed surface is locally parallel

to the two image planesto the two image planes– If not, unequal amounts of foreshortening in imagesIf not, unequal amounts of foreshortening in images– Alleviate by computing initial disparity, warping the Alleviate by computing initial disparity, warping the

images, iteratingimages, iterating• Other problems:Other problems:

– Not robust against noiseNot robust against noise– Similar pixels may not correspond to physical Similar pixels may not correspond to physical

featuresfeatures

Feature-Based CorrespondenceFeature-Based Correspondence

• Main idea: physically-significant features Main idea: physically-significant features should be preferred to matches between should be preferred to matches between raw pixel intensitiesraw pixel intensities

• Instead of correlation-like measures, use Instead of correlation-like measures, use a measure of the distance between a measure of the distance between feature descriptorsfeature descriptors

• Typical features: points, lines, and cornersTypical features: points, lines, and corners• Example: Marr-Poggio-Grimson algorithmExample: Marr-Poggio-Grimson algorithm

Marr-Poggio-Grimson AlgorithmMarr-Poggio-Grimson Algorithm

• Convolve images with Laplacian of Convolve images with Laplacian of Gaussian filters with decreasing widthsGaussian filters with decreasing widths

• Find zero crossings of the Laplacian Find zero crossings of the Laplacian along horizontal scanlines of the filtered along horizontal scanlines of the filtered imagesimages

• For each For each , match zero crossings with , match zero crossings with same parity and similar orientations in a same parity and similar orientations in a [–[–ww....ww] disparity range, with] disparity range, with 22w

Marr-Poggio-Grimson Algorithm Marr-Poggio-Grimson Algorithm (cont.)(cont.)

• Use disparities found at larger scales to Use disparities found at larger scales to control eye vergence and cause control eye vergence and cause unmatched regions at smaller scales to unmatched regions at smaller scales to come into correspondencecome into correspondence

Marr-Poggio-Grimson Algorithm Marr-Poggio-Grimson Algorithm (cont.)(cont.)

Marr-Poggio-Grimson Algorithm Marr-Poggio-Grimson Algorithm (cont.)(cont.)

Ordering ConstraintOrdering Constraint

• Order of matching features usually the Order of matching features usually the same in both imagessame in both images

• But not always: occlusionBut not always: occlusion

Dynamic ProgrammingDynamic Programming

• Treat feature correspondence as graph Treat feature correspondence as graph problemproblem

Left imageLeft imagefeaturesfeatures

Right image featuresRight image features11 22 33 44

11

2233

44

Cost of edges =Cost of edges =similarity ofsimilarity of

regions betweenregions betweenimage featuresimage features

Dynamic ProgrammingDynamic Programming

• Find min-cost path through graphFind min-cost path through graph

Left imageLeft imagefeaturesfeatures

Right image featuresRight image features11 22 33 44

11

2233

44

ReconstructionReconstruction

• Given pair of image points p and p', and Given pair of image points p and p', and focal points O and O', find preimage Pfocal points O and O', find preimage P

• In theory: find P by intersecting the rays In theory: find P by intersecting the rays R=Op and R'=Op'R=Op and R'=Op'

• In practice: R and R' won't actually In practice: R and R' won't actually intersect due to calibration and feature intersect due to calibration and feature localization errorslocalization errors

Reconstruction ApproachesReconstruction Approaches

• GeometricGeometric– Construct the line segment perpendicular to Construct the line segment perpendicular to

R and R' that intersects both rays and take R and R' that intersects both rays and take its mid-pointits mid-point

Reconstruction ApproachesReconstruction Approaches

• Image-space: find the point P whose Image-space: find the point P whose projection onto the images minimizes projection onto the images minimizes distance to desired correspondencesdistance to desired correspondences

• Nonlinear optimizationNonlinear optimization