lecture 09: stereo algorithms - penn state college of ...rtc12/cse486/lecture09.pdf · cse486, penn...
TRANSCRIPT
![Page 1: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/1.jpg)
CSE486, Penn StateRobert Collins
Lecture 09: Stereo Algorithms
![Page 2: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/2.jpg)
CSE486, Penn StateRobert Collins
Recall: Simple Stereo System
Camps, PSU
X
Y Z
located atlocated at(0,0,0)(0,0,0)
leftcamera
zz
xx
yy
(X,Y,Z)
Image coords of point (X,Y,Z) Left Camera:
TTxx
rightcameralocated atlocated at(T(Txx,0,0),0,0)
zz
xx
yy
Right Camera:
( , )
( , )
![Page 3: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/3.jpg)
CSE486, Penn StateRobert Collins
Recall: Stereo Disparity
Left camera
Right camera
Stereo Disparity
Important equation!
baseline
disparity
depth
![Page 4: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/4.jpg)
CSE486, Penn StateRobert Collins
Recall: Stereo Disparity
Important equation!
disparity
depth
Left camera
Right camera
Note: Depth andstereo disparity areinversely proportional
![Page 5: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/5.jpg)
CSE486, Penn StateRobert Collins
Stereo Example
Left Image Right Image
From Middlebury stereo evaluation pagehttp://www.middlebury.edu/stereo/
![Page 6: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/6.jpg)
CSE486, Penn StateRobert Collins
Stereo Example
Disparity values (0-64)Left image
Right image
Note how disparity is larger(brighter) for closer surfaces.
![Page 7: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/7.jpg)
CSE486, Penn StateRobert Collins
Computing Disparity
• Correspondence Problem:
– Determining which pixel in the right image
corresponds to each pixel in the left image.
– Disp = x_coord(left) - x_coord(right)
Camps, PSU
Recall our discussion of computing correspondencesof image patches ( Lecture 7 ).
SSD - sum of squared difference measureNCC - normalized cross correlation measure
![Page 8: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/8.jpg)
CSE486, Penn StateRobert Collins
Epipolar Constraint
Important Concept:
For stereo matching, we don’t have to search the whole 2D right image for a corresponding point.
The “epipolar constraint” reduces the search spaceto a one-dimensional line.
![Page 9: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/9.jpg)
CSE486, Penn StateRobert Collins
Left camera
Right camera
Recall : Simple Stereo System
zz
xx
yyzz
xx
yy
TTxx
Same Y Coord!
Epipolar line
P
![Page 10: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/10.jpg)
CSE486, Penn StateRobert Collins
Matching using Epipolar LinesLeft Image Right Image
Match Score Values
For a patch in left image
Compare with patches along same row in right image
![Page 11: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/11.jpg)
CSE486, Penn StateRobert Collins
Matching using Epipolar LinesLeft Image Right Image
Match Score Values
Select patch with highestmatch score.
Repeat for all pixels inleft image.
![Page 12: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/12.jpg)
CSE486, Penn StateRobert Collins Example: 5x5 windows
NCC match score
Computed disparities Ground truth
Black pixels: bad disparity values,or no matching patch in right image
![Page 13: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/13.jpg)
CSE486, Penn StateRobert Collins
Occlusions: No matches
Left image
Right image
![Page 14: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/14.jpg)
CSE486, Penn StateRobert Collins
Effects of Patch Size
5x5 patches 11x11patches
Smoother in some areas Loss of finer details
![Page 15: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/15.jpg)
CSE486, Penn StateRobert Collins
Adding Inter-Scanline Consistency
So far, each left image patch has been matchedindependently along the right epipolar line.
This can lead to errors.
We would like to enforce some consistencyamong matches in the same row (scanline).
![Page 16: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/16.jpg)
CSE486, Penn StateRobert Collins
Disparity Space Image
First we introduce the concept of DSI.The DSI for one row represents pairwise match scores between patches along that row in the left and right image.
Pixels along left scanline
Pix
els
alon
g ri
ght
scan
line
Pixel i
Pixel j
C(i,j) = Match scorefor patch centeredat left pixel i withpatch centered atright pixel j.
![Page 17: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/17.jpg)
CSE486, Penn StateRobert Collins
Disparity Space Image (DSI)Left Image Right Image
Dissimilarity Values(1-NCC) or SSD
![Page 18: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/18.jpg)
CSE486, Penn StateRobert Collins
Disparity Space Image (DSI)Left Image Right Image
Dissimilarity Values(1-NCC) or SSD
![Page 19: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/19.jpg)
CSE486, Penn StateRobert Collins
Disparity Space Image (DSI)Left Image Right Image
Dissimilarity Values(1-NCC) or SSD
![Page 20: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/20.jpg)
CSE486, Penn StateRobert Collins
Disparity Space Image (DSI)Left Image
Dissimilarity Values
DSI
Enter each vector ofmatch scores as a column in the DSI
![Page 21: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/21.jpg)
CSE486, Penn StateRobert Collins
Disparity Space ImageLeft scanline
Rig
ht s
canl
ine
![Page 22: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/22.jpg)
CSE486, Penn StateRobert Collins
Disparity Space Image
Invalid entries due to constraintthat disparity >= low value(0 in this case)
Invalid entries due to constraintthat disparity <= high value64 in this case)
Left scanlineR
ight
sca
nlin
e
![Page 23: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/23.jpg)
CSE486, Penn StateRobert Collins
Disparity Space ImageN cols in left scanline
M c
ols
in r
ight
sca
nlin
e
However, I’m going to keep the full image around, including invalid values(I think it is easier to understand thepixel coordinates involved)
If we rearrange the diagonal band of valid values into a rectangular array (in this caseof size 64 x N), that is what is traditionally known as the DSI
coordinate in left scanline (e.g. N)
Disparity(e.g. 64)
Disparity Space Image
![Page 24: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/24.jpg)
CSE486, Penn StateRobert Collins
DSI and Scanline ConsistencyAssigning disparities to all pixels in left scanline now amounts to finding a connected path through the DSI
Start
End
![Page 25: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/25.jpg)
CSE486, Penn StateRobert Collins
Lowest Cost Path
Want one with lowest “cost” (Lowest sum ofdissimilarity scores along the path)
We would like to choose the “best” path.
?
?
?
![Page 26: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/26.jpg)
CSE486, Penn StateRobert Collins
Constraints on Path
It is common to impose an ordering constrainton the path. Intuitively, the path is not allowedto “double back” on itself.
![Page 27: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/27.jpg)
CSE486, Penn StateRobert Collins
Ordering Constraint
Ordering constraint…
A
B CD
A B C D
A B C D
…and its failure
A B C
D
A B C D
D A B C
![Page 28: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/28.jpg)
CSE486, Penn StateRobert Collins
Dealing with Occlusions
… …Left scanline Right scanline
![Page 29: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/29.jpg)
CSE486, Penn StateRobert Collins
Dealing with Occlusions
… …Left scanline Right scanline
Match
Match
MatchOccluded fromright scanline
Occluded fromleft scanline
![Page 30: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/30.jpg)
CSE486, Penn StateRobert Collins
An Optimal Scanline Strategy
• We want to find best path, taking intoaccount ordering constraint and thepossibility of occlusions.
Algorithm we will discuss now is fromCox, Hingorani, Rao, Maggs, “A MaximumLikelihood Stereo Algorithm,” ComputerVision and Image Understanding, Vol 63(3), May 1996, pp.542-567.
![Page 31: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/31.jpg)
CSE486, Penn StateRobert Collins
Cox et.al. Stereo Matching
matchOccludedfrom left
Occludedfrom right
Three cases:– Matching patches. Cost = dissimilarity score– Occluded from right. Cost is some constant value.– Occluded from left. Cost is some constant value.
match
Occludedfrom left
Occludedfrom right
i-1,j-1 i-1,j
i,j-1 i,j
C(i,j)= min([ C(i-1,j-1) + dissimilarity(i,j) C(i-1,j) + occlusionConstant, C(i,j-1) + occlusionConstant]);
![Page 32: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/32.jpg)
CSE486, Penn StateRobert Collins
Cox et.al. Stereo Matching
Recap: want to find lowestcost path from upper left tolower right of DSI image.
At each point on the path wehave three choices: step left,step down, step diagonally.
Each choice has a well-definedcost associated with it.
This problem just screams out for Dynamic Programming!(which, indeed, is how Cox et.al. solve the problem)
End
Start
![Page 33: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/33.jpg)
CSE486, Penn StateRobert Collins
DP cost matrix(cost of optimal path from each point to END)
DSIReal Scanline Example
Every pixel in left column now is marked witheither a disparity value, or an occlusion label.
Proceed for every scanline in left image.
![Page 34: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/34.jpg)
CSE486, Penn StateRobert Collins
Example
Result of DP alg. Black pixels = occluded.
Result of DP alg Result without DP (independent pixels)
![Page 35: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/35.jpg)
CSE486, Penn StateRobert Collins
Occlusion Filling
Simple trick for filling in gaps caused by occlusion.
= left occluded
Fill in left occluded pixels with value from the nearest valid pixel preceding it in the scanline.
Similarly, for right occluded, look for valid pixel to the right.
![Page 36: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/36.jpg)
CSE486, Penn StateRobert Collins
Example
Result of DP alg with occlusion filling.
![Page 37: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/37.jpg)
CSE486, Penn StateRobert Collins
ExampleResult of DP alg with occlusion filling. Result without DP (independent pixels)
![Page 38: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/38.jpg)
CSE486, Penn StateRobert Collins
ExampleResult of DP alg with occlusion filling. Ground truth
![Page 39: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/39.jpg)
CSE486, Penn StateRobert Collins www.middlebury.edu/stereo/
![Page 40: Lecture 09: Stereo Algorithms - Penn State College of ...rtc12/CSE486/lecture09.pdf · CSE486, Penn State Robert Collins Disparity Space Image First we introduce the concept of DSI](https://reader033.vdocument.in/reader033/viewer/2022042314/5f0255d87e708231d403c37e/html5/thumbnails/40.jpg)
CSE486, Penn StateRobert Collins
State-of-the-Art Results
J. Sun, Y. Li, S.B. Kang, and H.-Y. Shum.“Symmetric stereo matching for occlusion handling”. IEEE Conference on Computer Vision and Pattern Recognition, June 2005.
Algorithm Results Ground truth