kece471 computer vision stereo - korea
TRANSCRIPT
![Page 1: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/1.jpg)
KECE471 Computer Vision
Stereo
Chang-Su Kim
Chapter 11, Computer Vision by Forsyth and PonceNote: Most contents were copied from the lecture notes of Prof. Kyeong Mu Lee in SNU
![Page 2: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/2.jpg)
Stereo
• Inferring depth information using two cameras like a
human
• Two eyes perceives three-dimension
Human eyes
Robot eyes
![Page 3: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/3.jpg)
Stereo
![Page 4: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/4.jpg)
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923
![Page 5: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/5.jpg)
Teesta suspension bridge-Darjeeling, India
![Page 6: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/6.jpg)
Stereo
• Inferring depth information using two eyes or cameras
• Two eyes perceive 3rd dimension
(a) (b)
![Page 7: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/7.jpg)
Applications
[Matthies,Szeliski,Kanade’88]
![Page 8: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/8.jpg)
Applications
![Page 9: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/9.jpg)
Applications
Background Substitution
![Page 10: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/10.jpg)
Binocular Stereo
![Page 11: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/11.jpg)
Pinhole Camera Model
Image plane
Focal length f
Center ofprojection
O
),,( ZYXP
),,( fyxP
),(),(),,(
Thus
Z
Yf
Z
XfyxZYX
Z
Yfy
Z
f
Y
y
Z
Xfx
Z
f
X
x
o
Y
X
Z
X
x
Z• 3D to 2D projection:
![Page 12: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/12.jpg)
Basic Stereo Model
1 2
Express as a function
of , , ,
Z
x x f B
Focal length f
),,( ZYXP
B line Base
),( 122 yxp
1O2O
),( 111 yxp 1o 2o
Left camera Right camera)(
and
121
21
pd
fB
xx
fBZ
f
x
Z
BX
f
x
Z
X
![Page 13: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/13.jpg)
Human Stereopsis: Reconstruction
Disparity: 𝑑 = 𝑟 − 𝑙 = 𝐷 − 𝐹.
𝑑 = 0
𝑑 < 0
![Page 14: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/14.jpg)
Finding Correspondence
along the same scan line
![Page 15: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/15.jpg)
Finding Correspondence
![Page 16: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/16.jpg)
General stereo
• What if two cameras are not parallel?
),,( ZYXP
2p
1O
2O
1p1o
2o
![Page 17: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/17.jpg)
Epipolar Geometry
Rrotation Tontranslati
1O 2O
),,( ZYXP
1X
1Y
1Z
1f
2p1p
2X
2Y
2Z
2f
![Page 18: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/18.jpg)
Epipolar Geometry
![Page 19: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/19.jpg)
Epipolar Geometry
• Epipolar Constraint
– A matching points lies on the associated epipolar line
– It reduces the correspondence problem to 1D search
along the epipolar line
– It reduces the cost and ambiguity of matching
![Page 20: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/20.jpg)
Rectification
• Simple case
– Cameras are parallel
– Focal lengths are the same
– Two image planes lie on the same plane
• Then, epipolar lines correspond to scan lines
• Rectification is a procedure to convert images so that the assumptions are satisfied
– It simplifies algorithms
– It improves efficiency
[KM Lee, Lecture Notes]
![Page 21: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/21.jpg)
• Reproject (warp) images so that epipolar
lines are aligned with the scan lines
Rectification
![Page 22: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/22.jpg)
Rectification
[Loop and Zhang, CVPR’99]
![Page 23: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/23.jpg)
Rectification
[Loop and Zhang, CVPR’99]
![Page 24: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/24.jpg)
Correspondence: What to Match?
• Objects?
– More identifiable, but difficult to compute
• Pixels?
– Easier to handle, but maybe ambiguous
• Edges?
• Collections of pixels (regions)?
![Page 25: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/25.jpg)
Correspondence: Photometric
Constraint
• Assume that the same world point has the
same intensity in both images.
– However, it is not true in general
• Noise
• Illumination
• Camera calibration
![Page 26: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/26.jpg)
Pixel Matching
For each scanline , for each pixel in the left image• compare with every pixel on same epipolar line in right image
• pick pixel with minimum match cost
• This will never work, so: match windows
What if ?
![Page 27: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/27.jpg)
Correspondence Using Window
Matching
SSD error
disparity
Left Right
scanline
![Page 28: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/28.jpg)
SSD
Left Right
Lw Rw
LI RI
LwRw
),( LL yx ),( LL ydx
m
m
• Two blocks 𝒘𝐿 and 𝒘𝑅
• 𝑆𝑆𝐷 = 𝒘𝐿 −𝒘𝑅2
![Page 29: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/29.jpg)
Normalization
• There can be differences in gain and
sensitivity
• Normalize the pixels in each window
𝒘 =𝒘− 𝜇𝟏
𝒘− 𝜇𝟏
• Minimizing SSD becomes maximizing NCC
(normalized cross correlation)
𝒘𝐿 − 𝒘𝑅2 = 2 − 2 𝒘𝐿 ⋅ 𝒘𝑅
![Page 30: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/30.jpg)
Left Right
LwRw
m
m
Lw
Lw
row 1
row 2
row 3
m
m
m
“Unwrap” image to form vector, using raster scan order
Each window is a vectorin an 𝑚2 dimensionalvector space.Normalization makesthem unit length.
Normalization
![Page 31: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/31.jpg)
Distance Metrics
Left Right
Lw Rw
LI RI
![Page 32: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/32.jpg)
Stereo Results
Images courtesy of Point Grey ResearchDisparity Map
![Page 33: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/33.jpg)
Problems with Window-Based
Matching
• Disparity within the window may not be
constant
• Blur across depth discontinuities
• Poor performance in textureless regions
• Erroneous results in occluded regions
![Page 34: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/34.jpg)
Window Size
W = 3 W = 20
• The results depend on the window size
• Some approaches have been developed to use an adaptive window size (try multiple sizes and select best match)
![Page 35: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/35.jpg)
[Szeliski, 1991]
Certainty Modeling
• Compute certainty map from correlations
input depth map certainty map
![Page 36: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/36.jpg)
Hierarchical Stereo MatchingD
ow
nsa
mpling
(Gauss
ian p
yra
mid
)
Dis
pari
ty p
ropagati
on
Allows faster computation
Deals with large disparity
ranges
(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
![Page 37: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/37.jpg)
Stereo Matching Using
Dynamic Programming
![Page 38: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/38.jpg)
Ordering Constraint• Points on the epipolar lines appear in the same order
• It may not be true in some cases, but can be assumed for most cases
• This is the basic assumption of the stereo matching using dynamic programming
Ordering constraint… …and its failure
![Page 39: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/39.jpg)
Occlusion and Disocclusion
… …
Left scanline Right scanline
![Page 40: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/40.jpg)
… …
Left scanline Right scanline
Match
Match
MatchOcclusion Disocclusion
Occlusion and Disocclusion
![Page 41: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/41.jpg)
Search over Correspondences
Three cases:
– Sequential – add cost of match (small if intensities agree)
– Occluded – add cost of no match (large cost)
– Disoccluded – add cost of no match (large cost)
Left scanline
Right scanline
Occluded Pixels
Disoccluded Pixels
![Page 42: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/42.jpg)
• Dynamic programming yields the optimal path, satisfying the ordering constraint
• Every segment on each scan line will be labeled as either matching or occlusion
– Diagonal arc: matching
– Horizontal arc: left occlusion
– Vertical arc: right occlusion
Occlusion
Left scanline
Occlu
sion
Rig
ht sca
nlin
e
Start
End
Dynamic Programming Approach
![Page 43: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/43.jpg)
Bellman’s Optimality Principle
Home
School
.5
.8
.7
.5
1.2
.8
.2
.3
.5
.8
1.2
1.0
1.3
![Page 44: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/44.jpg)
• Cost function 𝐶(𝑖, 𝑗): the optimal cost up to node (𝑖, 𝑗).
𝐶(𝑖, 𝑗) = min{
𝐶(𝑖 − 1, 𝑗 − 1) +matching cost,
𝐶(𝑖 − 1, 𝑗) +left occlusion penalty,
𝐶(𝑖, 𝑗 − 1) +right occlusion penalty
}
• While computing the cost, we record how node (𝑖, 𝑗) is connected to one of the three candidates
Left scanline
Rig
ht sca
nlin
e
Terminal
Dynamic Programming Approach
Occlu
sion
Occlusion
![Page 45: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/45.jpg)
• Raster-scan the nodes, computing optimal cost for each node.
Left scanline
Rig
ht sca
nlin
e
Terminal
Dynamic Programming Approach
Occlu
sion
Occlusion
![Page 46: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/46.jpg)
Left scanline
Rig
ht sca
nlin
e
Terminal
• Raster-scan the nodes, computing optimal cost for each node.
Dynamic Programming Approach
Occlu
sion
Occlusion
![Page 47: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/47.jpg)
Left scanline
Rig
ht sca
nlin
e
Terminal
• Raster-scan the nodes, computing optimal cost for each node.
Dynamic Programming Approach
Occlu
sion
Occlusion
![Page 48: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/48.jpg)
Left scanline
Rig
ht sca
nlin
e
Terminal
• Raster-scan the nodes, computing optimal cost for each node.
Dynamic Programming Approach
![Page 49: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/49.jpg)
• It’s done
Left scanline
Rig
ht sca
nlin
e
Terminal
Dynamic Programming ApproachOcclusion
Occlu
sion
![Page 50: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/50.jpg)
• It treats each scan line independently and thus may
generate streaking artifacts
• An error can propagate
Streaking artifacts
Dynamic Programming Approach
![Page 51: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/51.jpg)
• Enforcing inter-scanline continuity constraint• J.C. Kim, K.M. Lee, B.T. Choi, and S.U. Lee, “A dense stereo matching using two-pass
dynamic programming with generalized ground control points” CVPR 2005
• Y. Ohta and T. Kanade, “Stereo by Intra- and Inter-Scanline Search,” IEEE Trans.
PAMI, 7(2):139-154 (1985).
Dynamic Programming Approach
![Page 52: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/52.jpg)
Taxonomy and Categorization
• Four steps
1. Matching cost computation
2. Cost aggregation
3. Disparity computation and optimization
4. Disparity refinement
[Scharstein and Szeliski, 2002]
![Page 53: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/53.jpg)
Four Steps: Example
1. For every disparity, compute raw
matching costs
𝐸0 𝑥, 𝑦, 𝑑 = 𝜌(𝐼𝐿(𝑥 + 𝑑, 𝑦) − 𝐼𝑅(𝑥, 𝑦)
– 𝜌 𝑥 = 𝑥2
– 𝜌 𝑥 = |𝑥|
– Robust M-estimator 𝑟 ⋅ ⇒
• Why use a robust function?
• Occlusions, other outliers
[Szeliski, Lecture Notes]
![Page 54: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/54.jpg)
Four Steps: Example
2. Aggregate costs spatially
• Here, we are using a box filter
(efficient moving average
implementation)
• Alternatively, weighted average,
diffusion…
[Szeliski, Lecture Notes]
![Page 55: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/55.jpg)
Four Steps: Example
3. Choose winning disparity at each pixel
4. Interpolate to sub-pixel accuracy
d
E(d)
d*
[Szeliski, Lecture Notes]
![Page 56: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/56.jpg)
Cost Aggregation
• Shiftable window
• Variable windows, adaptive weights, and
segmentation-based
[Szeliski, Lecture Notes]
![Page 57: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/57.jpg)
Disparity Optimization
• Dynamic Programming
– Scanline optimization
– Evaluate best cumulative
cost at each pixel
[Szeliski, Lecture Notes]
![Page 58: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/58.jpg)
Disparity Optimization
• Cost function
𝐸 𝒅 = 𝐸data 𝒅 + 𝜆 ⋅ 𝐸smooth (𝒅)
• Recent Trend
– Belief propagation
– Graph-cut
SAD WTA Graph cut[Szeliski, Lecture Notes]
![Page 59: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/59.jpg)
Segmentation-Based Stereo Matching
![Page 60: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/60.jpg)
Middlebury Evaluation
• http://vision.middlebury.edu/
![Page 61: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/61.jpg)
Middlebury Evaluation
• http://vision.middlebury.edu/
![Page 62: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/62.jpg)
ETC
• Plane sweep stereo
• Multi-view stereo
![Page 63: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/63.jpg)
[Szeliski, Lecture Notes]
Plane Sweep Stereo
• Sweep family of planes through volume
virtual camera
composite
input image
?
input image
![Page 64: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/64.jpg)
Plane Sweep Stereo
• For each depth plane
– compute composite (mosaic) image — mean
– compute error image — variance
– convert to confidence and aggregate spatially
• Select winning depth at each pixel
[Szeliski, Lecture Notes]
![Page 65: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/65.jpg)
Multi-view Stereo
Figures by Carlos Hernandez
Input: calibrated images from several viewpoints
Output: 3D object model
[Seitz, Lecture Notes]
![Page 66: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/66.jpg)
Multi-view Stereo
error
depth
[Seitz, Lecture Notes]
![Page 67: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/67.jpg)
Merging Depth Maps
[Curless and Levoy 1996]
– compute weighted average of depth maps
set of depth maps(one per view)
merged surfacemesh
[Seitz, Lecture Notes]
![Page 68: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/68.jpg)
16 images (ring)47 images (ring)
Merging Depth Maps
317 images
(hemisphere)input image ground truth model
Goesele, Curless, Seitz, 2006
[Seitz, Lecture Notes]
![Page 69: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/69.jpg)
CONSISTENT STEREO
MATCHING
Example I
I-L. Jung, T.-Y. Chung, J.-Y. Sim, and C.-S. Kim, “Consistent stereo matching under varying radiometric conditions,” IEEE Trans. Multimedia, vol. 15, pp. 56-69, Jan. 2013.
![Page 70: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/70.jpg)
• Failures of color consistency assumption
– Corresponding pixels may have different colors
– Colors are affected by various illumination conditions
Different exposure conditions
Pseudo-Disparity Estimation
![Page 71: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/71.jpg)
• Idea
– Histogram = probability distribution of pixel values in
an image
– Cumulative histogram values = the ranks of pixel
brightness
– Corresponding pixels indicate the same scene point
• Their colors can be different
• But their ranks in each image should be almost the
same
Pseudo-Disparity Estimation
![Page 72: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/72.jpg)
• Joint CDF maps
– 𝐾0 : The joint CDF for the left view
– 𝐾1 : The joint CDF for the right view
Pseudo-Disparity Estimation
73/18
![Page 73: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/73.jpg)
Adaptive Color Transform
• Affine Color Mapping𝛾1 𝒑 − 𝒅𝒑 = 𝜓𝛾0 𝒑 + 𝜂𝟏
• Parameter Estimation
– Least squares
![Page 74: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/74.jpg)
Color Transform Results
![Page 75: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/75.jpg)
Consistent Stereo Matching
• Forward vs. inverse mappings
![Page 76: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/76.jpg)
Consistent Stereo Matching
• Reliability term for matching cost
computations
![Page 77: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/77.jpg)
Consistent Stereo Matching
• Reliability Term for Matching Cost
Computations
– The matching between 𝒑0 and 𝒑1 is disturbed
using 𝒑0, 𝒑1/4, 𝒑2/4, 𝒑3/4, and 𝒑1 as pivots
![Page 78: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/78.jpg)
Consistent Stereo Matching
• Consistency Term for Disparity Refinement
𝐸 𝐷 = 𝐸data 𝐷 + 𝜆smooth𝐸smooth 𝐷 + 𝜆consist𝐸consist(𝐷)
– Penalties for inconsistent disparities
![Page 79: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/79.jpg)
Stereo Matching Results
![Page 80: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/80.jpg)
Consistency Maps
Proposed
AW+GC
![Page 81: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/81.jpg)
View Synthesis Results
![Page 82: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/82.jpg)
Conclusions
• Rank-based pseudo-disparity
estimation for color matching
• Consistency Criterion
– Reliability term for matching cost computation
– Consistency term for disparity refinement
• Especially good for view synthesis
applications
• Computationally complicated
![Page 83: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/83.jpg)
MULTI-VIEW CORRESPONDENCE
MATCHING WITH ACTIVITY
VECTORS
Example II
S.-Y. Lee, J.-Y. Sim, C.-S. Kim, and S.-U. Lee, "Correspondence Matching of Multi-View Video Sequences Using Mutual Information Based Similarity Measure," to appear in IEEE Trans. Multimedia, 2013.
![Page 84: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/84.jpg)
Introduction
• Camera network
– Control unit
– Each camera has
• local processing
• Communication
• Wide-view stereo
![Page 85: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/85.jpg)
Motivation
• In order to handle the multi-view visual data, the geometry relation
between one view and another is required.
Google street view
86
3-D reconstruction
![Page 86: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/86.jpg)
Related Feature-based Matching
• SIFT: scale invariant feature transform– D. Lowe, “Distinctive image features from scale-invariant keypoints,” International
Journal of Computer Vision, vol. 60, no. 2, pp. 91–110, Nov. 2004
feature detection
feature descriptor
matching decision
Harris cornerLoGMSER
128-dimensional SIFT vector
minimum Euclidean distance
87
![Page 87: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/87.jpg)
Limitation of Feature-based
Matching
• Homogenous regions
?
![Page 88: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/88.jpg)
Proposed Algorithm
• Multi-view video sequences
– captured by fixed position cameras
– relatively long sequences
I= {𝐼𝑡 ∶ 𝑡 = 0,1, … , 𝑇 − 1} J= {𝐽𝑡 ∶ 𝑡 = 0,1, … , 𝑇 − 1}
![Page 89: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/89.jpg)
Activity Vector
J. M. McHugh, J. Konrad, V. Saligrama, and P. M. Jodoin, "Foreground-Adaptive Background Subtraction," IEEE Signal Processing
Letters, vol. 16, no. 5, pp. 390-393, May 2009.
𝐼0, 𝐼1, 𝐼2, 𝐼3…
Original frame
𝐼0𝐵, 𝐼1
𝐵, 𝐼2𝐵, 𝐼3
𝐵…
Binary frame(Moving object detection)
𝐴 𝐩
= (𝐼0𝐵 𝐩 , 𝐼1
𝐵 𝐩 , 𝐼2𝐵 𝐩 , 𝐼3
𝐵 𝐩 … , 𝐼𝑇−1𝐵 𝐩 )
Activity vector
![Page 90: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/90.jpg)
Mutual Information Based Similarity
• MIBS measure
𝐼 𝑋; 𝑌 = 𝑝𝑥,𝑦 log2𝑝𝑥,𝑦
𝑝𝑥𝑝𝑦
=
𝑚,𝑛∈{0,1}
𝑝(𝑚, 𝑛) log2𝑝(𝑚, 𝑛)
𝑝 𝑚 𝑝(𝑛)
=
𝑚,𝑛∈{0,1}
𝐾𝑚𝑛
𝑇log2
𝑇𝐾𝑚𝑛
𝐾𝑚∗𝐾∗𝑛
≜ 𝑆(𝐩, 𝐪)
S 𝐩, 𝐪 = 𝛼00𝐾00 + 𝛼01𝐾01 + 𝛼10𝐾10 + 𝛼11𝐾11
= 𝛼𝑚𝑛
Mutual information based similarity measure (MIBS measure)
![Page 91: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/91.jpg)
Mutual Information Based Similarity
• In static backgrounds
– A number of ‘0’ does not give much
information
1
0
𝐴(𝐩)
1
0
𝐴(𝐪)
Hamming distance = 0(equivalent tosimilarity = 6 (maximum)
MIBS measure = 0
![Page 92: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/92.jpg)
Mutual Information Based Similarity
• Matching criterion
MIBS Hamming distance
For given 𝐩 ∈ 𝐼, 𝐪∗ = argmax𝐪∈𝐽
𝑠(𝐩, 𝐪)
![Page 93: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/93.jpg)
Experimental Results: Dataset
Soccer
100,000
perspective
high
Road
172,000
translation
medium
ParkingLot
200,000
rotation
low
Library
150,000
zoom, rotation
medium
Jahayeon
100,000
rotation
high
Crossroad
66,000
perspective
high
ArtCollege
100,000
zoom, rotation
low
Desk
150,000
zoom
high
Hall
100,000
zoom, rotation
medium
Stair
126,000
rotation
medium
![Page 94: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/94.jpg)
Experimental Results
I
Ermis
MIBS
![Page 95: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/95.jpg)
Comparison to Other Measures
Similarity Definition
Hamming 𝑇 − 𝐾10 − 𝐾01
Jaccard-Needham
𝐾11𝐾11 + 𝐾10 + 𝐾01
Correlation 𝐾11𝐾00 − 𝐾10𝐾01
((𝐾10 + 𝐾11)(𝐾01 + 𝐾00)(𝐾11 + 𝐾01)(𝐾00 + 𝐾10))1/2
Yule 𝐾11𝐾00 − 𝐾10𝐾01𝐾11𝐾00 + 𝐾10𝐾01
Russel-Rao 𝐾11𝑇
Rogers-Tanmoto 𝐾11𝐾00𝐾11 + 𝐾00 + 2𝐾10 + 2𝐾01
Kulzinsky 𝐾11𝐾10 + 𝐾01
![Page 96: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/96.jpg)
MIBS
Jaccard-Needham Correlation
Yule Russell-Rao
Hamming
Rogers-Tanmoto Kulzinsky
Source
Ermis
![Page 97: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/97.jpg)
Comparison to Other Measures
Average error of correspondence matching to ground truth(unit: pixel)
Soccer ParkingLot Jahayeon
MIBS 7.365405 6.298014 0.613001
Hamming 66.217085 11.305671 6.058103
Ermis 64.555733 11.198556 5.454871
Jaccard-Needham 7.242479 6.899561 0.744769
Correlation 35.08293 15.982141 11.824747
Yule 74.904488 24.963444 7.806933
Russel-Rao 7.306032 7.27316 1.641155
Rogers 66.217085 11.305671 6.058103
Kulzinsky 7.242479 6.899561 0.744769
Dice 7.242479 6.899561 0.744769
![Page 98: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/98.jpg)
Overall System
Foreground object
detection
Adaptive activity area
Consistent pixel position
MRF optimization
![Page 99: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/99.jpg)
Adaptive Activity Area
• Gap between the objects and ground
– Discrepancy between a detected foreground object
and its true active area on the ground plane
– Camera pitch angle
true active area
![Page 100: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/100.jpg)
Adaptive Activity Area
• Adaptive activity area
– Bottom areas of the separated objects
– Experimentally, the ratio 𝜅 is set to 0.25
![Page 101: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/101.jpg)
Consistent Pixel Positions
• Bidirectional matching
: one of the regular grid points in I
Termination condition: 𝐩(𝑘+1) = 𝐩𝑘 or 𝑘 > 10
![Page 102: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/102.jpg)
Panoramic View Synthesis
Conventional ProposedI J
![Page 103: KECE471 Computer Vision Stereo - Korea](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b0640a2a5c4787621e2f/html5/thumbnails/103.jpg)
Conclusions
• Correspondence matching algorithm for
multi-view video sequences
– MIBS measure outperformed conventional
similarity measures
– System incorporates
• Adaptive activity area
• Consistent pixel positions