mutual information-based stereo matching combined with sift descriptor in log-chromaticity color...
TRANSCRIPT
![Page 1: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/1.jpg)
Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space
Yong Seok Heo, Kyoung Mu Lee, and Sang Uk LeeDepartment of EECS, ASRI, Seoul National University, 151-742, Seoul, Korea
IEEE Conference on Computer Vision and Pattern Recognition, 2009.
![Page 2: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/2.jpg)
Outline• Introduction
• System Overview
• Mutual Information as a Stereo Correspondence Measure
• Proposed Algorithm
• Experiments
• Conclusion
![Page 3: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/3.jpg)
Introduction
![Page 4: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/4.jpg)
Introduction• Radiometric variations (between two input images)
• Degrade the performance of stereo matching algorithms.
• Mutual Information :• Powerful measure which can find any global
relationship of intensities• Erroneous as regards local radiometric variations
![Page 5: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/5.jpg)
Introduction• Different camera exposures (global)
• Different light configurations (local) Conventional MI
Conventional MI
![Page 6: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/6.jpg)
Objective• To present a new method:• Based on mutual information combined with
SIFT descriptor• Superior to the state-of-the art algorithms
(conventional mutual information-based)
Mutual Information
global radiometric variations (camera exposure)
SIFT descriptor
local radiometric variations (light configuration)
+
![Page 7: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/7.jpg)
System Overview
![Page 8: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/8.jpg)
Proposed Alogorithm
![Page 9: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/9.jpg)
Mutual Information (as a Stereo Correspondence Measure)
![Page 10: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/10.jpg)
Energy Function• Energy Function:
• In MAP-MRF framework
• f: disparity map
![Page 11: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/11.jpg)
Mutual Information• Used as a data cost:
•Measures the mutual dependence of the two random variables
Disparity Map
Left / RightImage
![Page 12: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/12.jpg)
Mutual Information
• Entropy:
• Joint Entropy:
• P(i): marginal probability of intensity i
• P(iL,iR): joint probability of intensity iL and iR
Entropy Entropy Joint Entropy
1
0
1
0
1
0
![Page 13: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/13.jpg)
Mutual InformationSuppose you are reporting the result of rolling a
fair eight-sided die. What is the entropy?
→The probability distribution is f (x) = 1/8, for x =1··8 , Therefore entropy is:
= 8(1/8)log 8 = 3 bits
![Page 14: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/14.jpg)
Mutual Information
H( IL, IR )
H( IL ) H( IR )
H( IL | IR ) H( IR | IL )MI( IL; IR )
Entropy Entropy Joint Entropy
![Page 15: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/15.jpg)
Mutual Information
• Entropy:
• Joint Entropy:
• P(i): marginal probability of intensity i
• P(iL,iR): joint probability of intensity iL and iR
Entropy Entropy Joint Entropy
1
0
1
0
1
0
i1 i2
![Page 16: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/16.jpg)
Pixel-wise Mutual Information• Previous: Mutual Information of whole images• Difficult to use it as a data cost in an energy
minimization framework → Pixel-wise
![Page 17: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/17.jpg)
Pixel-wise Mutual Information
=
P(.) / P(., .) : marginal / joint probability
G(.) / G(., .) : 1D / 2D Gaussian function
![Page 18: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/18.jpg)
![Page 19: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/19.jpg)
Left Image Intensity
Right Image Intensity
![Page 20: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/20.jpg)
Conventional MI• Cannot handle the local radiometric variations
caused by light configuration change
• Collect correspondences in the joint probability assuming that there is a global transformation
• The shape of the corresponding joint probability is very sparse.
• Do not encode spatial information
![Page 21: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/21.jpg)
Conventional MI• Different camera exposures (global)
• Different light configurations (local)Conventional MI
Conventional MI
![Page 22: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/22.jpg)
Proposed Algorithm
![Page 23: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/23.jpg)
111
![Page 24: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/24.jpg)
Log-chromaticity Color Space• Transform the input color images to log-
chromaticity color space [5]
• To deal with local as well as global radiometric variations
• Used to establish a linear relationship between color values of input images
[5] Y. S. Heo, K.M. Lee, and S. U. Lee. Illumination and camera invariant stereo matching. In Proc. of CVPR, 2008
![Page 25: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/25.jpg)
111
![Page 26: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/26.jpg)
SIFT Descriptor• Robust and accurately depicts local gradient
information
• Computed for every pixel in the log-chromaticity color space
![Page 27: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/27.jpg)
Energy Function• Data Cost:
• Mutual Information:
• SIFT descriptor distance:
( )constant
Log-chromaticity intensity
![Page 28: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/28.jpg)
111
![Page 29: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/29.jpg)
Joint Probability Using SIFT DescriptorA joint probability is computed at each channel
by use of the estimated disparity map from the previous iteration.
Wrong disparity can induce an incorrect joint probability.
Incorporate the spatial information in the joint probability computation step
Adopt the SIFT descriptor
![Page 30: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/30.jpg)
Joint Probability Using SIFT Descriptor
• K-channel SIFT-weighted joint probability:
• : Euclidean distance• VL,K(P) / VR,K(P) : SIFT descriptors for the pixel P• l : SIFT descriptor size
T = 1 If TrueT = 0 If false
i1 i2
![Page 31: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/31.jpg)
Joint Probability Using SIFT Descriptor• A joint probability is computed at each channel • Use estimated disparity map from the previous
iteration.
• is governed by the constraint that corresponding pixels should have similar gradient structures.
![Page 32: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/32.jpg)
111
![Page 33: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/33.jpg)
Energy Function• Data Cost:
• Smooth Cost:
MI SIFT
![Page 34: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/34.jpg)
Energy Minimization• The total energy is minimized by the Graph-cuts
expansion algorithm[3].
[3] Y. Boykov, O. Veksler, and R. Zabih. Fast approximate energy minimization via graph cuts. IEEE Trans. PAMI, 23(11):1222–1239, 2001.
![Page 35: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/35.jpg)
Energy Minimization
![Page 36: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/36.jpg)
Experiments
![Page 37: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/37.jpg)
Experimental Results• The std. dev σ of the Gaussian function is 10,
τ = 30, l = 4*4*8 = 128
• The window size of the SIFT descriptor : 9X9
• λ = 0.1, μ = 1.1, VMAX=5
• The total running time of our method for most images does not exceed 8 minutes.
• Aloe image (size : 427 X 370 / disparity range : 0-70) is about 6 minutes on a PC with PENTIUM-4 2.4GHz CPU.
![Page 38: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/38.jpg)
MI vs. SIFT
MI SIFT MI + SIFT
17.6% 11.97 % 9.27 %
26.45% 17.87 % 11.83 %
L: illum(1)-exp(1) / R: illum(3)-exp(1)
MI SIFT MI + SIFT
Error Rate
Error Rate
![Page 39: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/39.jpg)
L: illum(1)-exp(1) / R: illum(3)-exp(1)
![Page 40: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/40.jpg)
Different ExposureLeft Image Right Image Ground Truth Proposed
Rank/BT NCC ANCC MI
![Page 41: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/41.jpg)
Different Light Source ConfigurationsLeft Image Right Image Ground Truth Proposed
111
Rank/BT NCC ANCC MI
![Page 42: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/42.jpg)
Different ExposureLeft Image Right Image Ground Truth Proposed
111
Rank/BT NCC ANCC MI
![Page 43: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/43.jpg)
Different Light Source ConfigurationsLeft Image Right Image Ground Truth Proposed
111
Rank/BT NCC ANCC MI
![Page 44: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/44.jpg)
Different ExposureLeft Image Right Image Ground Truth Proposed
111
Rank/BT NCC ANCC MI
![Page 45: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/45.jpg)
Different Light Source ConfigurationsLeft Image Right Image Ground Truth Proposed
111
Rank/BT NCC ANCC MI
![Page 46: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/46.jpg)
Exposure Exposure
Light Configuration Light Configuration
![Page 47: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/47.jpg)
Exposure Exposure
Light Configuration Light Configuration
![Page 48: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/48.jpg)
Conclusion
![Page 49: Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department](https://reader030.vdocument.in/reader030/viewer/2022032604/56649e5c5503460f94b54426/html5/thumbnails/49.jpg)
Conclusion• Propose a new stereo matching algorithm
based on :• mutual information (MI) combined with• SIFT descriptor
• Quite robust and accurate to local as well as global radiometric variations