![Page 1: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/1.jpg)
Sylvie CHAMBON
Pixel matching for binocular stereovision by propagation of feature point matches and randomized voting
schemeGuillaume GALES
Alain CROUZIL
Patrice DALLE
Ph.D. advisors
Ph. D. director
LaboratoryTCIGroupMITTDoctoral school
![Page 2: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/2.jpg)
Acquisition
Left image Right image
![Page 3: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/3.jpg)
Acquisition
Image gaucheImage droite
![Page 4: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/4.jpg)
Acquisition
Anaglyph
![Page 5: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/5.jpg)
Calibration
P
OgOdbaseline
XZ
Y
Right imageLeft image
jl
il jr
ir
x
l
zl
ylzr
x
r
yr
pli,j pr
i0,j0
![Page 6: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/6.jpg)
Pixel matching
P
OgOdbaseline
XZ
Y
Right imageLeft image
jl
il jr
ir
x
l
zl
ylzr
x
r
yr
pli,j pr
i0,j0
![Page 7: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/7.jpg)
Epipolar rectification
� 2 Mise en correspondance de pixels pour la stéréovision binoculaire
Plan épipolaire
Plan image gauche Plan image droit
P
Og
Centre optique gaucheOd
Centre optique droit
pgi,j
pdi�,j�
droites épipolaires conjuguées
Figure 2.5: Géométrie épipolaire.
Image gauche Image droite
Figure 2.6: Rectification épipolaire. – La première ligne présente un couple d’images etsa version rectifiée est sur la seconde ligne.
34
Left image Right image
Before rectification
After rectification
![Page 8: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/8.jpg)
Pixel matching
P
Right imageLeft imageXZ
Y
• pli,j pr
i,j0
jl
il
jr
ir
![Page 9: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/9.jpg)
Pixel matching
i
j
disparity vector
pri,j0
pli,j
pri,j0 = pl
i,j + dli,j
= pli,j +
0di,j
�
![Page 10: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/10.jpg)
Pixel matching
Disparity map
![Page 11: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/11.jpg)
3D reconstruction
![Page 12: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/12.jpg)
Di!culties in occluded areas
Left image Right image
![Page 13: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/13.jpg)
Di!culties near depth discontinuity areas
Left image Right image
![Page 14: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/14.jpg)
Di!culties near depth discontinuity areas
Left image Right image
![Page 15: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/15.jpg)
Outline
• Pixel matching methods‣ Local methods‣ Basic algorithm
‣ Seeds propagation methods‣ Global methods‣ Region-based methods
• Contributions‣ Evaluation of seed selection methods for propagation‣ Multi-measure propagation matching‣ Randomized voting scheme for matching
![Page 16: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/16.jpg)
Basic algorithm
Left image Right image
![Page 17: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/17.jpg)
Basic algorithm
Left image Right image
![Page 18: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/18.jpg)
Basic algorithm
Left image Right image
![Page 19: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/19.jpg)
Basic algorithm
Left image Right image
![Page 20: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/20.jpg)
Basic algorithm
Left image Right image
![Page 21: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/21.jpg)
Basic algorithm
Left image Right image
Search area
![Page 22: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/22.jpg)
Basic algorithm
Left image Right image
Corre
lation
sc
ore
Candidates
![Page 23: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/23.jpg)
Basic algorithm
Left image Right image
Corre
lation
scor
e
Candidates
![Page 24: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/24.jpg)
Basic algorithm
Left image Right image
Corre
lation
sc
ore
Candidates
![Page 25: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/25.jpg)
Result
![Page 26: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/26.jpg)
Threshold constraint
Threshold
Left image Right image
Corre
lation
scor
e
Candidates
![Page 27: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/27.jpg)
Symmetry constraint
Left image Right image
![Page 28: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/28.jpg)
Symmetry constraint
Left image Right image
![Page 29: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/29.jpg)
Result
left to right right to left
![Page 30: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/30.jpg)
Correlation measures
Left image Right image
CLASSICAL STATISTICSeg : SAD
CROSS CORRELATIONeg : NCC
DERIVATIVEeg : GC
NON PARAMETRICeg : CENSUS
ROBUSTeg : SMPD2
![Page 31: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/31.jpg)
Left image Right image
CLASSICAL STATISTICSeg : SAD
CROSS CORRELATIONeg : NCC
DERIVATIVEeg : GC
NON PARAMETRICeg : CENSUS
ROBUSTeg : SMPD2
Correlation measures
![Page 32: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/32.jpg)
Left image Right image
CLASSICAL STATISTICSeg : SAD
CROSS CORRELATIONeg : NCC
DERIVATIVEeg : GC
NON PARAMETRICeg : CENSUS
ROBUSTeg : SMPD2
Correlation measures
![Page 33: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/33.jpg)
Left image Right image
CLASSICAL STATISTICSeg : SAD
CROSS CORRELATIONeg : NCC
DERIVATIVEeg : GC
NON PARAMETRICeg : CENSUS
ROBUSTeg : SMPD2
Correlation measures
![Page 34: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/34.jpg)
Left image Right image
CLASSICAL STATISTICSeg : SAD
CROSS CORRELATIONeg : NCC
DERIVATIVEeg : GC
NON PARAMETRICeg : CENSUS
ROBUSTeg : SMPD2
Correlation measures
![Page 35: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/35.jpg)
Left image Right image
000100000... 000100000...
CLASSICAL STATISTICSeg : SAD
CROSS CORRELATIONeg : NCC
DERIVATIVEeg : GC
NON PARAMETRICeg : CENSUS
ROBUSTeg : SMPD2
Correlation measures
![Page 36: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/36.jpg)
Left image Right image
CLASSICAL STATISTICSeg : SAD
CROSS CORRELATIONeg : NCC
DERIVATIVEeg : GC
NON PARAMETRICeg : CENSUS
ROBUSTeg : SMPD2
Correlation measures
![Page 37: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/37.jpg)
• État de l’art : mise en correspondance de pixels‣ Méthodes locales
‣Méthodes par propagation de germes‣ Méthodes globales‣ Méthodes fondées sur les régions
• Contributions‣ Évaluation des méthodes de sélection de germes pour la propagation‣ Mise en correspondance par propagation multi-mesure‣ Mise en correspondance par sondage de régions
![Page 38: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/38.jpg)
Hypothesis
• We suppose we have a set of reliable matches called seeds computed by :‣ Feature point matching or‣ Highly constrained matching
• HYPOTHESIS : almost everywhere, two pixel neighboring pixels are the projections of two points belonging to a same surface
• CONSEQUENCES : ‣ Almost everywhere, two neighboring pixels have
almost the same disparity‣ Reduction of the search area to the neighboring of
seeds- Reduction of ambiguities- Reduction of computation time
The hypothesis is not true near depth discontinuities (in red)
![Page 39: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/39.jpg)
Principle
Left image Right image
![Page 40: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/40.jpg)
Principle
Left image Right image
Search area
![Page 41: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/41.jpg)
Principle
Left image Right image
![Page 42: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/42.jpg)
Principle
Left image Right image
![Page 43: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/43.jpg)
Principle
Left image Right image
![Page 44: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/44.jpg)
Two approaches of propagation
‣ SIMULTANEOUS APPROACH : at each iteration, all the seeds are taken into account to be propagated
‣ SEQUENTIAL APPROACH: at each iteration, one seed is selected to be propagated according to a predefined criteria (“best-first strategy”)
![Page 45: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/45.jpg)
• État de l’art : mise en correspondance de pixels‣ Méthodes locales‣ Méthodes par propagation de germes
‣Méthodes globales‣ Méthodes fondées sur les régions
• Contributions‣ Évaluation des méthodes de sélection de germes pour la propagation‣ Mise en correspondance par propagation multi-mesure‣ Mise en correspondance par sondage de régions
![Page 46: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/46.jpg)
Principle
Left image Right image Disparity map
Initialization
Right image
![Page 47: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/47.jpg)
Principle
Left image Right image Disparity map
Initialization
Image droite
![Page 48: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/48.jpg)
Principle
Left image Right image Disparity map
Initialization
Image droite
![Page 49: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/49.jpg)
Principle
Left image Right image Disparity map
Error minimization
![Page 50: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/50.jpg)
• État de l’art : mise en correspondance de pixels‣ Méthodes locales‣ Méthodes par propagation de germes‣ Méthodes globales
‣Méthodes fondées sur les régions• Contributions‣ Évaluation des méthodes de sélection de germes pour la propagation‣ Mise en correspondance par propagation multi-mesure‣ Mise en correspondance par sondage de régions
![Page 51: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/51.jpg)
Hypothesis
• HYPOTHESIS : the pixels within a same region are the projections of points belonging to a same surface
• CONSEQUENCE : the pixels belonging to a same (and small) region have almost the same disparity
The hypothesis is not true when objects with di!erent depths have the same color (in red)
![Page 52: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/52.jpg)
Disparity map by surface fitting
Robust estimation of the surface model
parameters for each region
Global optimization
Initial matching with a local method
Segmentation
Left image Right image
d
i
j
Principle
![Page 53: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/53.jpg)
Disparity map by surface fitting
Robust estimation of the surface model
parameters for each region
Global optimization
Initial matching with a local method
Segmentation
Left image Right image
d
i
j
Principle
![Page 54: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/54.jpg)
Disparity map by surface fitting
Robust estimation of the surface model
parameters for each region
Global optimization
Initial matching with a local method
Segmentation
Left image Right image
d
i
j
Principle
![Page 55: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/55.jpg)
METHOD
Local Easy to implement Bad results in di!cult areas
Propagation Reduction of ambiguitiesLow computation time Selection of the initial seeds
Global Take the whole information into account
Parametrization of the cost function terms
Region-based Best results Number of parameters
Summary
![Page 56: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/56.jpg)
Contributions
![Page 57: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/57.jpg)
• État de l’art : mise en correspondance de pixels‣ Méthodes locales‣ Méthodes par propagation de germes‣ Méthodes globales‣ Méthodes fondées sur les régions
• Contributions
‣Évaluation des méthodes de sélection de germes pour la propagation
‣ Mise en correspondance par propagation multi-mesure‣ Mise en correspondance par sondage de régions
![Page 58: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/58.jpg)
• For each pixel, a response value is computed
• Extrema selection• Post-processing
Feature point detection
FAMILLY DETECTORS
Derivative MORAVEC, HARRIS (3 versions), KITCHEN-ROSENFELD, MIC, SIFT,HARRIS-LAPLACE, HESSIAN-LAPLACE, SURF
Morphology FAST, SUSAN
Entropy KADIR
![Page 59: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/59.jpg)
Feature point detection
• For each pixel, a response value is computed
• Extrema selection• Post-processing
FAMILLY DETECTORS
Derivative MORAVEC, HARRIS (3 versions), KITCHEN-ROSENFELD, MIC, SIFT,HARRIS-LAPLACE, HESSIAN-LAPLACE, SURF
Morphology FAST, SUSAN
Entropy KADIR
![Page 60: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/60.jpg)
Feature point detection
• For each pixel, a response value is computed
• Extrema selection• Post-processing
FAMILLY DETECTORS
Derivative MORAVEC, HARRIS (3 versions), KITCHEN-ROSENFELD, MIC, SIFT,HARRIS-LAPLACE, HESSIAN-LAPLACE, SURF
Morphology FAST, SUSAN
Entropy KADIR
![Page 61: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/61.jpg)
Feature point detection
• For each pixel, a response value is computed
• Extrema selection• Post-processing
FAMILLY DETECTORS
Derivative MORAVEC, HARRIS (3 versions), KITCHEN-ROSENFELD, MIC, SIFT,HARRIS-LAPLACE, HESSIAN-LAPLACE, SURF
Morphology FAST, SUSAN
Entropy KADIR
![Page 62: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/62.jpg)
Feature point detection
Left image Right image
![Page 63: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/63.jpg)
Feature point detection
Left image Right image
![Page 64: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/64.jpg)
Feature point detection
Left image Right image
![Page 65: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/65.jpg)
Feature point detection
Left image Right image
![Page 66: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/66.jpg)
Feature point matching
• Feature point matching by computing distance measures between descriptors‣ Robust descriptors to large geometric transformations‣ Correlation measure between the neighboring of pixels (« detection-correlation »)
![Page 67: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/67.jpg)
Evaluation protocol
� 4.2 Détermination expérimentale des paramètres de sélection des germes
Image gauche Image droite Disparités théoriques
Tsukuba
Venus
Teddy
Cones
Table 4.1: Images testées. – Les couples d’images que nous avons testés sont ceux mis àdisposition sur le site : vision.middlebury.edu/stereo/.
109
Left image Right image Ground truth
Tsukuba
Venus
Teddy
Cones
![Page 68: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/68.jpg)
• Error measure
• Recall rate
Evaluation protocol
|d� dth| ⇥ 0.5
Tr =
number of correct seeds
number of seeds
![Page 69: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/69.jpg)
Evaluation protocol
i
j
R1R3
R2
• Distribution rate
Td =
number of regions holding at least one correct seed
number of regions
Td ⇡ 33%
![Page 70: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/70.jpg)
Td ⇡ 100%
Evaluation protocol
i
j
R1R3
R2
• Distribution rate
Td =
number of regions holding at least one correct seed
number of regions
![Page 71: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/71.jpg)
• Trade-o" A
Evaluation protocol
i
j
R1R3
R2
• Distribution rate
Td ⇡ 100%
Td =
number of regions holding at least one correct seed
number of regions
TOA = Tr ⇥ Td
![Page 72: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/72.jpg)
Parameters to optimize TOA
DETECTION PARAMETERS
Scale(s) SmallDETECTION PARAMETERS
Response threshold Very permissive
CORRELATION PARAMETERS
Correlation window size Big
CORRELATION PARAMETERS Search area Neighboring of
feature pointsCORRELATION PARAMETERS
Correlation score threshold Strict
Results
Results of the learning process of the parameters (simulated annealing)
![Page 73: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/73.jpg)
Results
1 Moravec2 Harris, variante 13 Harris, variante 24 Harris, variante 3
5 Kitchen et Rosenfeld
11 Harris-Laplace12 Hessian-Laplace
13 SURF14 Kadir
NCC
SMPD2
CENSUS
SAD
GC
� 4.3 Complémentarité des germes
Tré
Tra
10 20 30 40 50 60 70
60
70
80
90
98
975
55
8
2
97737
492
3
6
826
1
95
14
58
4
8
247
613
1
3
24
1414
10 14
3
661
141014
12
10
1112
12
1010
11
12
1112111113
13
131313
Figure 4.7: Vue d’ensemble des résultats pour le compromis A. – Chaque pastille cor-respond à un couple détecteur-corrélation. La position en abscisse correspond au taux derépartition moyen. La position en ordonnée correspond au taux de rappel moyen. Le nu-méro inscrit dans la pastille correspond au détecteur, voir § 4.2.1.2. Chaque couleur depastille correspond à une mesure de corrélation (bleu = SAD, orange = GC, vert = NCC,violet = SMPD2 et bleu = CENSUS). Les ellipses autour de chaque pastille indiquentl’écart-type. Là encore, on remarque bien que FAST-CENSUS (pastille bleue portant lenuméro 9) obtient les performances les plus intéressantes.
4.3 Complémentarité des germes
Nous venons de déterminer que le meilleur compromis moyen entre le taux de rappel etle taux de répartition peut être obtenu en utilisant le couple détecteur-corrélation FAST-CENSUS et atteint 41.6% (avec un taux de rappel de 85.6% et une répartition de 48%).Nous considérons ce résultat comme solution « de référence ». L’objectif de cette sectionest de dépasser ce résultat. Il est possible de perfectionner le compromis A dans les cassuivants :⌅ Par amélioration du taux de répartition tel que :⇥ le taux de rappel n’est pas dégradé ;⇥ le taux de rappel est dégradé mais cette perte est compensée par l’amélioration du
taux de répartition.
123
6 Beaudet7 MIC
8 SUSAN9 FAST10 SIFT
Tr
Td
![Page 74: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/74.jpg)
Complementarity of the seed sets
• FAST+CENSUS :• Improvement of the distribution rate by taking the union of two complementary seed sets
Left image Right image
Seed set 1
Seed set 2
TOA = 41.6% Tr = 85.6% Td = 48%
![Page 75: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/75.jpg)
• FAST+CENSUS :• Improvement of the distribution rate by taking the union of two complementary seed sets
Complementarity of the seed sets
Left image Right image
Seed set 1
Seed set 2
TOA = 41.6% Tr = 85.6% Td = 48%
![Page 76: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/76.jpg)
• FAST+CENSUS :• Improvement of the distribution rate by taking the union of two complementary seed sets
Complementarity of the seed sets
Ensemblede germes 1
Ensemblede germes 2
Left image Right image
Seed set 1
Seed set 2
TOA = 41.6% Tr = 85.6% Td = 48%
![Page 77: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/77.jpg)
• FAST+CENSUS :• Improvement of the distribution rate by taking the union of two complementary seed sets
Complementarity of the seed sets
Left image Right image
TOA = 41.6% Tr = 85.6% Td = 48%
![Page 78: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/78.jpg)
• Improvement of the recall rate‣ To determine the parameters of the detection-correlation to optimize the trade-o" B :
Complementarity of the seed sets
TOB = Tr ⇥ Coef. of cardinality
![Page 79: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/79.jpg)
Parameters to optimize TOA
Parameters to optimize TOB
DETECTION PARAMETERS
Scale(s) Small SmallDETECTION PARAMETERS
Response threshold Very permissive Very permissive
CORRELATION PARAMETERS
Correlation window size Big Big
CORRELATION PARAMETERS Search area Neighboring of
feature pointsCORRELATION PARAMETERS
Correlation score threshold Strict Very strict
Results
Results of the learning process of the parameters (simulated annealing)
![Page 80: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/80.jpg)
Results
(%) (%) (%)
Ref. 41.6 85.6 48
Track 1 52 (+10.4) 83.5 (-2.1) 61.1 (+13.1)
Track 2 53.7 (+12.1) 85.7 (+0.1) 61.9 (+13.9)
Track 3 43 (+2.2) 85.6 (+1.4) 49.2 (+1.6)
Track 4 51 (+9.4) 91 (+5.4) 56 (+8)
• Ref. : FAST-CENSUS• Improvement of the distribution rate :‣ Track 1 : FAST-CENSUS U SUSAN-CENSUS‣ Track 2 : KR-CENSUS U SURF-NCC U MORAVEC-CENSUS U HESSIAN-LAPLACE-CENSUS
U FAST-CENSUS• Improvement of the recall rate :‣ Track 3 : SUSAN-CENSUS U FAST-SAD‣ Track 4 : MIC-CENSUS U FAST-CENSUS U HESSIAN-LAPLACE-SMPD2 U HARRIS2-
CENSUS U KR-CENSUS U HARRIS-LAPLACE-SMPD2 U MIC-SMPD2 U HARRIS1-CENSUS U MORAVEC-CENSUS
TOA Tr Td
![Page 81: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/81.jpg)
• État de l’art : mise en correspondance de pixels‣ Méthodes locales‣ Méthodes par propagation de germes‣ Méthodes globales‣ Méthodes fondées sur les régions
• Contributions‣ Évaluation des méthodes de sélection de germes pour la propagation
‣Mise en correspondance par propagation multi-mesure
‣ Mise en correspondance par sondage de régions
![Page 82: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/82.jpg)
Goal
• The result of a propagation method is quasi-dense‣ The more the propagation goes on, the more the risk of propagating errors is
• Our goal is to improve the density of a propagation result with a high recall rate
![Page 83: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/83.jpg)
Principle
• Idea: to use a correlation measure according to the encountered di!culty‣ A « classic » correlation measure is used to start the propagation‣ A robust correlation measure is used in di!cult areas to resume the propagation
• A region constraint prevents the propagations into the neighboring regions
Step 1 : « classic » measure Step 2 : robust measure
![Page 84: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/84.jpg)
• Feature point matching• Union of the best seed sets• Completion by selecting highly constrained matches in empty regions• Validation
Experimental protocol
i
j
![Page 85: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/85.jpg)
Experimental protocol
i
jCompletion
• Feature point matching• Union of the best seed sets• Completion by selecting highly constrained matches in empty regions• Validation
![Page 86: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/86.jpg)
Experimental protocol
i
jValidation
• Feature point matching• Union of the best seed sets• Completion by selecting highly constrained matches in empty regions• Validation
![Page 87: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/87.jpg)
Experimental protocol
i
jValidation
• Feature point matching• Union of the best seed sets• Completion by selecting highly constrained matches in empty regions• Validation
![Page 88: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/88.jpg)
Results
(%) (%) (%)Ref. 41.6 85.6 48
Ref.+ 57.5 (+16.1) 87.1 (+1.5) 65.6 (+17.6)Track 1 52 83.5 61.1
Track 1+ 61.4 (+9.4) 84.9 (+1.4) 71.3 (+9.69)Track 2 53.7 85.7 61.9
Track 2+ 62.3 (+8.6) 90 (+4.3) 71.6 (+9.7)Track 3 43 85.6 49.2
Track 3+ 60 (+17) 86.6 (+1) 65.7 (+16.5)Track 4 51 91 56
Track 4+ 63.2 (+12.2) 91 (+0) 69.4 (+13.4)
TOA Tr Td
![Page 89: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/89.jpg)
Experimental protocol
• Density
• Acceptation rateD =
nombre de pixels propages
nombre total de pixels
Ta =
nombre de pixels propages acceptes
nombre total de pixels propages
![Page 90: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/90.jpg)
Results
Approach SequentialMeasure(s) GC-CENSUSThreshold (Strict, Permissive)
Initial seed set HIgh distrib. rateRegion constraint Permissive
Approach SequentialMeasure(s) CENSUSThreshold Strict
Initial seed set HIgh distrib. rateRegion constraint No
Approach basicMeasure CENSUSThreshold none
Approach Sequential or simultaneousMeasure(s) GC-C ou CENSUSThreshold ([Strict], none)
Initial seed set HIgh distrib. rateRegion constraint Permissive
Ta
D
94%
92%80%50%
Permissive or none
Strict
Left image Right image
Multi-measure propagation
![Page 91: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/91.jpg)
• État de l’art : mise en correspondance de pixels‣ Méthodes locales‣ Méthodes par propagation de germes‣ Méthodes globales‣ Méthodes fondées sur les régions
• Contributions‣ Évaluation des méthodes de sélection de germes pour la propagation‣ Mise en correspondance par propagation multi-mesure
‣Mise en correspondance par sondage de régions
![Page 92: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/92.jpg)
Goal
i
j
• To have a dense result with few errors‣ Use of homogeneous color region
• Estimation of disparities in occluded areas
![Page 93: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/93.jpg)
i
j
• To have a dense result with few errors‣ Use of homogeneous color region
• Estimation of disparities in occluded areas
Goal
![Page 94: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/94.jpg)
i
j
• To have a dense result with few errors‣ Use of homogeneous color region
• Estimation of disparities in occluded areas
Goal
![Page 95: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/95.jpg)
Cartes de disparité par modèle de surfaceDisparity maps by surface fitting
Randomized selections of 3 points in the disparity space for
each region
Voting
Initial matchingSegmentations
Left image Right image
...
...
Principle
![Page 96: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/96.jpg)
Principle
i
jd
![Page 97: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/97.jpg)
Principle
i
jd
![Page 98: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/98.jpg)
Principle
i
jd
![Page 99: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/99.jpg)
Principle
i
jd
![Page 100: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/100.jpg)
Principle
i
jd
![Page 101: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/101.jpg)
Principle
i
jd
![Page 102: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/102.jpg)
Principle
i
jd
![Page 103: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/103.jpg)
Segmentation 1 Segmentation 2 Segmentation 3 Segmentation 4
Principle
![Page 104: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/104.jpg)
Segmentation 1Segmentation 2Segmentation 3Segmentation 4
disparity
xi argmax
Principle
![Page 105: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/105.jpg)
Segmentation 1Segmentation 2Segmentation 3Segmentation 4
disparity
density
xi argmax
Principle
![Page 106: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/106.jpg)
� 6 Mise en correspondance par sondage de régions
t Cones Teddy Tsukuba Venus
0.5
0.75
1
1.5
2
Figure 6.4: Cartes de disparité obtenues par sondage des régions. – On marque en rougeles disparités erronées pour di érentes valeurs de t.
162
Cones Teddy Tsukuba Venus
Results
![Page 107: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/107.jpg)
Conclusion and perspectives
![Page 108: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/108.jpg)
Summary of the contributionsConclusion and perspectives
• Study of the seeds selection step for pixel matching by propagation‣ Analysis and comparison of 14 feature point detectors combined with 5 correlation
measures‣ Analysis of the complementarity of the di"erent seed sets‣ The union of di"erent sets can help to improve the results
• Multi-measure propagation‣ Good alternative to the local basic method but the choice of the initial seed set is
sensitive• Pixel matching by region-based randomized voting scheme‣ Dense results‣ Quality of the results comparable to the state of the art
![Page 109: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/109.jpg)
NEEDS ADVISED METHOD
Speed Propagation of seeds computed by FAST+GC
Speed and density Basic method with the correlation measure GC
Density Multi-measure propagation of seeds FAST+GC+CV Basic method with the correlation measure CENSUS (very slow)
High density Region-based randomized voting scheme started with SMPD2, 3 segmentations and about 50-75 random selections
Best results Region-based randomized voting scheme started with CENSUS, 4 segmentations and about 100 random selections
Summary of the contributionsConclusion and perspectives
CV : Completion and validation
![Page 110: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/110.jpg)
PerspectivesConclusion and perspectives
• Extension of the study to :‣ more matching constraints‣ “robust” descriptors‣ more segmentation techniques‣ di"erent image classes‣ more surface models
• Building a new feature point detector by machine learning (following the example of FAST) to give a response similar to the union of the best detectors
![Page 111: Pixel matching for binocular stereovision by propagation of feature points matches and region-based randomized voting scheme](https://reader034.vdocument.in/reader034/viewer/2022042713/54963a57b47959424d8b4f55/html5/thumbnails/111.jpg)