features, feature descriptors, matching
DESCRIPTION
Features, Feature descriptors, Matching. Jana Kosecka George Mason University. Computer Vision. Visual Sensing. Images I(x,y) – brightness patterns. image appearance depends on structure of the scene material and reflectance properties of the objects - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/1.jpg)
Features, Feature descriptors, Matching
Jana KoseckaGeorge Mason University
![Page 2: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/2.jpg)
MSRI Workshop, January 20052
Computer Vision Computer Vision
Visual Sensing Visual Sensing
Images I(x,y) – brightness patternsImages I(x,y) – brightness patterns
- image appearance depends on structure of the scene- material and reflectance properties of the objects- position and strength of light sources
![Page 3: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/3.jpg)
MSRI Workshop, January 20053
• photometric properties of the environment• geometric properties of the environment
What gives rise to images
![Page 4: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/4.jpg)
MSRI Workshop, January 20054
Basic ingredients
Radiance – amount of energy emitted along certain direction
Iradiance – amount of energy received along certain direction
BRDF – bidirectional reflectance distributionLambertian surfaces – the appearance depends only on radiance, not on the viewing direction
Image intensity for a Lambertian surface
![Page 5: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/5.jpg)
MSRI Workshop, January 20055
Challenges
![Page 6: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/6.jpg)
MSRI Workshop, January 20056
Image Primitives and Matching
Given an image point in left image, what is the (corresponding) point in the rightimage, which is the projection of the same 3-D point
![Page 7: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/7.jpg)
MSRI Workshop, January 20057
Image Primitives and Correspondence
Difficulties – ambiguities, large changes of appearance, due to changeof viewpoint, non-uniquess
![Page 8: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/8.jpg)
MSRI Workshop, January 20058
Correspondence
Lambertian assumption
Rigid body motion
Matching - Correspondence
radiance
![Page 9: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/9.jpg)
MSRI Workshop, January 20059
Translational model
Affine model
Transformation of the intensity values taking into account occlusions and noise
Local Deformation Models
![Page 10: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/10.jpg)
MSRI Workshop, January 200510
Matching and Correspondence
Motivated by problems Reconstruction of 3D scene from multiple views Object recognition using (constellation of) features modelsVarieties Small base-line matching Wide base-line matching – large view point changes For now assuming Lambertian assumption –
appearance of a local surface patch is independent of the viewpoint
![Page 11: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/11.jpg)
MSRI Workshop, January 200511
• Translational model
• RHS approximation by the first two terms of Taylor series
• Small baseline
• Brightness constancy constraint
Feature Tracking and Optical Flow
![Page 12: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/12.jpg)
MSRI Workshop, January 200512
• Integrate around over image patch
• Solve
Feature Tracking and Optical flow
![Page 13: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/13.jpg)
MSRI Workshop, January 200513
rank(G) = 0 blank wall problemrank(G) = 1 aperture problem rank(G) = 2 enough texture – good feature candidates
Conceptually:
In reality: choice of threshold is involved
Optical Flow, Feature Tracking
![Page 14: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/14.jpg)
MSRI Workshop, January 200514
Affine feature tracking
Intensity offsetContrast change
![Page 15: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/15.jpg)
MSRI Workshop, January 200515
• Qualitative properties of the motion fields
• Previous method - assumption locally constant flow
• Alternative regularization techniques (locally smooth flow fields, integration along contours)
Optical Flow
![Page 16: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/16.jpg)
MSRI Workshop, January 200516
• Compute eigenvalues of G• If smalest eigenvalue of G is bigger than - mark pixel as candidate feature point
• Alternatively feature quality function (Harris Corner Detector)
Point Feature Extraction
![Page 17: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/17.jpg)
MSRI Workshop, January 200517
Harris Corner Detector - Example
![Page 18: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/18.jpg)
MSRI Workshop, January 200518
Feature Selection
Compute Image Gradient Compute Feature Quality measure for each pixel
Search for local maxima
Feature Quality Function Local maxima of feature quality function
![Page 19: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/19.jpg)
MSRI Workshop, January 200519
Feature Tracking
Translational motion model
Closed form solution
1. Build an image pyramid 2. Start from coarsest level 3. Estimate the displacement at the coarsest level 4. Iterate until finest level
![Page 20: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/20.jpg)
MSRI Workshop, January 200520
Coarse to fine feature tracking
1. compute 2. warp the window in the second image by3. update the displacement 4. go to finer level 5. At the finest level repeat for several iterations
0
2
1
![Page 21: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/21.jpg)
MSRI Workshop, January 200521
Tracked Features
![Page 22: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/22.jpg)
MSRI Workshop, January 200522
Wide baseline matching
Point features detected by Harris Corner detector
![Page 23: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/23.jpg)
MSRI Workshop, January 200523
• Sum of squared differences
• Normalize cross-correlation
• Sum of absolute differences
Region based Similarity Metric
![Page 24: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/24.jpg)
MSRI Workshop, January 200524
NCC score for two widely separated views
NCC score
![Page 25: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/25.jpg)
MSRI Workshop, January 200525
Advanced matching techniques
( )
1. Selected salient image locations - points, pieces of countours2. Associate Local photometric descriptors 3. Invariance to image transformations + illumination changes
NCC - is not invariant with respect to image transformation
![Page 26: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/26.jpg)
MSRI Workshop, January 200526
Summary of the approach
Very good results in the presence of occlusion and clutter local information discriminant greyvalue information robust estimation of the global relation between
images for limited view point changes
Solution for more general view point changes wide baseline matching (different viewpoint, scale and
rotation) local invariant descriptors based on greyvalue
information
![Page 27: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/27.jpg)
MSRI Workshop, January 200527
Local descriptors
Greyvalue derivatives
Invariance to image rotation :
differential invariants [Koenderink87]
![Page 28: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/28.jpg)
MSRI Workshop, January 200528
Feature Detection and Matching
Detection of interest points/regions Harris detector (extension to scale and
affine invariance) Computation of descriptors for each point
(e.g. diff. invariants, steerable filters, SIFT descriptor) Similarity of descriptors (Euclidean distance, Mahalanobis
Distance)
![Page 29: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/29.jpg)
MSRI Workshop, January 200529
Keypoint Detector and SIFT Descriptor
Each image is characterized by a set of scale-invariant keypoints and their associated descriptors [D. Lowe,2000]
Keypoints - extrema in DOG pyramid
Descriptor – 8 bin orientation histograms computed
over 4 x 4 grid overlayed over pixel neighbourhood
and stacked together to form a 128 dim feature vector
![Page 30: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/30.jpg)
MSRI Workshop, January 200530
SIFT Keypoints
![Page 31: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/31.jpg)
MSRI Workshop, January 200531
Overview
Scale invariance is not sufficient for large baseline changes
State of the art on affine invariant points/regions
Affine invariant interest points
Application to recognition
![Page 32: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/32.jpg)
MSRI Workshop, January 200532
Scale invariant interest points
Invariant points + associated regions [Mikolajczyk & Schmid’01]
multi-scale Harris points
selection of points
at the characteristic scale
with Laplacian
Courtesy of Schimd’01
![Page 33: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/33.jpg)
MSRI Workshop, January 200533
Viewpoint changes
Locally approximated by an affine transformation
A
detected scale invariant region
projected region
Courtesy of Schimd’01
![Page 34: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/34.jpg)
MSRI Workshop, January 200534
Affine invariant Harris points
Localization & scale influence affine neighhorbood=> affine invariant Harris points (Mikolajczyk &
Schmid’02)
Iterative estimation of these parameters1. localization – local maximum of the Harris measure 2. scale – automatic scale selection with the Laplacian3. affine neighborhood – normalization with second
moment matrixRepeat estimation until convergence
Initialization with multi-scale interest points
![Page 35: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/35.jpg)
MSRI Workshop, January 200535
Alternative features/descriptors
Affine invariant regions (Tuytelaars et al.’00) ellipses fitted to intensity maxima parallelogram formed by interest points and edges
• Maximally stable regions (Matas et al. BMVC’02) regions stable across large range of thresholds, connected components of thresholded image descriptors – rotationaly and affine invariant and color moments
![Page 36: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/36.jpg)
MSRI Workshop, January 200536
Feature Matches
33 correct matches
Courtesy of Schimd’01
![Page 37: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/37.jpg)
MSRI Workshop, January 200537
Pieces of Countour/Line descriprors
Select salient pieces using scale invariant detection techniques Characterize either the intensity profile along contour/or local neighbourhood with sideness information – form the descriptor Type of suitable salient regions depends of the class of objects Computational model of visual attention can guide the process of selecting salient regions
![Page 38: Features, Feature descriptors, Matching](https://reader035.vdocument.in/reader035/viewer/2022062519/56815167550346895dbf95af/html5/thumbnails/38.jpg)
MSRI Workshop, January 200538
Additional changes of the appearance