Feature Detection and Matching

Reading: Szeliski Chapter 4 Pages: 208~266

http://staff.ustc.edu.cn/~xjchen99/teaching/teaching.html

How to UNDERSTAND?

COMPARE with WHAT we learn before?

Image Matching

by Diva Sian

by swashford

Harder Case

by Diva Sian by scgbt

Even Harder Case

“How the Afghan Girl was Identified by Her Iris Patterns” Read the story

Harder still?

NASA Mars Rover images

NASA Mars Rover imageswith SIFT feature matchesFigure by Noah Snavely

Answer below (look for tiny colored squares…)

Features

All is Vanity, by C. Allan Gilbert, 1873-1929Readings

• Szeliski, Ch 4.1• (optional) K. Mikolajczyk, C. Schmid,  A performance evaluation of local

descriptors. In PAMI 27(10):1615-1630- http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/mikolajczyk_

Image Matching

Image Matching

Invariant Local FeaturesFind features that are invariant to transformations

– geometric invariance: translation, rotation, scale– photometric invariance: brightness, exposure, …

Feature Descriptors

Advantages of Local Features

Locality – features are local, so robust to occlusion and clutter

Distinctiveness: – can differentiate a large database of objects

Quantity– hundreds or thousands in a single image

Efficiency– real-time performance achievable

Generality– exploit different types of features in different situations

More Motivation…

Feature points are used for:– Image alignment (e.g., mosaics)– 3D reconstruction– Motion tracking– Object recognition– Indexing and database retrieval– Robot navigation– … other

Features

• Point/patch,• Edge/curve• Region

Want Uniqueness

• Look for image regions that are unusual– Lead to unambiguous matches in other images

• How to define “unusual”?

Corner Detection• Basic idea: Find points where two edges meet—i.e., high gradient in two

directions

• “Cornerness” is undefined at a single pixel, because there’s only one gradient per point– Look at the gradient behavior over a small window

• Categories image windows based on gradient statistics– Constant: Little or no brightness change– Edge: Strong brightness change in single direction– Flow: Parallel stripes– Corner/spot: Strong brightness changes in orthogonal directions

Matching Criterion

• (weighted) Summed Square Difference – SSD

• Compare an image patch against itself

Local Measures of Uniqueness

Suppose we only consider a small window of pixels– What defines whether a feature is a good or bad

candidate?

Slide adapted from Darya Frolova, Denis Simakov, Weizmann Institute.

Feature Detection

“flat” region:no change in all directions

“edge”: no change along the edge direction

“corner”:significant change in all directions

Local measure of feature uniqueness– How does the window change when you shift it?– Shifting the window in any direction causes a big change

Slide adapted from Darya Frolova, Denis Simakov, Weizmann Institute.

Consider shifting the window W by (u,v)• how do the pixels in W change?• compare each pixel before and after by

summing up the squared differences (SSD)

• this defines an SSD “error” of E(u,v):

Feature Detection: Mathematics

W

Harris Detector: Mathematics

2

,

( , ) ( , ) ( , ) ( , )x y

E u v w x y I x u y v I x y

Change of intensity for the shift [u,v]:

IntensityShifted

intensityWindow function

orWindow function w(x,y) =

Gaussian1 in window, 0 outside

Auto-correlation

function

Auto-Correlation Function

Auto-Correlation Function

Good unique minimum 1D aperture problem No good peak

Taylor Series expansion of I:

If the motion (u,v) is small, then first order approx is good

Small Motion Assumption

Feature Detection: Mathematics

• Image gradient– Harris detector with a [-2,-1,0,1,2] filter for Ix– Gaussian filter

Plugging this into the formula on

Small Motion Assumption

2

,

( , ) ( , ) ( , ) ( , )x y

E u v w x y I x u y v I x y

Feature Detection: Mathematics

W

Feature Detection: MathematicsThis can be rewritten:

x-

x+

Auto-correlation matrix

2

2

( ) ( ) ( )( )

( ) ( ) ( )x i x i y i

ii x i y i y i

I I IA

I I I

x x xx

x x x

Feature Detection: Mathematics

For the example above• You can move the center of the blue window to anywhere on the

blue unit circle• Which directions will result in the largest and smallest E values?• We can find these directions by looking at the eigenvectors of H

Quick eigenvalue/eigenvector reviewThe eigenvectors of a matrix A are the vectors x that satisfy:

The scalar is the eigenvalue corresponding to x– The eigenvalues are found by solving:

– In our case, A = H is a 2x2 matrix, so we have

– The solution:

Once you know , you find x by solving

Feature Detection: MathematicsThis can be rewritten:

Eigenvalues and eigenvectors of H=A

• Define shifts with the smallest and largest change (E value)• x+ = direction of largest increase in E.

• + = amount of increase in direction x+

• x- = direction of smallest increase in E.

• - = amount of increase in direction x+

x-

x+

Feature Detection: Mathematics

( , ) ,u

E u v u v Hv

Intensity change in shifting window: eigenvalue analysis

1, 2 – eigenvalues of H

(max)-1/2

(min)-1/2

Ellipse E(u,v) = const

If we try every possible orientation n,the max. change in intensity is 2

Feature Detection: Mathematics

1

2

“Corner”1 and 2 are large, 1 ~ 2;E increases in all directions

1 and 2 are small;E is almost constant in all directions

“Edge” 1 >> 2

“Edge” 2 >> 1

“Flat” region

Classification of image points using eigenvalues of H:

Feature Detection: MathematicsHow are +, x+, -, and x+ relevant for feature detection?

• What’s our feature scoring function?

Feature Detection: MathematicsHow are +, x+, -, and x+ relevant for feature detection?

• What’s our feature scoring function?

Want E(u,v) to be large for small shifts in all directions• the minimum of E(u,v) should be large, over all unit vectors [u v]

• this minimum is given by the smaller eigenvalue (-) of H

Feature DetectionHere’s what you do

• Compute the gradient at each point in the image• Create the H matrix from the entries in the gradient• Compute the eigenvalues. • Find points with large response (- > threshold)

• Choose those points where - is a local maximum as features

Feature DetectionHere’s what you do

• Compute the gradient at each point in the image• Create the A matrix from the entries in the gradient• Compute the eigenvalues. • Find points with large response (- > threshold)

• Choose those points where - is a local maximum as features

[Shi and Tomasi 1994]

Harris Detector

Measure of corner response:

2det traceR H k H

1 2

1 2

det

trace

H

H

(k – empirical constant, k = 0.04-0.06)

Harris and Stephens 1988

• The trace is the sum of the diagonals, i.e., trace(H) = h11 + h22

• Very similar to - but less expensive (no square root)

Harris Detector: Mathematics

1

2 “Corner”

“Edge”

“Edge”

“Flat”

• R depends only on eigenvalues of H

• R is large for a corner

• R is negative with large magnitude for an edge

• |R| is small for a flat region

R > 0

R < 0

R < 0|R| small

Harris Detector

• The Algorithm:– Find points with large corner response function R

(R > threshold)– Take the points of local maxima of R

Harmonic Mean

• Smoother function in the region where

Brown, M., Szeliski, R., and Winder, S. (2005).

0 1

0 1

det

trf

H

H

0 1

Isocontours of Response

Harris Detector: Workflow

Harris Detector: Workflow

Compute corner response R

Harris Detector: Workflow

Find points with large corner response: R>threshold

Harris Detector: Workflow

Take only the points of local maxima of R

Harris Detector: Workflow

Example: Gradient Covariances

Full imageDetail of image with gradient covar-

iance ellipses for 3 x 3 windows

from Forsyth & Ponce

Corners are where both eigenvalues are big

Example: Corner Detection (for camera calibration)

courtesy of B. Wilburn

Example: Corner Detection

courtesy of S. Smith

SUSAN corners

Harris Detector: Summary• Average intensity change in direction [u,v] can be

expressed as a bilinear form:

• Describe a point in terms of eigenvalues of H:measure of corner response

• A good (corner) point should have a large intensity change in all directions, i.e. R should be large positive

( , ) ,u

E u v u v Hv

2

1 2 1 2R k

Outline of Feature Detection

1. Compute the horizontal and vertical derivatives of the image Ix and Iy by convolving the original image with derivatives of Gaussians

2. Compute the three images corresponding to the outer products of these gradients. (The matrix H is symmetric, so only three entries are needed.)

3. Convolve each of these images with a larger Gaussian.4. Compute a scalar interest measure using one of the formulas

discussed above.5. Find local maxima above a certain threshold and report

them as detected feature point locations.

Adaptive Non-Maximal Suppression (ANMS)

• Local maxima• Response value should be significantly (10%)

larger than all of its neighbors within a radius (r)

(a) Strongest 250 (b) Strongest 500

Adaptive Non-Maximal Suppression (ANMS)

(a) Strongest 250 (b) Strongest 500

(c) ANMS 250, r = 24 (d) ANMS 500, r = 16

Invariance

Suppose you rotate the image by some angle– Will you still pick up the same features?

What if you change the brightness?

Scale?

Invariance

• Repeatability of feature detector: frequency with which keypoints are detected in one image are found

within ε (ε=1.5) pixels of the corresponding location in a transformed image

Scale Invariance

• Detect features at a variety of scales

• Multiple resolutions in a pyramid

• Matching in all possible levels

Multi-Scale Oriented Patches

Scale invariant detection

Suppose you’re looking for corners

Feature Detection

Key idea: find scale that gives local maximum of f– f is a local maximum in both position and scale– Common definition of f: Laplacian– Difference between two Gaussian filtered images

with different sigmas

Feature Detection

• Stable features in both location and scale• Laplacian of Gaussian (LoG)

• Difference of Gaussian2 2 2 2

2 21 2

1 2

1 2

2 2

2 2

1 1 1

2

x y x y

DoG G G e e

Slide from Tinne Tuytelaars

Lindeberg et al, 1996

Slide from Tinne Tuytelaars

Lindeberg et al., 1996

Scale-Space Feature Detection

Empirical number of levels in each octave = 31.6

SIFT Feature Detection

• Scale Invariant Feature Transform

Feature Detector

• Harris and DoG detectors tend to respond at complementary locations.

Sample image Harris response DoG response

Rotation Invariance and orientation estimation

• Dominant orientation– Average gradient within a region around the keypoint

(larger window than detection window to make it more reliable)

– Orientation histogram

Affine Invariance

• For wide baseline stereo matching or object recognition

Affine Invariance

• For a small enough patch, any continuous image wrapping can be well approximated by an affine deformation

• Eigenvalue analysis for Hessian matrix or auto-correlation: using the principle axis and ratios to fit the affine coordinate frame

1 1 2 2, , , x x

(Lindeberg and Garding 1997; Baumberg 2000; Mikolajczyk and Schmid 2004; Mikolajczyk, Tuytelaars, Schmid et al. 2005; Tuytelaars and Mikolajczyk 2007)

Affine Invariance

• Maximally Stable Extremal Region (MSER)– Matas, Chum, Urban et al. (2004)

• Algorithm – Binary regions by thresholding the image at all possible

gray levels– Region area changes as the threshold changes– D_area/D_t is minimal maximal stable

Keypoint Detection

• Very active area– (Xiao and Shah 2003; Koethe 2003; Carneiro and Jepson 2005; Kenney, Zuliani, and

Manjunath 2005; Bay, Tuytelaars, and Van Gool 2006; Platel, Balmachnova, Florack et al. 2006; Rosten and Drummond 2006

• Invariance to transformation such as scaling, rotations, noise, and blur

• These experimental results, code, and pointers to the surveyed papers can be found on their Web site at http://www.robots.ox.ac.uk/vgg/research/affine/

(Mikolajczyk, Tuytelaars, Schmid et al. 2005)

4.1.2 Feature Descriptor

• TBD

SIFT Feature

• Scale Invariant Feature Transform (SIFT)• Detect features that densely cover the image

over the full range of scales and locations

Distinctive Image Features from Scale-Invariant Keypoints, David G. Lowe

Points and Patches

• Two approaches to finding feature points and their correspondences – detect in one, then track

• For video sequence application, tracking, stereo…

– Detect independently, then match • For image stitching, recognition …

Keypoint Detection and Matching

• Four steps:– Feature detection– Feature description– Feature matching– Feature tracking