features
DESCRIPTION
Features. CSE 455, Winter 2010 February 1 , 2010. Review From Last Time. Measuring height. 5.4. 5. Camera height. 4. 3.3. 3. 2.8. 2. 1. The cross ratio. A Projective Invariant Something that does not change under projective transformations (including perspective projection). - PowerPoint PPT PresentationTRANSCRIPT
Features
CSE 455, Winter 2010February 1, 2010
Neel Joshi, CSE 455, Winter 20102
Review From Last Time
Neel Joshi, CSE 455, Winter 2010
Measuring height
1
2
3
4
55.4
2.8
3.3Camera height
Neel Joshi, CSE 455, Winter 20104
The cross ratio
A Projective Invariant Something that does not change under projective transformations
(including perspective projection)
P1
P2
P3
P4
1423
2413
PPPPPPPP
The cross-ratio of 4 collinear points
Can permute the point ordering• 4! = 24 different orders (but only 6 distinct values)
This is the fundamental invariant of projective geometry
1i
i
i
i ZYX
P
3421
2431
PPPPPPPP
Neel Joshi, CSE 455, Winter 20105
Monocular Depth Cues
Stationary Cues: Perspective Relative size Familiar size Aerial perspective Occlusion Peripheral vision Texture gradient
Neel Joshi, CSE 455, Winter 20106
Camera calibration
Goal: estimate the camera parameters Version 1: solve for projection matrix
ΠXx
1************
ZYX
wwywx
• Version 2: solve for camera parameters separately– intrinsics (focal length, principle point, pixel size)– extrinsics (rotation angles, translation)– radial distortion
1000100010001
1000000
010000100001
/100010001
3
2
1
333231
232221
131211
ttt
rrrrrrrrr
fΠ
Neel Joshi, CSE 455, Winter 20107
Calibration using a reference object Place a known object in the scene
identify correspondence between image and scene compute mapping from scene to image
Issues• must know geometry very accurately• must know 3D->2D correspondence
Neel Joshi, CSE 455, Winter 2010
Alternative: multi-plane calibration
Images courtesy Jean-Yves Bouguet, Intel Corp.
Advantage• Only requires a plane• Don’t have to know positions/orientations• Good code available online!
– Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/ – Matlab version by Jean-Yves Bouget:
http://www.vision.caltech.edu/bouguetj/calib_doc/index.html– Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/
Neel Joshi, CSE 455, Winter 20109
Today
Features
What are Features? Edges Corners Generally “interesting” parts of an image
All is Vanity, by C. Allan Gilbert, 1873-1929Readings
• M. Brown et al. Multi-Image Matching using Multi-Scale Oriented Patches, CVPR 2005
Neel Joshi, CSE 455, Winter 201010
Edge detection
Convert a 2D image into a set of curves Extracts salient features of the scene More compact than pixels
Neel Joshi, CSE 455, Winter 201011
Image gradient The gradient of an image:
The gradient direction is given by:
• how does this relate to the direction of the edge?The edge strength is given by the gradient magnitude
The gradient points in the direction of most rapid increase in intensity
Neel Joshi, CSE 455, Winter 201012
The Canny edge detector
original image (Lena)
Neel Joshi, CSE 455, Winter 201013
The Canny edge detector
norm of the gradient
Neel Joshi, CSE 455, Winter 201014
The Canny edge detector
thresholding
Neel Joshi, CSE 455, Winter 201015
The Canny edge detector
thinning(non-maximum suppression)
Neel Joshi, CSE 455, Winter 201016
Image matching
by Diva Sian
by swashford
Neel Joshi, CSE 455, Winter 201017
Harder case
by Diva Sian by scgbt
Neel Joshi, CSE 455, Winter 201018
Even harder case
“How the Afghan Girl was Identified by Her Iris Patterns” Read the story
Neel Joshi, CSE 455, Winter 2010
Harder still?
NASA Mars Rover images
Neel Joshi, CSE 455, Winter 2010
NASA Mars Rover imageswith SIFT feature matches
Figure by Noah Snavely
Answer below (look for tiny colored squares…)
Neel Joshi, CSE 455, Winter 2010
Can we do image matching with Edges?
Neel Joshi, CSE 455, Winter 201022
Image Matching
Neel Joshi, CSE 455, Winter 201023
Image Matching
Neel Joshi, CSE 455, Winter 2010
Invariant local features
Find features that are invariant to transformations geometric invariance: translation, rotation, scale photometric invariance: brightness, exposure, …
Feature Descriptors
Neel Joshi, CSE 455, Winter 201025
Advantages of local features
Locality features are local, so robust to occlusion and clutter
Distinctiveness: can differentiate a large database of objects
Quantity hundreds or thousands in a single image
Efficiency real-time performance achievable
Generality exploit different types of features in different situations
Neel Joshi, CSE 455, Winter 201026
More motivation…
Feature points are used for: Image alignment (e.g., mosaics) 3D reconstruction Motion tracking Object recognition Indexing and database retrieval Robot navigation … other
Neel Joshi, CSE 455, Winter 201027
What makes a good feature?
Snoop demo
Neel Joshi, CSE 455, Winter 201028
Want uniqueness
Look for image regions that are unusual Lead to unambiguous matches in other images
How to define “unusual”?
Neel Joshi, CSE 455, Winter 201029
Local measures of uniqueness
Suppose we only consider a small window of pixels What defines whether a feature is a good or bad candidate?
Slide adapted form Darya Frolova, Denis Simakov, Weizmann Institute.
Neel Joshi, CSE 455, Winter 201030
Feature detection
“flat” region:no change in all directions
“edge”: no change along
the edge direction
“corner”:significant
change in all directions
Local measure of feature uniqueness How does the window change when you shift it? Shifting the window in any direction causes a big change
Slide adapted form Darya Frolova, Denis Simakov, Weizmann Institute.
Neel Joshi, CSE 455, Winter 201031
Consider shifting the window W by (u,v)• how do the pixels in W change?• compare each pixel before and after by
summing up the squared differences (SSD)• this defines an SSD “error” of E(u,v):
Feature detection: the math
W
Neel Joshi, CSE 455, Winter 201032
Taylor Series expansion of I:
If the motion (u,v) is small, then first order approx is good
Plugging this into the formula on the previous slide…
Small motion assumption
Neel Joshi, CSE 455, Winter 201033
Consider shifting the window W by (u,v)• how do the pixels in W change?• compare each pixel before and after by
summing up the squared differences• this defines an “error” of E(u,v):
Feature detection: the math
W
Neel Joshi, CSE 455, Winter 201034
Feature detection: the math
This can be rewritten:
For the example above• You can move the center of the green window to anywhere on the
blue unit circle• Which directions will result in the largest and smallest E values?• We can find these directions by looking at the eigenvectors of H
Neel Joshi, CSE 455, Winter 201035
Quick eigenvalue/eigenvector review The eigenvectors of a matrix A are the vectors x that satisfy:
The scalar is the eigenvalue corresponding to x The eigenvalues are found by solving:
In our case, A = H is a 2x2 matrix, so we have
The solution:
Once you know , you find x by solving
Neel Joshi, CSE 455, Winter 201036
Feature detection: the mathThis can be rewritten:
Eigenvalues and eigenvectors of H• Define shifts with the smallest and largest change (E value)• x+ = direction of largest increase in E. • + = amount of increase in direction x+
• x- = direction of smallest increase in E. • - = amount of increase in direction x+
x-
x+
Neel Joshi, CSE 455, Winter 201037
Feature detection: the math
How are +, x+, -, and x+ relevant for feature detection?• What’s our feature scoring function?
Neel Joshi, CSE 455, Winter 201038
Feature detection: the math
How are +, x+, -, and x+ relevant for feature detection?• What’s our feature scoring function?
Want E(u,v) to be large for small shifts in all directions• the minimum of E(u,v) should be large, over all unit vectors [u v]• this minimum is given by the smaller eigenvalue (-) of H
Neel Joshi, CSE 455, Winter 201039
Feature detection summary
Here’s what you do• Compute the gradient at each point in the image• Create the H matrix from the entries in the gradient• Compute the eigenvalues. • Find points with large response (- > threshold)• Choose those points where - is a local maximum as features
Neel Joshi, CSE 455, Winter 201040
Feature detection summary
Here’s what you do• Compute the gradient at each point in the image• Create the H matrix from the entries in the gradient• Compute the eigenvalues. • Find points with large response (- > threshold)• Choose those points where - is a local maximum as features
Neel Joshi, CSE 455, Winter 201041
The Harris operator
- is a variant of the “Harris operator” for feature detection
• The trace is the sum of the diagonals, i.e., trace(H) = h11 + h22
• Very similar to - but less expensive (no square root)• Called the “Harris Corner Detector” or “Harris Operator”• Lots of other detectors, this is one of the most popular
Neel Joshi, CSE 455, Winter 201042
The Harris operator
Harris operator
Neel Joshi, CSE 455, Winter 201043
Harris detector example
Neel Joshi, CSE 455, Winter 201044
f value (red high, blue low)
Neel Joshi, CSE 455, Winter 201045
Threshold (f > value)
Neel Joshi, CSE 455, Winter 201046
Find local maxima of f
Neel Joshi, CSE 455, Winter 201047
Harris features (in red)
Neel Joshi, CSE 455, Winter 201048
Infinite Images
Neel Joshi, CSE 455, Winter 201049
Image Analogies