ee 576 - image featuresisl.ee.boun.edu.tr/courses/ee576/lectures/sunum/featurespres.pdf · outline...
TRANSCRIPT
Outline
Introduction
EE 576 - Image Features
H.I. Bozma
Electric Electronic EngineeringBogazici University
March 11, 2020
H.I. Bozma EE 576 - Image Features
Outline
Introduction
IntroductionLocal FeaturesLocal Structure MatrixHarris Corner DetectorKLT DetectorGlobal Features
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Features
◮ Local: Derived directly parts of the image◮ Results of filtering operations (edges)◮ Corners◮ KLT◮ SIFT
◮ Global: Derived from the whole of the image◮ Histogram◮ Bag-of-words◮ GIST
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Edges
Changes in image brightness is to capture important events andchanges in properties of the world. In particular, edges encode:
◮ Changes in material properties
◮ Discontinuities in depth,
◮ Discontinuities in surface orientation,
◮ Variations in scene illumination.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Edges
◮ Intensity: I =√
R2x + R2
y
◮ Angle: θ = tan−1 Ry
Rx
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Sobel
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Canny
1. Apply Gaussian filter to smooth the image in order to removethe noise
2. Find the intensity gradients of the image
3. Apply non-maximum suppression to get rid of spuriousresponse to edge detection
4. Apply double threshold to determine potential edges
5. Suppress all the edges that are weak and not connected tostrong edges.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Hough Transform
Assume edge detection & thresholding of the edge magnitudeConsider a pixel x = [x1, x2]
A straight line passing thruax1 + b = x2There are many lines passing thru x
∃ line, many x that satisfy the line equation
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Hough Transform (cont)
Consider x as parameters and a, b as variables.
b = −x1a+ x2 (1)
A line in (a, b) space parametrized by xAnother x gives another line
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Hough Transform (cont)
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Hough Transform (cont)
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Corners
◮ Corners – Image features characterized by their high intensitychanges in the horizontal and vertical directions.
◮ Object is square =⇒ Its four corners are very good interestpoints.
◮ Corners are used in shape analysis and motion analysis.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Corner
Figure: Left: An edge, Right: A corner.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Edges vs Corners
◮ Edges are locations where the variation of I in a certaindirection is high, while the variation in the orthogonaldirection is low. For example, in an edge oriented along the x2axis, ∂I
∂x1is high while ∂I
∂x2is low.
◮ Corners are local image features characterized by locationswhere variations of intensity function in both x1 and x2directions are high – both ∂I
∂x1and ∂I
∂x2are high.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Corner Detection
◮ Directly from the grayscale images or
◮ As the intersection of digital contours (usually lines) if thesecontours have been found.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Corner Detectors
The two most commonly used corner detectors to-date are:
◮ the Harris corner detector
◮ the Kanade-Lucas-Tomasi (KLT) corner detector
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Local Structure Matrix C
A common property : Both operate with C
C =
[
∂2I∂2x1
∂2I∂x2∂x1
∂2I∂x2∂x1
∂2I∂2x2
]
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Practical Computation of C
The partial derivatives – Replaced, respectively, by theconvolutions with a 1D horizontal and vertical discrete masks.
◮ The derivatives of the intensity function are first calculated ineach point.
◮ If necessary, the image is smoothed before taking thederivatives
◮ Then, C is obtained.
◮ Finally, each of the entries is smoothed (integrated) byGaussian filter of selected size σ.
◮ Often, a simple box (averaging) filter is used instead of theGaussian.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Eigenvalues of C
◮ λ1 and λ2 be the two eigenvalues of the matrix C.
◮ As C is symmetric and positive semi-definite, both theeigenvalues are non-negative.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Eigenvalues and Eigenvectors
◮ In a uniform and homogeneous region, λ1, λ2 = 0,
◮ At a step edge, λ1 > λ2 = 0. The corresponding eigenvectorfor λ1 is associated with the direction that is orthogonal tothe edge.
◮ At a corner, λ1 ≥ λ2 > 0. The larger are the values of λ1 andλ2, the higher are the contrasts of the edges orthogonal to thedirections of the corresponding eigenvectors.
◮ Hence, the eigenvectors encode the edge directions whileeigenvalues encode the edge magnitudes.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Corner Detection II
In summary, a corner should be marked at a location where thesmaller eigenvalue λ2 is large enough.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Algorithm
In Harris corner detector, the local structure matrix is smoothed bya Gaussian filter.
CH = wG (σ) ∗[
∂2I∂2x1
∂2I∂x2∂x1
∂2I∂x2∂x1
∂2I∂2x2
]
wG is an isotropic Gaussian filter with standard deviation σ
The operation ∗ denotes convolution.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
◮ A measure of the corner response at each pixel coordinates xis defined by:
h(x) = det(CH(x))− κtrace(CH(x))2
◮ κ is a parameter. Note that κ < 0.25, then h(x) > 0
◮ A corner is detected if h(x) > τ .
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Additional Considerations
◮ To prevent too many corner features occurring togetherclosely, a non-maximal suppression process on the cornerresponse image
◮ Thresholding process.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Harris Detector Parameters
1. κ
2. The radius, d, of the neighbourhood region for suppressingweak corners,
3. The threshold value τ
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Mathematical Formulation
Let λ1 = λ, then λ2 = αλ where 0 ≤ α ≤ 1
h(x) = λ1λ2 − κ(λ1 + λ2)2
= αλ2 − κ(1 + α)2λ2
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
◮ If h(x) ≥ 0 → κ ≤ α
(1+α)2.
◮ Two equally strong edges → α = 1 → κ = 14 .
◮ If one of the edges is very weak – α << 1, κ < α << 1.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
κ Parameter
◮ κ - A measure of the lower bound for the ratio between themagnitude of the weaker edge and that of the stronger edge.
◮ Larger value of κ – A less sensitive detector and yields lesscorners
◮ Smaller value of κ – A more sensitive detector and yields morecorners.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Corners detected by the Harris corner detector
Figure: σ = 1, τ = 1500 (threshold), and d = 5 (neighbourhood radiusfor non-maximal suppression). Left: κ = 0.1, Right= κ = 0.15.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Two Parameters
1. The threshold on λ2, denoted by τ ,
2. The linear size of a square window (neighbourhood) D.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
KLT Algorithm
1. Compute ∂I∂x1
and ∂I∂x2
on the image.
2. For each image point p:
2.1 Form the matrix C over a D × D neighbourhood N of p asfollows:
CK =∑
y∈N
[
∂2I
∂2x1(y) ∂
2I∂x2∂x1
(y)∂2I
∂x2∂x1(y) ∂
2I∂2x2
(y)
]
2.2 Compute λ2 - the smaller eigenvalue of C;2.3 If λ2 > τ , save the p into a list, L.
3. Sort L in decreasing order of λ2
4. Scan the sorted list from top to bottom. For each currentpoint, p, delete all points appearing further in the list whichbelong to the neighbourhood of p.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Output of KLT Corner Detector
◮ A list of corner points that have λ2 > τ
◮ the neighbourhood N of these points do not overlap.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
◮ No simple criteria for determining the size of d – commonlytaken to be 2 ≤ d ≤ 10.
◮ The summation operation over the neighbourhood N →Uniform smoothing for the image gradients around eachimage point.
◮ The parameters τ and d for the KLT corner detector –similar to the parameter κ and the parameter d used in thenon-maximal suppression of the Harris corner detector.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Corners detected by the KTL detector
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Corners detected by Left: Harris, Right: the KTL detector.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
SIFT Keypoint Detector [1]
◮ SIFT features → Represent local image measurements:◮ Relatively invariant to image translation, scaling and rotation◮ Partially invariant to changes in illumination and local image
deformations.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
SIFT Algorithm
A number of Gaussian filters whose standard deviations {σ1, σ2}differ by a fixed scale factor. That is, σj+1 = kσjk - A constant scalar set to
√2
The input image I → Convolved with a number of Gaussian filtersThe convolutions yield a small number of smoothed images,denoted by G (x ;σ1),G (x ;σ2), · · ·
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Smoothing with varying σ
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
SIFT Algorithm - 2
The adjacent smoothed images – Subtracted to yield a smallnumber (3 or 4) of DoG (Difference-of-Gaussian) imagesD(x ; j) = G (x ;σj+1)− G (x ;σj)Repeat on the subsampled smoothed images → A number of DoGimages over the scale space.Each point in these DoG images is then examined.Keypoint: Marked at a location where the point is a localminimum or maximum of its 8 neighbours on the same scale andof its 9 neighbours on the scales above and below.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
DOGs
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
SIFT Algorithm - 3
Examine the keypoints for possible elimination via checking forpoints that have poorly determined locations but have high edgeresponses.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
SIFT Algorithm - 4 Elimination Stage
◮ The 2× 2 Hessian matrix (The second partial derivatives) ofthe local image intensity around each keypoint is computed as:
H =
[
Dx1x1 I Dx2x1 I
Dx1x2 I Dx2x2 I
]
◮ For poorly defined peaks in the DoG function, the principalcurvature across the edge would be much larger than theprincipal curvature along it which can be computed via solvingfor the eigenvalues of H.
◮ The eigenvalues of H are proportional to the principalcurvatures of D. It turns out that the ratio r of the twoeigenvalues is sufficient for SIFT’s purposes - where r isdefined as the ratio of the larger eigenvalue to that of thesmaller eigenvalue.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
SIFT Algorithm - 4 Elimination Stage (cont)
◮ The trace of H Dx1x1 I + Dx2x2 I , gives us the sum of the twoeigenvalues
◮ The determinant Dx1x1 IDx2x2 I − Dx2x1 IDx1x2 I yields theproduct.
◮ R = Tr(H)2
Det(H) = (r+1)2
r
◮ R is minimum when the eigenvalues are equal to each other→ The higher the absolute difference between the twoeigenvalues, which is equivalent to a higher absolute differencebetween the two principal curvatures of D, the higher thevalue of R.
◮ Hence, R >(rth+1)2
rth=⇒ Poor localization and hence reject.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
SIFT Algorithm - 5
• A set of keypoint locations• The surrounding intensity and gradient information around eachkeypoint – Examined for the coding of the SIFT descriptor at thatkeypoint.• Histograms computed in a 16 x 16 region around the keypointsuch that each histogram contains samples from a 4 x 4 subregionof the original neighborhood region.• The image gradients are computed and classified into 8orientations for a 4× 4 histogram array giving a SIFT descriptor of128 elements long for each keypoint
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Encoding Keypoints
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Comparative Analysis
◮ Unlike the Harris and KLT corner detectors, the SIFTkeypoints do not often coincide at corner features.
◮ However, the SIFT descriptor of each keypoint does provideuseful information for wide baseline stereo matching and forobject recognition.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Histogram
Figure: Images of an object from two different viewpoints and theirhistograms.
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Bag-of-Words [2]
◮ Image representation - A collection of words, disregardinggrammar and even word order.
◮ Words – Obtain many samples and then cluster to get asmaller set
◮ Frequency of occurrence of each word - Used as a feature fortraining a classifier
◮ To vector quantize the descriptors into clusters which will bethe visual ‘words’ for text retrieval → Vocabulary
◮ An image → As an histogram of words
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Bag-of-Words
→
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Spatial Envelope [3]
naturalnessopennessroughnessexpansionruggednessSpectral Spatial domain
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Hull Census Transform (HCT)
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
Bubble Space [4]
Robot base
x = [c α]T
Base space X = R2 × S1
Pan-tilt movement:
f = [f1f2]T
Camera state space F ⊂ S2
Bubble space B = X × FRobot’s base + its viewing directions
H.I. Bozma EE 576 - Image Features
Outline
Introduction
Local Features
Local Structure Matrix
Harris Corner Detector
KLT Detector
Global Features
D. G. Lowe, “Object recognition from local scale-invariantfeatures,” in ICCV, 1999, pp. 1150–1157.
J. Sivic and A. Zisserman, “Efficient visual search of videoscast as text retrieval,” Pattern Analysis and Machine
Intelligence, IEEE Transactions on, vol. 31, no. 4, pp.591–606, April 2009.
A. Torralba, K. P. Murphy, W. T. Freeman, and M. A. Rubin,“Context-based vision system for place and objectrecognition,” Computer Vision, IEEE Int. Conf. on, vol. 1, p.273, 2003.
O. Erkent and H. I. Bozma, “Bubble Space and PlaceRepresentation in Topological Maps,” The Int. J. of Rob. Res.,vol. 32, no. 6, pp. 671 – 688, 2013.
H.I. Bozma EE 576 - Image Features