cs-f441: selected topics from computer science (deep ... › dl › lecture-06-kt-cv.pdfcs-f441:...
TRANSCRIPT
CS-F441: SELECTED TOPICS FROM COMPUTER
SCIENCE (DEEP LEARNING FOR NLP & CV)
Lecture-KT-08: Canny Edge Detector, Feature Engineering
Dr. Kamlesh Tiwari,Assistant Professor,
Department of Computer Science and Information Systems,BITS Pilani, Rajasthan-333031 INDIA
Oct 30, 2019 (Campus @ BITS-Pilani July-Dec 2019)
Recap: Sharpening Filter
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 2 / 31
Recap: Sharpening
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 3 / 31
Recap: Image gradient
The gradient of an image: Of = [ ∂f∂x ,
∂f∂y ]
Points in the direction of most rapid increase in intensity
where strength is ||Of || =√( ∂f∂x )
2 + ( ∂f∂y )
2
and direction is θ = tan−1( ∂f∂y /
∂f∂x )
Associative property of convolutionddx
(f ∗ h) = f ∗ ddx
h
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 4 / 31
Recap: Edges
An edge is a place of rapid change in the image intensity function
How to differentiate a digital image (filter ?)
∂f∂x
[x , y ] = F [x + 1, y ]− F [x , y ]
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 5 / 31
Recap: Roberts
Simple approximation to the first derivative.
It makes the edge point only, not the information about the edgeorientation.It works well in binary images.The Roberts method finds edges using the Roberts approximationto the derivative.√
(I(i , j)− I(i − 1, j − 1))2 + (I(i , j − 1)− I(i − 1, j))2[0 1−1 0
] [1 00 −1
]
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 6 / 31
Recap: Effects of Noise
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 7 / 31
Recap: Prewit Operator
Performs smoothening (by averaging) and then finds the magnitude ofderivatives and applies threshold on them.
The mask used is as below
hx =
1 0 −11 0 −11 0 −1
hy =
1 1 10 0 0−1 −1 −1
Bigger size mask is
hx =
−3 −1 1 3−3 −1 1 3−3 −1 1 3−3 −1 1 3
hy =
3 3 3 31 1 1 1−1 −1 −1 −1−3 −3 −3 −3
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 8 / 31
Recap: Sobel OperatorCommon approximation of derivative of Gaussian
Edge magnitude is√
s2x + s2
y and the direction is tan−1(sy/sx)
Edge is detected by using threshold
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 9 / 31
Recap: Non-maximum supression
Check if pixel is local maximum along gradient direction
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 10 / 31
Recap: Non-maximum supression
Check if pixel is local maximum along gradient direction
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 10 / 31
Recap: Non-maximum supression
Check if pixel is local maximum along gradient direction
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 10 / 31
Recap: Non-maximum supression
Check if pixel is local maximum along gradient direction
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 10 / 31
Recap: Laplacian of Gaussian (Marr-Hildreth)It is high-pass, and non-directional filter. Uses second derivatives 0 −1 0
−1 4 −10 −1 0
−1 −1 −1−1 8 −1−1 −1 −1
Apply LoG to remove noise.
0 0 1 0 00 1 2 1 01 2 −16 2 10 1 2 1 00 0 1 0 0
1 Detect all zero crossings{+,−}, {−,+}, {+,0,−},{−,0,+}
2 Use threshold to keep onlystrong ones
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 11 / 31
Canny Edge Detector [1/3]
Standard edge detector: Maximizes probability of detecting real edgeswhile minimizing the probability of false detection of non-edge points.
Five separate steps:
1. Smoothing: Blurring of the image to remove noise. (by applying aGaussian filter). The kernel of a Gaussian filter with a standarddeviation of σ = 1.4 is given by
1159
2 4 5 4 24 9 12 9 45 12 15 12 54 9 12 9 42 4 5 4 2
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 12 / 31
Canny Edge Detector [1/3]
Standard edge detector: Maximizes probability of detecting real edgeswhile minimizing the probability of false detection of non-edge points.
Five separate steps:
1. Smoothing: Blurring of the image to remove noise. (by applying aGaussian filter). The kernel of a Gaussian filter with a standarddeviation of σ = 1.4 is given by
1159
2 4 5 4 24 9 12 9 45 12 15 12 54 9 12 9 42 4 5 4 2
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 12 / 31
Canny Edge Detector [2/3]
2. Finding gradients: The edges should be marked where thegradients of the image has large magnitudes (by applyingSobel-operator).
3. Non-maximum suppression: Only local maxima should bemarked as edges.
1 Round the gradient direction to nearest 45◦, corresponding to theuse of an 8-connected neighborhood.
2 Compare the edge strength of the current pixel with the edgestrength of the pixel in the positive and negative gradient direction.i .e. if the gradient direction is north (theta = 90◦), compare with thepixels to the north and south.
3 If the edge strength of the current pixel is largest; preserve thevalue of the edge strength. If not, suppress (i .e. remove) the value.
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 13 / 31
Canny Edge Detector [2/3]
2. Finding gradients: The edges should be marked where thegradients of the image has large magnitudes (by applyingSobel-operator).
3. Non-maximum suppression: Only local maxima should bemarked as edges.
1 Round the gradient direction to nearest 45◦, corresponding to theuse of an 8-connected neighborhood.
2 Compare the edge strength of the current pixel with the edgestrength of the pixel in the positive and negative gradient direction.i .e. if the gradient direction is north (theta = 90◦), compare with thepixels to the north and south.
3 If the edge strength of the current pixel is largest; preserve thevalue of the edge strength. If not, suppress (i .e. remove) the value.
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 13 / 31
Canny Edge Detector [3/3]
4. Double thresholding: Edge pixels stronger than the highthreshold are marked as strong; edge pixels weaker than the lowthreshold are suppressed and edge pixels between the twothresholds are marked as weak. Strong edges are white, whileweak edges are grey. Edges with a strength below boththresholds are suppressed
5. Edge tracking by hysteresis: Final edges are determined bysuppressing all edges that are not connected to a very certain(strong) edge. Strong edges are interpreted as certain edges, andare included in the final edge image. Weak edges are included ifand only if they are connected to strong edges.
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 14 / 31
Canny Edge Detector [3/3]
4. Double thresholding: Edge pixels stronger than the highthreshold are marked as strong; edge pixels weaker than the lowthreshold are suppressed and edge pixels between the twothresholds are marked as weak. Strong edges are white, whileweak edges are grey. Edges with a strength below boththresholds are suppressed
5. Edge tracking by hysteresis: Final edges are determined bysuppressing all edges that are not connected to a very certain(strong) edge. Strong edges are interpreted as certain edges, andare included in the final edge image. Weak edges are included ifand only if they are connected to strong edges.
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 14 / 31
Canny Edge Detector
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 15 / 31
Humans are the best
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 16 / 31
General Object Detection Pipeline
1 Image acquisition2 Enhancement3 Feature Extraction4 Feature Matching5 Decision
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 17 / 31
Desired Properties of Feature
1 Repeatability: The same feature can be found in several imagesdespite geometric and photometric transformations
2 Saliency: Each feature has a distinctive description3 Compactness and efficiency: Many fewer features than image
pixels4 Locality: A feature occupies a relatively small area of the image;
robust to clutter and occlusion
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 18 / 31
What feature you would you choose?
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 19 / 31
Local measures of uniquenessSuppose we only consider a small window of pixelsWhat defines whether a feature is a good or bad candidate?
Measure feature uniqueness using How does the window changewhen you shift it?
flat region: no change in all directionsedge: no change along the edge directioncorner: significant change in all directions 1
1Slide adapted from Darya Frolova, Denis Simakov, Weizmann Institute
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 20 / 31
What is the change?
If we shift the window W by (u, v) then SSD is
E(u, v) =∑
(x ,y)∈W
[I(x + u, y + v)− I(x , y)]2
Recall Taylor series expansion
I(x + u, y + v) = I(x , y) +∂I∂x
u +∂I∂y
v + ...
For small u and v we may neglect higher order terms so
E(u, v) =∑
(x ,y)∈W
[Ixu + Iyv ]2 =∑
(x ,y)∈W
[[
Ix Iy] [ u
v
]]2
where Ix = ∂I∂x and Iy = ∂I
∂y
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 21 / 31
Change continued...
E(u, v) =∑
(x ,y)∈W
[[
Ix Iy] [ u
v
]]2
E(u, v) =∑
(x ,y)∈W
[[
u v] [ I2
x Ix IyIy Ix I2
y
] [uv
]]2
E(u, v) =∑
(x ,y)∈W
[[
u v]
H[
uv
]]2
Move the window on blue circle.Which directions will have maximalE? Does eigenvectors of H helpsuggesting?
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 22 / 31
Change continued...
E(u, v) =∑
(x ,y)∈W
[[
Ix Iy] [ u
v
]]2
E(u, v) =∑
(x ,y)∈W
[[
u v] [ I2
x Ix IyIy Ix I2
y
] [uv
]]2
E(u, v) =∑
(x ,y)∈W
[[
u v]
H[
uv
]]2
Move the window on blue circle.Which directions will have maximalE? Does eigenvectors of H helpsuggesting?
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 22 / 31
Change continued...
E(u, v) =∑
(x ,y)∈W
[[
Ix Iy] [ u
v
]]2
E(u, v) =∑
(x ,y)∈W
[[
u v] [ I2
x Ix IyIy Ix I2
y
] [uv
]]2
E(u, v) =∑
(x ,y)∈W
[[
u v]
H[
uv
]]2
Move the window on blue circle.Which directions will have maximalE? Does eigenvectors of H helpsuggesting?
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 22 / 31
Change continued...
E(u, v) =∑
(x ,y)∈W
[[
Ix Iy] [ u
v
]]2
E(u, v) =∑
(x ,y)∈W
[[
u v] [ I2
x Ix IyIy Ix I2
y
] [uv
]]2
E(u, v) =∑
(x ,y)∈W
[[
u v]
H[
uv
]]2
Move the window on blue circle.Which directions will have maximalE?
Does eigenvectors of H helpsuggesting?
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 22 / 31
Change continued...
E(u, v) =∑
(x ,y)∈W
[[
Ix Iy] [ u
v
]]2
E(u, v) =∑
(x ,y)∈W
[[
u v] [ I2
x Ix IyIy Ix I2
y
] [uv
]]2
E(u, v) =∑
(x ,y)∈W
[[
u v]
H[
uv
]]2
Move the window on blue circle.Which directions will have maximalE? Does eigenvectors of H helpsuggesting?
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 22 / 31
Eigenvalue/eigenvector review
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 23 / 31
Feature Detection
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 24 / 31
Feature Detection
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 25 / 31
Feature PointsDo the following
Compute gradient of each point in the imageCreate the H matrix using gradientsCompute EigenvaluesFind points with large response (λ > threshold)Choose those points where λ is a local maximum as feature point
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 26 / 31
Corner Response Function
f =λ1λ2
λ1 + λ2
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 27 / 31
Harris Operator
Do the following:1 Compute E for each image window to get their cornerness scores.2 Find points whose surrounding window gave large corner response
(f > threshold)3 Take the points of local maxima, i.e., perform non-maximum
suppression
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 28 / 31
Example of Harris Feature
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 29 / 31
Properties of the Harris corner detector
Rotation Invarient? YESScale Invarient? NO
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 30 / 31
Thank You!
Thank you very much for your attention2!
Queries ?
2https://www.cs.cornell.edu/courses/cs6670/2011sp/lectures/lec02 filter.pdf
STCS-DL4NLP&CV (CS-F441) Campus @ BITS-Pilani Lecture-KT-08 (Oct 30, 2019) 31 / 31