lecture 6: finding features (part 1/2)vision.stanford.edu/.../lectures/lecture6_detectors...fei-fei...
TRANSCRIPT
Lecture 6 - Fei-Fei Li
Lecture6:FindingFeatures(part1/2)
Dr.JuanCarlosNieblesStanfordAILab
ProfessorFei-FeiLiStanfordVisionLab
10-Oct-161
Lecture 6 - Fei-Fei Li
Whatwewilllearntoday?
â˘âŻ LocalinvariantfeaturesââŻMoOvaOonâ⯠Requirements,invariances
â˘âŻ KeypointlocalizaOonââŻHarriscornerdetector
â˘âŻ ScaleinvariantregionselecOonââŻAutomaOcscaleselecOonââŻDifference-of-Gaussian(DoG)detector
â˘âŻ SIFT:animageregiondescriptor
10-Oct-162
Nextlecture(#7)
Lecture 6 - Fei-Fei Li
Whatwewilllearntoday?
â˘âŻ LocalinvariantfeaturesââŻMoOvaOonâ⯠Requirements,invariances
â˘âŻ KeypointlocalizaOonââŻHarriscornerdetector
â˘âŻ ScaleinvariantregionselecOonââŻAutomaOcscaleselecOonââŻDifference-of-Gaussian(DoG)detector
â˘âŻ SIFT:animageregiondescriptor
10-Oct-163
Somebackgroundreading:RickSzeliski,Chapter4.1.1;DavidLowe,IJCV2004
Lecture 6 - Fei-Fei Li
by Diva Sian
by swashford
Slid
e cr
edit
: St
eve
Seit
z
10-Oct-165
Imagematching:achallengingproblem
Lecture 6 - Fei-Fei Li
AnswerBelow(LookforOnycoloredsquares)
NASAMarsRoverimageswithSIFTfeaturematches(FigurebyNoahSnavely) Sl
ide
cred
it:
Stev
e Se
itz
10-Oct-168
Lecture 6 - Fei-Fei Li
MoOvaOonforusinglocalfeaturesâ˘âŻ GlobalrepresentaOonshavemajorlimitaOonsâ˘âŻ Instead,describeandmatchonlylocalregionsâ˘âŻ Increasedrobustnessto
â⯠Occlusions
â⯠ArOculaOon
â⯠Intra-categoryvariaOons
θqĎ
dq
Ď
θ
d
10-Oct-169
Lecture 6 - Fei-Fei Li
GeneralApproachN
pix
els
N pixels
Similarity measure Af
e.g. color
Bf
e.g. color
B1B2
B3A1
A2 A3
Tffd BA <),(
1. Find a set of distinctive key- points
3. Extract and normalize the region content
2. Define a region around each keypoint
4. Compute a local descriptor from the normalized region
5. Match local descriptors
Slid
e cr
edit
: Ba
stia
n Le
ibe
10-Oct-1610
Lecture 6 - Fei-Fei Li
CommonRequirementsâ˘âŻ Problem1:
â⯠Detectthesamepointindependentlyinbothimages
No chance to match!
We need a repeatable detector! Slid
e cr
edit
: D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1611
Lecture 6 - Fei-Fei Li
CommonRequirementsâ˘âŻ Problem1:
â⯠Detectthesamepointindependentlyinbothimages
â˘âŻ Problem2:â⯠Foreachpointcorrectlyrecognizethecorrespondingone
We need a reliable and distinctive descriptor!
?
Slid
e cr
edit
: D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1612
Lecture 6 - Fei-Fei Li
Invariance:GeometricTransformaOons
Slid
e cr
edit
: St
eve
Seit
z
10-Oct-1613
Lecture 6 - Fei-Fei Li
LevelsofGeometricInvariance
Slid
e cr
edit
: Ba
stia
n Le
ibe
10-Oct-1614
CS131 CS231a
Lecture 6 - Fei-Fei Li
Invariance:PhotometricTransformaOons
â˘âŻ OfenmodeledasalineartransformaOon:â⯠Scaling+Offset
Slid
e cr
edit
: Ti
nne
Tuyt
elaa
rs
10-Oct-1615
Lecture 6 - Fei-Fei Li
Requirementsâ˘âŻ RegionextracOonneedstoberepeatableandaccurate
â⯠InvarianttotranslaOon,rotaOon,scalechangesâ⯠Robustorcovarianttoout-of-plane(ďż˝affine)transformaOonsâ⯠RobusttolighOngvariaOons,noise,blur,quanOzaOon
â˘âŻ Locality:Featuresarelocal,thereforerobusttoocclusionandcluier.
â˘âŻ QuanOty:Weneedasufficientnumberofregionstocovertheobject.
â˘âŻ DisOncOveness:TheregionsshouldcontainâinteresOngâstructure.
â˘âŻ Efficiency:Closetoreal-Omeperformance.
Slid
e cr
edit
: Ba
stia
n Le
ibe
10-Oct-1616
Lecture 6 - Fei-Fei Li
ManyExisOngDetectorsAvailable
â˘âŻ Hessian&Harris [Beaudetâ78],[Harrisâ88]â˘âŻ Laplacian,DoG [Lindebergâ98],[Loweâ99]â˘âŻ Harris-/Hessian-Laplace [Mikolajczyk&Schmidâ01]â˘âŻ Harris-/Hessian-Affine [Mikolajczyk&Schmidâ04]â˘âŻ EBRandIBR [Tuytelaars&VanGoolâ04]â˘âŻ MSER [Matasâ02]â˘âŻ SalientRegions [Kadir&Bradyâ01]â˘âŻ OthersâŚ
â˘âŻ Thosedetectorshavebecomeabasicbuildingblockformanyrecentapplica8onsinComputerVision.
Slid
e cr
edit
: Ba
stia
n Le
ibe
10-Oct-1617
Lecture 6 - Fei-Fei Li
KeypointLocalizaOon
â˘âŻ Goals:
â⯠RepeatabledetecOonâ⯠PreciselocalizaOonâ⯠InteresOngcontent
�Lookfortwo-dimensionalsignalchanges
Slid
e cr
edit
: Ba
stia
n Le
ibe
10-Oct-1618
Lecture 6 - Fei-Fei Li
FindingCorners
â˘âŻ Keyproperty:â⯠Intheregionaroundacorner,imagegradienthastwoormoredominantdirecOons
â˘âŻ Cornersarerepeatableanddis8nc8ve
C.Harris and M.Stephens. "A Combined Corner and Edge Detector.â Proceedings of the 4th Alvey Vision Conference, 1988.
Slid
e cr
edit
: Sv
etla
na L
azeb
nik
10-Oct-1619
Lecture 6 - Fei-Fei Li
CornersasDisOncOveInterestPointsâ˘âŻ Designcriteria
ââŻWeshouldeasilyrecognizethepointbylookingthroughasmallwindow(locality)
â⯠Shifingthewindowinanydirec8onshouldgivealargechangeinintensity(goodlocaliza8on)
âedgeâ: no change along the edge direction
âcornerâ: significant change in all directions
âflatâ region: no change in all directions Sl
ide
cred
it:
Alyo
sha
Efro
s
10-Oct-1620
Lecture 6 - Fei-Fei Li
HarrisDetectorFormulaOon
â˘âŻ Changeofintensityfortheshif[u,v]:E(u,v) = w(x, y) I (x +u, y + v)â I (x, y)"# $%
2
x ,yâ
Intensity Shifted intensity
Window function
or Window function w(x,y) =
Gaussian 1 in window, 0 outside
Slid
e cr
edit
: Ri
ck S
zelis
ki
10-Oct-1621
Lecture 6 - Fei-Fei Li
HarrisDetectorFormulaOonâ˘âŻ Thismeasureofchangecanbeapproximatedby:
whereMisa2�2matrixcomputedfromimagederivaOves:
âĽâŚ
â¤â˘âŁ
âĄâ
vu
MvuvuE ][),(
M
Sumoverimageregionâtheareawearecheckingforcorner
Gradient with respect to x, times gradient with respect to y
2
2,( , ) x x y
x y x y y
I I IM w x y
I I I⥠â¤
= ⢠âĽâ˘ âĽâŁ âŚ
â
Slid
e cr
edit
: Ri
ck S
zelis
ki
10-Oct-1622
Lecture 6 - Fei-Fei Li
HarrisDetectorFormulaOon
Ix Image I IxIy Iy
whereMisa2�2matrixcomputedfromimagederivaOves:
M
Sumoverimageregionâtheareawearecheckingforcorner
Gradient with respect to x, times gradient with respect to y
2
2,( , ) x x y
x y x y y
I I IM w x y
I I I⥠â¤
= ⢠âĽâ˘ âĽâŁ âŚ
â
Slid
e cr
edit
: Ri
ck S
zelis
ki
10-Oct-1623
Lecture 6 - Fei-Fei Li
WhatDoesThisMatrixReveal?â˘âŻ First,letâsconsideranaxis-alignedcorner:
â˘âŻ Thismeans:â⯠DominantgradientdirecOonsalignwithxoryaxisâ⯠IfeitherÎťiscloseto0,thenthisisnotacorner,solookforlocaOonswherebotharelarge.
â˘âŻ Whatifwehaveacornerthatisnotalignedwiththeimageaxes?
âĽâŚ
â¤â˘âŁ
âĄ=
âĽâĽâŚ
â¤
â˘â˘âŁ
âĄ=
ââââ
2
12
2
00Îť
Îť
yyx
yxx
IIIIII
M
Slid
e cr
edit
: D
avid
Jac
obs
10-Oct-1624
Lecture 6 - Fei-Fei Li
WhatDoesThisMatrixReveal?â˘âŻ First,letâsconsideranaxis-alignedcorner:
â˘âŻ Thismeans:â⯠DominantgradientdirecOonsalignwithxoryaxisâ⯠IfeitherÎťiscloseto0,thenthisisnotacorner,solookforlocaOonswherebotharelarge.
â˘âŻ Whatifwehaveacornerthatisnotalignedwiththeimageaxes?
âĽâŚ
â¤â˘âŁ
âĄ=
âĽâĽâŚ
â¤
â˘â˘âŁ
âĄ=
ââââ
2
12
2
00Îť
Îť
yyx
yxx
IIIIII
M
Slid
e cr
edit
: D
avid
Jac
obs
10-Oct-1625
Lecture 6 - Fei-Fei Li
GeneralCase
â˘âŻ SinceMissymmetric,wehave
â˘âŻ WecanvisualizeMasanellipsewithaxislengthsdeterminedbytheeigenvaluesandorientaOondeterminedbyR
RRM âĽâŚ
â¤â˘âŁ
âĄ= â
2
11
00Îť
Îť
Direction of the slowest change
Direction of the fastest change
(�max)-1/2
(�min)-1/2
adap
ted
from
Dar
ya F
rolo
va,
Den
is S
imak
ov
(Eigenvalue decomposition)
10-Oct-1626
Lecture 6 - Fei-Fei Li
InterpreOngtheEigenvaluesâ˘âŻ ClassificaOonofimagepointsusingeigenvaluesofM:
�1
âCornerâ ďż˝1andďż˝2arelarge,ďż˝1 ~ ďż˝2;EincreasesinalldirecOons
ďż˝1andďż˝2aresmall;EisalmostconstantinalldirecOons âEdgeâ
�1 >> �2
âEdgeâ ďż˝2 >> ďż˝1
âFlatâregion
Slid
e cr
edit
: Kr
iste
n G
raum
an
�2
10-Oct-1627
Lecture 6 - Fei-Fei Li
CornerResponseFuncOon
â˘âŻ FastapproximaOonâ⯠AvoidcompuOngtheeigenvalues
â⯠ι:constant(0.04to0.06)
�2
âCornerâ θ > 0
âEdgeâ θ < 0
âEdgeâ θ < 0
âFlatâregion
θ = det(M )âÎą trace(M )2 = Îť1Îť2 âÎą(Îť1 +Îť2 )2
�1 Slid
e cr
edit
: Kr
iste
n G
raum
an
10-Oct-1628
Lecture 6 - Fei-Fei Li
WindowFuncOonw(x,y)
â˘âŻ OpOon1:uniformwindowâ⯠Sumoversquarewindow
â⯠Problem:notrotaOoninvariant
â˘âŻ OpOon2:SmoothwithGaussianâ⯠Gaussianalreadyperformsweightedsum
â⯠ResultisrotaOoninvariant
1 in window, 0 outside
2
2,( , ) x x y
x y x y y
I I IM w x y
I I I⥠â¤
= ⢠âĽâ˘ âĽâŁ âŚ
â
Gaussian
2
2,
x x y
x y x y y
I I IM
I I I⥠â¤
= ⢠âĽâ˘ âĽâŁ âŚ
â
2
2( ) x x y
x y y
I I IM g
I I IĎ
⥠â¤= â⢠âĽ
⢠âĽâŁ âŚ
Slid
e cr
edit
: Ba
stia
n Le
ibe
10-Oct-1629
Lecture 6 - Fei-Fei Li
Summary:HarrisDetector[Harris88]â˘âŻ Computesecondmomentmatrix
(autocorrelaOonmatrix)1. Image derivatives
Ix Iy
2. Square of derivatives
Ix2 Iy2 IxIy
3. Gaussian filter g(sI) g(Ix2) g(Iy2) g(IxIy)
R
2
2
( ) ( )( , ) ( )
( ) ( )x D x y D
I D Ix y D y D
I I IM g
I I IĎ Ď
Ď Ď ĎĎ Ď
⥠â¤= â⢠âĽ
⢠âĽâŁ âŚ
2 2 2 2 2 2( ) ( ) [ ( )] [ ( ) ( )]x y x y x yg I g I g I I g I g IÎą= â â +θ = det[M (Ď I ,Ď D )]âÎą[trace(M (Ď I ,Ď D ))]
2
4. Cornerness function â two strong eigenvalues
5. Perform non-maximum suppression
Slid
e cr
edit
: Kr
ysti
an M
ikol
ajcz
yk
10-Oct-1630
Lecture 6 - Fei-Fei Li
HarrisDetector:Workflow
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1631
Lecture 6 - Fei-Fei Li
HarrisDetector:Workflow-computercornerresponsesθ
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1632
Lecture 6 - Fei-Fei Li
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
HarrisDetector:Workflow-Takeonlythelocalmaximaofθ,whereθ>threshold
10-Oct-1633
Lecture 6 - Fei-Fei Li
HarrisDetector:Workflow-ResulOngHarrispoints
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1634
Lecture 6 - Fei-Fei Li
HarrisDetectorâResponses[Harris88]
Effect: A very precise corner detector.
Slid
e cr
edit
: Kr
ysti
an M
ikol
ajcz
yk
10-Oct-1635
Lecture 6 - Fei-Fei Li
HarrisDetectorâResponses[Harris88]
Slid
e cr
edit
: Kr
ysti
an M
ikol
ajcz
yk
10-Oct-1636
Lecture 6 - Fei-Fei Li
HarrisDetectorâResponses[Harris88]
Slid
e cr
edit
: Kr
iste
n G
raum
an
â˘âŻ Resultsarewellsuitedforfindingstereocorrespondences
10-Oct-1637
Lecture 6 - Fei-Fei Li
HarrisDetector:ProperOes
â˘âŻ TranslaOoninvariance?
Slid
e cr
edit
: Kr
iste
n G
raum
an
10-Oct-1638
Lecture 6 - Fei-Fei Li
HarrisDetector:ProperOes
â˘âŻ TranslaOoninvarianceâ˘âŻ RotaOoninvariance?
Ellipse rotates but its shape (i.e. eigenvalues) remains the same
Corner response θ is invariant to image rotation
Slid
e cr
edit
: Kr
iste
n G
raum
an
10-Oct-1639
Lecture 6 - Fei-Fei Li
HarrisDetector:ProperOes
â˘âŻ TranslaOoninvarianceâ˘âŻ RotaOoninvarianceâ˘âŻ Scaleinvariance?
Not invariant to image scale!
All points will be classified as edges!
Corner
Slid
e cr
edit
: Kr
iste
n G
raum
an
10-Oct-1640
Lecture 6 - Fei-Fei Li
Whatwehavelearnedtoday?
â˘âŻ LocalinvariantfeaturesââŻMoOvaOonâ⯠Requirements,invariances
â˘âŻ KeypointlocalizaOonââŻHarriscornerdetector
â˘âŻ ScaleinvariantregionselecOonââŻAutomaOcscaleselecOonââŻDifference-of-Gaussian(DoG)detector
â˘âŻ SIFT:animageregiondescriptor
10-Oct-1641
Nextlecture(#7)
Somebackgroundreading:RickSzeliski,Chapter14.1.1;DavidLowe,IJCV2004
Lecture 6 - Fei-Fei Li
ApplicaOon:ImageSOtching
Slid
e cr
edit
: D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1643
Lecture 6 - Fei-Fei Li
ApplicaOon:ImageSOtching
Slid
e cr
edit
: D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1644
â˘âŻ Procedure:ââŻDetectfeaturepointsinbothimagesâ⯠FindcorrespondingpairsââŻUsethesepairstoaligntheimages
Lecture 6 - Fei-Fei Li
ApplicaOon:ImageSOtching
Slid
e cr
edit
: D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1645
â˘âŻ Procedure:ââŻDetectfeaturepointsinbothimagesâ⯠FindcorrespondingpairsââŻUsethesepairstoaligntheimages