lecture: face recognitionvision.stanford.edu/teaching/cs131_fall1718/files/13_lda... ·...
TRANSCRIPT
Lecture 12 -Stanford University
Lecture:FaceRecognition
JuanCarlosNiebles andRanjayKrishnaStanfordVisionandLearningLab
07-Nov-171
Lecture 12 -Stanford University
Whatwewilllearntoday• Introductiontofacerecognition• TheEigenfaces Algorithm• LinearDiscriminantAnalysis(LDA)
07-Nov-172
TurkandPentland,Eigenfaces forRecognition,JournalofCognitiveNeuroscience 3 (1):71–86.
P.Belhumeur,J.Hespanha,andD.Kriegman."Eigenfaces vs.Fisherfaces:RecognitionUsingClassSpecificLinearProjection".IEEETransactionsonpatternanalysisandmachineintelligence 19 (7):711.1997.
Lecture 12 -Stanford University
CourtesyofJohannesM.Zanker
07-Nov-17
“Faces”inthebrain
3
Lecture 12 -Stanford University
“Faces”inthebrainfusiform facearea
07-Nov-17
Kanwisher,etal.1997
4
Lecture 12 -Stanford University
DetectionversusRecognition
07-Nov-175
Detectionfindsthefacesinimages RecognitionrecognizesWHOthepersonis
Lecture 12 -Stanford University
• Digitalphotography
FaceRecognition
07-Nov-176
Lecture 12 -Stanford University
FaceRecognition• Digitalphotography• Surveillance
07-Nov-177
Lecture 12 -Stanford University
• Digitalphotography• Surveillance• Albumorganization
FaceRecognition
07-Nov-178
Lecture 12 -Stanford University
FaceRecognition• Digitalphotography• Surveillance• Albumorganization• Persontracking/id.
07-Nov-179
Lecture 12 -Stanford University
• Digitalphotography• Surveillance• Albumorganization• Persontracking/id.• Emotionsandexpressions
FaceRecognition
07-Nov-1710
Lecture 12 -Stanford University
• Digitalphotography• Surveillance• Albumorganization• Persontracking/id.• Emotionsandexpressions
• Security/warfare• Tele-conferencing• Etc.
FaceRecognition
07-Nov-1711
Lecture 12 -Stanford University
TheSpaceofFaces
• Animageisapointinahighdimensionalspace– Ifrepresentedingrayscale intensity,
anNxMimageisapointinRNM
– E.g.100x100image=10,000dim
07-Nov-1712
Slidecredit:ChuckDyer,SteveSeitz,Nishino
Lecture 12 -Stanford University
100x100imagescancontainmanythingsotherthanfaces!
07-Nov-1713
Lecture 12 -Stanford University
TheSpaceofFaces
• Animageisapointinahighdimensionalspace– Ifrepresentedingrayscale intensity,
anNxMimageisapointinRNM
– E.g.100x100image=10,000dim
• However,relativelyfewhighdimensionalvectorscorrespondtovalidfaceimages
• Wewanttoeffectivelymodelthesubspaceoffaceimages
07-Nov-1714
Slidecredit:ChuckDyer,SteveSeitz,Nishino
ɸ1
Lecture 12 -Stanford University
Wherehaveweseensomethinglikethisbefore?
07-Nov-1715
ɸ1
Lecture 12 -Stanford University
Imagespace
Facespace
•Maximizethescatter ofthetrainingimagesinfacespace
• Computen-dimsubspacesuchthattheprojectionofthedatapointsontothesubspacehasthelargestvariance amongalln-dimsubspaces.
07-Nov-1716
Lecture 12 -Stanford University
• So,compressthemtoalow-dimensionalsubspacethatcaptureskeyappearancecharacteristicsofthevisualDOFs.
KeyIdea
• USEPCAforestimatingthesub-space(dimensionalityreduction)
•ComparetwofacesbyprojectingtheimagesintothesubspaceandmeasuringtheEUCLIDEANdistancebetweenthem.
07-Nov-1717
Lecture 12 -Stanford University
Whatwewilllearntoday• Introductiontofacerecognition• TheEigenfaces Algorithm• LinearDiscriminantAnalysis(LDA)
07-Nov-1718
TurkandPentland,Eigenfaces forRecognition,JournalofCognitiveNeuroscience 3 (1):71–86.
P.Belhumeur,J.Hespanha,andD.Kriegman."Eigenfaces vs.Fisherfaces:RecognitionUsingClassSpecificLinearProjection".IEEETransactionsonpatternanalysisandmachineintelligence 19 (7):711.1997.
Lecture 12 -Stanford University
Eigenfaces:keyidea
• Assumethatmostfaceimageslieonalow-dimensionalsubspacedeterminedbythefirstk(k<<d)directionsofmaximumvariance
• UsePCAtodeterminethevectorsor“eigenfaces”thatspanthatsubspace
• Representallfaceimagesinthedatasetaslinearcombinationsofeigenfaces
07-Nov-1719
M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991
Lecture 12 -Stanford University
Trainingimages:x1,…,xN
07-Nov-1720
Lecture 12 -Stanford University
Topeigenvectors:ɸ1,…,ɸk
Mean: μ
07-Nov-1721
Lecture 12 -Stanford University
VisualizationofeigenfacesPrincipal component (eigenvector) ɸk
μ + 3σkɸk
μ – 3σkɸk
07-Nov-1722
Lecture 12 -Stanford University
Eigenface algorithm• Training
1. Aligntrainingimagesx1,x2,…,xN
2. Computeaverageface
3. Computethedifferenceimage(thecentereddatamatrix)
07-Nov-1723
Notethateachimageisformulatedintoalongvector!
µ =1N
xi∑
Lecture 12 -Stanford University
Eigenface algorithm4. Computethecovariancematrix
5. ComputetheeigenvectorsofthecovariancematrixΣ6. Computeeachtrainingimagexi ‘sprojectionsas
7. Visualizetheestimatedtrainingfacexi
07-Nov-1724
xi → xic ⋅φ1, xi
c ⋅φ2,..., xic ⋅φK( ) ≡ a1,a2, ... ,aK( )
xi ≈ µ + a1φ1 + a2φ2 +...+ aKφK
Lecture 12 -Stanford University 07-Nov-1725
6. Computeeachtrainingimagexi ‘sprojectionsas
7. Visualizethereconstructedtrainingfacexi
xi → xic ⋅φ1, xi
c ⋅φ2,..., xic ⋅φK( ) ≡ a1,a2, ... ,aK( )
xi ≈ µ + a1φ1 + a2φ2 +...+ aKφK
a1φ1 a2φ2 aKφK...
Eigenface algorithm
Reconstructedtrainingface
𝑥"
Lecture 12 -Stanford University
Eigenvalues (variancealongeigenvectors)
07-Nov-1726
Lecture 12 -Stanford University
• OnlyselectingthetopKeigenfacesà reducesthedimensionality.• Fewereigenfaces resultinmoreinformationloss,andhenceless
discriminationbetweenfaces.
ReconstructionandErrors
K =4
K =200
K =400
07-Nov-1727
Lecture 12 -Stanford University
Eigenface algorithm
• Testing1. Takequeryimaget2. Projectintoeigenface space andcomputeprojection
3. CompareprojectionwwithallNtrainingprojections• Simplecomparisonmetric:Euclidean• Simpledecision:K-NearestNeighbor
(note:this“K”referstothek-NNalgorithm,isdifferentfromthepreviousK’sreferringtothe#ofprincipalcomponents)
07-Nov-1728
t→ (t −µ) ⋅φ1, (t −µ) ⋅φ2,..., (t −µ) ⋅φK( ) ≡ w1,w2,...,wK( )
Lecture 12 -Stanford University
Eigenfaces looksomewhatlikegenericfaces.
07-Nov-1729
Visualizationofeigenfaces
Lecture 12 -Stanford University
Shortcomings• Requirescarefullycontrolleddata:
– Allfacescenteredinframe– Samesize– Somesensitivitytoangle
• Alternative:– “Learn”onesetofPCAvectorsforeachangle– Usetheonewithlowesterror
• Methodiscompletelyknowledgefree– (sometimesthisisgood!)– Doesn’tknowthatfacesarewrappedaround3Dobjects(heads)
– Makesnoefforttopreserveclassdistinctions
07-Nov-1730
Lecture 12 -Stanford University
SummaryforEigenface
Pros• Non-iterative,globallyoptimalsolution
Limitations
• PCAprojectionisoptimalforreconstruction fromalowdimensionalbasis,butmayNOTbeoptimalfordiscrimination…
07-Nov-1731
Lecture 12 -Stanford University
Besidesfacerecognitions,wecanalsodoFacialexpressionrecognition
07-Nov-1732
Lecture 12 -Stanford University
Happinesssubspace(methodA)
07-Nov-1733
Lecture 12 -Stanford University
Disgustsubspace(methodA)
07-Nov-1734
Lecture 12 -Stanford University
FacialExpressionRecognitionMovies(methodA)
07-Nov-1735
Lecture 12 -Stanford University
Whatwewilllearntoday• Introductiontofacerecognition• TheEigenfaces Algorithm• LinearDiscriminantAnalysis(LDA)
07-Nov-1736
TurkandPentland,Eigenfaces forRecognition,JournalofCognitiveNeuroscience 3 (1):71–86.
P.Belhumeur,J.Hespanha,andD.Kriegman."Eigenfaces vs.Fisherfaces:RecognitionUsingClassSpecificLinearProjection".IEEETransactionsonpatternanalysisandmachineintelligence 19 (7):711.1997.
Lecture 12 -Stanford University
Whichdirectionwillisthefirstprinciplecomponent?
07-Nov-1737
Lecture 12 -Stanford University
Fischer’sLinearDiscriminantAnalysis
• Goal:findthebestseparationbetweentwoclasses
07-Nov-1738
SlideinspiredbyN.Vasconcelos
Lecture 12 -Stanford University
DifferencebetweenPCAandLDA
• PCApreservesmaximumvariance
• LDApreservesdiscrimination– Findprojectionthatmaximizesscatterbetweenclassesandminimizesscatterwithinclasses
07-Nov-1739
Lecture 12 -Stanford University
IllustrationoftheProjection
Poor Projection
x1
x2
x1
x2
• Using two classes as example:
Good
07-Nov-1740
Lecture 12 -Stanford University 07-Nov-1741
Basicintuition:PCAvs.LDA
Lecture 12 -Stanford University
07-Nov-17
42
LDAwith2variables• WewanttolearnaprojectionWsuchthattheprojectionconvertsallthe
pointsfromxtoanewspace(Forthisexample,assumem==1):
• Lettheperclassmeansbe:
• Andtheperclasscovariancematricesbe:
• Wewantaprojectionthatmaximizes:
𝑧 = 𝑤&𝑥
Lecture 12 -Stanford University
Fischer’sLinearDiscriminantAnalysis
07-Nov-1743
SlideinspiredbyN.Vasconcelos
Lecture 12 -Stanford University
07-Nov-17
44
LDAwith2variablesThefollowingobjectivefunction:
Canbewrittenas
Lecture 12 -Stanford University
LDAwith2variables
• Wecanwritethebetweenclassscatteras:
• Also,thewithinclassscatterbecomes:
07-Nov-1745
SlideinspiredbyN.Vasconcelos
Lecture 12 -Stanford University
• Wecanpluginthesescattervaluestoourobjectivefunction:
• Andourobjectivebecomes:
LDAwith2variables
07-Nov-1746
SlideinspiredbyN.Vasconcelos
Lecture 12 -Stanford University
LDAwith2variables
• Thescattervariables
07-Nov-1747
SlideinspiredbyN.Vasconcelos
Lecture 12 -Stanford University
2S
1S
BS
21 SSSW +=
x1
x2Withinclassscatter
Betweenclassscatter
07-Nov-1748
Visualization
Lecture 12 -Stanford University
LinearDiscriminantAnalysis(LDA)
• Maximizingtheratio
• Isequivalenttomaximizingthenumeratorwhilekeepingthedenominatorconstant,i.e.
• AndcanbeaccomplishedusingLagrangemultipliers,wherewedefinetheLagrangian as
• Andmaximizewithrespecttobothwandλ
07-Nov-1749
SlideinspiredbyN.Vasconcelos
Lecture 12 -Stanford University
LinearDiscriminantAnalysis(LDA)• Settingthegradientof
Withrespecttowtozerosweget
or
• Thisisageneralizedeigenvalueproblem
• Thesolutioniseasywhen exists
07-Nov-1750
SlideinspiredbyN.Vasconcelos
Lecture 12 -Stanford University
LinearDiscriminantAnalysis(LDA)
• Inthiscase
• AndusingthedefinitionofSB
• Notingthat(μ1-μ0)Tw=αisascalarthiscanbewrittenas
• andsincewedon’tcareaboutthemagnitudeofw
07-Nov-1751
SlideinspiredbyN.Vasconcelos
Lecture 12 -Stanford University 07-Nov-1752
LDAwithNvariablesandCclasses
Lecture 12 -Stanford University
Variables• N Sample images:
• C classes:
• Average of each class:
• Average of all data:
{ }Nxx ,,1 !
å==
N
kkxN 1
1µ
å=Î ikx
ki
i xN c
µ 1
07-Nov-1753
Lecture 12 -Stanford University
ScatterMatrices
• Scatter of class i:
å=
=c
iiW SS
1
( )( )å=
--=c
i
TiiiB NS
1
µµµµ
• Within class scatter:
• Between class scatter:
( )( )Tikx
iki xxSik
µµc
--= åÎ
07-Nov-1754
Lecture 12 -Stanford University
Mathematical Formulation• Recall that we want to learn a projection W
such that the projection converts all the points from x to a new space z:
• After projection:– Between class scatter– Within class scatter
• So, the objective becomes:
WSWS BT
B =~
WSWS WT
W =~
WSW
WSW
SS
WW
TB
T
W
Bopt WW
max arg~~
max arg ==
07-Nov-1755
𝑧 = 𝑤&𝑥
Lecture 12 -Stanford University
MathematicalFormulation
• Solvegeneralizedeigenvectorproblem:
WSW
WSWW
WT
BT
opt Wmax arg=
miwSwS iWiiB ,,1 !== l
07-Nov-1756
Lecture 12 -Stanford University
Mathematical Formulation
• Solution: Generalized Eigenvectors
• Rank of Wopt is limited– Rank(SB) <= |C|-1– Rank(SW) <= N-C
miwSwS iWiiB ,,1 !== l
07-Nov-1757
Lecture 12 -Stanford University
PCAvs.LDA
• Eigenfaces exploitthemaxscatterofthetrainingimagesinfacespace
• Fisherfaces attempttomaximisethebetweenclassscatter,whileminimisingthewithinclassscatter.
07-Nov-1758
Lecture 12 -Stanford University 07-Nov-1759
Basicintuition:PCAvs.LDA
Lecture 12 -Stanford University
Results:Eigenface vs.Fisherface
• Variation in Facial Expression, Eyewear, and Lighting
• Input: 160imagesof16people• Train: 159images• Test: 1image
With glasses
Without glasses
3 Lighting conditions
5 expressions
07-Nov-1760
Lecture 12 -Stanford University
Eigenface vs.Fisherface
07-Nov-1761
Lecture 12 -Stanford University
Whatwehavelearnedtoday• Introductiontofacerecognition• TheEigenfaces Algorithm• LinearDiscriminantAnalysis(LDA)
07-Nov-1762
TurkandPentland,Eigenfaces forRecognition,JournalofCognitiveNeuroscience 3 (1):71–86.
P.Belhumeur,J.Hespanha,andD.Kriegman."Eigenfaces vs.Fisherfaces:RecognitionUsingClassSpecificLinearProjection".IEEETransactionsonpatternanalysisandmachineintelligence 19 (7):711.1997.