understanding faces 12/1/15 some slides from amin sadeghi, lana lazebnik, silvio savarese, fei-fei...
TRANSCRIPT
Computational Photography CS498dwh
Understanding Faces12/1/15Some slides from Amin Sadeghi, Lana Lazebnik, Silvio Savarese, Fei-Fei LiChuck Close, self portrait
Lucas by Chuck CloseDetection, Recognition, and Transformation of Faces1Exams back on ThursdayFace detection and recognition
Detection
RecognitionSally3Applications of Face RecognitionDigital photography
Applications of Face RecognitionDigital photographySurveillance
Applications of Face RecognitionDigital photographySurveillanceAlbum organization
Consumer application: iPhoto 2009
http://www.apple.com/ilife/iphoto/7Consumer application: iPhoto 2009Can be trained to recognize pets!
http://www.maclife.com/article/news/iphotos_faces_recognizes_cats8Face detection
Detection
What does a face look like?
What does a face look like?
What makes face detection hard?Expression
What makes face detection hard?Viewpoint
What makes face detection hard?Occlusion
What makes face detection hard?Distracting background
What makes face detection and recognition hard?Coincidental textures
Consumer application: iPhoto 2009Things iPhoto thinks are faces
17How to find faces anywhere in an image?Filter Image with a face?
19Normalize mean and standard deviation
Train a Filter
SVM19Face detection: sliding windows
Filter/Template
Multiple scales
What features?Exemplars(Sung Poggio 1994)
Edge (Wavelet) Pyramids(Schneiderman Kanade 1998)
Intensity Patterns (with NNs)(Rowley Baluja Kanade 1996)
Haar Filters(Viola Jones 2000)How to classify?Many waysNeural networksAdaboostSVMsNearest neighbor
Face classifierTraining LabelsTraining ImagesClassifier TrainingTrainingImage FeaturesImage FeaturesTestingTest ImageTrained ClassifierTrained ClassifierFacePrediction
Face recognition
Detection
RecognitionSallyFace recognitionTypical scenario: few examples per face, identify or verify test exampleWhats hard: changes in expression, lighting, age, occlusion, viewpointBasic approaches (all nearest neighbor)Project into a new subspace (or kernel space) (e.g., Eigenfaces=PCA)Measure face featuresMake 3d face model, compare shape+appearance (e.g., AAM)
Simple techniqueTreat pixels as a vector
Recognize face by nearest neighbor
Most recent research focuses on faces in the wild, recognizing faces in normal photosClassification: assign identity to faceVerification: say whether two people are the same
Important stepsDetectAlignRepresentClassifyState-of-the-art Face RecognizersExample of recent approach
DeepFace: Closing the Gap to Human-Level Performance in Face VerificationTaigman, Yang, Ranzato, & Wolf (Facebook, Tel Aviv), CVPR 2014Following slides adapted from Daphne Tsatsoulis Face Alignment1. Detect a face and 6 fiducial markers using a support vector regressor (SVR)
2. Iteratively scale, rotate, and translate image until it aligns with a target face
3. Localize 67 fiducial points in the 2D aligned crop
4. Create a generic 3D shape model by taking the average of 3D scans from the USF Human-ID database and manually annotate the 67 anchor points
5.Fit an affine 3D-to-2D projection and use it to frontally warp the face
Train DNN classifier on aligned facesArchitecture (deep neural network classifier)Two convolutional layers (with one pooling layer)3 locally connected and 2 fully connected layers> 120 million parameters
Train on dataset with 4400 individuals, ~1000 images eachTrain to identify face among set of possible people
Face matching (verification) is done by comparing features at last layer for two faces
Results: Labeled Faces in the Wild DatasetPerforms similarly to humans!(note: humans would do better with uncropped faces)
Experiments show that alignment is crucial (0.97 vs 0.88) and that deep features help (0.97 vs. 0.91)
Transforming facesFigure-centric averagesNeed to AlignPositionScaleOrientation
Antonio Torralba & Aude Oliva (2002)Averages: Hundreds of images containing a person are averaged to reveal regularities in the intensity patterns across all the images.33Another usage is to show a significant trend.----- Meeting Notes (10/4/11 17:12) -----Aligned through translation and scaleaverage
33How do we average faces?
http://www2.imm.dtu.dk/~aam/datasets/datasets.html34
morphingMorphing
image #1image #2
warp
warp
35Cross-Dissolve vs. Morphing
Average ofAppearance VectorsImages from James HaysAverage ofShape Vectorshttp://www.faceresearch.org/demos/vector----- Meeting Notes (10/4/11 17:12) -----the demo
36Aligning FacesNeed to AlignPositionScaleOrientationKey-points The more key-points, the finer alignment
Images from Alyosha Efros
37Appearance Vectors vs. Shape Vectors200*150 pixels (RGB)Vector of200*150*3Dimensions
AppearanceVector43 coordinates (x,y)ShapeVectorVector of43*2DimensionsYou can think about faces in different ways. You can represent this way or that way. These are complements.38Average of two Faces
Input face keypointsPairwise average keypoint coordinatesTriangulate the facesWarp: transform every face triangleAverage the pixels39Average of multiple faces
1. Warp to mean shape2. Average pixelshttp://graphics.cs.cmu.edu/courses/15-463/2004_fall/www/handins/brh/final/http://www.faceresearch.org/demos/averageAverage Men of the world
Average Women of the world
You will notice that women from every place is very attractive because the ..
42Subpopulation meansOther Examples:Average KidsHappy MalesEtc.http://www.faceresearch.org
Average maleAverage female
Average kid
Average happy maleWe might be curious what different populations of faces look like.43How to represent variations?Training imagesx1,,xN
44PCAGeneral dimensionality reduction techniqueFinds major directions of variation
Preserves most of variance with a much more compact representationLower storage requirements (eigenvectors + a few numbers per face)Faster matching/retrieval
Principal Component AnalysisGiven a point set , in an M-dim space, PCA finds a basis such thatThe most variation is in the first basis vectorThe second most, in the second vector that is orthogonal to the first vectorThe third
x1x0x1x01st principal component2nd principal component
1st principal component2nd principal componentThe goal of PCA is to find orthogonal vectors that the first r vectors have the highest variance of all.----- Meeting Notes (10/4/11 17:12) -----Write it more mathematicallysay what PCA is solving46PCA in MATLABx=rand(3,10);%10 3D examples mu=mean(x,2);x_norm = x-repmat(mu,[1 n]);x_covariance = x_norm*x_norm';[U, E] = eig(x_covariance)
U = 0.74 0.07 -0.66 0.65 0.10 0.74 -0.12 0.99 -0.02
E = 0.27 0 0 0 0.63 0 0 0 0.94Principal Component AnalysisChoosing subspace dimension r:look at decay of the eigenvalues as a function of rLarger r means lower expected error in the subspace data approximation
rM1eigenvalues
First r < M basis vectors provide an approximate basis that minimizes the mean-squared-error (MSE) of reconstructing the original points
Eigenfaces example (PCA of face images)Top eigenvectors: u1,ukMean:
49Visualization of eigenfaces (appearance variation)
Principal component (eigenvector) uk + 3kuk 3kuk50Face SpaceHow to find a set of directions to cover all space?
We call these directions BasisIf number of basis faces is large enough to span the face subspace:Any new face can be represented as a linear combination of basis vectors.
Limitations of Global appearance method: not robust to misalignment, background variation
52Can represent face in appearance or shape space200*150 pixels (RGB)
AppearanceVector43 coordinates (x,y)ShapeVectorYou can think about faces in different ways. You can represent this way or that way. These are complements.53First 3 Shape Bases with PCA
Mean appearancehttp://graphics.cs.cmu.edu/courses/15-463/2004_fall/www/handins/brh/final/
----- Meeting Notes (10/4/11 17:12) -----show both extremes54Manipulating facesHow can we make a face look more female/male, young/old, happy/sad, etc.?http://www.faceresearch.org/demos/transform
Current facePrototype 2Prototype 155----- Meeting Notes (10/4/11 17:24) -----Add a matlab thing to show the differences. show how to use PCA in matlab. Manipulating facesWe can imagine various meaningful directions.Current faceMasculineFeminineHappySad56Psychological Attributes
Human Perception
Prince Charles, Woody Allen, Bill Clinton, Saddam Hussein, Richard Nixon and Princess Diana59
Jim Carrey (left) and Kevin Costner60
Nixon and Winona Ryder61
Which face is more attractive?
originalbeautifiedhttp://leyvand.com/beautification2008/64----- Meeting Notes (10/4/11 17:12) -----The goal is to keep the same person but more attractice.System Overview
http://leyvand.com/beautification2008/65Things to rememberFace Detection: train face vs. non-face model and scan over multi-scale image
Face Recognition: detect, align, compute features, and compute similarity
Represent faces with an appearance vector and a shape vector
Use PCA for compression or to model main directions of variance
Can transform faces by moving shape vector in a given direction and warping
66