automatic face recognition: state of the...
TRANSCRIPT
Automatic Face Recognition: State of the Art
Anil K. Jain(with Unsang Park, Brendan Klare, Hyun-Cheol Choi)
Department of Brain & Cognitive EngineeringKorea University
Department of Computer Science & EngineeringMichigan State University
Cameras Everywhere
1M CCTV cameras in London & 4M in U.K.; average Briton is seen by 300 cameras/day; 400Kcameras in Beijing provide 100% coverage of public places; 150K cameras in Seoul
Given a query face (probe), identify it from a target population (gallery)
MATCH
Probe Gallery
Automated Face Recognition
1:1 vs. 1 to N matching
Computing Similarity
"This recognition problem is made difficultby the great variability in head rotation andtilt, lighting intensity and angle, facialexpression, aging, etc.” Bledsoe, Chan and Bisson (1964-66)
Used 20 inter-point distances for matching
Pose, lighting, expression (PIE)
Occlusion
Aging
Intra-class Variability
www.marykateandashley.com news.bbc.co.uk/hi/english/in_depth/americas/2000/us_elections
Inter-class Similarity
• Why face recognition?• Applications• Matching algorithms• State of the art performance• Current research
Outline
Why Face?
• Face recognition: common human experience
• Social interaction: expression, emotion, intent, age
• Multidisciplinary nature: Cognitive science, HCI,
graphics, computer vision,..
• Easy to capture: covert acquisition
• Applications: surveillance; border crossing;
deduplication; entertainment,…
1,130 papers with “face recognition” in the title published in 2009 alone
Bertillon System (1882)
H.T. F. Rhodes, Alphonse Bertillon: Father of Scientific Detection, Harrap, 1956
Value of photographing prisoners was recognised by the Habitual Criminal Act, U.K., 1869
Matching 700K faces against 51M gallery (Florida DMV) found 5K duplicates
Detecting Multiple Enrollment
Border Crossing
SmartGate, Australia
HK-Schenzen border crossing
http://www.flir.com/US/
Surveillance
Entertainment
Determine viewer demographics
http://www.tstore.co.kr/userpoc/game/viewProduct.omp?insDpCatNo=DP03002&insProdId=0000028419
&prodGrdCd=PD004401&t_top=DP000503Virtual makeover
Tae Hee Kim
How Automated FR Works
Face Detection
Feature Extraction Matching
Image Normalization
Visible to Shortwave Infrared (SWIR) Spectrum (Bourlai et al., 2010)
Face Sensing
2D (still, video), 3D (shape, texture), multipsectral
Face Detection
*Theo Pavlidis, http://home.att.net/~t.pavlidis/comphumans/comphuman.htm
Challenge: Representation
How to learn salient features?
Pose-dependent
Algorithms
Pose-invariant
Pose-dependency
Matching features
Appearance-based -- Elastic Bunch Graph Matching
Local feature-based
Hybrid
Viewer-centered Images
-- Active Appearance Model
Object-centered Models
-- Morphable Model
Face representation
PCA, LDA LFA
Taxonomy of Face Recognition
LBP, Gabor
Holistic Features
…EigenFaces
Fisherfaces
Reconstructed face
Input face
…
PCA LDA
Minimize reconstruction error Maximize between-class to within-class scatter
56.4 38.6 -19.7 9.8 -45.9 19.6 -21.4 14.2
18.3 35.6 -17.5 -27.6 60.6 -20.8 41.9 -9.6
• Local Binary Patterns– Represent a local face
region as distribution of LBP features • normalized histogram
– Performs better than holistic methods
Local Features
0
5
1
5
2
5
Histogram of LBP feature values
Ojala et al., “Multiresolution gray-scale and rotation invariant texture classification with local binary patterns,” IEEE TPAMI, 2002
Face Descriptors
Figure 1: An attribute classifier is trained torecognize the presence or absence of adescribable aspect (65) of visual appearance.Responses for 13 attribute classifiers are shownfor a pair of images of Halle Berry.
Figure 2: A number of simile classifiers aretrained to recognize the similarities of parts offaces to 60 reference people (Rj). Theresponses to 13 simile classifiers are shown fora pair of images of Harrison Ford.
N. Kumar et al., “Attribute and Simile Classifiers for Face Verification,” ICCV, 2009
Amazon Mechanical Turk used to obtain ~1000 training examples/attribute; $5k
Performance: State-of-Art
Best performance: high resolution 2D (controlled lighting) & 3D images
• Goal: GAR=98% @ FAR=0.1%, an order of magnitude better than FRVT2002
• Four input face modalities
• Large image database (up to 100K)
Face Recognition Vendor Test (FRVT 2006)
P.J. Phillips, “FRVT 2006 and ICE 2006, Large-Scale Results,” March, 2007. http://www.frvt.org/frvt2006/
High res., controlled lighting, neutral
Controlled lighting, smiling (400 IPD)
Uncontrolled lighting, smiling (190 IPD) 3D shape + texture
Video Surveillance Trial
~60% true ID: German Federal Police at Mainz Train Station (2007)
Human vs. Machine• O’Toole (2007) compared humans and 7 algorithms
on face pairs; 3 algorithms surpassed avg. human performance on difficult pairs; six on easy pairs
• Ding (2010): TH algorithm was better than 4500 customs inspectors on easy pairs in operational data
Easy Pair Difficult Pair
O’Toole et al., “Face recog. alg. surpass humans matching faces over changes in illumination,” TPAMI, 2007Ding et al. “Computers do better than experts matching faces in a large population”, IEEE ICCI, 2010
Current Research
Face MarksPeriocular Age Invariance
Sketch RecognitionFace Individuality
IR Face Recognition Avatar Recognition
Face at a Distance
Baseline performance: FaceVACS from Cognitec
Age Invariance
• Facial shape and texture change over time • Applications
– Age specific access control (vending machines)
– Missing children, multiple enrollment
Age 18 Age 31 Age 29Age 17
Databases: FG-NET and MORPH
Age Invariant Face Recognition
Training set(age-separated images)
Feature extraction & subspace learning
Learn appearance aging pattern
3D aging model
……
……
Build ensemble of subspaces: Minimize within-subject to between-subject variation
},,,{' 10 Nϕϕϕ =Φ
Φ
ΦΦ
Φ
)(
)1()(
)1(
M
M
MLBP
MLBP
SIFT
SIFT
Approach #1: aging invariant subspace learning
Approach #2: appearance aging model
28Aging simulation
Input
+
Age
i
Gallery
Im
ag
es
Pro
be I
mag
es
FaceVACS and generative model fail; discriminative
approach succeeds
Discriminative approach fails; FaceVACS and generative model
succeed
Matching Results
All three methods fail; fusion of generative and discriminative models
succeeds
Park, Tong & Jain, "Age Invariant Face Recognition", IEEE Trans. PAMI, 2010
Facial Sketch
30
Sketch to Mug shot Matching
Sketch drawn based on eye witness description
Klare, Li, and Jain, "Matching Forensic Sketches to Mug shot Photos," IEEE Trans. PAMI, 2010 (To Appear)
Mug shots in the Michigan police database
TRAINING
(sketch, photo) pairs
Patch featuresGroup patch vectors
into slices
12 34 . N...56
Learndiscriminantprojection for
each slice
0
5
1
5
2
5
Overlapping patches
MATCHINGProbeSketch
GalleryPhotos
0
5
1
5
2
5
Discriminantprojection
Feature extraction and grouping into slices
Nearest neighbor classifier
Matching Sketch to Photo
Forensic Sketch Matching
Successful Matches Failed Matches
Periocular
Periocular region(220x225)
Park, Jillela, Ross and Jain, " Periocular Biometrics in the Visible Spectrum", IEEE Trans. Inf. Forensics & Security 2010
• Performance on FRGC (3,400 images of 568 subjects)• No occlusion: 88% vs. 99.8% for FaceVACS• With occlusion: 81% vs. 40% for FaceVACS
1700x2270
Face Marks
Some marks are distinctive & permanent
Large birth mark Large birth mark Gang tattoo
Recognition with Face Marks
FaceVACS fails at rank-1 Recognition succeeds with FaceVACS + face marks
Matching with face marks
⊗Score fusion
≠ =
≠ =
Park & Jain, "Face Matching and Retrieval Using Soft Biometrics," IEEE Trans. on Inf. Forensics and Security, 2010
Face Recognition At A Distance• PTZ camera system
– Acquires high resolution face images (up to 10m)– Two static cameras control the PTZ camera– 96% recognition accuracy (20 probe and 10K gallery subjects)
PTZ view
Tracking
Motion segmentation
Choi, Park, and Jain, " PTZ Camera Assisted Face Acquisition, Tracking & Recognition," BTAS, 2010
Apsaras of Angkor Wat
Hindu temple built in 1,150AD; French explorers discovered the hidden ruins ~1890
Do they represent different ethnicities?
140 landmarks
Facial components allow domain expertsto assign different weights
Facial Landmarks
Klare, Mallapragada, Jain, Davis, "Clustering Face Carving: Exploring the Devatas of Angkor Wat", ICPR, 2010
Procustes Alignment
Remove variations in translation, rotation and scaling; fit a Point Distribution model (PDM) to each facial component; compute weighted sum of component similarities
Generating Clusters
Faces in Virtual World• Rise in criminal activity in Second Life
– Al-Qaeda recruitment [1], cyber crime [2], identity theft
• Search virtual world for avatars on watch list
Nood and Attema, The Second Life of Virtual Reality, http://www.epn.net/interrealiteit/EPN-REPORT-The_Second_Life_of_VR.pdfCole, Osama bin Laden's “SecondLife", Salon, http://www.salon.com/opinion/feature/2008/02/25/avatars/Yampolskiy, Klare and Jain, “Face recognition in virtual world”, working paper, 2010
Human-Like Faces
CourtesyFrank Hegel
Summary
• Face recognition is a topic of great interest to several disciplines
• Progress in automatic face recognition driven by: searching large face databases in real-time with high accuracy and low cost; humans are not necessarily the best for this task
• Excellent performance in constrained environments: frontal pose, neutral expression, controlled illumination & background, small age gap
• Unconstrained FR will require: better sensing & modeling, additional cues, contextual information,..