who is there? face recognition applicationsbiometrics.cse.msu.edu/presentations/aniljain_w... · d....
TRANSCRIPT
![Page 1: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/1.jpg)
Who Is There? Face Recognition Applications
Anil K. Jain Department of Computer Science
Michigan State University biometrics.cse.msu.edu
Sept 3, 2015
![Page 2: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/2.jpg)
Why Face Recognition?
2
Identity: ABC Age: ~ 40 Gender: Male Ethnicity: White Hair: Short, Brown Moustache: Yes Beard: Yes Mole: Yes Scar: Yes
Other attributes: expression, emotion,..
![Page 3: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/3.jpg)
Identify Repeat Offenders Habitual Criminal Act of the British Parliament (1869)
H.T. F. Rhodes, Alphonse Bertillon: Father of Scientific Detection, Harrap, 1956
![Page 4: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/4.jpg)
Growing Popularity
• Universality – Everyone has a face; covert, touchless, remote
acquisition; large legacy databases
• Applications – De-duplication, search, surveillance, targeted
ads, social media, mobile phones
• Hardware – Face matcher: ~1.5 million templates/sec/core
• Benchmarks – FERET (1993-98), FRVT (2000-2013), MBGC
(2010), LFW (2007), YTF (2011), IJB-A (2015)
![Page 5: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/5.jpg)
Verification: 1 to 1 Comparison
Same Person? What is the confidence?
![Page 6: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/6.jpg)
Search: 1 to N Comparisons
Probe Gallery
MATCH
Closed-set v. Open-set search
6
![Page 7: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/7.jpg)
Semi-automated Search
Law Enforcement
Who is this?
One of them?
Search System
Gallery
Forensic expert
![Page 8: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/8.jpg)
State of the Art: Verification LFW (2007) IJB-A (2015) FRGC v2.0 (2006) MBGC (2010)
0%
20%
40%
60%
80%
100%
FRGC v2.0 MBGC LFW IJB-A
TAR at 0.1% FAR
D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv, July 28, 2015
LFW Standard Protocol
99.77% (Accuracy) 3,000 genuine & 3,000
imposter pairs; 10-fold CV
LFW BLUFR Protocol
88% TAR @ 0.1% FAR 156,915 genuine, ~46M
imposter pairs; 10-fold CV
![Page 9: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/9.jpg)
State of the Art: Search
PCSO (mugshots) LFW (web) IJB-A (web)
Closed set (CMC @ Rank 1) 0.864 0.602 0.676
Closed set (CMC @ Rank 10) 0.989 0.786 0.764
Open set (FNIR @ 10% FPIR) 0.211 0.412 0.278
Open set (FNIR @ 1% FPIR) 0.333 0.645 0.414
Experimental Setting: Gallery size, # genuine queries, # imposter queries
1M 3K 7K 80M 3.3K 4K 80M 11K 4K
![Page 10: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/10.jpg)
De-duplication: One Person, One ID
Face-based scrubbing of 13.5M
records (>30M photos) in Michigan DMV database; photos of different subjects in the same record!
Courtesy: Pete Langenfeld, Michigan State Police
![Page 11: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/11.jpg)
Driver License Information
2009 driver license photo Gallery: 34 million (30M DMV photos, 4M mugshots)
Courtesy: Pete Langenfeld, MSP
Routine Traffic Stop
![Page 12: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/12.jpg)
1 2 3 4 5
6 7 8 9 10
Top-10 retrievals
Gallery: 34 million (30M DMV photos, 4M mugshots)
Smile makes a difference!
Courtesy: Pete Langenfeld, MSP
Routine Traffic Stop
![Page 13: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/13.jpg)
Border Crossing
SmartGate, Australia & NZ HK-Schenzen border
International Border Crossing
ePassports from eligible countries Fusion of face & fingerprint
![Page 14: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/14.jpg)
Locating Intruders
Courtesy: Allen Ganz, NEC
![Page 15: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/15.jpg)
Passenger Verification Chengdu Train Station
Face image must match photo on ID card
![Page 16: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/16.jpg)
Face Search: Tag Recommendation
![Page 17: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/17.jpg)
What if Tag is Not available?
Need a reject option
![Page 18: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/18.jpg)
Video Surveillance
German Federal Police face recognition trial at Mainz Train Station (2007); ~60% TAR (during daylight) @ FAR of 0.1% (22,673 persons passed through the monitored area/day)
Growing market for CCTV projected to reach $25.6 billion by 2018
![Page 19: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/19.jpg)
Identifying Suspects 30M CCTV cameras in China & growing
Enrollment
Monitor in the console room Railway Station Entrance 3D Face Capture at Police Station
![Page 20: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/20.jpg)
Age and Gender Estimation
Blue: correct estimates; red: incorrect estimates; yellow: incorrect ground truth
Improved search, targeted advertisement, personalization
![Page 21: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/21.jpg)
Mobile Face Unlock
Uploaded: Dec 6, 2011 YouTube
![Page 22: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/22.jpg)
How to Increase Application Coverage?
![Page 23: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/23.jpg)
Application Requirements
• Operating environment
• Accuracy (FRR < 2% @ FAR = 0.01%)
• Speed/throughput
• Template size
• Security (e.g., spoof detection)
• Usability
• Cost
![Page 24: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/24.jpg)
Faces in a crowd
Face Detection
![Page 25: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/25.jpg)
Invariant & Salient Representation
http://www.theguardian.com/theguardian/2010/dec/05/barack-obama-doppelganger-ilham-anas
![Page 26: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/26.jpg)
Robust Matching
Images of one subject in NIST IJB-A data, overlaid with V-J detector & dlib landmarks
![Page 27: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/27.jpg)
Use of Micro Features: Scars, Marks & Tattoos
Detroit police linked at least six armed robberies after matching a tipster’s description of the suspect’s distinctive tattoo
Identical twins Tear drop tattoo
![Page 28: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/28.jpg)
Capacity of Face Recognition
How many distinct identities can be resolved?
![Page 29: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/29.jpg)
Summary
• We cannot trust credentials to answer the Who Is There? question
• Machine face recognition is a hard problem
• Successful applications rely on constrained image capture and cooperative subjects
• Are the deployed systems (e.g., de-duplication) meeting the requirements?
• Face information needs to be augmented with behavioral (contextual) cues
![Page 30: Who Is There? Face Recognition Applicationsbiometrics.cse.msu.edu/Presentations/AnilJain_W... · D. Wang, C. Otto and A. K. Jain, "Face Search at Scale: 80 Million Gallery", arXiv,](https://reader033.vdocument.in/reader033/viewer/2022051920/600daf28f7270e79b9403cb8/html5/thumbnails/30.jpg)
Identical Quadruplets
Haircuts help to avoid confusion among the four six-year-old twins
http://www.cbsnews.com/8301-503543_162-57508537-503543/chinese-mom-shaves-numbers-on-quadruplets-heads/