![Page 1: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/1.jpg)
Session IIFace Anti-Spoofing Generalization
Host: Yaojie Liu
ComputerVision Lab
![Page 2: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/2.jpg)
Training-Testing Difference
The testing scenarios are different with the training phase.
• Environment (Lighting, Indoor/outdoor, etc.)
• Camera/Image quality
• Subjects (Age, Race, etc.)
• Spoof types
2
![Page 3: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/3.jpg)
Training-Testing Difference
The testing scenarios are different with the training phase.
• Environment (Lighting, Indoor/outdoor, etc.)
• Camera/Image quality
• Subjects (Age, Race, etc.)
• Spoof types
3
Cross-database DomainAdaption
![Page 4: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/4.jpg)
Training-Testing Difference
The testing scenarios are different with the training phase.
• Environment (Lighting, Indoor/outdoor, etc.)
• Camera/Image quality
• Subjects (Age, Race, etc.)
• Spoof types
4
Unknown Spoof Detection
![Page 5: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/5.jpg)
Outline
• Cross-database domain adaption
• Unknown attack detection
• Testing protocols & evaluation metrics
5
![Page 6: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/6.jpg)
Cross-database Domain Adaption
• Enforce features to be domain-invariant• Domain adaption [1,2]
• Metric learning [3,5,6]
• Meta learning [7,8]
6
1. Learning Generalizable and Identity-Discriminative Representations for Face Anti-Spoofing, TIFS, 20182. Unsupervised Domain Adaptation for Face Anti-Spoofing, TIFS 20183. Multi-adversarial Discriminative Deep Domain Generalization, CVPR, 20194. Domain Adaptation in Multi-Channel Autoencoder based Features for Robust Face Anti-Spoofing, ICB 20195. Improving Cross-database Face Presentation Attack Detection via Adversarial Domain Adaptation, ICB 20196. Single-Side Domain Generalization for Face Anti-Spoofing, CVPR 20207. Regularized Fine-grained Meta Face Anti-spoofing, AAAI 20208. Learning Meta Model for Zero- and Few-shot Face Anti-spoofing, AAAI 2020
![Page 7: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/7.jpg)
Learning Generalizable and Identity-Discriminative Representations for Face Anti-Spoofing
• Learn face anti-spoofing and face recognition at the same time
• Apply a Fast Domain Adaption (FDA) to remove the bias of different domain
• Share the weights of face anti-spoofing and face recognition
7
1. Li et. al., Learning Generalizable and Identity-Discriminative Representations for Face Anti-Spoofing, TIFS, 2018
![Page 8: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/8.jpg)
Learning Generalizable and Identity-Discriminative Representations for Face Anti-Spoofing
• Fast Domain Adaption (FDA)• Style transfer network
• Content loss + Style (domain) loss
8
1. Li et. al., Learning Generalizable and Identity-Discriminative Representations for Face Anti-Spoofing, TIFS, 2018
![Page 9: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/9.jpg)
Metric learning
• Adversarial learning• learn target features such that discriminator cannot
correctly predict the domain
• remove unrelated features
• Triplet loss• learn target features such that live samples from
different domains are similar
• find shared features
9
1. Multi-adversarial Discriminative Deep Domain Generalization, CVPR, 20192. Improving Cross-database Face Presentation Attack Detection via Adversarial Domain Adaptation, ICB 20193. Single-Side Domain Generalization for Face Anti-Spoofing, CVPR 2020
![Page 10: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/10.jpg)
Improving Cross-database Face Presentation Attack Detection via Adversarial Domain Adaptation
• Pretrain a source encoder/decoder
• Classify with k-NN classifier
10
1. Wang et. al., Improving Cross-database Face Presentation Attack Detection via Adversarial Domain Adaptation, ICB, 2019
![Page 11: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/11.jpg)
Multi-adversarial Deep Domain Generalization forFace Presentation Attack Detection
• Feature generator• extract features for face anti-spoofing
• adversarial-trained to remove domain information
• Depth estimation• improve the discriminativeness
• Dual-force triplet mining• enforce a smaller intra-class distance
• enforce a larger inter-class distance
• cross domain
13
1. Shao et. al., Multi-adversarial Discriminative Deep Domain Generalization for Face Presentation Attack Detection, CVPR, 2019
![Page 12: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/12.jpg)
Multi-adversarial Deep Domain Generalization forFace Presentation Attack Detection
• M1, M2, M3: domain specified features
• G: generalized features
• G and D1, D2, D3 compete
14
1. Shao et. al., Multi-adversarial Discriminative Deep Domain Generalization for Face Presentation Attack Detection, CVPR, 2019
![Page 13: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/13.jpg)
Single-Side Domain Generalization for Face Anti-Spoofing• The parameter sharing feature generator is trained to make the feature distributions of different
domains undistinguishable for the real faces but not for the fake ones under the single-side adversarial learning.
15
1. Single-Side Domain Generalization for Face Anti-Spoofing, CVPR 2020
![Page 14: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/14.jpg)
Dual-force Triplet Mining
• In one domain • Minimize live-to-live distance between different
subjects
• Maximize live-to-spoof distance between different subjects
• Cross domains• Minimize live-to-live distance between different
subjects
• Maximize live-to-spoof distance between different subjects
• Anchor as live
19
1. Shao et. al., Multi-adversarial Discriminative Deep Domain Generalization for Face Presentation Attack Detection, CVPR, 2019
![Page 15: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/15.jpg)
Dual-force Triplet Mining
• In one domain • Minimize live-to-live distance between different
subjects• Maximize live-to-spoof distance between
different subjects
• Cross domains• Minimize live-to-live / spoof-to-spoof distance
between different subjects only• Maximize live-to-spoof / spoof-to-spoof distance
between different domains
• Triplet with live (d1,d2,d3), spoof (d1), spoof (d2), spoof(d3)
20
1. Single-Side Domain Generalization for Face Anti-Spoofing, CVPR 2020
![Page 16: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/16.jpg)
Domain Adaptation in Multi-Channel Autoencoder based Features for Robust Face Anti-Spoofing
• Use multi-modality data (RGB, NIR, and Depth) instead of RGB only
• Domain Adaption: fine-tuning (RGB → NIR-Depth)
23
1. George et. al., Biometric Face Presentation Attack Detection with Multi-Channel Convolutional Neural Network, TIFS 2019
![Page 17: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/17.jpg)
Domain Adaptation in Multi-Channel Autoencoder based Features for Robust Face Anti-Spoofing
24
1. George et. al., Biometric Face Presentation Attack Detection with Multi-Channel Convolutional Neural Network, TIFS 2019
![Page 18: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/18.jpg)
Meta Learning
• Meta-learning, also known as “learning to learn”, intends to design models that can learn new skills or adapt to new environments rapidly with a few training examples.
25
1. Regularized Fine-grained Meta Face Anti-spoofing, AAAI 20202. Learning Meta Model for Zero- and Few-shot Face Anti-spoofing, AAAI 2020
![Page 19: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/19.jpg)
Meta Learning for FAS
• Tackle cross-database testing: Train on multiple domains, test on one domain
26
1. Regularized Fine-grained Meta Face Anti-spoofing, AAAI 20202. Learning Meta Model for Zero- and Few-shot Face Anti-spoofing, AAAI 2020
![Page 20: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/20.jpg)
Meta Learning
• A learner to handle all meta learning tasks
• Training set (meta-train set +meta-test set), testing set• E.g., domain 1,2,3 → train, domain 4 → test
• Meta-task 1: domain 1,2 →meta-train, domain 3 →meta-test
• Meta-task 2: domain 1,3 →meta-train, domain 2 →meta-test
• Meta-task 3: domain 2,3 →meta-train, domain 1 →meta-test
27
1. Learning Meta Model for Zero- and Few-shot Face Anti-spoofing, AAAI 2020
Learner
Meta Learner for task1
Meta Learner for task2
Meta Learner for task3
Meta Learner for task4
![Page 21: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/21.jpg)
Meta Learning
• A learner to handle all meta learning tasks
• Training set (meta-train set +meta-test set), testing set
• Choose meta tasks
• Update meta learner (inner update) meta-train losses
• Compute meta-test losses
• Update learner with meta-test losses
28
1. Learning Meta Model for Zero- and Few-shot Face Anti-spoofing, AAAI 2020
Learner
Meta Learner for task1
Meta Learner for task2
Meta Learner for task3
Meta Learner for task4
![Page 22: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/22.jpg)
Meta Learning
• A learner to handle all meta learning tasks
• Training set (meta-train set +meta-test set), testing set
• Choose meta tasks
• Update meta learner (inner update) meta-train losses
• Compute meta-test losses
• Update learner with meta-test losses + meta-train losses
29
1. Regularized Fine-grained Meta Face Anti-spoofing, AAAI 2020
![Page 23: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/23.jpg)
Cross-database Domain Adaption
• Enforce features to be domain-invariant• Domain adaption [1,2]
• Metric learning [3,5,6]
• Meta learning [7,8]
30
1. Learning Generalizable and Identity-Discriminative Representations for Face Anti-Spoofing, TIFS, 20182. Unsupervised Domain Adaptation for Face Anti-Spoofing, TIFS 20183. Multi-adversarial Discriminative Deep Domain Generalization, CVPR, 20194. Domain Adaptation in Multi-Channel Autoencoder based Features for Robust Face Anti-Spoofing, ICB 20195. Improving Cross-database Face Presentation Attack Detection via Adversarial Domain Adaptation, ICB 20196. Single-Side Domain Generalization for Face Anti-Spoofing, CVPR 20207. Regularized Fine-grained Meta Face Anti-spoofing, AAAI 20208. Learning Meta Model for Zero- and Few-shot Face Anti-spoofing, AAAI 2020
![Page 24: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/24.jpg)
Unknown Attack Detection
• One-class classifier• One-class SVM
• Gaussian Mixture Model
• AutoEncoder
• Zero-shot learning
31
1. An Anomaly Detection Approach to Face Spoofing Detection: A New Formulation and Evaluation Protocol, IEEE Access, 20172. Unknown Presentation Attack Detection with Face RGB Images, ICB, 20183. Deep Anomaly Detection for Generalized Face Anti-Spoofing, CVPRW, 20194. Deep Tree Learning for Zero-shot Face Anti-Spoofing, CVPR 2019
![Page 25: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/25.jpg)
An Anomaly Detection Approach to Face Spoofing Detection: A New Formulation and Evaluation Protocol
A very comprehensive study on various hand-crafted feature and classifiers.
• Feature: LBP-TOP, LPQ-TOP, BSIF-TOP, Image quality measures
• Classifier: SVM1, SVM2, LDA2, Sparse representation classifier (SRC)1, SRC 2
• Dataset: CASIA-FASD, Replay-attack, MSU-MFSD
32
1. Arashlool et. al., An Anomaly Detection Approach to Face Spoofing Detection: A New Formulation and Evaluation Protocol, 2017
![Page 26: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/26.jpg)
An Anomaly Detection Approach to Face Spoofing Detection: A New Formulation and Evaluation Protocol
A very comprehensive study on various hand-crafted feature and classifiers.
• Feature: LBP-TOP, LPQ-TOP, BSIF-TOP, Image quality measures
• Classifier: SVM1, SVM2, LDA2, Sparse representation classifier (SRC)1, SRC 2
• Dataset: CASIA-FASD, Replay-attack, MSU-MFSD
• Conclusion: neither the two-class systems nor the one-class approaches perform well
enough
33
1. Arashlool et. al., An Anomaly Detection Approach to Face Spoofing Detection: A New Formulation and Evaluation Protocol, 2017
![Page 27: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/27.jpg)
Unknown Presentation Attack Detection with Face RGB Images
A very comprehensive study on various hand-crafted feature and classifiers.
• Feature: Color LBP
• Classifier: SVM1, Auto Encoder, GMM
• Dataset: CASIA-FASD, Replay-attack, MSU-MFSD
34
1. Xiong et. al., Unknown Presentation Attack Detection with Face RGB Images, ICB, 2018
![Page 28: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/28.jpg)
Unknown Presentation Attack Detection with Face RGB Images
Xiong et. al., Unknown Presentation Attack Detection with Face RGB Images, ICB, 2018
35
A very comprehensive study on various hand-crafted feature and classifiers.
• Feature: Color LBP
• Classifier: SVM1, Auto Encoder, GMM
• Dataset: CASIA-FASD, Replay-attack, MSU-MFSD
• Conclusion: improve the performance
• NN+LBP works best on C+R+M protocols
• AE+LBP works best on Oulu protocols
![Page 29: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/29.jpg)
Deep Anomaly Detection for Generalized Face Anti-Spoofing
• Deep metric learning
• Triplet Focal loss
• Focus on the harder cases
36
1. Perez-Cabo et. al., Deep Anomaly Detection for Generalized Face Anti-Spoofing, CVPRW, 2019
![Page 30: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/30.jpg)
Literature and Issues
• Limited Spoof Types1,2
• Only model the live distribution1,2
37
1. S. R. Arashloo et. al. An anomaly detection approach to face spoofing detection: a new formulation and evaluation protocol.2. F. Xiong and W. Abdalmageed. Unknown presentation attack detection with face RGB images. BTAS 2018
“This is live face!”
Live
Known Spoof
Unknown Spoof
![Page 31: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/31.jpg)
What if More Spoof Types?
38
LiveHalf Mask Silicone Transparent Papercraft Mannequin
Replay3D Mask Attacks
PrintObfuscation Imperson. Cosmetic Funny Eye Paperglasses Partial Paper
Makeup Attacks Partial Attacks
![Page 32: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/32.jpg)
Deep Tree Learning for Zero-shot Face Anti-Spoofing
• Previous methods only model the live
• Learning semantic spoof attributes
39
1. Liu et. al., Deep Tree Learning for Zero-shot Face Anti-Spoofing, CVPR 2019
Attr2
Live
Known Spoof
Unknown Spoof
Attr n Attr3
Attr4 Attr1
![Page 33: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/33.jpg)
Deep Tree Networks (DTN)
40
TRU
CRU
TRU
CRU
TRU
CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
SFL SFLSFL SFL SFL SFLSFL SFL
256× 256 × 6(RGB+HSV)
TreeNodes
LeafNodes
Convolutional Residual Unit
Tree Routing Unit
Supervised Feature Learning
Attr1 Attr2 Attr3 … Attr8
![Page 34: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/34.jpg)
Deep Tree Networks (DTN)
41
TRU
CRU
TRU
CRU
TRU
CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
SFL SFLSFL SFL SFL SFLSFL SFL
256× 256 × 6(RGB+HSV)
![Page 35: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/35.jpg)
Deep Tree Networks (DTN)
42
TRU
CRU
TRU
CRU
TRU
CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
SFL SFLSFL SFL SFL SFLSFL SFL
256× 256 × 6(RGB+HSV)
𝜑 𝑥 < 0
![Page 36: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/36.jpg)
Deep Tree Networks (DTN)
43
TRU
CRU
TRU
CRU
TRU
CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
SFL SFLSFL SFL SFL SFLSFL SFL
256× 256 × 6(RGB+HSV)
𝜑 𝑥 ≥ 0
![Page 37: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/37.jpg)
Deep Tree Networks (DTN)
44
TRU
CRU
TRU
CRU
TRU
CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
SFL SFLSFL SFL SFL SFLSFL SFL
256× 256 × 6(RGB+HSV)
𝜑 𝑥 ≥ 0
![Page 38: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/38.jpg)
Deep Tree Networks (DTN)
45
TRU
CRU
TRU
CRU
TRU
CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
SFL SFLSFL SFL SFL SFLSFL SFL
256× 256 × 6(RGB+HSV)
![Page 39: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/39.jpg)
Supervised Feature Learning
Classification
Binary Mask Regression
46
TRU
CRU
TRU
CRU
TRU
CRU
TRU
CRU
CRU
CRU CRU CRU
SFL
256× 𝟐𝟓𝟔 × 𝟔(RGB+HSV)
TRU TRU TRU
CRU CRU CRU CRU CRU CRU CRU
SFL SFL SFL SFL SFL SFL SFL
![Page 40: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/40.jpg)
Supervised Feature Learning
Binary Mask Regression
47
![Page 41: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/41.jpg)
Training TRU
48
TRU
CRU
TRU
CRU
TRU
CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
SFL SFLSFL SFL SFL SFLSFL SFL
256× 256 × 6(RGB+HSV)
𝒴
𝒳
𝒵
𝒪
𝒗
Feature Space
![Page 42: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/42.jpg)
Training TRU
49
TRU
CRU
TRU
CRU
TRU
CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
TRU
CRU
CRU CRU
SFL SFLSFL SFL SFL SFLSFL SFL
![Page 43: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/43.jpg)
Tree Routing Unit (TRU)
• Routing Function
• Based on eigen-analysis of visiting set
• We optimize:
50
TRU
CRU
TRU
CRU
TRU
CRU
TRU
CRU
CRU
CRU CRU CRU
SFL
256× 𝟐𝟓𝟔 × 𝟔(RGB+HSV)
TRU TRU TRU
CRU CRU CRU CRU CRU CRU CRU
SFL SFL SFL SFL SFL SFL SFL
𝒴
𝒳
𝒵
𝒪
𝒗
Feature Space
![Page 44: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/44.jpg)
t-SNE Results
53
![Page 45: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/45.jpg)
Databases and testing protocols
54
Database Sensors Print/Replay Mask Makeup # Spoof Type # Subjects # Videos Year
Replay-Attack RGB X 3 50 1200 2012
CASIA-FASD RGB X 3 50 600 2012
3DMAD RGB, Depth X 1 17 510 2014
MSU-MFSD RGB X 3 55 280 2015
MSU-USSA RGB X 8 1000 9,000 (I) 2016
HKBU MAR RGB X 2 35 1008 2016
MiW RGB X 3 434 1604 2017
OULU-NPU RGB X 4 55 4950 2017
SiW RGB X 6 165 4478 2018
SiW-M RGB X X X 13 493 1630 2019
CASIA-SURF RGB, NIR, Depth X 1000 21000 2019
WMCA RGB, NIR, Depth, Thermal X X 7 72 1679 2019
CelebA-Spoof RGB X X 4 10,177 625,537 (I) 2020
![Page 46: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/46.jpg)
Replay Attack Database
• Photo/video attacks
• Controlled/adverse sessions
55
Database Sensors Print/Replay Mask Makeup # Spoof Type # Subjects # Videos Year
Replay-Attack RGB X 3 50 1200 2012
Chingovska et. al., On the Effectiveness of Local Binary Patterns in Face Anti-spoofing, BIOSIG, 2012
![Page 47: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/47.jpg)
CASIA-FASD Database
• Three different image quality
• Eye cut to counter the eye-blinking methods
• Warp paper to counter the motion methods
56
1. Zhang et. al., A Face Antispoofing Database with Diverse Attacks, ICB, 2012
Database Sensors Print/Replay Mask Makeup # Spoof Type # Subjects # Videos Year
CASIA-FASD RGB X 3 50 600 2012
![Page 48: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/48.jpg)
MSU-MFSD Database
• Two capture devices
• Build-camera in MacBook Air 13 (640*480)
• Front camera in Google Nexus 5 Android phone (720x480)
• Mostly used with CASIA and Replay
57
1. Wen et. al., Face Spoof Detection with Image Distortion Analysis, TIFS 2015
Database Sensors Print/Replay Mask Makeup # Spoof Type # Subjects # Videos Year
MSU-MFSD RGB X 3 55 280 2015
![Page 49: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/49.jpg)
MSU-USSA Database
• Live images from Internet
• Higher resolution compared with MFSD• Front-facing camera in the Google Nexus 5 Android
phone (1280 × 960).• Rear-facing camera in the Google Nexus 5 Android
phone (3264 × 2448)
• Spoof from 8 devices
58
1. Patel et. al., Secure Face Unlock: Spoof Detection on Smartphones, TIFS 2016
Database Sensors Print/Replay Mask Makeup # Spoof Type # Subjects # Videos Year
MSU-MFSD RGB X 3 55 280 2015
![Page 50: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/50.jpg)
OULU-NPU Database
• 6 camera, 1080P resolution
• Comprehensive evaluation protocols
59
1. Boulkenafet et. al., OULU-NPU: A Mobile Face Presentation Attack Database with Real-World Variations, FG, 2017
Database Sensors Print/Replay Mask Makeup # Spoof Type # Subjects # Videos Year
OULU-NPU RGB X 4 55 4950 2017
![Page 51: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/51.jpg)
SiW Database
• Pose, illumination, expression
• More subjects
• Comprehensive evaluation protocols
60
1. Liu et. al., Learning Deep Models for Face Anti-Spoofing: Binary or Auxiliary Supervision, CVPR, 2018
Database Sensors Print/Replay Mask Makeup # Spoof Type # Subjects # Videos Year
SiW RGB X 6 165 4478 2018
![Page 52: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/52.jpg)
CASIA-SURF Database
• Multi modalities
• More subjects/videos
61
1. Zhang et. al., CASIA-SURF: A Large-scale Multi-modal Benchmark for Face Anti-spoofing, CVPR 2019
Database Sensors Print/Replay Mask Makeup # Spoof Type # Subjects # Videos Year
CASIA-SURF RGB, NIR, Depth X 1000 21000 2019
![Page 53: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/53.jpg)
3DMAD Database
• Multi modalities
• More subjects/videos
62
1. Erdogmus et. al., Spoofing in 2D Face Recognition with 3D Masks and Anti-spoofing with Kinect, BTAS 2013
Database Sensors Print/Replay Mask Makeup # Spoof Type # Subjects # Videos Year
3DMAD RGB, Depth X 1 17 510 2014
![Page 54: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/54.jpg)
HKBU MAR Database
63
1. Liu et. al., rPPG Correspondence Feature for 3D Mask Face Presentation Attack Detection, ECCV 20182. Liu et. al., 3D Mask Face Anti-spoofing with Remote Photoplethysmography, ECCV 20163. Liu et. al., A 3D Mask Face Anti-spoofing Database with RealWorld Variations, CVPRW 2016
Database Sensors Print/Replay Mask Makeup # Spoof Type # Subjects # Videos Year
HKBU MAR RGB X 2 35 1008 2016
![Page 55: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/55.jpg)
SiW-M Database
• More spoof types
• Leave-one-out testing protocols
• Include hard live and spoof samples
64
1. Liu et. al., Deep Tree Learning for Zero-shot Face Anti-Spoofing, CVPR 2019
Database Sensors Print/Replay Mask Makeup # Spoof Type # Subjects # Videos Year
SiW-M RGB X X X 13 493 1630 2019
![Page 56: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/56.jpg)
CelebA-Spoof Database
• Rich variations and annotations
65
1. Zhang et. al., CelebA-Spoof: Large-Scale Face Anti-Spoofing Dataset with Rich Annotations , ECCV 2020
Database Sensors Print/Replay Mask Makeup # Spoof Type # Subjects # Videos Year
CelebA-Spoof RGB X X 4 10,177 625,537 (I) 2020
![Page 57: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/57.jpg)
CelebA-Spoof Database
• Testing protocols less challenging
• Better to design new protocols or
do cross-database testing
66
1. Zhang et. al., CelebA-Spoof: Large-Scale Face Anti-Spoofing Dataset with Rich Annotations , ECCV 2020
Database Sensors Print/Replay Mask Makeup # Spoof Type # Subjects # Videos Year
CelebA-Spoof RGB X X 4 10,177 625,537 (I) 2020
![Page 58: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/58.jpg)
Evaluation metrics
• Area Under the Curve (AUC)• 0.5 → useless model• <0.7 → sub-optimal performance• 0.7 – 0.8 → good performance• > 0.8 → excellent performance• 1 → perfect
• EER
• APCER / BPCER / ACER
• TPR at FPR = x (e.g. x = 0.2%)
67
![Page 59: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/59.jpg)
Evaluation metrics
• Area Under the Curve (AUC)
• EER• False pos rate = False neg rate
• APCER / BPCER / ACER
• TPR at FPR = x (e.g. x = 0.2%)
68
![Page 60: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/60.jpg)
Evaluation metrics
• Area Under the Curve (AUC)
• EER
• APCER / BPCER / ACER• ISO standard
• APCER: Attack Presentation Classification Error Rate
• BPCER: Bona Fide Presentation Classification Error Rate
• ACER: (APCER+BPCER)/2
• TPR at FPR = x (e.g. x = 0.2%)
69
![Page 61: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/61.jpg)
Evaluation metrics
• Area Under the Curve (AUC)
• EER
• APCER / BPCER / ACER
• TPR at FPR = x (e.g. x = 0.2%)
70
![Page 62: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/62.jpg)
Evaluation metrics
• We recommend:• EER
• APCER / BPCER / ACER
• TPR at FPR = x (e.g. x = 0.2%)
71
![Page 63: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/63.jpg)
Summary• Direct FAS
• Auxiliary FAS
• Temporal FAS
• Generative FAS
• Cross-domain FAS
• Unknow attack FAS
![Page 64: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/64.jpg)
Problem 1: Training-Testing Difference
• Cross-domain and unknown attack performances are still poor• EER for intra-testing: ~ 0% – 5%
• EER for inter-testing: ~ 15% - 50%
• How cross-domain testing contribute to real-world applications?
73
![Page 65: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/65.jpg)
Problem 2: Explainablity
• Spatial explainablity
• Temporal explainablity
• Spoofing process explainablity
• Research on camera and imaging
74
![Page 66: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/66.jpg)
Problem 3: New Attacks
• Can we transfer our knowledge of FAS to other attacks?
• Face/Generic adversarial attacks
• Face /Generic manipulation attacks
• Counter attacks to current methods
• 3D mask attacks with flashing light → rPPG methods
75
![Page 67: Session II Face Anti-Spoofing Generalizationcvlab.cse.msu.edu/pdfs/IJCB2020_Tutorial_Session_2.pdf · •Learn face anti-spoofing and face recognition at the same time •Apply a](https://reader033.vdocument.in/reader033/viewer/2022053108/6079a5fba7054729f51b8738/html5/thumbnails/67.jpg)
End of Session II
7 Minutes Break
ComputerVision Lab