![Page 1: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/1.jpg)
Interacting with Face
Jian Sun
Visual Computing Group
Microsoft Research Asia
![Page 2: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/2.jpg)
Face is unique
• Face-to-face interaction/communication is essential
![Page 3: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/3.jpg)
Has worked for millions of years
![Page 4: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/4.jpg)
Convey nonverbal clues
![Page 5: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/5.jpg)
Exchange feeling
![Page 6: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/6.jpg)
Psychological & physical connection
![Page 7: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/7.jpg)
Computer Face is challenging
![Page 8: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/8.jpg)
Face understanding is the key
Face Understanding (low-level, high-level)
Application-dependent
![Page 9: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/9.jpg)
Face understanding is the key
Where?
Direction?
Emotion?
Who?
![Page 10: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/10.jpg)
Low-level
face understanding
• Face Detection
– Detect 2D frontal face
• Face Alignment
– Locate face features
• Face Segmentation
– Extract the whole face region
• 3D Face Pose Tracking
– Track the face pose and orientation
![Page 11: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/11.jpg)
Low-level
face understanding
• Face Detection
– Detect 2D frontal face
• Face Alignment
– Locate face features
• Face Segmentation
– Extract the whole face region
• 3D Face Pose Tracking
– Track face pose and orientation
![Page 12: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/12.jpg)
Low-level
face understanding
• Face Detection
– Detect 2D frontal face
• Face Alignment
– Locate face features
• Face Segmentation
– Extract the whole face region
• 3D Face Pose Tracking
– Track face pose and orientation
![Page 13: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/13.jpg)
Low-level
face understanding
• Face Detection
– Detect 2D frontal face
• Face Alignment
– Locate face features
• Face Segmentation
– Extract the whole face region
• 3D Face Pose Tracking
– Track the face pose and orientation
![Page 14: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/14.jpg)
Low-level
face understanding
• Face Detection
– Detect 2D frontal face
• Face Alignment
– Locate face features
• Face Segmentation
– Extract the whole face region
• 3D Face Pose Tracking
– Track face pose and orientation
![Page 15: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/15.jpg)
Low-level
face understanding
• Face Detection
– Detect 2D frontal face
• Face Alignment
– Locate face features
• Face Segmentation
– Extract the whole face region
• 3D Face Pose Tracking
– Track face pose and orientation
![Page 16: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/16.jpg)
Low-level
face understanding
• Face Detection
– Detect 2D frontal face
• Face Alignment
– Locate face features
• Face Segmentation
– Extract the whole face region
• 3D Face Pose Tracking
– Track face pose and orientation
![Page 17: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/17.jpg)
Face understanding framework
detection alignment and
pose tracking
𝑥1𝑥2⋮𝑥𝑛
representation
Low-level Representation:
High-level Understanding:
Male or Female? Smiling or not? Same or not?
Identify, age, race, expression, emotion (sad, happy…), …
…
![Page 18: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/18.jpg)
Knowing where you are
• Focus/Exposure control
![Page 19: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/19.jpg)
Knowing where you are
• For TV - personal settings, parental control, and
energy saving
![Page 20: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/20.jpg)
Knowing where you are
• “Auto Self Portrait”
![Page 21: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/21.jpg)
Face Computer Face
Face Understanding (low-level, high-level)
Application-dependent
![Page 22: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/22.jpg)
Protect Privacy
• Facial region segmenation
![Page 23: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/23.jpg)
Protect Privacy
• Background blur
![Page 24: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/24.jpg)
Protect Privacy
• Virtual background
![Page 25: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/25.jpg)
Knowing facial 3D pose
![Page 26: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/26.jpg)
Knowing facial 3D pose
(in Microsoft LifeCam)
![Page 27: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/27.jpg)
Knowing facial expression
• Chatting room
– Hang out with friends before & after playing games
• Live meeting
• Facebook, Youtube
– Funny greetings, avatar comments
![Page 28: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/28.jpg)
Knowing facial expression
Input Image
2D Points from AAM Tracker
Action Units
…
Avatar Key
Frames
…
Frown Normal
Pouty Others Stretched
Happy Neutral Unhappy
Face State Analysis
Frown State
Mouth Shape State
Mouth Emotion State
![Page 29: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/29.jpg)
Knowing facial expression
![Page 30: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/30.jpg)
Knowing identities
![Page 31: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/31.jpg)
Knowing identities - face recognition
• Recognition feature is learned from data
Learning-based local
descriptor
Concatenated patch
histogram
PCA dimension
reduction
Zhimin Cao, Qi Yin, Xiaoou Tang, and Jian Sun. Face Recognition with Learning
based Descriptor. CVPR 2010.
![Page 32: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/32.jpg)
![Page 33: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/33.jpg)
Try it out
![Page 34: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/34.jpg)
Xbox Kinect
• Body skeleton tracking using a depth camera
![Page 35: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/35.jpg)
Kinect Identity
• A skeleton a game character / a player profile
• But, when player leaves/back or tracking fails/resumes
– New player or an existing payer? Who is who?
![Page 36: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/36.jpg)
![Page 37: Interacting with Face · 2018-01-04 · Knowing identities - face recognition •Recognition feature is learned from data Learning-based local descriptor Concatenated patch histogram](https://reader034.vdocument.in/reader034/viewer/2022042923/5f6fb77e0e4274112a3eb5ff/html5/thumbnails/37.jpg)
Summary
• Interacting with face is unique
• Face understanding is the key