face tracking and person action recognition - update
DESCRIPTION
Face Tracking and Person Action Recognition - Update. Sascha Schreiber. Overview. Recapitulation of methodology for action recognition Face tracking with I-Condensation Recognition performance comparison on actions from the m4 dataset Kalman filtering of occluded gestures Outlook. - PowerPoint PPT PresentationTRANSCRIPT
M4 meeting@Prague 22-23.01.2004
Institute for Human-Machine CommunicationMunich University of Technology
Sascha Schreiber
Face Tracking and Person Face Tracking and Person Action Recognition - UpdateAction Recognition - Update
Sascha Schreiber2/14
Institute for Human-Machine CommunicationMunich University of Technology
• Recapitulation of methodology for action recognition
• Face tracking with I-Condensation
• Recognition performance comparison on actions from the m4 dataset
• Kalman filtering of occluded gestures
• Outlook
Overview
Sascha Schreiber3/14
Institute for Human-Machine CommunicationMunich University of Technology
Person Action Recognition
Extraction of person locations
Temporal segmentation
Feature calculation
Classification of segments
Face detection/Blob tracking
Global Motion Features
Bayesian Information Criterion
Hidden Markov Models
Actions, timestamps
Sascha Schreiber4/14
Institute for Human-Machine CommunicationMunich University of Technology
Person Action Recognition
Extraction of person Extraction of person locationslocations
Temporal segmentation
Feature calculation
Classification of segments
Face detection/Blob trackingFace detection/Blob tracking
Global Motion Features
Bayesian Information Criterion
Hidden Markov Models
Actions, timestamps
Sascha Schreiber5/14
Institute for Human-Machine CommunicationMunich University of Technology
Face Tracking
Particle filtering with ICondensation( ) ( ){( , ), 1,..., }i it tx i N ( ) ( ){( , ), 1,..., }i it tx i N • N weighted particles
( ) ( ) ( )1 ( | )i i i
t t t tp y x ( ) ( ) ( )1 ( | )i i i
t t t tp y x • Updating using their likelihood
• Sampling from prediction density
- Standard Condensation sampling
- Sampling from importance function for reinitialisation
- Importance sampling with weighting correction factor
Introduction of importance function: skin color distribution
Automatic initialization by pyramid sampling and MLP classification
Sascha Schreiber6/14
Institute for Human-Machine CommunicationMunich University of Technology
Performance of Face Tracking
Standard Condensation ICondensation
Demonstration of difference between:
Sascha Schreiber7/14
Institute for Human-Machine CommunicationMunich University of Technology
Person Action Recognition
Extraction of person locations
Temporal segmentation
Feature calculation
Classification of segmentsClassification of segments
Face detection/Blob tracking
Global Motion Features
Bayesian Information Criterion
Hidden Markov ModelsHidden Markov Models
Actions, timestamps
Sascha Schreiber8/14
Institute for Human-Machine CommunicationMunich University of Technology
IDIAP training data (TRN 01-30), IDIAP test data (TST 01-30)
Continuous HMMs (6 states, 3 mixtures)
Sit down Stand up NoddingShaking head
Writing Pointing Score
Sit down 9 0 0 0 0 1 90%
Stand up 1 9 0 0 1 3 64%
Nodding 2 1 215 58 6 8 74%
Shaking head
0 0 22 16 4 1 37%
Writing 0 0 38 20 468 24 85%
Pointing 0 0 2 0 0 70 97%
Overall 80%
Recognition Performance m4
Sascha Schreiber9/14
Institute for Human-Machine CommunicationMunich University of Technology
IDIAP training data (TRN 01-30), IDIAP test data (TST 01-30)
Discrete HMMs (6 states, codebook 1500)
Sit down Stand up NoddingShaking head
Writing Pointing Score
Sit down 9 0 0 0 0 1 90%
Stand up 3 10 0 0 0 1 71%
Nodding 1 3 251 2 33 0 87%
Shaking head
0 0 30 2 11 0 5%
Writing 0 0 28 0 515 7 94%
Pointing 4 1 4 0 6 57 79%
Overall 86%
Recognition Performance m4
Sascha Schreiber10/14
Institute for Human-Machine CommunicationMunich University of Technology
Occluded Gestures
Classification result
Classification
Compensationof occlusion
Stream-segmentation
Feature-extraction
Smoothed featurestream
Featurestream
Segmented Featurestream
Video-stream
Occlusion
Scenario: Person walking on front of a tracked object
Sascha Schreiber11/14
Institute for Human-Machine CommunicationMunich University of Technology
Occluded Gestures
Application for Kalman filtering:
• Calculation of an estimate
1ˆ ˆk kx Ax
Time update equation
ˆ ˆ ˆ( ) k k k k kx x K z HxMeasurement update equation
• Discrete-time process: 1 1 k k kx Ax w
k k kz Hx v
ˆkx
Sascha Schreiber12/14
Institute for Human-Machine CommunicationMunich University of Technology
Occluded Gestures
xkˆ)1(
yk Kalman-
filterKalman-
filterKalman-
filter
xkn
ˆ)(
xkˆ)2(
• N action-specialized Kalman-Filters, each trained for a special gesture to be recognized by the HMM
Improving featurestream by smoothing with :
xkˆ
yk Kalman-
filter
• One general Kalman-Filter for the disturbed featurestream
Sascha Schreiber13/14
Institute for Human-Machine CommunicationMunich University of Technology
Performance of Kalman filtering
Score
Featurestream unoccluded & unfiltered 79,86%
Featurestream occluded & unfiltered 56,75%
Featurestream occluded & filtered (general) 57,98%
Featurestream occluded & filtered (specialized) 60,12%
IDIAP training data (TRN 01-30), IDAP test data (TST 01-30)
Continuous HMMs (6 states, 3 mixtures)
Sascha Schreiber14/14
Institute for Human-Machine CommunicationMunich University of Technology
• Implementation of extended Kalman filter• Head orientation tracking• Integration of face recognition into particle filter• Further improvement of action detection on m4 data• Connection to Meeting Segmentation / Multimodal
Recognizer
Outlook
M4 meeting@Prague 22-23.01.2004
Institute for Human-Machine CommunicationMunich University of Technology
Face Tracking and Person Face Tracking and Person Action Recognition - UpdateAction Recognition - Update
Sascha Schreiber