iti presentation seminar... · q final q, = ·,,,,, 0.1· 0.5, if 5 1, if 5 35 0.02· 1.7, if 35....

CeRTH Informatics & Telematics Institute

ITI Presentation

Activity related recognition in AmIenvironments

Mr. Anastasios Drosou

(Dr. Tzovaras’ Group)

CeRTHITI

Outline

• Introduction• Activity Detection• Activity related recognition

– Human Tracking– Reaching Part– Grasping Part

• Classifiers• Results• Conclusions

ITI Presentation, Thessaloniki 16 March 2011 2

CeRTHITI

Definition of Biometrics

“Biometrics measure the unique physical or behavioural characteristics of individuals as a means

to recognize or authenticate their identity.”

Establish someone’s identity based on who he/she is rather than on what he/she poseses (e.g. ID cards,) or what he/she remembers (e.g. password).

• Verification problem: Is the user Mr. X? 1:1 comparison.• Identification: Who is the user? 1:many comparison.

3ITI Presentation, Thessaloniki 16 March 2011

CeRTHITI

Types of Biometrics

4

Description Types Pros Cons

Physiological Biometrics

A physical attributeunique to a person

Fingerprint, palm, iris, geometry, retina, facial characteristics, etc.

High Accuracy. Obtrusive,Uncomfortable,Easy to spoof,Intolerant of changes over time.

Behavioural Biometrics

Traits that are learned or acquired from a person

Facial Dynamics, Gait, Voice, Key-stroke Patterns, Activity related signals, etc.

Unobtrusive,Difficult to spoof,Continuous & on-the-moveauthentication,Rare changes.

Lower accuracy, yet...

ITI Presentation, Thessaloniki 16 March 2011

CeRTHITI

Motivation

We claim recognition potential because:

1. Personal behavioural patterns in everyday movements (i.e. gait, grimaces, standard movements, etc)

2. Physiology of the human body (height, arm length, finger lengths, etc.)

3. Bodymetric restrictions (i.e. impairments, etc.)4. Minimum Jerk Model (Flash & Hogan)5. Minimum Torque Change Model (Uno, Kawato & Suzuki)6. End state comfort effect (Rosenbaum et al.)7. …


CeRTHITI

Case study:Prehension biometrics

• Response of the person to specific stimuli.• Work related, everyday activities with no special

protocol.• Continuous authentication/recognition.• Multi-activity concept for human authentication.

6

2 stages of a prehension movement:i. a fast initial movement (reaching)ii. a slow approach movement (grasping)


CeRTHITI

7

Characteristics of a biometric that must be present in order to use the system for authentication purposes– Universality

• Universality: Every person should possess this characteristic• In practice, this may not be the case• Otherwise, population of non-universality must be small < 1%

– Uniqueness• Uniqueness: No two individuals possess the same characteristic. • Genotypical – Genetically linked (e.g. identical twins will have same biometric)• Phenotypical – Non-genetically linked, different perhaps even on same individual• Establishing uniqueness is difficult to prove analytically• May be unique, but “uniqueness” must be distinguishable

– Permanence• Permanence: The characteristic does not change in time, that is, it is time invariant• At best this is an approximation• Degree of permanence has a major impact on the system design and long term operation of biometrics. (e.g. enrollment,

adaptive matching design, etc.)• Long vs. short-term stability

– Collectability - Measurability• Collectability: The characteristic can be quantitatively measured.• In practice, the biometric collection must be:

– Non-intrusive– Reliable and robust– Cost effective for a given application

• Measurability: The trait can be measured with simple technical instruments– Performance – accuracy, speed, and robustness of technology used– Circumvention – The trait is easily measured with minimal discomfort

….then it can potentially serve as a biometric for a given application.

System-Level Criteria

CeRTHITI

Outline





CeRTHITI

Motion Templates

9

• Motion Energy Image (MEI)– Describes the motion energy for a given view of action.– Binary image.

• Motion History Image (MHI)– Intensity image.

1

0

),,(),,(−

=

−=τ

iT ityxDtyxE

if ( , , ) 1( , , )

(0, ( , , 1) 1) otherwiseTT

D x y tMHI x y t

max MHI x y tτ =

= − −


CeRTHITI

Radon Transforms (I)

10

0 20 40 60 80 100 1200

0.005

0.01

0.015

0.02

0.025

1. Face detection2. Centre of Face3. RIT Extraction

0 01

1( ) ( co s( ), sin ())J

jRIT t MHI r j u t y j u t

Jθ θ θ

=

∆ = + ∆ ∆ + ∆ ∆∑for 1,..., with 360 /ot T T θ= = ∆

• Radial Integration Transform (RIT):


CeRTHITI

0 01

1( ) ( cos( ), sin( ))T

tCIT t MHI x k t y k t

Tρ ρ θ ρ θ

=

∆ = + ∆ ∆ + ∆ ∆∑for 1,..., with 360 /ok K T θ= = ∆

• Circular Integration Transform (CIT):

11

0 10 20 30 40 50 60 70 800

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1. Face detection2. Centre of Face3. CIT Extraction

Radon Transforms (II)


CeRTHITI

12

Similarity Metrics

,

E(( )( ))cov( , )( , )corr x yµ µ

ρσ σ σ σ

− −= = = x y

x yx y x y

x yx y

2 21

( , ) ( ) ( )nE i ii

D x y=

= − = −∑x y x y

• Euclidian Distance

• Correlation Factor - Coefficients


CeRTHITI

• Camera based tracking:– Custom ITI Tracker– OpenNI Library Tracker

• Ascension Technology Corp. Magnetic Tracker (ground truth).

• Cyberglove® finger tracking.

Human Body Tracking


CeRTHITI

Face Detection & Tracking

14

• Viola & Jones’s real-time face detection method:– T weak classifiers form a strong one,– integral Image,– haar like features (subtracting the pixel

values in the dark from the bright rectangles),– AdaBoost algorithm.

• Mean Shift algorithm:– Bases on spatial and colour information

between sequential frames.– Bhattacharyya distance


CeRTHITI

Background Extraction –Skin Colour Filtering –

Motion Detection

15

• Location of the face • Disparity images (stereoscopic Camera)

• No training required.• Computational inexpensive.• Simple approach - explicit rules.• Skin cluster boundaries of RGB & HSV.

( ) ( ) ( )D I S I B I= ∩

1

2, if ( ( , )) 1( , )

(0, ( , ) 1), otherwiset

tt

M I x yMHI x y

max MHI x y−

== −

1 2( ) ( ) ( )tM I D I D I≡ −


CeRTHITI

Movement’s Tracking


CeRTHITI

ITI Body Tracker Evaluation


CeRTHITI

Primesense® TOF Sensor - OpenNI Open Source Library

• Real time & accurate.• Human body form recognition and

segmentation ecognizes the human • Human body tracking by simultaneousl

tracking of 48 essential points of the human body in the 3D space.

• Trained machine learning algorithmby millions of manually taggedimages of people in different poses.

• OpenNI manages to adjust the most appropriate skeleton model to each human body in terms of size and pose.


CeRTHITI

Cyberglove® finger tracker

• 4 DoF for each finger• 3 phalanxes for each finger• 3 DoF for the orientation of the

whole palm• Time-stamped Data


CeRTHITI

Outline





CeRTHITI

Feature Extraction (I)

21

x

y

*The diameter of the dots is anti proportional to the

depth of the tracked point.

HeadRight HandLeft Hand


CeRTHITI

Phone Conversation Trajectories

22

Subject 1 Subject 2 Subject 3

1

2


CeRTHITI

Office Panel Trajectories

23

Subject 1 Subject 2 Subject 3

1

2


CeRTHITI

Warping the trajectory

24

Enrollment Authentication

Client:

|| |||| ||

Head Phone

Head Phone

P PqP P ′

−=

−

( ') ( )P D q PD= Impostor:


CeRTHITI

Feature Extraction (I)

25

• Exact location of the head/hand 3D position, at time t (timestamp tracking).

• Dynamic Spatial Cost

dtds

v zyxzyx

,,,, = , ,

, ,x y z

x y z

dva

dt=

2)1(

2)1(

2)1( )()()()1()( −−− −+−+−+−= ttttttpp zzyyxxtStS


CeRTHITI

Feature Extraction (II)

26

• Curvature - Derivative

• Torsion - Derivativeabc

csbsassPi

))()((4)(

^^^^

* −−−=κ gdba

PPP iiis +++

−= −+

22)()(3)( 1

*1

** κκκ

)(6

)(6(

21)( **

*

iii Pgmm

HPdef

HPκκ

τ⋅

+⋅

=−+

ghdbaPPPrPPP iisiii

is +++++−

= −+

222))(6/)()(()()(4)(

***1

*1

** κκττττ


CeRTHITI

Spatiotemporal Activity Surface

1, ,( , ) min ( , ) iRP k K i kf d RP sθ φ = …=

3. Triangulation

1. PCA on 3D Trajectories2. 3-dimensional manifold trajectories

using time as 3rd dimension


CeRTHITI

Intra-similarity vS. Inter-variance of AS


CeRTHITI

Static Anthropometric –Attributed Graph Matching

B,A,, 71== iEVG

7 Attributed Edges8 Nodes

(AGM) problem:

Supports: • Full-graph matching (n`=n)• Sub-graph matching (n`>n)


( )1min || ' ||s q

j r j jjpW B PB+=

−∑

0 0' ( )Tj j jB P B P Mε= +

CeRTHITI

Confidence Factor –Ergonomy

13

missHead missRHand missLHandq

frames

N N NfN

+ += −

, ·q final qf b f=

, ,

,

, ,

0.1· 0.5, if 51, if 5 35

0.02· 1.7, if 35

torso object torso object

torso object

torso object torso object

d d cmb cm d cm

d d cm

+ <= ≤ ≤− + >


CeRTHITI

Outline





CeRTHITI

Feature Extraction

32

• Angle between the phalanxes of each finger, at time t (timestamp tracking).

• Dynamic Travel Cost


1 20 2 , ,..., aF θ θ θ=

ddtθθω =

2

2

d dadt dtθω θ

= =

* 2

2

[ ( )]( ( ), ) ( )(1 )j j j j j

j j j

k a T T aV a t T

r s−

= +

*( ( )) ( 1), 0j j j j jT a t k ln a k= + ≥Joint’s optimal time

The cost of moving joint j through an angle of size in the given time

CeRTHITI

Dynamic Time Warping (DTW) Classifier

• Based on dynamic Programming• No training required• Simple Implementation• Fast


( 1) ( 1)1

( , ) ( , ) ( , ) ( , )k

k k k k k k m mm

D x y D x y c x y c x y− −=

= + =∑

• The problem of finding the optimal warping path can be reduced to finding this sequence of nodes (xk; yk), which minimizes D(xk; yk) along the complete path.• The main aim is to find the path for which the least cost is associated.

1 1( , )

L L

c i ji j

S V p q= =

=∑∑

· ( , )M c minD S D L L=

Area size

Total dissimilarity

CeRTHITI

• Fully connected, left-to-right, five-state (N=5) HMM.

• The triplet:

– πj is the probability of the jth state being the first state among all the trajectories,

– aij is the probability of the jth state occurring immediately after the ith state, – bj denotes the PDF of the jth state.

• The observational data from each state of the HMM are generated according to a PDF dependent on the instant of tth state, .

• Given HMMs for the L enrolled subjects and the new trajectory vectors, we assign user label m as the HMM that maximizes the likelihood (ML principle).

Hidden Markov Model (HMM) Classifier

35

, , j ij jbλ π α=

( )j tb O

S4 S3

S2

S1

S5


CeRTHITI

Spherical Harmonics

| |2 1 ( )!( , ) ( )( ( ) ( ))4 ( )!

m ml l

l l mY P cos cos m isin ml m

θ φ θ φ φπ+ −

= ++

2

0 0( , ) ( , ) ( , )

i i

l m mm RP l RP lc f Y d f Y d d

π πθ φ θ φ θ φ θ φ

Ω= Ω =∫ ∫ ∫

*l

l

Km

l lm K

c c=−

= ∑

1 1 2 21

N

tot j j N Nj

S w S w S w S w S=

= = + +…+∑

*l yz

ly

cc

µσ−

=

Rotational Invariance:

Z-Normalization:

Fusion among Reference Points:


Spherical Harmonics series

Spherical Harmonics coeffs.

CeRTHITI

ACTIBIO dataset (AmI indoor environment):• 29 Subjects,• 2 Time sessions - 8 repetitions in total,• Annotated frame sequences,• 5 cameras in total:

– 1 stereo camera (BumblebeeXB3 Point Grey Inc), – 2 usb cameras (Lateral-Zenithal)– …

Database


CeRTHITI

Outline





CeRTHITI

Activity Detection Results


Events Phone Panel Mic. Panel Drink Hands Up

Phone 93.1% 0% 0% 6.9% 0%Panel 0% 89.7% 10.3% 0% 0%Mic. Panel 0% 3.44% 3.44% 93.1% 100%Drink 0% 10.3% 86.2% 3.44% 0%

CeRTHITI

Tracking Results

40

Subject’s No. Confidence Score Subject’s No. Confidence ScoreSubject 1 0.835227 Subject 3 0.818681Subject 3 0.937500 Subject 4 0.846774Subject 5 0.933333 Subject 6 0.927778

… … … …Subject 27 0.935897 Subject 28 0.720588Subject 29 0.866667


CeRTHITI

Authentication HMM Results

41

Phone Conv. (EER=15%)

Office Panel (EER=9.8%)


CeRTHITI

Authentication Performance

ITI Internal Database (15 Subjects

ACTIBIO Database (29 Subjects)

ACTIBIO Database (29 Subjects)

Score Level Fusion:

0.25* 0.75*total Phone PanelS S S= +


CeRTHITI

Recognition Performance (HMM)

Score Level Fusion:

0.25* 0.75*total Phone PanelS S S= +ITI Presentation, Thessaloniki 16 March 2011

CeRTHITI

Spherical Harmonics Results


(EER=11.4%)

(EER=16.7%)

(EER=15%)

CeRTHITI

Conclusions

• Recognition using just a common stereoscopic camera.• Unobtrusive, on the move, continuous authentication.• Significant recognition potential just by the spatial

information of the trajectories.• Higher authentication rates are expected, given the

relative entropies.• Rotational invariance from spherical harmonics analysis.• Privacy enabled method, since no obtrusive information

is stored.• Activity-related biometric authentication provided very

promising results and is expected to maximize the performance of a multimodal biometric system.


iti presentation seminar... · q final q, = ·,,,,, 0.1· 0.5, if 5 1, if 5 35 0.02· 1.7, if 35....

Documents