Activity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and Recognition
1
Dimitrios TzovarasDimitrios Tzovaras
Centre of Research & Technology Centre of Research & Technology -- HellasHellasInformatics & Telematics InstituteInformatics & Telematics Institute
• What is activity recognition?• Some indicative examples• Activity recognition for authentication• Gait as a biometrics• Gait recognition – Potential
OutlineOutline
• Gait recognition – Potential• State-of-the-art approaches• Gait recognition in realistic applications• Improvement of Gait recognition using soft biometrics• Conclusions
2
ActivityActivity recognitionrecognitionAims to recognize the actions and goals of one ormore agents from a series of observations on theagents' actions and the environmental conditions.
Introduction Activity RecognitionActivity Recognition
3
ActivityActivity recognitionrecognition approachesapproaches::In real life activity recognition, systems typicallyfollow a hierarchical approach:
Introduction Activity RecognitionActivity Recognition
� Lower levels : Background-foreground segmentation,� Lower levels : Background-foreground segmentation,tracking and object detection
� Mid-level: Action recognition modules
� High-level: Reasoning engines, which encode theactivity semantics based on the lower level action-primitives.
4
MidMid--level,level, ModellingModelling andand recognisingrecognising activitiesactivities
Introduction Activity RecognitionActivity Recognition
Simple activities Complex activities
Non-
Approaches
Volumetric ParametricGraphical
SyntacticKnowledge
ParametricVolumetric Parametric
ModelsSyntactic
Based
•2D-templates(MHI, MEI)• 3D objects
models• Manifold
LearningMethods
•Space-timefiltering•Constellation of parts•Sub-volume
matching•Tensors
• HMM• Conditional
RandomFields
• Linear DynamicalSystems
• Non-linearDynamicalSystems
• Belief nets (Dynamic BayesianNet)
• Propagationnets
• Grammars (Context FreeGrammars)
•StochasticGrammars(StochasticContext FreeGrammars)•Attribute Grammars
• Logic BasedApproaches
• Constraint Satisfaction
• Ontologies
(Adapted from Turaga et al., 2008)5
MidMid--level,level, ModellingModelling andand recognisingrecognising activitiesactivities::
Introduction Activity RecognitionActivity Recognition
� Non parametric approaches : Extract a set of featuresfrom each frame of the video. The features are then matched
to a stored template.
� Volumetric approaches : Do not extract features on a� Volumetric approaches : Do not extract features on aframe by frame basis. A video is considered as a 3D volume ofpixel intensities and extend standard image features to the 3D
case.
� Parametric approaches: Impose a model on the temporaldynamics of the motion. The particular parameters for a classof actions is then estimated from training data.
(Turaga et al., 2008)6
MidMid--level,level, ModellingModelling andand recognisingrecognising activitiesactivities::
Introduction Activity RecognitionActivity Recognition
� Graphical models : Probabilistic model for which a graphdenotes the conditional independence structure between
random variables.
� Syntactic approaches: Syntactic approaches such asGrammars express the structure of a process using a set ofGrammars express the structure of a process using a set ofproduction rules.
� Knowledge and Logic-based approaches: Rely onformal logical rules to describe common-sense domainknowledge to describe activities. Logical rules are useful to:
a) express domain knowledge as input by a user orb) present the results of high-level reasoning in an intuitive andhuman-readable format.
(Turaga et al., 2008)7
Two main approaches :• methods based on various sensors placed on
the subject to extract meaningful features• methods based on video analysis to detect
human activity (one of the most promising andchallenging applications of computer vision)
Introduction Activity RecognitionActivity Recognition
challenging applications of computer vision)
8
Approaches of Video based action Approaches of Video based action recognitionrecognition
Introduction Activity RecognitionActivity Recognition
Direct mapping of image cuesModel based
Global/holistic Patch -based
9
e.g.• Joint angles• Joint positions
Global/holistic
e.g. • Silhouettes• Edge images • Optical flow
Patch -based
e.g.• Spatio-temporal
silent points(Oikonomou et al,. 2006)
(From Bobick & Davis, 2001)
(From Sheikh et al, 2005)
Introduction Video based Activity RecognitionVideo based Activity Recognition
Model-based approach
Activities: (a) walking, (b) sitting, (c) standing, (d) running
Representation of an action in 4 -space
Modeling the action
Matching algorithm
Classify activity
For each action Ai, i ���� 1,2,…N do
10
(Sheikh et al., 2005)
Representation of an action in 4 -space
Action in XYZ space Action in XYT space
For each action Ai, i ���� 1,2,…N do
1. Normalisation. Compute a similarity transform, transforming the mean of the point to the origin and making the average distance of the points from the origin equal to (Separately for each action instance)
2. Compute Subspace Angle between W and Wt:• Compute Orthogonal Bases: Uses SVD to reliably compute
orthonormal bases of W΄and Wt, and• Compute Projection: Using the iterative procedure described in
Bjork and Golup (1973) for j � 1, .... p
• Find Angle: Compute
2
΄W~
tW~
Select
Where W΄ corresponds to the projection of an action instance, and matrices W1, W2, … WN each modeling the N different actions.
Alignment Minimisation
Global/holistic approach: Activity recognition in o ffice
Introduction Video based Activity RecognitionVideo based Activity Recognition
11
Alignment Minimisation
(Rusu et al., 2009)
Activity recognition in a vehicle
Head orientation
Hands vertical position
Hands horizontalposition
Activities: (a) drive-forward, (b) backup, (c) shift-gear, (d) turn-left, (e) turn-right (f) touch-radio
Introduction Video based Activity RecognitionVideo based Activity Recognition
12
Head orientation
Trace of covariance matrix from hand blob
Hand blob’s distances from steering wheel, transmission lever, and instrument panel
(Park & Trivedi, 2005)
Introduction VariousVarious sensors basedsensors based Activity RecognitionActivity Recognition
Activity Recognition from Accelerometer Data
Activities: (a) Standing, (b) Walking, (c) Running, (d) Climbing up stairs, (e) Climbing down stairs, (f) Sit-ups, (g) Vacuuming, (h) Brushing teeth.
13
Data Collection Apparatus
(Ravi et al., 2005)
� Content Based Video Analysis: With video sharingwebsites experiencing relentless growth, it has becomenecessary to develop efficient indexing and storageschemes to improve user experience. Most commercially
Introduction Activity RecognitionActivity Recognition
Applications of activity recognition (1/4):Applications of activity recognition (1/4):
viable application: summarization and retrieval ofconsumer content (i.e. sports videos) (Chang, 2002).
� Animation and Synthesis: The gaming and animationindustry rely on synthesizing realistic humans and humanmotion.
14
� Interactive Applications and Environments:Essential for designing human-machine interfaces withvital applications:
a) Context aware computing to support cognitively impaired
Introduction Activity RecognitionActivity Recognition
Applications of activity recognition (2/4):Applications of activity recognition (2/4):
a) Context aware computing to support cognitively impairedpeople
b) Health monitoring and fitness
c) Seamless services provisioning based on the location andactivity of people.
d) Smart rooms that can react to a users gestures (i.e. atcommunity-dwelling for older people).
15
� Security and Surveillance: Automatic recognition ofabnormalities in a camera’s field of view.
� Abnormal activities: Activities that occur rarely and have notbeen expected in advance
Introduction Activity RecognitionActivity Recognition
Applications of activity recognition (3/4):Applications of activity recognition (3/4):
been expected in advance
� Example:• In working Environment:
ـ Raising handsـ Lying down
• In community-dwellingـ Falling down (backwards/forwards)ـ Lying downـ No movementـ Raising hands
(Hu et al., 2009)16
� Behavioural biometrics: Biometrics involves study ofapproaches and algorithms for uniquely recognisinghumans based on physical or behavioural cues. Regardingactivity related biometrics , although gait has been
Introduction Activity RecognitionActivity Recognition
Applications of activity recognition (4/4):Applications of activity recognition (4/4):
excessively studied, to our knowledge nono otherother bodybodymotionsmotions havehave beenbeen reportedreported toto bebe utilisedutilised asas biometricbiometricforfor authenticationauthentication purposespurposes ..
17
� Biometric purposes - Activity Related AuthenticationExtracting an activity-related signature that characterises theindividual by the way he/she responses to a stimulus (e.g.during working in an office).
Examined Activities:
CERTH/ITI proposed activity recognition system
CERTH/ITI Activity recognition systemCERTH/ITI Activity recognition system
Examined Activities:� Talking to panel� Answering to a phone call
� Security purposesDetecting abnormal activities that denote danger (e.g. in theworking environment/office)
Examined Activities:� Raising hands
18
�Activity recognition Approaches– Motion Energy/History Images– Feature Extraction (RIT/CIT)
CERTH/ITI proposed activity recognition system MethodsMethods
ApproachesApproaches
�Activity-related authentication– Tracking three points of interest (head, hands)– Authentication based on HMMs
19
Motion ImagesMotion Images
• Motion Energy Image (MEI)– Describes the motion energy for a given view of action.– Binary image.
U1
),,(),,(−
−=τ
ityxDtyxE
CERTH/ITI proposed activity recognition system MethodsMethods
• Motion History Image (MHI)– Intensity image.
U0
),,(),,(=
−=i
T ityxDtyxE
−−=
)1)1,,(,0max(),,(
tyxHtyxHT
τ 1),,( =tyxDotherwise
if
,
,
20
Motion Images Motion Images –– Activity RecognitionActivity Recognition
CERTH/ITI proposed activity recognition system MethodsMethods
Motion Energy Image (MEI)
Motion History Image (MHI)
21
MEI Templates of Six MicroMEI Templates of Six Micro--ActivitiesActivities
CERTH/ITI proposed activity recognition system MethodsMethods
Pick up or put back Phone Conversation Activate Alarm
Talking to the microphone Raising Hands Drink from Glass
22
Flow ChartFlow Chart
Grab frames from office activities.
Extraction of the Motion History
Image.
Extraction of the Motion Energy
Image.
0.01
0.015
0.02
0.025
CERTH/ITI proposed activity recognition system MethodsMethods
Get difference from sequential
frames.
Convert difference images to grayscale.
RIT Extraction+
CIT Extraction
Activityrecognized.
Comparisonof eucledian distance &
Correlation Factor
0 20 40 60 80 100 1200
0.005
0 10 20 30 40 50 60 70 800
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
TrueFalse
Prototype MHI/MEI
23
RIT ExtractionRIT Extraction
1. Face detection
2. Centre of Face
3. RIT Extraction
CERTH/ITI proposed activity recognition system MethodsMethods
0 2 0 4 0 6 0 8 0 1 0 0 1 2 00
0 . 0 0 5
0 . 0 1
0 . 0 1 5
0 . 0 2
0 . 0 2 5
24
1. Face detection
2. Centre of Face
3. CIT Extraction
CERTH/ITI proposed activity recognition system MethodsMethods
CIT ExtractionCIT Extraction
0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 00
0 . 1
0 . 2
0 . 3
0 . 4
0 . 5
0 . 6
0 . 7
0 . 8
0 . 9
1
• Euclidian Distance
MetricsMetrics
∑ =−=−++−+−=
n
i iinnE yxyxyxyxD1
22222
211 )()(...)()(
CERTH/ITI proposed activity recognition system MethodsMethods
• Correlation Factor - Coefficients
yx
yx
yxyx
yxyxyxcorr
σσ
µµ
σσρ
)))(((),cov(),( ,
−−Ε===
26
RHLH
+
Face Tracking
BodymetricRestriction
s
Skin ColorDetection
DepthInformation
Motion Detection
Face Detection
Authentication System OverviewAuthentication System Overview
CERTH/ITI proposed activity recognition system MethodsMethods
+
SignatureProcessing
HMM Training
HMM Classifier
pre-Interpolation
Smoothing TrimmingKalmanFiltering
uniform Interpolation
Signature ExtractionSignature Extraction
CERTH/ITI proposed activity recognition system MethodsMethods
Interpolation Filtering Interpolation
Activity: Office Panel Activity: Phone Conversation
Trajectory’s Features for Trajectory’s Features for authenticationauthentication
CERTH/ITI proposed activity recognition system MethodsMethods
Uniform Interpolation(losing speed information)
Trajectory Features Extraction (curvature,
torsion)
29
information)
-0.08 -0.06 -0.04 -0.02 0 0.02 0.04 0.06 0.08-0.25
-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
Trajectory’s Features for authenticationTrajectory’s Features for authentication
40
50
60
70
80
abc
csbsassPi
))()((4)(
^^^^
* −−−=κ
gdba
PPP ii
is +++−
= −+
22
)()(3)( 1
*1
** κκ
κ
-9 -8 -7 -6 -5 -4 -3 -2 -1 0 1
x 108
-1500
-1000
-500
0
500
1000
-500 -400 -300 -200 -100 0 100 2000
10
20
30
40
50
60
70
80
0 10 20 30 40 50 60 70-1500
-1000
-500
0
500
1000
Motion symmetry perception:a) Curvature sub-signature b) Torsion sub-signature
CERTH/ITI proposed activity recognition system MethodsMethods
0 10 20 30 40 50 60 700
10
20
30
Trajectory Features Extraction (curvature,
torsion)
2/)(^
cbas ++=)(
6)(
6(2
1)(
***
iii Pgmm
H
Pdef
HP
κκτ
⋅+
⋅=
−+
ghdba
PPPrPPP iisiii
is +++++−
= −+
222
))(6/)()(()()(4)(
***1
*1
** κκττττ
gdba +++ 22
3
1
1
1
1
!3
1
222
111
111+
+++
+++
−−− ⋅∆==
HV
zyx
zyx
zyx
zyx
abcabcdef
iii
iii
iii
iii
ghdbar +−−+= 3222
0 10 20 30 40 50 60 70-500
-400
-300
-200
-100
0
100
200
0 10 20 30 40 50 60 70-9
-8
-7
-6
-5
-4
-3
-2
-1
0
1x 10
8
,whereas
30
Hidden Markov ModelHidden Markov Model
• HMM is characterized by:1. N, the number of states in the model2. M, the number of distinct observation symbols per state3. The state probability distribution A={aij}4. The observation symbol probability distribution5. The initial state distribution
CERTH/ITI proposed activity recognition system MethodsMethods
5. The initial state distribution
• Three basic Problems:1. Evaluation Problem – how to compute the probability that the
observed sequence was produced by the HMM.2. Determination of a best sequence of model states – since no
correct state, find the optimal solution.3. Training problem – optimization of model parameters so to best
account for the observed signal
31
HMM ClassificationHMM Classification• Activity – related signature of a person for a specific
action is the set of stored Hidden Markov Model parameters, trained with action sample(s) of the particular user
• Enrolment: HMM Signatures are created by training an HMM using a limited training set (3 or more samples of an action) for a given person.
CERTH/ITI proposed activity recognition system MethodsMethods
an action) for a given person.
• Selected experimentally: No of states=5, No of distinct observation symbols (3D position of head and moving head)
• Authentication:ـ Segmented action is fed to the claimed user’s stored HMMـ Action’s similarity to the user’s behavior is evaluated (log –likelihood calculation) using forward – backward algorithmـ Acceptance/Rejection is based on thresholding the calculated likelihood32
C l i e n t C l i e n t -- I m p o s t o rI m p o s t o r
Phone Conversation
CERTH/ITI proposed activity recognition system ResultsResults
Office Panel
Merged Signatures
Phone
Conversation
SNR = 28
Results FAR Results FAR –– FRR (1/2)FRR (1/2)
CERTH/ITI proposed activity recognition system ResultsResults
Office Panel
SNR = 28
Results FAR Results FAR –– FRR (2/2)FRR (2/2)
SNR = 28
CERTH/ITI proposed activity recognition system ResultsResults
Merged
Signature
Summarising…Summarising…• Three (3) activities tested up to now with very promising
results.Hands Up Phone Conversation Talking to Mic. Panel
CERTH/ITI proposed activity recognition system ConclusionsConclusions
• Body Tracking - Signature Extraction – Trajectories –Identification
0 10 20 30 40 50 60 70-500
-400
-300
-200
-100
0
100
200
0 10 20 30 40 50 60 70-9
-8
-7
-6
-5
-4
-3
-2
-1
0
1x 10
8
36
ActivityActivity--relatedrelated biometricbiometric authentiauthenti--cationcation providedprovided veryvery promisingpromising resultsresultsandand isis expectedexpected toto maximizemaximize thethe
CERTH/ITI proposed activity recognition system ConclusionsConclusions
andand isis expectedexpected toto maximizemaximize thetheperformanceperformance ofof aa multimodalmultimodal biometricbiometricsystemsystem..
37
Gait…� is the personal, idiosyncratic way in which
people walk or move on foot
� has received significant attention as a biometric,in the last 10-15 years
(Nixon & Carter, 2006; Rahati et al., 2008; Boulgouris et al., 2005)(Nixon & Carter, 2006; Rahati et al., 2008; Boulgouris et al., 2005)
Introduction Gait as a biometricGait as a biometric
� has demonstrated high recognition rates(Zhang et al., 2007; Bouchrika & Nixon, 2007)(Zhang et al., 2007; Bouchrika & Nixon, 2007)
38
General Gait Authentication SystemGeneral Gait Authentication System– Enrolment procedure
Introduction Gait as a biometricGait as a biometric
Gait Database
Camera
Data Acquisition
Module
Human Detection & Extraction
Module
Gait Feature Extraction
Module
Data Storage Module
Data Acquisition
Layer
Data Pre-processing
Layer
Feature Extraction
Layer
Data Storage Layer
39
General Gait Authentication SystemGeneral Gait Authentication System– Authentication procedure
Introduction Gait as a biometricGait as a biometric
Gait Database
Camera
Data Acquisition
Module
Human Detection & Extraction
Module
Gait Feature Extraction
Module
Data Retrieval Module
Data Acquisition
Layer
Data Pre-processing
Layer
Feature Extraction
Layer
Data Storage Layer
Authentication Module
Authentication Results
Authentication Layer
40
Approaches to the gait recognitionApproaches to the gait recognition
Featured based (model free):Focus on the spatiotemporal information contained in the silhouette images (low level measurements).
Model based:Serve as prior knowledge to predict motion parameters, to interpret human dynamics, or to constrain the estimation of low-
Introduction Gait as a biometricGait as a biometric
measurements).
(From Li et al. 2004)
constrain the estimation of low-level image measurements.
(From Wang et al. 2004)
41
Features for gait recognition Features for gait recognition FeaturedFeatured--based method (1/2)based method (1/2)
• Width of the outer contour of the silhouette (Kale et al. 2004)
• Entire binary silhouette (Kale et al. 2004)
• Boundary vector variations from the centre of silhouette (Wang et al. 2003)
• Statistical moments from body parts ellipses (Lee et al. 2002)
Introduction Gait as a biometricGait as a biometric
• Height, stride length and cadence (BenAbdaker et al., 2002)
42
• Height, distance between head and pelvis, max distance between pelvis and feet, and the distance between feet (Johnson & Bobick 2001)
• Self similarity plot (BenAbdelkader et al., 2001)
Where is the bounding box of the person in frame t1, and Ot1, Ot2, …, OtN are the scaled image templates of the person.
• Horizontal and vertical projections( ) ( )
( ) ( )
( ) ( )
1
1
, , 1,...,
, , 1,...,
1, if , is a foreground pixel
=
=
= =
= =
∑
∑
c
r
N
h ry
N
v cr
P x S x y x N
P x S x y y N
x y
Introduction Gait as a biometricGait as a biometric
Features for gait recognition Features for gait recognition FeaturedFeatured--based method (2/2)based method (2/2)
(Liu et al. 2002)( ) ( )1, if , is a foreground pixel
,0, otherwise
=
x yS x y
• Angular transform ( ) ( ) ( ) ( )
( )
2 2
,
1, c c
x y
S x y x x y yN
θφθ
θ
θ φ∈
Α = − + −∑
where θ is an angle, Φθ
is the set of the pixels in
the circular sector and Νθ
is the
cardinality of Φθ(Boulgouris et al. 2004).
the circular sector , , and is 2 2
θ θθ θ
∆ ∆ − +
43
• Absolute joint positions (Zhang et al. 2004)
• Limb angles (Zhang et al. 2004, Goffredo et al. 2008, Lu et al. 2006)
• Lengths of various body parts (Lu et al. 2006)
• Widths/thickness of body parts (Lu et al. 2003)
Features for gait recognition Features for gait recognition ModelModel--based method (1/2)based method (1/2)
Introduction Gait as a biometricGait as a biometric
• Widths/thickness of body parts (Lu et al. 2003)
• Height (Han et al. 2006)
Human parts proportions (Winter 2004)(From Desseree & Legrand, 2005)44
• Area of a body component
• Vector distance between the gravity centres of a body component and the
Features for gait recognition Features for gait recognition ModelModel--based method (2/2)based method (2/2)
Introduction Gait as a biometricGait as a biometric
centres of a body component and the whole body
(Huang & Boulgouris, 2009)
• Orientation of a body component(Huang & Boulgouris, 2009)
• Similarity based on body components (Boulgouris & Chi, 2007)
45
Whole body models
•14 rigid bodies (Wang et al. 2004)
• 5 rigid bodies (Zhang et al. 2007; Cheng-Chang et al. 2007)
Creating a human modelCreating a human model
Leg model models
Introduction Gait as a biometricGait as a biometric
Leg model models
• 3 rigid bodies (thigh, shank, foot) (Desseree et al.2005)
• 2 rigid bodies (thigh and shank) (Yam et al. 2002)
• 1 rigid body (the thigh) (Cunado et al. 2003)
46
Constrains:
• Gait is symmetrical and periodical (healthy population)
� adequate to define the model for only one leg
� the same model can describe either leg
Introduction Gait as a biometricGait as a biometric
Creating a human modelCreating a human model
� the same model can describe either leg(Yam et al. 2004)
• Dependency of the neighbouring joints:
• shoulder-elbow, thigh-knee, knee-ankle
• lower limb is driven by the upper limb (Wang et al. 2004)
• Minimum and maximum values of the joint angles
47
Overcoming the dependency to the gait direction
• Based on pinhole imaging model and weak-perspective projection, the gait direction is obtained and thus the projection function:
Problems in gait recognitionProblems in gait recognition
Introduction Gait as a biometricGait as a biometric
projection function:
)tansin(cos αθθ +=
f
lHL
48(Han et al. 2006)
• Recognition rate: 50-70%
Introduction Gait as a biometricGait as a biometric
Proven references to be used for gait invariant ana lysis
• Simple Reference: 45 deg and 135 deg angle views achievebetter results
• Two references: Combinations of orthogonal references suchas combination of 0 and 90° are effective
Problems in gait recognitionProblems in gait recognition
as combination of 0 and 90° are effective
49(Makihara et al 2006)
CERTH/ITI proposed gait authentication system Gait as a biometricGait as a biometric
CERTH/ITICERTH/ITI GaitGait RecognitionRecognition ApproachApproach --FunctionalitiesFunctionalities
o Live capturing frames / Reading frames fromdatabase
o Human Extraction based on original frames
o Live Enrolment and Authentication based on liveo Live Enrolment and Authentication based on livecapturing and Human Extraction
o Offline Enrolment and Authentication on thedatabase
50
Gait Recognition Algorithm Block DiagramGait Recognition Algorithm Block Diagram
CERTH/ITI proposed gait authentication system MethodsMethods
51
Dynamic background update and silhouette Dynamic background update and silhouette extractionextraction� Background image is updated every time a new frame arrives using an
interpolation rule:α * Current Background + (1-α) * Current Frame
where α is an adaptive weight factor that changes in every frame.
� Depending on chosen segmentation algorithm, we could have:the same α for all frame pixels, based in global illumination changes or
CERTH/ITI proposed gait authentication system MethodsMethods
the same α for all frame pixels, based in global illumination changes orevery pixel in frame could have its own weight factor based on its local illumination changes.
Update Background:α * Current Background + (1- α) * Current Frame
α
1-α
CurrentFrame
CurrentBackground
Silhouette
52
PrePre--Processing StageProcessing StageSilhouette RepresentationPre-processing of foreground silhouettes
(Noise removal)
Usage of DepthUsage of Depth
CERTH/ITI proposed gait authentication system MethodsMethods
Binary Silhouette (a),3D Radial Distributed Silhouette (b), and
3D Geodesic Distributed Silhouette (c)
(a) (c)(b)
53
Gait Cycle Estimation: Algorithm I Gait Cycle Estimation: Algorithm I o Human Height (calculated in the Human Extraction
stage).o Number of foreground pixels in the lower half of the
human silhouettes.
CERTH/ITI proposed gait authentication system MethodsMethods
Period : Number of frames in one cycle (NFC) is calculated by observing the peaks.
54
For
egro
und
Pix
els
Gait Cycle Estimation: Algorithm II Gait Cycle Estimation: Algorithm II
CERTH/ITI proposed gait authentication system MethodsMethods
• Period is estimated by calculating the boundingbox’s width in each frame and getting the localminima for the entire sequence
• Local Minima that are too close (<5 Frames) arediscarded.
),1)(( amesnumberOfFrnnxWidthBoundingBoLocalMinCi ∈=
55
Utilization of Gait Energy Images (GEI)Utilization of Gait Energy Images (GEI)
CERTH/ITI proposed gait authentication system MethodsMethods
Gait frame Gait Energy Image
∑=
⋅=CycleEnd
CycleStarti
igaitFrametCycleLengh
GEI )(1
56
Signature Extraction StageSignature Extraction Stage
CERTH/ITI proposed gait authentication system MethodsMethods
),()1,;()1,;( 2
1
0
1
01 yxfMpyKNpxKQ m
N
x
M
ynnm −−=∑∑
−
=
−
=
57
Mapping of gait characteristics into gait transform sMapping of gait characteristics into gait transform s
Double support position
CERTH/ITI proposed gait authentication system MethodsMethods
Length of stride mapping in the RIT Transform
Mid-stance position
58
Distance between:�Pelvis and feet�Pelvis and head
Mapping of gait characteristics into gait transform sMapping of gait characteristics into gait transform s
CERTH/ITI proposed gait authentication system MethodsMethods
Gait dynamics parameters of the RIT Transform
Height estimation +
59
Hand movement
Mapping of gait characteristics into gait transform sMapping of gait characteristics into gait transform s
CERTH/ITI proposed gait authentication system MethodsMethods
Hand movement detection using the CIT Transform
60
Mapping of gait characteristics into gait transform sMapping of gait characteristics into gait transform s
CERTH/ITI proposed gait authentication system MethodsMethods
61
CERTH/ITI proposed gait authentication system MethodsMethods
Template Matching using temporal correlationTemplate Matching using temporal correlation
For each gait cycle compute the distance between the stored (gallery) and the claimed (probe) gait signature
∑ ∑= =
+−=Np
i
FVecSize
nlii
lntVecGalleryFeanobFeatVecD
1 0
2) )()(Pr (min
CERTH/ITI proposed gait authentication system MethodsMethods
Template Matching using time warping (LTN)Template Matching using time warping (LTN)
� The probe frame is determined by linearly compensating the cycle length differences.
� Each gallery frame (X) is compared with a probe frame (Y). If gallery cycle has Gfeature vectors, Probe cycle has P feature vectors then:
Y = X * P / G + wwhere w is a search-for-best-matching frame window: -k <= w <= k, where k is a small
positive number, i.e. k = 2.
GallerySequence
Probe-ImpostorSequence
Probe-ClientSequence
CERTH/ITI proposed gait authentication system MethodsMethods
Reliability Factor and Quality measurement of the g ait signalReliability Factor and Quality measurement of the g ait signal
� Quality of the gait signal using QRIT metric
� Estimates the quality of the gait signal using the coefficient values of the Radial Integration Transform to the interval [0, 360o]
Detection of noisy region(s) using the coefficients of the RIT transform
Although it has been reported that the side viewsilhouettes contain most discriminative information,the problem of gait direction dependency stillremains.
CERTH/ITI proposed gait authentication system MethodsMethods
New approach:• Rotation of silhouettes in order to
synthesize the side view– Challenge: some pixels may have to be guessed.
remains.
65
WalkingWalking AngleAngle DeterminationDetermination
•VV
Using the 3D data from the stereoscopic camera, the walking direction can be estimated.
• The head position is extracted, and
CERTH/ITI proposed gait authentication system MethodsMethods
•=∂ −
21
211cosVV
VV• The head position is extracted, and the mean head’s 3D point is estimated at the first and the last frame of each gait cycle.
• The walking angle, with respect to the camera, is calculated using the formula.
66
SilhouetteSilhouette rotationrotation
The 3D coordinates of each silhouette pixel are extracted using the disparity data from the stereoscopic camera. This way a 3D point cloud is generated, which is rotated using the following formula.
CERTH/ITI proposed gait authentication system MethodsMethods
the following formula.
∂∂−
∂∂
⋅=
100
0)cos()sin(
0)sin()cos(
irotated
i PP
The new point cloud is now reprojected on the camera to create a new silhouette, which is used to extract the gait features67
Further improvement: Further improvement: Disparity Data Smoothing Disparity Data Smoothing
The disparity data from the stereoscopic camera are, in general, quite noisy.
� The data are denoised using a Gaussian filter in order to achieve better results
CERTH/ITI proposed gait authentication system MethodsMethods
68
Gait Recognition Algorithm with Disparity Data Gait Recognition Algorithm with Disparity Data Refinement Block DiagramRefinement Block Diagram
CERTH/ITI proposed gait authentication system MethodsMethods
( )),(),(1
),(',
jiDyxgk
jiDcr
∗⋅=
69
Further improvement: Use of soft biometrics (1/2)Further improvement: Use of soft biometrics (1/2)
Height and stride length were utilized to augment the information obtained by the gait recognition system.
CERTH/ITI proposed gait authentication system MethodsMethods
70
CERTH/ITI proposed gait authentication system MethodsMethods
Probabilistic Soft Biometrics framework
Further improvement: Use of soft biometrics (2/2)Further improvement: Use of soft biometrics (2/2)
71
Height-coefficient
Stride-coefficient
Calculated similarly to height-coefficient
� Database data from two sessions� 1st session consists of 75 Persons (3 covariates – hat, briefcase,
shoe)� 2nd session consists of 53 Persons, 48 common with the 1st
session (3 covariates – hat, briefcase, coat)
Database HUMABIODatabase HUMABIO
CERTH/ITI proposed gait authentication system MethodsMethods
Normal Hat CoatBriefcase
� Capture Details� Indoor scenario similar to the airport pilot� Four experiments were defined
(Hat-Exp.A, Briefcase-Exp.B, Shoe-Exp.C and Time-Exp.D)
72
� Detection Error Trade-off curves (DET) for estimating the Equal Error Rates (using z-Norm scores) for each feature extractor and the weighted algorithm on all experiments
30
40
50
60
70
80
90
100DET Curves for RIT transform (using z-Norm scores)
Fal
se R
ejec
tion
Rat
e
Exp.A (HAT)
Exp.B (BF)
Exp.C (Slipper-Pantofle)
Exp.D (Tim e)
30
40
50
60
70
80
90
100DET Curves for CIT transform (using z-Norm scores)
Fal
se R
ejec
tion
Rat
e
Exp.A (HAT)
Exp.B (BF)
Exp.C (Slipper-Pantofle )
Exp.D (Time)
EER
CERTH/ITI proposed gait authentication system ResultsResults
Verification results on the HUMABIO databaseVerification results on the HUMABIO database
0 10 20 30 40 50 60 70 80 90 1000
10
20
30
False Acceptance Rate 0 10 20 30 40 50 60 70 80 90 1000
10
20
30
False Acceptance Rate
Fal
se R
ejec
tion
Rat
e
0 10 20 30 40 50 60 70 80 90 1000
10
20
30
40
50
60
70
80
90
100
DET Curves for Krawtchouk transform (using z-Norm s cores)
False Acceptance Rate
Fal
se R
ejec
tion
Rat
e
Exp.A (HAT)
Exp.B (BF)
Exp.C (Slipper-Pantofle)
Exp.D (Time)
0 10 20 30 40 50 60 70 80 90 1000
10
20
30
40
50
60
70
80
90
100DET Curves for RCK-G (using z-Norm scores)
False Acceptance Rate
Fal
se R
ejec
tion
Rat
e
Exp.A (HAT)
Exp.B (BF)
Exp.C (Slipper-Pantofle )
Exp.D (Time)
73
Recognition results comparison on the USF “Gait Recognition results comparison on the USF “Gait Challenge” databaseChallenge” database
CERTH/ITI proposed gait authentication system ResultsResults
References1. Sudeep Sarkar,P.Jonathon Phillips, Zonguy Liu, Isidro Robledo, Patrick Grother, Kevin Bowyer, The Human Gait Challenge Problem Data Sets,
Performance, and Analysis
2. R. Collins, R. Gross, and J.Shi, “Silhouette Based Human Identification from Body Sh ape and Gait ,” Proc. Int’l Conf.Automatic Face and Gesture Recognition
3. Nikolaos V. Boulgouris,, Konstantinos N. Plataniotis, Dimitrios Hatzinakos, “Gait recognition using linear time normalization”
4. Nikolaos V.Boulgouris, Konstantinos N.Plataniotis, Dimitris Hatzinakos, An Angular Transform of Gait Sequences for Gait Ass isted Recognition
5. Amit Kale, Aravind Sundaresan, A. N. Rajagopalan, Naresh P. Cuntoor, Amit K. Roy-Chowdhury, Volker Krüger,and Rama Chellappa, “Identification of Humans Using Gait”
Best ScoreComputational intensive
Training (increase enrolment time)
74
� Database data from two sessions
� 1st session consists of 28 Persons (5 covariates – coat, briefcase, shoe, stopped during walking, diagonally)
� 2nd session consists of the same 28 Persons after one month, (1 covariate – stop during walking)
Database ACTIBIODatabase ACTIBIO
CERTH/ITI proposed gait authentication system MethodsMethods
(1 covariate – stop during walking)
� Capture Details
� Indoor scenario� The subject was not walking in a strictly straight line
Normal With stopBriefcase75
View angle
ACTIBIO Database – Gait recorded paths
CERTH/ITI proposed gait authentication system MethodsMethods
Different Angles Straight and random paths
Random paths with stops andpressing buttons to control panel
Database Actibio – Multiple views (High Sync)
CERTH/ITI proposed gait authentication system MethodsMethods
Results on the ACTIBIO databaseResults on the ACTIBIO database
CERTH/ITI proposed gait authentication system ResultsResults
Briefcase experiment
Coat experiment
Normal walking experiment
Wearing slippers or socks experiment78
Results on the ACTIBIO databaseResults on the ACTIBIO database
CERTH/ITI proposed gait authentication system ResultsResults
During walking stopping experiment Changing view angle experiment
79
Results on the ACTIBIO databaseResults on the ACTIBIO database
CERTH/ITI proposed gait authentication system ResultsResults
�������� The improvement of the results with the use of the soft The improvement of the results with the use of the soft biometrics is obvious.biometrics is obvious.
Using the GEI algorithm Using the GEI & SoftBiometrics
80
• The gait recognition algorithm deals with therandom-path walking scenario, which is a brandnew and novel approach in the whole researchfield of gait recognition / authentication.
• The experimental results demonstrate that the
ConclusionsConclusions
81
• The experimental results demonstrate that thepresented gait recognition system perform betterthan the state-of-the-art algorithms.
• The use of soft biometric enhance theperformance of the gait recognition system.
• Novel methods were presented to improve thestate-of-the-art gait recognition systems.
• Innovative biometrics were developed based onthe users motion when responding to specificstimuli and demonstrated a very promising
ConclusionsConclusions
82
stimuli and demonstrated a very promisingauthentication potential.
�� SecuritySecurity systemssystems usingusing activityactivity relatedrelatedbiometricsbiometrics cancan clearlyclearly enhanceenhance safetysafety inincriticalcritical applicationsapplications..
Activity and Gait Recognition GroupActivity and Gait Recognition Group
Researcher B'Dr. D. Tzovaras
Postdoctoral Research Fellow (Prof. in 1 year)Dr. K. Moustakas
Postdoctoral Research FellowMrs. L. Mademli
Research AssistantMr. G. Stavropoulos
Dr. K. Moustakas
MSc. Research AssistantMr. D. Ioannidis
Mr. G. Stavropoulos
PhD CandidateMr. A. Drosou
• Book Chapters– S. Argyropoulos, D. Ioannidis, D. Tzovaras, and M. G. Strintzis, “Distributed source coding for
biometrics: A case study on gait recognition,” in Biometrics: theory, methods, and applications, N.V. Boulgouris, K. Plataniotis, E. Micheli-Tzanakou, Eds., IEEE/Wiley, 2009, ch. 22, pp. 559-578.
• Journals– D. Ioannidis, D. Tzovaras, I. G. Damousis, S. Argyropoulos, and K. Moustakas, “Gait recognition
using compact feature extraction transforms and depth information,” IEEE Trans. on InformationForensics and Security, vol. 2, no. 3, pp. 623–630, Sep. 2007.
– S. Argyropoulos, D. Tzovaras, D. Ioannidis, and M. G. Strintzis, "A Channel Coding Approach forHuman Authentication From Gait Sequences," IEEE Trans. on Information Forensics andSecurity, vol. 4, no. 3, pp. 428-440, Sep. 2009.
• Conferences
PublicationsPublications
• Conferences– D. Ioannidis, D. Tzovaras, K. Moustakas, "Gait Identification using the 3D Protrusion Transform,"
Image Processing, 2007. ICIP 2007. IEEE International Conference on , vol.1, no., pp.I -349-I -352, Sept. 16 2007-Oct. 19 2007
– S. Argyropoulos, D. Tzovaras, D. Ioannidis, and M. G. Strintzis, "Gait authentication usingdistributed source coding," in IEEE Int. Conf. on Image Processing (ICIP 2008), San Diego, CA,Oct. 2008, pp. 3108-3111.
– K. Moustakas, D. Tzovaras, G. Stavropoulos, “Gait Recognition Using Geometric Features andSoft Biometrics”, submitted for publication in IEEE Signal Processing Letters.
– A. Drosou, K. Moustakas, D. Ioannidis, D. Tzovaras, “On the potential of activity-relatedrecognition”, submitted for publication in International Conference on Computer Vision Theory andApplications.
Activity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and RecognitionActivity Recognition: Gait Analysis and Recognition
Thank you for your
85
Dimitrios TzovarasDimitrios Tzovaras
Centre of Research & Technology Centre of Research & Technology -- HellasHellasInformatics & Telematics InstituteInformatics & Telematics Institute
Thank you for your attention