1

Similarity-based matching for face authentication

Christophe Rosenberger

Luc Brun

ICPR 2008

2

outline

Introduction Image description Dissimilarity measure Pattern modeling Experimental results

Introduction

Propose in this paper a face authentication method based on a similarity measure

Find interest key points SIFT descriptor is used

A graph is then built where nodes correspond to these keypoints Model the authentication problem as a graph matching process

3

4

Image description

SIFT (Scale Invariant Feature Transform) Applied locally on key-points Based upon the image gradients in a local neighborhood SIFT detector

Scale-space extrema detection Keypoint localization

SIFT descriptor Orientation assignment Keypoint descriptor

5

SIFT detector

Detection of scale-space extrema DoG(Difference-of-Gaussian) filter for scale space

Keypoint localization (local extrema)

X is selected if it is larger or smaller than all 26 neighbors

Gradient orientation

Gradient magnitude

6

SIFT detector & descriptor Accurate keypoint localization

Reject points with low contrast (flat) Elimination edge responses

Orientation assignment To make the feature

invariant to rotation Find local major

orientation Orientation histogram for a window size

Local image descriptor 16x16 array of locations in scale space Create array of orientation histograms 8 orientations x 4x4 histogram array (128D)

Dissimilarity measure

Each individual is described by a set of invariant features Y (I) = {ki = (si, xi, yi)}, i = 1 : N(I) Si is the SIFT invariant descriptor

(xi, yi) its position in the original image I N(I) the number of detected keypoints in the image I

Database image {Y (Ii), i = 1 : M}

Dissimilarity between two sets of points such as Y (I1) and Y (I2) Often modeled as a weighted bipartite graph matching problem

Greedy algorithm Hungarian method Hybrid method

7

Greedy algorithm Weighted bipartite graph G(U,V,E,W) Given the graph G , scan the set of edges and find an edge (u,v)

with the smallest weight Match u with v and remove the edges incident on u and those

incident on v Repeat the above step until every node in U is matched to some

node in V

8

105

25

2012

3

49

6

3

4

5

Greedy algorithm

The greedy algorithm does not always produce a matching of minimum weight

Hungarian method A well-known algorithm for finding a minimum weight matching in

a bipartite graph Node in U by u1, u2,… un

In V by v1, v2,… vn

x ij = 1 if ui is matched with vj and 0 otherwise

subject to constraints

9

1

16

0

2

n

i

n

jijijxwMin

1 1

nixn

jij 2,1,1

1

njxn

iij 2,1,1

1

Each node in U ismatched to exactly one node in V and vice versa

A

B D

C 16

0

A

B

C

D

Hungarian method - ExampleR1 R2 R3 R4

B1 8 17 3 23

B2 39 4 11 20

B3 13 2 41 6

B4 22 8 9 2

10

R1 R2 R3 R4

B1 8 17 3 23 -3

B2 39 4 11 20 -4

B3 13 2 41 6 -2

B4 22 8 9 2 -2

Step 1First for each row we subtract the row

minimum from the rest of the rowR1 R2 R3 R4

B1 5 14 0 20

B2 35 0 7 16

B3 11 0 39 4

B4 20 6 7 0

-5 -0 -0 -0

Step 2Then for each column we subtract the

column minimum from the rest of the column

R2

B1

B2

B3

B4

R1

R3

R4

Hungarian method - Example

11

R1 R2 R3 R4

B1 0 20 0 26

B2 24 0 1 16

B3 0 0 33 4

B4 9 6 1 0

R1 R2 R3 R4

B1 0 14 0 20

B2 30 0 7 16

B3 6 0 39 4

B4 15 6 7 0

Step 3Cover the min number of rows or columns required to cover all of the zeros in the matrix

If the # of line = the # of rows/columns in the matrix then we are finished

Otherwise, we have to go to the next step

Step 41.Find the min value not covered (B3-R1) 2.subtracted from each uncovered entry and added to each entry covered by both a vertical and horizontal line (B1-R2 & B1-R4)3. Step 3 is then repeated

Result:Blue1 - Red3Blue2 - Red2Blue3 - Red1Blue4 – Red4

R2

B1

B2

B3

B4

R1

R3

R4

3

4

13

2

R1 R2 R3 R4

B1 0 14 0 20

B2 30 0 7 16

B3 6 0 39 4

B4 15 6 7 0

R1 R2 R3 R4

B1 0 14 0 20

B2 30 0 7 16

B3 6 0 39 4

B4 15 6 7 0

Hybrid method

12

We formulated the point matching problem in terms of minimum weight matching in a bipartite graph

In a real world situation, this is not always appropriate Feature points extracted from two corresponding images First image are denoted by a and b Second image are denoted by c and d

One might argue that since b and c both have the same feature value of 100,we should match them together and completely ignore a and d

Hybrid method

Feature points forming tight clusters in the feature space can be avoided by a simple combination of the greedy method and the Hungarian method

Combined algorithm We first apply the Hungarian method to get a matching M1

Apply the greedy algorithm to get a matching M2

form their intersection M=M1∩M2

13A bipartite graph Hungarian method Greedy algorithm Combinedmatching

Dissimilarity measure

Such a weighted bipartite graph matching method Optimizes a global matching score May thus be sensible to outliers

We thus rather use the following matching method Given two points and , we say that x is

associated to y iff :

C is an arbitrary threshold d(., .) denotes the Euclidean distance between SIFT descriptors y’ denotes any point of Y (I2) whose distance to x is minimal but greater

than d(x,y) 14

1IYx 2IYy

yxCdyxdandzxdyxd IYz ,,,min,2

zxdyxd yxdzxdIYz ,min, ),(),(,2

Dissimilarity measure The significance of the necessary gap between d(x, y) and d(x, y’) is

encoded by the constant C y is associated to x

x is the closest point from y among Y (I1) second smallest value of this distance d(y, x’) satisfies d(y, x) < Cd(y, x’)

X is matched to Y iff x is associated to y and y is associated to x

If Y (I1) and Y (I2) do not define any matching point Both sets have an infinite distance

Otherwise The set of matching points M1 of Y1 and M2 of Y2 may be computed

sequentially If we cannot find two couples of points within Y (I1) × Y (I2) with a

similar distance, the relationship “is matched to “ induces a bijective mapping Φ from M1 to M2

15

Dissimilarity measure

First step The above procedure classify both sets Y (I1) and Y (I2)

Dissimilar (case M1 = M2 =ø) A set of N = |M1| = |M2| couples (x, Φ(x)) of matched points

Second step Take into account the spatial relationship of the points among M1 & M2

Use one of the four pattern representations Using the locations of each matched keypoint Weighted graph

Edges are weighted by the similarity of the adjacent points

The matching between M1 and M2 being encoded by the application Φ Encode the adjacency matrices of both graphs A line or a column encoding the adjacency relationships

Point x of the first matrix Φ(x) in the second matrix 16

Graph Adjacency matrix

Pattern modeling

Spatial relationship between keypoints Using graph representations Connect keypoints by considering their ”local neighborhoods” Each keypoint corresponds to a vertex in the neighborhood

graph which connects “close” vertices Epsilon-neighborhood graph (ENG)

Two vertices are connected if the distance of the corresponding keypoints is less than epsilon

A completely connected graph (CCG) epsilon greater than the maximal distance between keypoints.

17

Pattern modeling Symmetric k-nearest neighbor graph (SKNG)

Two vertices V1, V2 are connected if V1 is among the k-nearest neighbors of V2 or vice versa

Mutual k-nearest neighbor graph (MKNG) Two vertices V1, V2 are connected if V1 is among the k-nearest

neighbors of V2 and vice versa

Given both adjacency matrices The quality of the match is estimated

using the squared Euclidean

distance which provides a global

measure of the dissimilarity between both graphs

18

Experimental results

Database AR face database This face database was created by Aleix Martinez and Robert

Benavente in the Computer Vision Center (CVC) at the U.A.B It contains over 4,000 color images 126 people's faces (70 men and 56 women)

26 image per individual

Images feature frontal view faces Different facial expressions, illumination conditions, and occlusions (sun

glasses and scarf).

19

Experimental results

Tested the four graph representations Two parameters have to be set

C is used in the first step of the keypoints matching C induce a restrictive selection of matching points

K for the three graph representations It corresponds to the number of neighbors considered in the graph

definition

The dissimilarity measure gives a coefficient that can be interpreted as a confidence criterion on the fact that two faces belong to different individuals

Gives the experimental results on the AR face database

on a ROC curve20

Experimental results

We use the FAR (False Acceptance Rate) and FRR (False Rejection Rate) criteria to analyse the behavior of an algorithm

21

The representation based on a Completely connected graph (CCG) gives the worst resultThis means that the selection of edges is necessary

Mutual k-nearest neighbor graph (MKNG) with k=5 and C=0.8 provides

Conclusion

We proposed in this paper a face authentication method based on a similarity measure

A graph representation based on SIFT invariant keypoints is used

The graph definition allows us to take into account their spatial relationships

22