SUBMITTED IEEE TRANSACTION ON IMAGE PROCESSING 1

Enhanced Local Texture Feature Sets for FaceRecognition under Difficult Lighting Conditions

Xiaoyang Tan and Bill Triggs

Abstract— Making recognition more reliable under uncon-trolled lighting conditions is one of the most important chal-lenges for practical face recognition systems. We tackle this bycombining the strengths of robust illumination normalization,local texture based face representations, distance transform basedmatching, kernel-based feature extraction and multiple featurefusion. Specifically, we make three main contributions: (i) wepresent a simple and efficient preprocessing chain that eliminatesmost of the effects of changing illumination while still preservingthe essential appearance details that are needed for recognition;(ii) we introduce Local Ternary Patterns (LTP), a generalizationof the Local Binary Pattern (LBP) local texture descriptor thatis more discriminant and less sensitive to noise in uniformregions, and we show that replacing comparisons based on localspatial histograms with a distance transform based similaritymetric further improves the performance of LBP/LTP basedface recognition; and (iii) we further improve robustness byadding Kernel PCA feature extraction and incorporating richlocal appearance cues from two complementary sources – Gaborwavelets and LBP – showing that the combination is considerablymore accurate than either feature set alone. The resulting methodprovides state-of-the-art performance on three data sets that arewidely used for testing recognition under difficult illuminationconditions: Extended Yale-B, CAS-PEAL-R1, and Face Recogni-tion Grand Challenge version 2 experiment 4 (FRGC-204). Forexample, on the challenging FRGC-204 data set it halves the errorrate relative to previously published methods, achieving a FaceVerification Rate of 88.1% at 0.1% False Accept Rate. Furtherexperiments show that our preprocessing method outperformsseveral existing preprocessors for a range of feature sets, datasets and lighting conditions.

I. INTRODUCTION

Face recognition has received a great deal of attention fromthe scientific and industrial communities over the past severaldecades owing to its wide range of applications in informationsecurity and access control, law enforcement, surveillanceand more generally image understanding [44]. Numerous ap-proaches have been proposed, including (among many others)eigenfaces [33,37], fisherfaces [5] and laplacianfaces [15],neural networks [21,34], elastic bunch graph matching [40],wavelets [40], and kernel methods [41]. Most of these methodswere initially developed with face images collected underrelatively well controlled conditions and in practice they havedifficulty dealing with the range of appearance variations thatcommonly occur in unconstrained natural images due to illu-

Xiaoyang Tan is with the Department of Computer Science and Technol-ogy, Nanjing University of Aeronautics and Astronautics, P.R. China. BillTriggs is with the CNRS and Laboratoire Jean Kuntzmann, BP 53, 38041Grenoble Cedex 9, France. The work was financed by the European Unionresearch project CLASS and part of it was undertaken at INRIA Grenoble.Corresponding author: Xiaoyang Tan (x.tan@nuaa.edu.cn).

mination, pose, facial expression, ageing, partial occlusions,etc.

This paper focuses mainly on the issue of robustness tolighting variations. For example, a face verification systemfor a portable device should be able to verify a client atany time (day or night) and in any place (indoors or out-doors). Unfortunately, facial appearance depends strongly onthe ambient lighting and – as emphasized by the recentFRVT and FRGC trials [30] – this remains one of the majorchallenges for current face recognition systems. Traditionalapproaches for dealing with this issue can be broadly classifiedinto three categories: appearance-based methods, normaliza-tion, and feature-based methods. In direct appearance-basedapproaches, training examples are collected under differentlighting conditions and directly (i.e. without undergoing anylighting preprocessing) used to learn a global model of thepossible illumination variations, for example a linear subspaceor manifold model, which then generalizes to the variationsseen in new images [6,4,22,9,42]. Direct learning of thiskind makes few assumptions but it requires a large number oftraining images and an expressive feature set, otherwise it isessential to include a good preprocessor to reduce illuminationvariations (c.f . fig.14).

Normalization based approaches seek to reduce the image toa more “canonical” form in which the illumination variationsare suppressed. Histogram equalization is one simple example,but purpose-designed methods often exploit the fact that (onthe scale of a face) naturally-occuring incoming illuminationdistributions typically have predominantly low spatial frequen-cies and soft edges so that high frequency information in theimage is predominantly signal (i.e. intrinsic facial appearance).For example, the Multiscale Retinex (MSR) method of Jobsonet al. [18] cancels much of the low frequency informationby dividing the image by a smoothed version of itself. Wanget al. [38] use a similar idea (with a different local filter)in the Self Quotient Image (SQI). More recently, Chen etal. [10] improved SQI by using Logarithmic Total Variation(LTV) smoothing, and Gross & Brajovic (GB) [13] developedan anisotropic smoothing method that relies on the iterativeestimation of a blurred version of the original image. Thesemethods are quite effective but their ability to handle spatiallynon-uniform variations remains limited. Shan et al. [31] andShort et al. [32] give comparative results for these and relatedmethods.

The third approach extracts illumination-insensitive featuresets [8,1,2,3,40,14] directly from the given image. These fea-ture sets range from geometrical features [8] to image deriva-tive features such as edge maps [1], Local Binary Patterns

2 SUBMITTED IEEE TRANSACTION ON IMAGE PROCESSING

Fig. 1. (Lower curve) Degradation of the performance of LBP descriptorswith nearest-neighbour classification under the increasingly extreme illumina-tion conditions of subsets 1-5 of the Yale database [5]. Example images areshown on the horizontal axis. (Upper curve) Adding our preprocessing chaingreatly improves the performance under difficult illumination.

Fig. 2. The stages of our full face recognition method.

(LBP) [2,3], Gabor wavelets [40], and local autocorrelationfilters [14].

Although such features offer a great improvement on rawgray values, their resistance to the complex illumination vari-ations that occur in real-world face images is still quite lim-ited. For example, even though LBP features are completelyinvariant to monotonic global gray-level transformations, theirperformance degrades significantly under changes of lightingdirection and shadowing – see fig. 1. Similar results applyto the other features. Nevertheless, it is known that completeillumination invariants do not exist [9] so one must contentoneself with finding representations that are more resistant tothe most common classes of natural illumination variations.

In this paper, we propose an integrative framework thatcombines the strengths of all three of the above approaches.The overall process can be viewed as a pipeline consisting ofimage normalization, feature extraction and subspace repre-sentation, as shown in fig. 2. Each stage increases resistanceto illumination variations and makes the information neededfor recognition more manifest. The method centres on a richset of robust visual features that is selected to capture asmuch as possible of the available information. A well-designedimage preprocessing pipeline is prepended to further enhancerobustness. The features are used to construct illumination-insensitive subspaces, thus capturing the residual statistics ofthe data with relatively few training samples (many fewer thantraditional raw-image-based appearance based methods suchas [22]).

We will investigate several aspects of this framework:

1) The relationship between image normalization and

feature sets. Normalization is known to improve theperformance of simple subspace methods (e.g. PCA) orclassifiers (e.g. nearest neighbors) based on image pixelrepresentations [31,38,10], but its influence on moresophisticated feature sets has not received the attentionthat it deserves. A given preprocessing method may ormay not improve the performance of a given featureset on a given data set. For example, for Histogramof Oriented Gradient features combining normalizationand robust features is useful in [11], while histogramequalization has essentially no effect on LBP descriptors[3], and in some cases preprocessing actually hurtsperformance [12] – presumably because it removes toomuch useful information. Here we propose a simpleimage preprocessing chain that appears to work well fora wide range visual feature sets, eliminating many of theeffects of changing illumination while still preservingmost of the appearance details needed for recognition.

2) Robust feature sets and feature comparison strate-gies. Current feature sets offer quite good performanceunder illumination variations but there is still room forimprovement. For example, LBP features are known tobe sensitive to noise in near-uniform image regions suchas cheeks and foreheads. We introduce a generalizationof LBP called Local Ternary Patterns (LTP) that ismore discriminant and less sensitive to noise in uniformregions. Moreover, in order to increase robustness tospatial deformations, LBP based representations typi-cally subdivide the face into a regular grid and comparehistograms of LBP codes within each region. This issomewhat arbitrary and it is likely to give rise toboth aliasing and loss of spatial resolution. We showthat replacing histogramming with a similarity metricbased on local distance transforms further improves theperformance of LBP/LTP based face recognition.

3) Fusion of multiple feature sets. Many current patternrecognition systems use only one type of feature. How-ever in complex tasks such as face recognition, it is oftenthe case that no single class of features is rich enoughto capture all of the available information. Finding andcombining complementary feature sets has thus becomean active research topic, with successful applications inmany challenging tasks including handwritten characterrecognition [16] and face recognition [26]. Here weshow that combining two of the most successful localface representations, Gabor wavelets and Local BinaryPatterns (LBP), gives considerably better performancethan either alone. The two feature sets are complimen-tary in the sense that LBP captures small appearancedetails while Gabor wavelets encode facial shape over abroader range of scales.

To demonstrate the effectiveness of the proposed method wegive results on the Face Recognition Grand Challenge version2 experiment 4 dataset (“FRGC-204”), and on two other facedatasets chosen to test recognition under difficult illuminationconditions. FRGC-204 is a challenging large-scale dataset con-taining 12 776 training images, 16 028 controlled target images

3

Fig. 3. (Top) the stages of our image preprocessing pipeline, and (bottom)an example of the effect of the three stages – from left to right: input image;image after Gamma correction; image after DoG filtering; image after robustcontrast normalization.

and 8 014 uncontrolled query images. To the best of knowledgethis is the first time that a preprocessing method has beensystematically evaluated on such a large-scale database, andour method achieves very significant improvements, achievinga Verification Rate of 88.1% at 0.1% False Acceptance Rate.

The rest of the paper is organized as follows: Section IIpresents our preprocessing chain, Section III introduces ourLTP local texture feature sets, Section IV describes ourmultiple-feature fusion framework, Section V reports exper-imental results, and Section VI concludes. A preliminarydescription of the methods was presented in the conferencepapers [35,36].

II. ILLUMINATION NORMALIZATION

A. The Preprocessing Chain

This section describes our illumination normalizationmethod. This is a preprocessing chain run before featureextraction that incorporates a series of stages designed tocounter the effects of illumination variations, local shadowingand highlights while preserving the essential elements of visualappearance. Fig. 3 illustrates the three main stages and theireffect on a typical face image. Although it was motivatedby intuition and experimental studies rather than biology, theoverall chain is reminiscent of the first few stages of visualprocessing in the mammalian retina and LGN. In detail, thestages are as follows.

Gamma Correction is a nonlinear gray-level transformationthat replaces gray-level I with Iγ (for γ > 0) or log(I) (forγ = 0), where γ ∈ [0, 1] is a user-defined parameter. Thisenhances the local dynamic range of the image in dark orshadowed regions while compressing it in bright regions andat highlights. The underlying principle is that the intensityof the light reflected from an object is the product of theincoming illumination L (which is piecewise smooth for themost part) and the local surface reflectance R (which carriesdetailed object-level appearance information). We want torecover object-level information independent of illumination,and taking logs makes the task easier by converting theproduct into a sum: for constant local illumination, a givenreflectance step produces a given step in log(I) irrespectiveof the actual intensity of the illumination. In practice a fulllog transformation is often too strong, tending to over-amplifythe noise in dark regions of the image, but a power law with

exponent γ in the range [0, 0.5] is a good compromise1. Herewe use γ = 0.2 as the default setting.

Difference of Gaussian (DoG) Filtering. Gamma correctiondoes not remove the influence of overall intensity gradientssuch as shading effects. Shading induced by surface structureis a potentially useful visual cue but it is predominantly lowspatial frequency information that is hard to separate fromeffects caused by illumination gradients. High pass filteringremoves both the useful and the incidental information, thussimplifying the recognition problem and in many cases in-creasing the overall system performance. Similarly, suppress-ing the highest spatial frequencies potentially reduces bothaliasing and noise without destroying too much of the under-lying recognition signal. DoG filtering is a convenient way toachieve the resulting bandpass behaviour. Fine details remaincritically important for recognition so the inner (smaller)Gaussian is typically quite narrow (σ0 ≤ 1 pixel), while theouter one might have σ1 of 2–4 pixels or more, dependingon the spatial frequency at which low frequency informationbecomes misleading rather than informative. Given the stronglighting variations in our datasets we find that σ1 ≈ 2 typicallygives the best results, but values up to about 4 are not toodamaging and may be preferable for datasets with less extremelighting variations. LBP and LTP features do seem to benefitfrom a little smoothing (σ0 ≈ 1), perhaps because pixel basedvoting is sensitive to aliasing artifacts. Below we use σ0 = 1.0and σ1 = 2.0 by default2.

We implement the filters using explicit convolution. Tominimize boundary effects, if the face is part of a largerimage the gamma correction and prefilter should be run on anappropriate region of this before cutting out the face image.Otherwise, extend-as-constant boundary conditions should beused: using extend-as-zero or wrap-around (FFT) boundaryconditions significantly reduces the overall performance, inpart because it introduces strong gradients at the image bordersthat disturb the subsequent contrast equalization stage. Priorgamma normalization is still required: if DoG is run withoutthis, the resulting images suffer from reduced local contrast(and hence loss of visual detail) in shadowed regions.

Masking. If facial regions (hair style, beard, . . . ) that arefelt to be irrelevant or too variable need to be masked out,the mask should be applied at this point. Otherwise, eitherstrong artificial gray-level edges are introduced into the DoGconvolution, or invisible regions are taken into account duringcontrast equalization.

Contrast Equalization. The final stage of our preprocessingchain rescales the image intensities to standardize a robust

1Shot noise – the dominant noise source in modern CCD sensors –is proportional to the square root of illuminance so γ = 0.5 makes itapproximately uniform:

√I + ∆I ≈

√I + ∆I

2√

I=√I + const. for

∆I ∝√I .

2Curiously, for some datasets it also helps to offset the center of the largerfilter by 1–2 pixels relative to the center of the smaller one, so that the finalprefilter is effectively the sum of a centered DoG and a low pass spatialderivative. The best direction for the displacement is somewhat variable buttypically diagonal. The effect is not consistent enough to be recommendedpractice, but it might repay further investigation.

4 SUBMITTED IEEE TRANSACTION ON IMAGE PROCESSING

0 10 20 30 40 50 600

2

4

6

8

10

12

14

sequence no. of uniform pattern.

nu

mb

er

of

un

ifo

rm p

att

ern

image a

image b

0 10 20 30 40 50 600

5

10

15

20

25

sequence no. of uniform pattern.

nu

mb

er

of

un

ifo

rm p

att

ern

image a

image b

Fig. 4. (Top) Two images of the same subject from the FRGC-204dataset. (Bottom) The LBP histograms of the marked image regions, (left)without preprocessing, (right) after preprocessing. Note the degree to whichpreprocessing reduces the variability of the histograms of these relativelyfeatureless but differently illuminated facial regions.

measure of overall contrast or intensity variation. It is im-portant to use a robust estimator because the signal typicallycontains extreme values produced by highlights, small darkregions such as nostrils, garbage at the image borders, etc.One could use (for example) the median of the absolute valueof the signal for this, but here we have preferred a simple andrapid approximation based on a two stage process:

I(x, y) ← I(x, y)(mean(|I(x′, y′)|a))1/a

(1)

I(x, y) ← I(x, y)(mean(min(τ, |I(x′, y′)|)a))1/a

(2)

Here, a is a strongly compressive exponent that reduces theinfluence of large values, τ is a threshold used to truncate largevalues after the first phase of normalization, and the mean isover the whole (unmasked part of the) image. By default weuse α = 0.1 and τ = 10.

The resulting image is well scaled but it can still con-tain extreme values. To reduce their influence on subsequentstages of processing, we apply a final nonlinear mappingto compress over-large values. The exact functional form isnot critical. Here we use the hyperbolic tangent I(x, y) ←τ tanh(I(x, y)/τ), thus limiting I to the range (−τ, τ).

B. Robustness and Computation Time

To illustrate the need for preprocessing we demonstrate itseffect on part of an LBP histogram feature set. Fig. 4 (top)shows a matching target-query pair chosen randomly fromthe FRGC-204 dataset. We chose a relatively featureless –and hence not particularly informative – face region (whitesquares) and extracted its LBP histograms, both without (bot-tom left) and with (bottom right) image preprocessing. Withoutpreprocessing the two histograms are both highly variable and

None HE MSR GB LTV TTFig. 5. Examples of the effects of the different preprocessing methods.Rows 1-5 respectively show images of one subject from subsets 1-5 of theYale-B data set, and rows 6-8 show images of different subjects from theCAS-PEAL data set, with from left to right: (None) no preprocessing; (HE)Histogram Equalization; (MSR) Multiscale Retinex; (GB) Gross & Brajovicmethod; (LTV) Logarithmic Total Variation; (TT) Our preprocessing method.

very different, but preprocessing significantly reduces thesedifferences, quantitatively decreasing the χ2 inter-histogramdistance from 93.4 to 25.0.

Run time is also a critical factor in many applications. Ourmethod uses only simple closed-form image operations so itis much more efficient than ones that require expensive itera-tive optimizations such as Logarithmic Total Variation (LTV,[10]) and anisotropic diffusion (GB, [13]). Our (unoptimizedMatlab) implementation takes only about 50 ms to process a128×128 pixel face image on a 2.8 GHz P4, allowing facepreprocessing to be performed in real time and thus providingthe ability to handle large face databases. In comparison, thecurrent implementation of GB is about 5 times slower andLTV is about 300 times slower.

C. Competing Methods

Below we will use recognition rates under several fea-ture sets and data sets to compare the performance of ourpreprocessing chain with that of several competing methods.We will not describe the methods tested in detail owing tolack of space, but briefly they are: Histogram Equalization

5

(HE); Multiscale Retinex (MSR) [18]; Gross & Brajovic’sanisotropic smoothing (GB) [13]; and Logarithmic Total Varia-tion (LTV) [10]. The implementations of these algorithms werebased in part on the publicly available Torch3Vision toolbox(http://torch3vision.idiap.ch) with its default or recommendedparameter settings. We would also like to thank Terrence Chenfor making his implementation of LTV [10] available to us.

To illustrate the effects of the different preprocessors, fig. 5shows some example images from the Yale-B and CAS-PEALdata sets, with the corresponding preprocessor outputs. Asthe images suggest, and the experiments below confirm, pointtransformations such as Histogram Equalization are not veryeffective at removing spatial effects such as shadowing. Incontrast, GB and our method TT (which are the best perform-ers below) remove much of the smooth shading informationand hence emphasize local appearance. The LTV imagesappear washed out owing to the presence of small but intensespecularities and dark peaks in the output. This is partly adisplay issue – MATLAB normalizes images based on theirextreme values – but we have not corrected it to emphasizethat many feature sets and image comparison metrics are alsosensitive to such peaks. In contrast, in GB the peaks tendto diffuse away, while in our method they undergo strongnonlinear compression.

III. LOCAL TERNARY PATTERNS

A. Local Binary Patterns (LBP)

Ojala et al. [28] introduced Local Binary Patterns (LBP)as a means of summarizing local gray-level structure. TheLBP operator takes a local neighborhood around each pixel,thresholds the pixels of the neighborhood at the value of thecentral pixel and uses the resulting binary-valued image patchas a local image descriptor. It was originally defined for 3×3neighborhoods, giving 8 bit integer LBP codes based on the8 pixels around the central one. Formally, the LBP operatortakes the form

LBP (xc, yc) =∑7n=0 2n s(in − ic) (3)

where in this case n runs over the 8 neighbors of the centralpixel c, ic and in are the gray-level values at c and n, and s(u)is 1 if u ≥ 0 and 0 otherwise. The LBP encoding process isillustrated in Fig. 6.

Fig. 6. Illustration of the basic LBP operator.

Two extensions of the original operator were made in [29].The first defined LBP’s for neighborhoods of different sizes,thus making it feasible to deal with textures at different scales.The second defined the so-called uniform patterns: an LBP is‘uniform’ if it contains at most one 0-1 and one 1-0 transitionwhen viewed as a circular bit string. For example, the LBPcode in Fig. 6 is uniform. Uniformity is important becauseit characterizes the patches that contain primitive structural

information such as edges and corners. Ojala et al. observedthat although only 58 of the 256 8-bit patterns are uniform,nearly 90 percent of all observed image neighbourhoods areuniform and many of the remaining ones contain essentiallynoise. Thus, when histogramming LBP’s the number of binscan be reduced significantly by assigning all non-uniformpatterns to a single bin, typically without losing too muchinformation.

B. Local Ternary Patterns (LTP)

LBP’s have proven to be highly discriminative features fortexture classification [28] and they are resistant to lightingeffects in the sense that they are invariant to monotonicgray-level transformations. However because they thresholdat exactly the value of the central pixel ic they tend to besensitive to noise, particularly in near-uniform image regions.Many facial regions are relatively uniform and it is legitimateto investigate whether the robustness of the features can beimproved in these regions.

This section extends LBP to 3-valued codes, Local TernaryPatterns (LTP), in which gray-levels in a zone of width ±taround ic are quantized to zero, ones above this are quantizedto +1 and ones below it to −1, i.e. the indicator s(u) isreplaced by a 3-valued function:

s′(u, ic, t) =

1, u ≥ ic + t0, |u− ic| < t−1, u ≤ ic − t

(4)

and the binary LBP code is replaced by a ternary LTP code.Here t is a user-specified threshold (so LTP codes are moreresistant to noise, but no longer strictly invariant to gray-leveltransformations). The LTP encoding procedure is illustratedin Fig. 7. Here the threshold t was set to 5, so the toleranceinterval is [49, 59].

Fig. 7. Illustration of the basic LTP operator.

When using LTP for visual matching we could use 3n

valued codes, but the uniform pattern argument also applies inthe ternary case. For simplicity, the experiments below use acoding scheme that splits each ternary pattern into its positiveand negative halves as illustrated in Fig. 8, subsequentlytreating these as two separate channels of LBP descriptorsfor which separate histograms and similarity metrics arecomputed, combining the results only at the end of thecomputation.

LTP’s bear some similarity to the texture spectrum (TS)technique from the early 1990’s [39]. However TS did not in-clude preprocessing, thresholding, local histograms or uniformpattern based dimensionality reduction and it was not testedon faces.

6 SUBMITTED IEEE TRANSACTION ON IMAGE PROCESSING

Fig. 8. Splitting an LTP code into positive and negative LBP codes.

C. Distance Transform based Similarity Metric

Ahonen et al. [2] introduced an LBP based method for facerecognition that divides the face into a regular grid of cells andhistograms the uniform LBP’s within each cell, finally usingnearest neighbor classification in the χ2 histogram distancefor recognition:

χ2(p, q) =∑i

(pi − qi)2

pi + qi(5)

Here p, q are image region descriptors (histogram vectors).This method gave excellent results on the FERET dataset.

However subdividing the face into a regular grid seemssomewhat arbitrary: the cells are not necessarily well alignedwith facial features, and the partitioning is likely to causeboth aliasing (due to abrupt spatial quantization of descrip-tor contributions) and loss of spatial resolution (as positionwithin each grid cell is not coded). Given that the overallgoal of coding is to provide illumination- and outlier-robustvisual correspondence with some leeway for small spatialdeviations due to misalignment, it seems more appropriateto use a Hausdorff-distance-like similarity metric that takeseach LBP or LTP pixel code in image X and tests whether asimilar code appears at a nearby position in image Y , with aweighting that decreases smoothly with image distance. Sucha scheme should be able to achieve discriminant appearance-based image matching with a well-controllable degree ofspatial looseness.

We can achieve this using Distance Transforms [7]. Givena 2-D reference image X , we find its image of LBP or LTPcodes and transform this into a set of sparse binary images bk,one for each possible LBP or LTP code value k (i.e. 59 imagesfor uniform codes). Each bk specifies the pixel positions atwhich its particular LBP or LTP code value appears. We thencalculate the distance transform image dk of each bk. Eachpixel of dk gives the distance to the nearest image X pixelwith code k (2D Euclidean distance is used in the experimentsbelow). The distance or similarity metric from image X toimage Y is then:

D(X,Y ) =∑

pixels (i, j) of Y w(dkY (i,j)X (i, j)) (6)

Here, kY (i, j) is the code value of pixel (i, j) of image Y andw() is a user-defined function3 giving the penalty to include for

3w is monotonically increasing for a distance metric and monotonicallydecreasing for a similarity one. In D, note that each pixel in Y is matchedto the nearest pixel with the same code in X . This is not symmetric betweenX and Y even if the underlying distance d is, but it can be symmetrized ifdesired.

Fig. 9. From left to right: a binary layer, its distance transform, and thetruncated linear version of this.

a pixel at the given spatial distance from the nearest matchingcode in X. In our experiments we tested both Gaussiansimilarity metrics w(d) = exp−(d/σ)2/2 and truncatedlinear distances w(d) = min(d, τ). Their performance issimilar, with truncated distances giving slightly better resultsoverall. For 120×120 face images in which an iris or nostrilhas a radius of about 6 pixels and overall global face alignmentis within a few pixels, our default parameter values were σ = 3pixels and τ = 6 pixels.

Fig. 9 shows an example of a binary layer and its distancetransforms. For a given target the transform can be computedand mapped through w() in a preprocessing step, after whichmatching to any subsequent image takes O(number of pixels)irrespective of the number of code values.

IV. A FRAMEWORK FOR ILLUMINATION-INSENSITIVEFACE RECOGNITION

This section details our robust face recognition frameworkintroduced in Section I (c.f . Fig. 2). The full method incorpo-rates the aforementioned preprocessing chain and LBP or LTPfeatures with distance transform based comparison. Hovever,as mentioned above, face recognition is a complex task forwhich it is useful to include multiple types of features, andwe also need to build a final classification stage that can handleresidual variability and learn effective models from relativelyfew training samples.

The selection of an expressive and complementary set offeatures is crucial for good performance. Our initial experi-ments suggested that two of the most successful local appear-ance descriptors, Gabor wavelets [20,40,25] and LBP (or itsextension LTP), were promising candidates for fusion. LBP isgood at coding fine details of facial appearance and texture,whereas Gabor features encode facial shape and appearanceover a range of coarser scales4. Both representations arerich in information and computationally efficient, and theircomplementary nature makes them good candidates for fusion.

In face recognition, it is widely accepted that discriminantbased approaches offer high potential performance and im-proved robustness to perturbations such as lighting variations(e.g. [5]) and that kernel methods provide a well-foundedmeans of incorporating domain knowledge in the discriminant.In particular, Kernel Linear Discriminant Analysis (KLDA[27]) has proven to be an effective method of extracting dis-criminant information from a high dimensional kernel featurespace under subspace constraints such as those engendered by

4Gabor features have also been used as a preprocessing stage for LBPfeature extraction [43].

7

lighting variations [25]. We use Gaussian kernels k(p, q) =e−dist(p,q)/(2σ2), where dist(p, q) is ‖p−q‖2 for Gabor waveletsand χ2 histogram distance (5) for LBP feature sets.

We now summarize our modified KLDA method. Let K bethe empirical kernel matrix of the training set, with eigende-composition K = UΛUT where Λ is the diagonal matrix ofnonzero eigenvalues. U, the associated matrix of normalizedeigenvectors, doubles as a basis for the span of total scattermatrix SφT. Let Φ be the matrix of the centered training setwith respect to the empirical feature space. The matrix thatprojects the training set onto the range space R(SφT) is thenΦ U Λ−1/2. This is used to obtain the projected within-classscatter matrix SΦ

W and between-class scatter matrix SΦB, from

which a basis V for kernel discriminative subspace is obtainedby the following eigendecomposition:

(SΦW + ε I)−1 SΦ

B V = Ξ V (7)

where ε is a small positive regularization constant and I theidentity matrix. The optimal projection matrix P is then:

P = Φ U Λ−1/2 V (8)

A test feature vector ztest can be projected into the optimaldiscriminant space by

Ωtest = PT φ(ztest) = (U Λ−1/2 V)T ktest (9)

where P is the optimal projection matrix givenby (8) and ktest ∈ RM is a vector with entriesK(zi

m, ztest) = 〈φ(zim), φ(ztest)〉, where φ(zi

m) are themapped training samples and M the number of trainingsamples. The projected test feature vector Ωtest is thenclassified using the nearest neighbour rule and the cosine‘distance’

dcos(Ωtest,Ωtemplate) = − ΩTtest Ωtemplate

‖Ωtest‖ ‖Ωtemplate‖(10)

where Ωtemplate is a face template in the gallery set. Othersimilarity metrics such as L1, L2 or Mahalanobis distancescould be used, but [25] found that the cosine distance per-formed best among the metrics it tested on this database, andour initial experiments confirmed this.

When a face image is presented to the system, its Gaborwavelet and LBP features are extracted, separately projectedinto their optimal discriminant spaces (9) and used to com-pute the corresponding distance scores (10). Each score s isnormalized using the ‘z-score’ method [17]

z =s− µσ

(11)

where µ, σ are respectively the mean and standard deviationof s over the training set.

Finally the two scores zGabor and zLBP are fused at thedecision level. Notwithstanding suggestions that it is moreeffective to fuse modalities at an earlier stage of process-ing [17], our earlier work found that although feature-level anddecision-level fusion both work well, decision-level fusion isbetter in this application [36]. Kittler et al. [19] investigateda number of different fusion schemes including product, sum,min and max rules, finding that the sum rule was the most

Fig. 10. The overall architecture of our multi-feature subspace based facerecognition method.

resilient to estimation errors and gave the best performanceoverall. Thus we fuse the Gabor and LBP similarity scores us-ing the simple sum rule: zGabor +zLBP. The resulting similarityscore is input to a simple Nearest Neighbor (NN) classifier tomake the final decision.

Fig. 10 gives the overall flowchart of the proposed method.We emphasize that it includes a number of elements thatimprove recognition in the face of complex lighting variations:(i) we use a combination of complementary visual features– LBP and Gabor wavelets; (ii) the features are individuallyboth robust and information-rich; (iii) preprocessing – whichis usually ignored in previous work on these feature sets[2,3,40,24] – greatly improves robustness; (iv) the inclusion ofkernel subspace discriminants increases discriminativity whilecompensating for any residual variations. As we will showbelow, each of these factors contributes to the overall systemperformance and robustness.

V. EXPERIMENTS

We illustrate the effectiveness of our methods by presentingexperiments on three large-scale face data sets with difficultlighting conditions: Extended Yale B, CAS-PEAL-R1, andFace Recognition Grand Challenge version 2 Experiment 4.For each data set we use its standard evaluation protocol inorder to facilitate comparison with previous work.

We divide the results into two sections, the first focusingon nearest neighbour classification with various LBP/LTPbased feature sets and distance metrics, and the second onKLDA based classifiers with combinations of LBP and Gaborfeatures. Note that unlike subspace based classifiers such asKLDA, the Nearest Neighbour methods do not use a separatetraining set – they simply compare probe images directly togallery ones using a given (not learned) feature set and distancemetric. They are thus simpler, but in general less discriminantthan methods that learn subspaces, feature sets or distancemetrics.

In both cases we compare several different preprocessingmethods. The benefits of preprocessing are particulary markedfor Nearest Neighbour classifiers. We only show results forLBP/LTP here, but additional experiments showed that ourpreprocessing method substantially increases the performanceof Nearest Neighbour classifiers for a wide variety of otherimage descriptors including pixel or Gabor based linear orkernelized eigen- or Fisher-faces under a range of descriptornormalizations and distance metrics.

8 SUBMITTED IEEE TRANSACTION ON IMAGE PROCESSING

Fig. 11. Example images from the three data sets used for testing: (top) frontal images of a subject from Extended Yale-B – columns 1–5 respectivelycontain samples from illumination subsets 1–5; (middle) a subject from the CAS-PEAL probes, with illumination ranging from left to right and from belowto above; (bottom) a subject from FRGC-204 – the first row shows controlled gallery images and the second one uncontrolled query images. In each case weshow raw (geometrically normalized) images on the left, and the corresponding output of our standard preprocessing method on the right. As the experimentsbelow confirm, preprocessing greatly reduces the influence of lighting variations, although it can not completely remove the effects of hard shadowing.

A. Data Sets

Fig. 11 shows some example images from our three datasets,with the corresponding output of our standard preprocessingchain.

Extended Yale-B. The Yale Face Dataset B [5] containing10 people under 64 different illumination conditions has beena de facto standard for studies of recognition under variablelighting over the past decade. It was recently updated to theExtended Yale Face Database B [22], containing 38 subjectsunder 9 poses and 64 illumination conditions. In both casesthe images are divided into five subsets according to the anglebetween the light source direction and the central camera axis(12, 25, 50, 77, 90). For our experiments, the images withthe most neutral light sources (‘A+000E+00’) were used as

the gallery, and all frontal images of each of the standardsubsets 1–5 were used as probes (in all, 2 414 images of 38subjects). The Extended Yale-B set contains only 38 subjectsand it has little variability of expression, ageing, etc. Howeverits extreme lighting conditions still make it a challenging taskfor most face recognition methods.

CAS-PEAL-R1. The CAS-PEAL-R1 face database contains30 863 images of 1 040 individuals (595 males and 445females, predominantly Chinese). The standard experimentalprotocol [12] divides the data into a training set, a galleryset and six frontal probe sets. There is no overlap betweenthe gallery and any of the probes. The gallery contains oneimage taken under standard conditions for each of the 1 040subjects, while the six probes respectively contain images with

9

TABLE IDEFAULT PARAMETER SETTINGS FOR OUR METHODS.

Procedure Parameter ValueGamma Correction γ 0.2DoG Filtering σ0 1

σ1 2Contrast Equalization α 0.1

τ 10LTP t 0.1-0.2LBP/LTP χ2 cell size 8×8σ for KLDA kernels with LBP σ 105

the following basic classes of variations: expression, lighting,accessories, background, distance and ageing. Here we use thelighting probe, which contains 2 243 images. The illuminationconditions are somewhat less extreme than those of Yale-B,but the induced shadows are substantially sharper, presumablybecause the angular light sources were less diffuse. This makesit harder for all high-pass based preprocessors to separateshadows from facial details.

When training KLDA on CAS-PEAL-R1, we use the stan-dard CAS-PEAL-R1 protocol and training set, which contains4 frontal images each of 300 subjects who were randomlyselected from the full 1 040-subject data set.

FRGC-204. The Face Recognition Grand Challenge version2 Experiment 4 data set [30] is the largest data set studiedhere. It contains 12 776 training images, 16 028 target imagesand 8 014 query images. The targets were obtained under con-trolled conditions but the probes were captured in uncontrolledindoor and outdoor settings, including many images of poorquality that pose a real challenge to any recognition method.FRGC-204 is the most challenging data set studied here, owingto its large size and to the wide range of natural variationsthat it contains including large lighting variations, ageing andimage blur.

We use the standard FRGC experimental protocol based onthe Biometric Experimentation Environment (BEE) evaluationtool5, reporting performance in terms of Receiver OperatingCharacteristic (ROC) curves of Face Verification Rate (FVR)versus False Accept Rate (FAR). BEE allows three typesof curves to be generated – ROC-I, ROC-II and ROC-III– corresponding respectively to images collected within asemester, within a year, and across years. Below we report onlyROC-III, the most challenging and most commonly reportedresults. To facilitate comparison with previous publications onFRGC-204, we only used Liu’s 6 388-image subset [25] of thefull FRGC-204 training set for training.

B. Experimental Settings

We restrict attention to geometrically aligned frontal faceviews but allow lighting, expression and identity to vary.Geometric alignment includes conversion to 8 bit gray-scale,rigid image scaling and rotation to place the centers of the twoeyes at fixed positions, and image cropping to 128×128 pixels.

5This evaluates the entire (16028×8014) all-pairs similarity matrix be-tween the query images and the targets – a very expensive calculation thatrequires more than 128 million face comparisons.

The eye coordinates are those supplied with the original datasets.

Unless otherwise noted, the parameter settings listed intable I apply to all experiments. The exact setting of thepreprocessor parameters is not critical: the method givessimilar results over a broad range of settings.

C. Results for Nearest Neighbour Classification

Fig. 12 (top) shows the extent to which nearest neighbourbased LBP face recognition can be improved by combiningthree of the enhancements proposed here: using preprocessing(PP); replacing LBP with LTP; and replacing local his-togramming and the χ2 histogram distance with the DistanceTransform based similarity metric (DT). On Extended Yale-B (top left), the absolute recognition rate is increased byabout 23.5% relative to standard unpreprocessed LBP/χ2.Preprocessing alone boosts the performance by 20.2% (from75.5% to 95.7%). Replacing LBP with LTP improves therecognition rate to 98.7% and adding DT further improvesit to 99.0%. Similarly, on CAS-PEAL-R1 (top right), ourpreprocessor improves the performance by over 20.0% forLBP, and replacing LBP/χ2 with LTP/DT improves it byanother 5.0%. In each case our preprocessing method is themost effective tested, followed by GB.

Fig. 12 (bottom) illustrates how the various feature sets andpreprocessing methods degrade with the increasingly extremeillumination of Extended Yale-B sets 1–5. Even without imagepreprocessing, our system performs quite well under themild lighting changes of subsets 1–3. However preprocessingis required for good performance under the more extremeconditions of subsets 4–5. For the most difficult subset 5,preprocessing improves the performance by 43.1%, whileincluding either LTP or the distance transform respectivelyincreases performance over PP+LBP/χ2 by about 10.0% and8.0%. Again our preprocessing method predominates, althoughLTV catches up as the lighting becomes more difficult andequals our method on subset 5. In contrast, GB does well onthe easier subsets but has trouble with subsets 4 and 5.

To aid comparison with previous work, note that on the(older and smaller) 10 subject Standard Yale-B set ourPP+LTP/DT method gives perfect results for all 5 illuminationsubsets. In contrast, on subsets 2–4: Harmonic Image Exem-plars gives 100, 99.7, 96.9% [42]; nine points of light gives100, 100, 97.2% [23]; and Gradient Angle gives 100, 100,98.6% [9]. None of these authors test on the most difficultset, 5.

D. Results for KLDA Subspace based Classifiers

CAS-PEAL-R1. The above Nearest Neighbour based classi-fiers give almost perfect results on Extended Yale B, but thebest of them only scores 49.2% on CAS-PEAL-R1. This is inline with the state of the art – the best method tested in [12],LGBPHS [43], scored slightly more than 50%, while pixelbased eigenfaces and fisherfaces respectively scored only 8.2%and 21.8% – but it is not very satisfying in absolute terms.CAS-PEAL-R1 is more difficult both because it contains 27

10 SUBMITTED IEEE TRANSACTION ON IMAGE PROCESSING

50

55

60

65

70

75

80

85

90

95

100

None HE MSR LTV GB TT

Rec

ogni

tion

Rat

e (%

)

Gabor/NN LBP+CHI2/NN LTP+CHI2/NN LBP+DT/NN LTP+DT/NN

0

5

10

15

20

25

30

35

40

45

50

None HE MSR LTV GB TT

Rec

ogni

tion

Rat

e (%

)

LBP+CHI2/NN LTP+CHI2/NN LBP+DT/NN LTP+DT/NN

Fig. 12. (Top) Overall nearest-neighbour recognition rates (%) on (left) Extended Yale-B and (right) CAS-PEAL-R1, using the proposed LBP based andGabor features and various preprocessing methods. (Bottom) Breakdown of error rates on the five Extended Yale-B subsets for (left) the various feature setswith our standard preprocessing, and (right) the various preprocessing methods with LTP/DT features.

times more subjects than Extended Yale B, and because ithas a greater degree of intrinsic variability owing to its morenatural image capture conditions (less perfectly controlledillumination, pose, expression, etc.).

To do better, we replaced the Nearest Neighbour classifierwith a kernel subspace (KLDA) based one and also generalizedthe feature set to contain both LBP and Gabor features.See section IV for a description of the resulting recognitionframework. Fig. 13 (bottom left) shows the resulting overallface search performance (recognition rate within the first rresponses). Including both Gabor and LBP features increasesthe rank-1 recognition rate by 30% relative to LBP featuresalone and by 10% relative to Gabor features alone, which sug-gests that the two feature sets do indeed capture different andcomplementary information. The resulting rank-1 recognitionrate of 72.7% is more than 20% higher than the previous bestmethod on this data set [43].

Fig. 13 (top left) presents rank-1 recognition rates on CAS-PEAL for the various preprocessing methods and featuresets. The combination of LBP and Gabor features givesbetter performance than the individual features under all sixpreprocessors (including ‘None’). For Gabor features, MSR,GB, LTV and TT (our method) all significantly improvethe performance relative to no preprocessing, whereas forLBP/LTP features, only our method has a clear positive effect(perhaps due to its inclusion of DoG filtering, which enhancessmall facial details). Histogram equalization (HE) actually

reduces the performance for both feature sets, and LTV alsoreduces it for LBP/LTP features. Our preprocessor is the bestmethod overall, beaten only by GB for pure Gabor features,and GB is again the second best.

FRGC-204. Similar conclusions hold for the FRGC-204 dataset. Fig. 13 (bottom right) shows that the proposed Gabor+LBPmethod increases the FVR at 0.1% FAR from about 80% foreither Gabor or LBP features alone to 88.1%. This exceeds thestate of the art on FRGC-204 [25] by over 12%, thus halvingthe error rate on this important data set.

Fig. 13 (top right) shows how the various preprocessingmethods affect the performance of several combinations ofvisual features and learning methods on FRGC-204. ReplacingNearest Neighbours with KLDA greatly improves the perfor-mance of both LBP and Gabor features under all prepro-cessors, and in each case the combination of Gabor+LBPoutperforms either of the corresponding individual features.Gabor+LBP also outperforms [25] for all preprocessors exceptLTV. In general, the inclusion of KLDA and/or multiplefeatures decreases the performance differences between thedifferent preprocessing methods (or no preprocessing at all),except that the LTV preprocessor uniformly decreases the per-formance of all KLDA methods. Overall TT (our preprocessor)still does best, although under KLDA, MSR is marginallybetter than TT on LBP features and unpreprocessed imagesperform surprisingly well for both LBP and LBP+Gaborfeatures (but not for Gabor alone).

11

Gabor/NN LBP/NN Gabor/KLDA LBP/KLDA LBP+Gabor/KLDA0

10

20

30

40

50

60

70

80

Rec

ogni

tion

Rat

e (%

)

None HE LTV MSR GB TT

0

10

20

30

40

50

60

70

80

90

Ver

ifica

tion

Rat

e (%

)

LBP/NN Gabor/NN LBP/KLDA Gabor/KLDA LBP+Gabor/KLDA

None HE LTV MSR GB TT

Fig. 13. Performance of the full KLDA-based methods on (left) CAS-PEAL-R1 and (right) FRGC-204. (Top) Recognition rates for several combinationsof visual features and learning methods, under various preprocessing options, for (left) CAS-PEAL-R1, (right) FRGC-204 (FVR at 0.1% FAR). (Bottom left)Search performance (% of cases with the correct subject within the first N matches) on CAS-PEAL-R1 for the KLDA Gabor, LBP and Gabor+LBP methods.(Bottom right) ROC-III face recognition performance on FRGC-204 for the KLDA Gabor, LBP and Gabor+LBP methods. The BEE baseline and Liu’s method[25] are also shown for comparison.

Fig. 14 shows the influence of training set size on FRGCvalidation rates for the KLDA based methods. Keeping all222 subjects while reducing the number of training imagesper subject has relatively little effect on performance untilthere are fewer than about 10 images per subject. Conversely,reducing the number of subjects while keeping a fixed numberof images per subject causes a much more rapid deterioration.This suggests that (with our robust descriptors) the principaldegree of variation in this dataset is identity not lighting relatedappearance changes.

Finally, we very briefly illustrate the contributions of theindividual stages of our preprocessing chain on the FRGC-204 data set for various features and learning methods6. Fig 15illustrates the effect of removing each of the four main stagesof preprocessing in turn while leaving the remaining stagesin place (the comparison is thus against our full preprocessor,

6We only present a small selection of our experimental results on prepro-cessing owing to lack of space. In general the experiments show that undernearest neighbour classification, each stage of preprocessing is beneficialfor a broad range of features and distance metrics including pixel-basedrepresentations such as eigen- or fisher-faces, local filters such as Gaborfeatures and texture histograms such as LBP/LTP.

not against no preprocessing). In general, each stage of prepro-cessing is beneficial and (not shown) the results are cumulativeover the stages, but the benefits are much greater for NearestNeighbour classifiers than for KLDA ones. The only casein which omitting a single stage of preprocessing actuallyimproves the results is for DoG filtering with LBP featuresunder KLDA, and the improvement in this case is slight (andto be contrasted with the large decrease that occurs when DoGis omitted from LBP under Nearest Neighbour classification).Also note that the last two stages of preprocessing involvemonotone gray-level transformations and hence (as expected)have no effect on LBP features. We nevertheless include themin our default preprocessor because they cause no harm forLBP and have a very beneficial effect on Gabor wavelets anda number of other feature sets for which we do not showresults including LTP and pixel-based features such as eigen-and Fisher-faces.

E. Discussion

The substantial performance gains produced by replacingnearest neighbour classification with KLDA on CAS-PEAL-

12 SUBMITTED IEEE TRANSACTION ON IMAGE PROCESSING

222 / 2 222 / 4 222 / 7.3 222 / 14.4 222 / 28.8 111 / 29.3 55 / 29.610

20

30

40

50

60

70

80

90

Training Set − # subjects / # images per subject

Ver

ifica

tion

Rat

e (%

)

NoPP / Gabor TT / Gabor NoPP / LBP TT / LBP NoPP / LBP+Gabor TT / LBP+Gabor

Fig. 14. Influence of the size of the training set on FRGC validation rates, for KLDA based methods with the Gabor, LBP and combined LBP+Gabor featuresets, with (‘TT’) and without (‘NoPP’) preprocessing. The full FRGC training set contains 222 subjects with an average of about 29 images per subject (‘222/ 28.8’ – the fifth group in the plot). The first four groups show that if we use all 222 subjects but reduce the number of training images by randomly selectingrespectively 2, 4, an average of about 7 or an average of about 14 images per subject, the performance gradually decreases, but quite good results can still beobtained with about 10 images per subject. In contrast, the final two groups show that reducing the number of subjects quickly reduces the performance, evenif we use all of the available images for them. In general, the performance differences between the different feature sets and preprocessing methods increaseas the amount of training data is reduced: KLDA is quite effective at reducing these differences, but only when it has sufficient training data.

Gabor/NN LBP/NN Gabor/KLDA LBP/KLDA LBP+Gabor/KLDA0

10

20

30

40

50

60

70

80

90

Ver

ifica

tion

Rat

e (%

)

No Gamma correctionNo DoG filteringNo equalization for variationsEqualization but no tanh compressionFull TT preprocessor

Fig. 15. Influence of the individual stages of our preprocessing chain. Forvarious features and learning methods on the FRGC-204 data set, we comparethe recognition rates (%) with our full preprocessing method to rates wheneach of the four main steps of preprocessing is removed in turn, leaving theremaining steps in place.

R1 and FRGC-204 are welcome but not particularly surpris-ing. These data sets have many sources of natural variationbesides illumination (other imaging conditions, expression,ageing, etc.) and their galleries contain very few examplesof each individual, giving Nearest Neighbour methods basedon generic (non-learned) features and distance metrics littleopportunity to generalize across these variations.

On the other hand, like Nearest Neighbours, KLDA is

based on an underlying image similarity metric: the featurespace distance embedded in its Gaussian kernel. Given theextent to which preprocessing improves Nearest Neighboursby providing a more illumination-resistant distance metric forcomparisons, one might have hoped for analogous improve-ments under KLDA. In this respect – even allowing for thefact that facial lighting variations can be described quite wellby rather low-dimensional models, c.f . e.g. [5,6,4] – KLDA’sability to compensate for the absence of preprocessing issomewhat surprising. Presumably, even though the suppliedtraining data was not designed to systematically span the spaceof lighting variations, KLDA implicitly learns a nonlineardescriptor space lighting model that is more accurate thanthe default models that are implicitly embodied in the variouspreprocessors tested, thus producing a more accurate implicit“projection to an illumination invariant description”. Sayingthis another way, rather than comparing each incoming exam-ple to a relatively large but stable set of illumination-invariantsupport vectors (nearby training examples after preprocessing),it seems to be better to compare them to a smaller butmore variable set of non-invariant support vectors with similarlighting (nearby unpreprocessed training examples).

Regarding the choice of preprocessor (if any), our method(TT) seems to be the best overall: not only is it fast (at leasta factor of 10 faster than GB and LTV) and very simple toimplement, but it provides the best performance among themethods tested in almost all of our experiments. GB is thesecond choice for Nearest Neighbour or KLDA classificationon datasets with relatively mild illumination variations (CAS-PEAL-R1, FRGC-204 and Extended Yale-B subsets 1-3). LTV

13

performs poorly on these sets, but becomes competitive forNearest Neighbour classification on sets with extreme lightingvariations such as Yale-B subsets 4-5 – c.f . Fig. 12 (bottomright).

VI. SUMMARY AND CONCLUSIONS

We have presented new methods for face recognition underuncontrolled lighting based on robust preprocessing and anextension of the Local Binary Pattern (LBP) local texture de-scriptor. There are following main contributions: (i) a simple,efficient image preprocessing chain whose practical recogni-tion performance is comparable to or better than current (oftenmuch more complex) illumination normalization methods;(ii) a rich descriptor for local texture called Local TernaryPatterns (LTP) that generalizes LBP while fragmenting lessunder noise in uniform regions: (iii) a distance transformbased similarity metric that captures the local structure andgeometric variations of LBP/LTP face images better than thesimple grids of histograms that are currently used; and (iv)a heterogeneous feature fusion-based recognition frameworkthat combines two popular feature sets – Gabor wavelets andLBP – with robust illumination normalization and a kernelizeddiscriminative feature extraction method. The combination ofthese enhancements gives the state of the art performance onthree well-known large-scale face datasets that contain widelyvarying lighting conditions.

Moreover, we empirically make comprehensive analysis andcomparison with several state of the art illumination normal-ization methods on the large-scale FRGC-204 dataset, and in-vestigate their connections with robust descriptors, recognitionmethods and image quality. This provides new insights intothe role of robust preprocessing methods played in dealingwith difficult lighting conditions and thus being useful in thedesignation of new methods for robust face recognition.

REFERENCES

[1] Y. Adini, Y. Moses, and S. Ullman, “Face recognition: Theproblem of compensating for changes in illumination direction,”IEEE Trans. Pattern Analysis & Machine Intelligence, vol. 19,no. 7, pp. 721–732, 1997.

[2] T. Ahonen, A. Hadid, and M. Pietikainen, “Face recognitionwith local binary patterns,” in European Conf. Computer Vision,Prague, 2005, pp. 469–481.

[3] ——, “Face description with local binary patterns: Applicationto face recognition,” IEEE Trans. Pattern Analysis & MachineIntelligence, vol. 28, no. 12, 2006.

[4] R. Basri and D. Jacobs, “Lambertian reflectance and linear sub-spaces,” IEEE Trans. Pattern Analysis & Machine Intelligence,vol. 25, no. 2, pp. 218–233, February 2003.

[5] P. Belhumeur, J. Hespanha, and D. Kriegman, “Eigenfaces vs.Fisherfaces: Recognition using class specific linear projection,”IEEE Trans. Pattern Analysis & Machine Intelligence, vol. 19,no. 7, pp. 711–720, 1997.

[6] P. Belhumeur and D. Kriegman, “What is the set of imagesof an object under all possible illumination conditions,” Int. J.Computer Vision, vol. 28, no. 3, pp. 245–260, 1998.

[7] G. Borgefors, “Distance transformations in digital images,”Computer Vision, Graphics & Image Processing, vol. 34, no. 3,pp. 344–371, 1986.

[8] R. Brunelli and T. Poggio, “Face recognition: Features versusTemplates,” IEEE Trans. Pattern Analysis & Machine Intelli-gence, vol. 15, no. 10, pp. 1042–1052, 1993.

[9] H. Chen, P. Belhumeur, and D. Jacobs, “In search of illumina-tion invariants,” in CVPR, 2000, pp. I: 254–261.

[10] T. Chen, W. Yin, X. Zhou, D. Comaniciu, and T. Huang, “Totalvariation models for variable lighting face recognition,” IEEETrans. Pattern Analysis & Machine Intelligence, vol. 28, no. 9,pp. 1519–1524, 2006.

[11] N. Dalal and B. Triggs, “Histograms of oriented gradients forhuman detection,” in CVPR, Washington, DC, USA, 2005, pp.886–893.

[12] W. Gao, B. Cao, S. Shan, X. Chen, D. Zhou, X. Zhang, andD. Zhao, “The CAS-PEAL large-scale chinese face databaseand baseline evaluations,” IEEE Trans. Systems, Man and Cy-bernetics, Part A, vol. 38, no. 1, pp. 149–161, 2008.

[13] R. Gross and V. Brajovic, “An image preprocessing algorithmfor illumination invariant face recognition,” in AVBPA, 2003,pp. 10–18.

[14] F. Guodail, E. Lange, and T. Iwamoto, “Face recognition systemusing local autocorrelations and multiscale integration,” IEEETrans. Pattern Analysis & Machine Intelligence, vol. 18, no. 10,pp. 1024–1028, 1996.

[15] X. He, X. Yan, Y. Hu, P. Niyogi, and H. Zhang, “Facerecognition using Laplacianfaces,” IEEE Trans. Pattern Analysis& Machine Intelligence, vol. 27, no. 3, pp. 328–340, 2005.

[16] Y. S. Huang and C. Y. Suen, “A method of combining multipleexperts for the recognition of unconstrained handwritten nu-merals,” IEEE Trans. Pattern Analysis & Machine Intelligence,vol. 17, no. 1, pp. 90–94, 1995.

[17] A. Jain, K. Nandakumar, and A. Ross, “Score normalization inmultimodal biometric systems,” Pattern Recognition, vol. 38,no. 12, pp. 2270–2285, 2005.

[18] D. Jobson, Z. Rahman, and G. Woodell, “A multiscale retinexfor bridging the gap between color images and the humanobservation of scenes,” IEEE Trans. Image Processing, vol. 6,no. 7, pp. 965–976, 1997.

[19] J. Kittler, M. Hatef, R. Duin, and J. Matas, “On combining clas-sifiers,” IEEE Trans. Pattern Analysis & Machine Intelligence,vol. 20, no. 3, pp. 226–239, 1998.

[20] M. Lades, J. C. Vorbruggen, J. Buhmann, J. Lange, C. von derMalsburg, R. P. Wurtz, and W. Konen, “Distortion invariantobject recognition in the dynamic link architecture,” IEEETransactions Computers, vol. 42, no. 3, pp. 300–311, 1993.

[21] S. Lawrence, C. Lee Giles, A. Tsoi, and A. Back, “Facerecognition: A convolutional neural-network approach,” IEEETrans. Neural Networks, vol. 8, no. 1, pp. 98–113, 1997.

[22] K. Lee, J. Ho, and D. Kriegman, “Acquiring linear subspaces forface recognition under variable lighting,” IEEE Trans. PatternAnalysis & Machine Intelligence, vol. 27, no. 5, pp. 684–698,2005.

[23] ——, “Nine points of light: Acquiring subspaces for facerecognition under variable lighting,” in CVPR, 2001, pp. I:519–526.

[24] C. Liu, “Gabor-based kernel pca with fractional power polyno-mial models for face recognition,” IEEE Trans. Pattern Analysis& Machine Intelligence, vol. 26, no. 5, pp. 572–581, 2004.

[25] ——, “Capitalize on dimensionality increasing techniques forimproving face recognition grand challenge performance,” IEEETrans. Pattern Analysis & Machine Intelligence, vol. 28, no. 5,pp. 725–737, 2006.

[26] C. Liu and H. Wechsler, “A shape- and texture-based enhancedfisher classifier for face recognition,” IEEE Trans. Image Pro-cessing, vol. 10, no. 4, pp. 598–608, 2001.

[27] S. Mika, G. Ratsch, J. Weston, B. Scholkopf, and K.-R. Muller,“Fisher discriminant analysis with kernels,” in Neural Networksfor Signal Processing IX, Y.-H. Hu, J. Larsen, E. Wilson, andS. Douglas, Eds. Piscataway, NJ: IEEE, 1999, pp. 41–48.

[28] T. Ojala, M. Pietikainen, and D. Harwood, “A comparativestudy of texture measures with classification based on featuredistributions,” Pattern Recognition, vol. 29, no. 1, pp. 51–59,1996.

14 SUBMITTED IEEE TRANSACTION ON IMAGE PROCESSING

[29] T. Ojala, M. Pietikainen, and T. Maenpaa, “Multiresolutiongray-scale and rotation invarianat texture classification withlocal binary patterns,” IEEE Trans. Pattern Analysis & MachineIntelligence, vol. 24, no. 7, pp. 971–987, 2002.

[30] P. J. Phillips, P. J. Flynn, W. T. Scruggs, K. W. Bowyer,J. Chang, K. Hoffman, J. Marques, J. Min, and W. J. Worek,“Overview of the face recognition grand challenge,” in CVPR,San Diego, CA, 2005, pp. 947–954.

[31] S. Shan, W. Gao, B. Cao, and D. Zhao, “Illumination nor-malization for robust face recognition against varying lightingconditions,” in AMFG, Washington, DC, USA, 2003, p. 157.

[32] J. Short, J. Kittler, and K. Messer, “A comparison of photometricnormalization algorithms for face verification,” in IEEE Int.Conf. Automatic Face & Gesture Recognition, 2004, pp. 254–259.

[33] L. Sirovich and M. Kirby, “Low dimensional procedure for thecharacterization of human faces,” J. Optical Society of America,vol. 4, no. 3, pp. 519–524, 1987.

[34] X. Tan, S. Chen, Z.-H. Zhou, and F. Zhang, “Recognizingpartially occluded, expression variant faces from single trainingimage per person with SOM and soft kNN ensemble,” IEEETrans. Neural Networks, vol. 16, no. 4, pp. 875–886, 2005.

[35] X. Tan and B. Triggs, “Enhanced local texture feature sets forface recognition under difficult lighting conditions,” in AMFG,2007, pp. 168–182.

[36] ——, “Fusing gabor and LBP feature sets for kernel-based facerecognition,” in AMFG, 2007, pp. 235–249.

[37] M. Turk and A. Pentland, “Eigenfaces for recognition,” Journalof Cognitive Neurscience, vol. 3, no. 1, pp. 71–86, 1991.

[38] H. Wang, S. Li, and Y. Wang, “Face recognition under varyinglighting conditions using self quotient image,” in IEEE Int.Conf. Automatic Face & Gesture Recognition, 2004, pp. 819–824.

[39] L. Wang and D. He, “Texture classification using texture spec-trum,” Pattern Recognition, vol. 23, pp. 905–910, 1990.

[40] L. Wiskott, J.-M. Fellous, N. Kruger, and C. von der Malsburg,“Face recognition by elastic bunch graph matching,” IEEETrans. Pattern Analysis & Machine Intelligence, vol. 19, no. 7,pp. 775–779, 1997.

[41] J. Yang, A. F. Frangi, J.-Y. Yang, D. Zhang, and Z. Jin, “KPCAplus LDA: A complete kernel fisher discriminant frameworkfor feature extraction and recognition,” IEEE Trans. PatternAnalysis & Machine Intelligence, vol. 27, no. 2, pp. 230–244,2005.

[42] L. Zhang and D. Samaras, “Face recognition under variablelighting using harmonic image exemplars,” in CVPR, vol. 01,Los Alamitos, CA, USA, 2003, pp. 19–25.

[43] W. Zhang, S. Shan, W. Gao, and H. Zhang, “Local GaborBinary Pattern Histogram Sequence (LGBPHS): A novel non-statistical model for face representation and recognition,” inICCV, Beijing, China, 2005, pp. 786–791.

[44] W. Zhao, R. Chellappa, P. J. Phillips, and A. Rosenfeld, “Facerecognition: A literature survey,” ACM Computing Surveys,vol. 34, no. 4, pp. 399–485, 2003.