Do You See What I See? A More Realistic Eyewitness Sketch Recognition

Hossein Nejati and Terence SimNational University of Singapore

School of Computing, National University of Singapore, Singapore 117417{nejati,dcssimt}@nus.edu.sg

Elisa Martinez-MarroquinLa Salle School of Engineering - Ramon Llull University

St Joan de la Salle, 42. 08022 - Barcelona. Spainelisa@salle.url.edu

Abstract

Face sketches have been used in eyewitness testimoniesfor about a century. However, 30 years of research showsthat current eyewitness testimony methods are highlyunreliable. Nonetheless, current face sketch recognitionalgorithms assume that eyewitness sketches are reliableand highly similar to their respective target faces. Asproven by psychological findings and a recent work onface sketch recognition, these assumptions are unrealisticand therefore, current algorithms cannot handle real worldcases of eyewitness sketch recognition.

In this paper, we address the eyewitness sketch recog-nition problem with a two-pronged approach. We proposea more reliable eyewitness testimony method, and an ac-companying face sketch recognition method that accountsfor realistic assumptions on sketch-photo similarities andindividual eyewitness differences. In our eyewitness testi-mony method we first ask the eyewitness to directly drawa sketch of the target face, and provide some ancillaryinformation about the target face. Then we build a drawingprofile of the eyewitness by asking him/her to draw a setof face photos. This drawing profile implicitly containsthe eyewitness’ mental bias. In our face sketch recognitionmethod we first correct the sketch for the eyewitness’ biasusing the drawing profile. Then we recognize the resultingsketch based on an optimized combination of the detectedfeatures and ancillary information. Experimental resultsshow that our method is 12 times better than the leadingcompeting method at Rank-1 accuracy, and 6 times betterat Rank-10. Our method also maintains its superiority asgallery size increases.

I.. Introduction

In 1984, based on the testimony of five eyewitnesses,Kirk Bloodsworth was convicted of the rape and murderof a nine-year-old girl and sentenced to the gas chamber.After Bloodsworth served nine years in prison, DNAtesting proved him to be innocent [1]. Such devastatingmistakes by eyewitnesses are not rare, and more than 75%of the convictions overturned through DNA testing sincethe 1990s were based on eyewitness testimony [2].

The eyewitness testimony for forensic applications hashad a long history, with roots that go back to the beginningof the nineteenth century [3]. It has been an importanttool for law enforcement agencies in difficult situationswhere no other clue is available. However, more than 30years of psychological studies show that current eyewitnesstestimony methods are highly susceptible to error andshould be reformed [4], [2], [5] (see figure 1 for someexamples). For example, in the very first step of currenteyewitness testimony methods (ETMs), the eyewitnessprovides a verbal description of the target face, not basedon the real appearance of the face, but on own mentalnorm and bias [6]. Moreover, this verbal description itselfdegrades the eyewitness’ visual memory as well as therecognizability of the target face, in a phenomenon knownas verbal overshadowing [7]. The eyewitness’ memorycan also be altered due to misleading information aris-ing from viewing similar faces or answering subjectivequestions raised by the police [8], [9]. Finally, piecewisereconstruction of the target face, specially in compositesketches, degrades the eyewitness’ mental image of thetarget face and the recognition result [10]. These problemsare basically because human memory is fragile, malleable,and susceptible to suggestion [8], which in turn render

Figure 1. Some examples of unreliable artistic sketches (twoleft columns from http://depletedcranium.com) and compositesketches (two right columns [11]).

current ETMs unreliable. Despite this, current methodsare still widely used by police departments all over theworld. This is mainly due to the lack of viable alternativemethods.

The resulting sketch from an eyewitness testimony,which we term the forensic sketch (which is drawn by apolice artist, or produced by a photo-composite software),should be matched against the police database of faces.There exist several automatic face sketch recognition al-gorithms (FSRs) proposed in the literature to perform thismatching, with some reporting recognition accuracies ashigh as 92% [12]. The main problem with all but oneof these FSRs is that they were tested on exact artisticsketches, instead of forensic sketches. These exact artisticsketches bear great similarity with their target faces, includ-ing exact point-to-point geometry, skin texture, lighting,and even hairstyle (see figure 2, and the next section).In contrast, forensic sketches usually differ greatly fromthe target face, due to verbal overshadowing, alterationsto the memory, and personal biases, etc. (compare figures1 and 2). Even if existing FSRs were applied on forensicsketches, they reliability would still be suspect, becausethese forensic sketches are unreliable in the first place, asmany psychological studies have shown (see [7], [4], [3],[2], [5]). As the saying goes, garbage in, garbage out.

To summarize:1) Current ETMs are very unreliable. This unreliability

also propagates to any process which uses forensicsketches, e.g. FSRs.

2) Current FSRs are also unreliable due to both the useof unreliable input, and to significantly overestimat-ing the sketch/target face similarity.

In this work we propose two methods to address both ofthese gaps:

1) We propose to employ a modified ETM procedurethat is based on the neural processes involved in

the Exception Report Model of face recognitionpropounded recently by Unnikrishnan [13], to avoidthe said pitfalls.

2) We propose an accompanying FSR for our ETMwhich accounts for individual differences amongeyewitnesses, and relies on more realistic assump-tions of sketch-target face similarity.

In the ETM part, we first ask the eyewitness to directlydraw a sketch of the target face with whatever drawingskills he/she has. We call this non-artistic sketch, the MainSketch. By having the eyewitness draw the Main Sketchby him/herself, we avoid verbal overshadowing, memorydegradation (due to questions, viewing other faces, etc),and piecewise reconstruction. However, as the result ofthis ETM is a non-artistic face sketch, we should processthis crude representation of the target face to find theidentity-specific features. In addition to the Main Sketch,we propose to ask the eyewitness to provide AncillaryInformation about the target face, such as its gender, skintone, ethnicity, etc (which is also practiced in real scenar-ios). Finally, we acquire an individual drawing profile forthe eyewitness by asking him/her to sketch a set of facephotos (in our experiments we asked eyewitnesses to drawat least 4 of these face photos).

In the FSR part, we first remove the noise and eye-witness’ mental bias from the Main Sketch. To do this,we estimate the true shape of the target face, from theMain Sketch, based on the eyewitness’ drawing profile−bylearning individual drawing and feature representationstyle. We then linearly combine all features by optimallyweighting different parts of the estimated shape and eachof the Ancillary Information. Finally, we use this linearcombination to match the modified Main Sketch against agallery of face photos.

We test our algorithm which we call Non-ArtisticFace Sketch Recognition (NAFSR), on a dataset of 100sketches, hand drawn by 10 eyewitnesses, from 249 facephotos of the Multi-PIE face dataset [14]. We comparedthe results of NAFSR with a PCA classification algorithm,an FSR introduced in [15], and a SIFT-based classification(used in [16]). In this comparison, NAFSR shows a signif-icant improvement over the other methods. These resultsshow that with a more reliable ETM, and by accounting foreyewitness bias in the FSR, we can recognize eyewitnesssketches more accurately..

II.. Related WorksThe resulting sketch of an eyewitness testimony, the

forensic sketch, is a representation of the target face whichshould be matched against the police face database of crim-inals. Several different face sketch recognition algorithms(FSRs) are proposed in the literature for recognizing thisrepresentation of the target face. Among the first to propose

Figure 2. Examples of the use of exact artistic sketches inprevious works. Note the extent of similarity between photos andsketches.

an FSR algorithm were Tang et al. [15]. They proposedan eigenface transformation to transfer gallery face photosto sketch-like images. This transformation decreases thedifference between the faces and the sketches, and resultsin better performance in PCA classification. The sketch-like images were then matched against an artistic sketchgallery using a PCA-based algorithm, with a reportedrecognition accuracy of 89%.

Another photo-to-sketch transformation method is pro-posed by Li et al. [17] in which eigenface transformationis similarly employed. In this method, a sketch-photo pairimage is concatenated into a single vector to learn the PCAclassifier towards an eigenspace with correlation to boththe sketch and the real face. However, the test results ofthis work are limited to transformed version of the originalimage, and no real sketches.

Liu et al. [12] further improved the photo-to-sketchtransformation using a non-linear transformation in whichthe photo is divided into patches and replacing each patchby the most similar patch from the sketch gallery. Thepatch similarity is calculated using a PCA-based classifier.The result of this transformation is then matched againstan artistic sketch gallery, using Non-linear discriminantanalysis with a reported accuracy of 92%.

Another improvement to patch based the photo-to-sketch transformation is introduced by Wang and Tang [18]in which a trained multi-scale Markov random field stitchesand warps the patches into a final sketch. This final sketchis then used for sketch-to-sketch matching in order to findthe target face, by a PCA-based algorithm.

While most of previous works have focused on photo-to-sketch transforming, Xiao et al. [19], [20] proposed anapproach which exploits a sketch-to-photo transformation.In this work, the authors reconstructed a photo-like imagefrom the provided artistic sketches, in order to transformthe problem into a photo-to-photo matching problem. Inthis work, embedded hidden Markov model (EHMM) isused to reconstruct each sketch, with the most similar

patches from the photo database. The resulting photo-likeimage is then classified using PCA.

Although the above algorithms are proposed to addressthe forensic sketch recognition problem, all of them havebeen tested on exact artistic sketches. As shown in figure2, these exact artistic sketches have significant similari-ties to their target faces (including exactly similar facialcomponent shape, illumination and shading, skin texture,and even hairstyle). In contrast, as figure 1 illustrates, areal forensic sketch from current eyewitness testimonies isvery likely to be significantly different, due to verbal over-shadowing, perception biases, piecewise reconstruction, etc(see [7], [4], [3], [2], [5]). Therefore, although the testresults of the above methods show that they can accuratelyrecognize exact artistic sketches, they provide no clueabout their performance on recognizing forensic sketches.

The only work which has stepped further by testingforensic sketches is [16]. In this work, Klare et al. em-ployed a fusion of SIFT features and multiscale localbinary patterns to recognize some forensic sketches as wellas exact artistic sketches. The interesting result of this workis that this algorithm reported to have 99.47% accuracyin matching exact artistic sketches; but when tested onforensic sketches, its accuracy dramatically decreased to16.33% (rank-1) and remained less than 33% even in rank-50. These results support our argument that even if analgorithm can accurately recognize exact artistic sketches,it does not necessarily provide reliable results in recog-nizing forensic sketches. This work also tested a state-of-art face recognition software [21] (as a representativefor conventional face recognition algorithms) on matchingforensic sketches, with its accuracy reported to be as lowas 2.04 and 8.16% in rank-1 and rank-50 respectively.

Based on the discussions in this section, we concludethree main points. First, although previous FSRs havereported high accuracy rates, they seem unreliable inrecognizing real eyewitness sketches. Second, exact artisticsketches are not a proper estimation of real forensicsketches. And third, conventional face recognition methodscannot be applied to match forensic sketches.

III.. Our MethodThe problem of face sketch recognition has two main

parts: (1) The eyewitness testimony method (ETM) inwhich a sketch is created, (2) the recognizing the resultingsketch from the ETM. Without a reliable ETM, the secondpart also provides unreliable results. Therefore, as all ofthe previous works have ignored the unreliability of currentETMs and merely focused on the final sketch, they couldnot accurately match forensic sketches.

Here we present a Non-Artistic Face Sketch Recogni-tion (NAFSR) approach to address the face sketch recog-nition problem by accounting for both of its parts. We de-

Figure 3. Symbolic representation of the NAFSR (top), and previous FSRs (bottom). Top: the ETM section, the eyewitness providesthe Ancillary Information (AUX), the Main Sketch, and the drawing profile, based on his/her mental normal distribution (MND). TheFSR section: (1) Support Vector Regressors (SVRs) training to map MND to system normal distribution (SND), based on the drawingprofile; (2) target shape estimation from the Main Sketch, (3) unusuality detection based on SND, (4) point difference measurement,(5) AUX difference measurement, (6) feature difference normalization, (7) optimized linear combination. Bottom: (1) transformation(sketch-to-photo or photo-to-sketch), (2) feature extraction, (3) possible normalization, (4) feature comparison. Note that in the previousmethods, the ETM section is ignored, thus the process of creation of the sketch remained unknown.

scribe a psychologically plausible and reliable eyewitnesstestimony method, accompanied by a sketch recognitionmethod, to faithfully retrieve and recognize eyewitness’memory of the target face (figure 3).

A.. Proposed Eyewitness Testimony Method

The basic task of an eyewitness testimony is to retrievethe eyewitness’ mental image of a face, and encode itinto a usable format for other processes such as FSRs.Information retrieval is a critical step in eyewitness testi-mony in which memory disturbances should be avoidedand individual differences should be accounted for. Inthe ETM part, we gather various types of informationregarding each target face. First, we ask the eyewitnessto draw a sketch of the target face by himself, called theMain Sketch, that includes outlines of facial features andfacial marks (e.g. wrinkles, moles, or scars). Examplesof these Main Sketches are presented in figure 4. Byasking the eyewitness to draw the Main Sketch himself,we avoid introducing changes to the eyewitness memory.The Main Sketch is therefore more likely to be drawnbased on genuine memory of the target face, in contrastwith traditional forensic sketches which are drawn based

on eyewitness’ verbal description (see section I). On theother hand, the Main Sketch is a crude and non-artisticrepresentation of the target face, which requires processingbefore being used (describe in section III-B).

In addition to the Main Sketch, we ask the eyewitness toprovide Ancillary Information about the target face includ-ing skin color, iris color, hair color, estimated age, ethnic-ity, and gender. For each class of the Ancillary Information,we provide a set of predefined classes from which theeyewitness can choose. We ask the eyewitness to choosethe skin color from Fitzpatrick Scale color pallet (veryfair, white, beige, beige with a brown tint, dark brown,black) [22], the iris color from Martin–Schultz scale colorpallet (gray, blue, green, brown, dark brown, black, red)[23], and the hair color from Fischer–Saller scale colorpallet (brown, black, blond, auburn, red, gray/white) [24].For estimated age, we group ages in groups of 5 years(e.g. 1-5, 6-10, 11-15...). For the ethnicity, we group racesinto Caucasian, American Indian, Latino, African, MiddleEastern, Indian, and East Asian. And finally for the genderwe have male and female.

Finally, we obtain a drawing profile for each eyewitnessby asking him/her to sketch a set of face photos. Here we

Figure 4. Examples of the Main Sketches used in our method,delivered directly by the eyewitnesses.

asked the eyewitnesses to draw at least 4 faces from aset of 30 face photos. The three outputs of the ETM isillustrated in figure 3.

B.. Proposed Face Sketch RecognitionWe define the sketch recognition process as finding

photo(s) with the smallest the difference score to a givensketch, in the photo dataset (see figure 3, FSR section).The difference score is defined based on two of the threeinputs from the ETM: the Main Sketch, the AncillaryInformation. However, as shown in figure 4, the MainSketch is a non-artistic representation of the target face,and requires processing before being used in the differencescore. Therefore we use the third input from the ETM, thedrawing profile, to estimate the true shape of the targetface from the Main Sketch.

1) Target Face Shape Estimation: The Exception Re-port Model (ERM) of face recognition, recently proposedby Unnikrishnan [13], suggests that the human brainemploys a modified norm based model in which attentionis focused exclusively on the unusual features for rapid andeffortless recognition of the target face. ERM is probablythe underlying mechanism that plays a role in perceiving,remembering and recognizing faces [13]. While previousmodels of face recognition require as many as 200 to250 features to characterize a face, ERM focuses attentionexclusively on features that are significantly different fromthe average or modal face. Thus, only a few (<10) unusualfeatures are required to characterize the individuality of agiven face. ERM can be quicker and require less mental ef-fort, because the neural processes employed by this modelare more economical than those employed by previousmodels of face recognition, which require exhaustive andhighly accurate multiple feature-by-feature comparisonsbetween different faces.

Motivated by ERM, we can separately account for

normal and unusual facial features presented in the MainSketches. The unusual nature of a facial feature is de-termined on the basis of the normal distribution of thatfeature. However, the normal distribution in the eyewit-ness’ mind (MND) can be different from a systematicallycalculated normal distribution (SND) based on the photodataset. Moreover, two different eyewitnesses may havedifferent MNDs based on their personal life experiences.When two individuals have different MNDs, they maydifferently perceive and remember a particular face as theymay identify a different set of unusual features in thatface. Similarly, when two eyewitnesses draw a face, theymay present different features as normal or unusual. Inaddition, personal drawing styles should also be considered[25]. Thus, to be able to estimate “what the eyewitnessmeans” from “what the eyewitness draws”, we shouldcalculate an individually tailored mapping function, fromeach eyewitness’ MND, to our system’s SND .

In order to learn the mapping function from MND toSND, we learn the user’s individual drawing style basedon his/her drawing profile. For each sketch-photo pair inthe drawing profile, we first scale and rotate all sketchesto the same size and angle based on the position of eyecenters. We then (using a simple Active Shape Model)fit 16 Piecewise Cubic Hermite Interpolating Polynomialsplines [26] to the outlines of 7 facial components, theeyes, the eyebrows, the nose, the mouth, and the jaw-line(2 to each eye, 2 to each eyebrow, 2 to the mouth, 3 tothe nose, and 3 to the jaw-line). Each of these 16 splinesare then divided into four parts (quarter splines), and eachpart is sampled in 25 equally distributed data points. Thenwe use these data points to train Support Vector Machineregressors for each quarter spline (i.e. 4 mapping functionfor each spline, 64 mapping function for the entire sketch).Finally, we estimate the true shape of the target face, bymapping each point in the Main Sketch by its respectivequarter spline mapping function. The training SVRs andtarget face shape estimation is indicated in processes 1 and2 in figure 3. After removing individual biases from theMain Sketch, we proceed to sketch recognition procedure.

2) Recognizing the Main Sketch: In order to recognizethe Main Sketch (matching process), we define a differencescore between a given sketch and a photo, based on theestimated target shape and the Ancillary Information. How-ever, note that each of these features has a different nature,range, and significance to the matching process. Therefore,we should use a proper difference measurement for eachfeature (figure 3, processes 3, 4, and 5), normalize thesemeasurements (figure 3, process 6), and finally combinethem into a single score (figure 3, process 7).

The difference measurement of sketch and photo pointsis indicated by sum of squared error (SSE). We calculatethree different SSEs, one for normal points, one for un-

Figure 5. Comparing sketch facial marks and image edge pixels,in the same region (i.e. a same size rectangle as the boundingbox of the sketch facial mark, centered at the same distance fromthe nearest facial component).

usual points, and one for relative distances between facialcomponents. We label a sketch/photo point as unusualif its value lies in a distance more than sigma, σ, fromthe mean, µ, in its respective normal PDF (figure 3,process 3) - the normal PDFs are calculated based on ourface dataset (Multi-PIE [14]). We calculate the relativedistances between the facial components by comparingtheir distances from the centers of each of the eyes (aswe scale and rotate each sketch based on eye positions).

The difference measurement for each of the skin color,iris color, hair color, ethnicity, and gender features betweena sketch and a photo is assigned heuristically based oncommon understanding of these features. The differencemeasurement for each of these features is represented byan n×n matrix, where n is the number of possible valuesfor that feature. Each cell of this matrix indicates thedifference between a certain eyewitness assigned value anda certain face photo value in the respective feature space,in terms of integers in range [0, n].

Finally, the difference measurement of facial marks isindicated by minimum squared error (MSE) between facialmark pixels and edge pixels in the same photo region. Thesame region is the same size rectangle as the bounding boxof the facial mark, centered at the same distance from thenearest facial component (see figure 5). To avoid noise, were-size each photo to 128× 128 pixels, apply Sobel edgedetection, and re-size back the resulting edge image to itsnormalized size, and then calculate MSE of the facial markpixels and the edge pixels.

After calculating all feature differences, we normalizeeach feature difference by dividing each one by its re-spective standard deviation over the entire dataset (figure3, process 6). Note that the spline differences require yetanother normalization. Because of the differences in splinesizes, points do not cover the same amount of area fromone spline to another (the eyes are much smaller than thejaw-line). Therefore, we have to normalize the effect ofeach spline based on its axis lengths too. Equations 1 to 3

illustrate the first regularization step:

∆unus = (

16∑i=1

100∑j=1

diff(S_pointj , P_pointj)

diami)(1)

...∆mark =

∑diff(S_marki, P_edgei) (2)

∆_set(S, P ) =

{∆norm

σnorm,

∆unus

σunus, ...,

∆mark

σmark

}(3)

where ∆unus is unusual points feature difference; diff isthe respective difference measurement function; diami isthe diameter length of the ith spline; ∆mark is the facialmark feature difference; σnorm, σunus, and σmark arestandard deviations for normal points, unusual points, andfacial marks respectively. Finally, ∆_set is the normalizedset of feature differences between sketch S and photo Pafter the normalization.

We finally optimally combine all features by calculatingthe coefficient matrix A that minimizes the sum squarederror E, given a set of sketch features S = {s1, · · · , sk}and a set of photo features P = {p1, · · · , pk}:

E =k∑

i=1

‖ A(diffi(si, pi) ‖2

final_score = ‖ A∆_set(S, P ) ‖2

Where final_score is the difference measurement be-tween sketch S and photo P (figure 3, process 7).

IV.. Experimental ResultsIn this section we present the experimental results of

applying NAFSR, on 100 non-artistic sketches. We alsocompare our classification results with PCA, the methodpresented by Tang et al. [15], and SIFT classification.Almost all previous methods have used a PCA-based al-gorithm; therefore, we can consider the PCA accuracies asa generalized measurement of their recognition accuracy.The FSR introduced in [15] is reported to have 71%rank-1 (and 96% rank-10) accuracy in recognizing exactartistic sketches. The SIFT classification is reported to haveaccuracy rate of 97% (rank-1) for exact artistic sketchesand 16.33% (rank-1) for forensic sketches [16].

The photo dataset is the 249 faces of the first sessionof the Multi-PIE face dataset [14], having a mixture ofgender, age, and ethnicity. Each photo is analyzed bySTASM [27] to detect facial component outlines. For eachphoto, all Ancillary Information except age are assignedmanually. The age feature is assigned to the averageestimated age of 16 individuals (none are eyewitnesses),in terms of 5-year bins.

The sketch dataset consists of 100 non-artistic sketchesof random target faces from the photo dataset, drawn by10 eyewitnesses, while looking at the face photos, with

Figure 6. CMC curves for PCA on Main Sketches, PCA onestimated target face shape, Tang et al. [15], SIFT (used in [16]),and NAFSR.

Rank-1 % Rank-10 % Rank 50 %PCA Main 0 0 5

PCA Estimated 0 4 17Tang et al. 1 6 28

SIFT 1 9 19NAFSR 12 60 93

Table I. Recognition performance of PCA on Main Sketches,PCA on estimated target face shapes, Tang et al. [15], SIFT (usedin [16]), and NAFSR.

unlimited time to deliver (similar to all previous works).We use a simple active shape model to detect facialcomponent outlines, and normalize all photos and sketchesbased on the eye positions. We estimate the target faceshape of each sketch, by treating the rest of the sketchesfrom the same eyewitness as the drawing profile.

Figure 6 and table I compare the results of NAFSR,PCA, the FSR proposed in [15], and SIFT classification.The input to these methods is the Main Sketch. In additionwe also test the PCA on the estimated target shapes.The CMC curves show the accuracies up to rank-50(50 guesses); however, in real cases the possible numberof guesses can be around 10. Note that the traditionalrecognition accuracy is the rank-1 value in the CMC curve.

Based on the results, PCA performs poorly in recogniz-ing both the Main Sketches and estimated target shapes,with its rank-50 accuracy remains below 10% and 20%respectively. Similarly, Tang et al. and SIFT classificationshave poor recognition accuracies. In contrast, the recogni-tion accuracy of NAFSR starts with 12% in rank-1 with arapid growth as rank increases (60% in rank-10).

Figure 7 demonstrates how the rank-1 recognition ac-curacies of NAFSR, Tang et al., and SIFT decrease as thegallery size increases. This figure indicates the scalabilityof our method to be used in datasets of very large size.

It should be noted that the direct comparison betweenthese methods is not completely possible as the input andassumptions are different. All previous methods have used

Figure 7. Drop in Rank-1 accuracy of NAFSR, Tang et al., andSIFT on the Main Sketches, as the gallery size increases.

artistic sketch as their input and assumed high sketch-photosimilarity. In contrast, we use non-artistic sketch as input,directly drawn by the eyewitnesses and we have a realisticassumption for differences between these sketches and thetarget faces. The non-artistic sketches have a very largermodality difference to face photos, than the exact artisticsketches. Therefore it can be expected beforehand thatprevious methods perform poorly. However, consideringthe reliability of these non-artistic sketches (see sectionIII-A), our results show that if processed carefully, theycan be used to find the target face. And the key toa proper processing of these sketches, lies in assessingthe procedure of creating them (ETM) - which is totallyignored by previous works.

V.. Summary and ConclusionMore than 30 years of psychological studies show that

forensic sketches are highly unreliable due to problemssuch as verbal overshadowing in the very first steps ofeyewitness testimony methods (ETMs) [7], and piecewisereconstruction in the next steps [10]. However, no practicalalternative ETM is successfully proposed and police de-partments continue to use the unreliable traditional meth-ods. We showed that the previous works for recognizingforensic sketches are unreliable in handling real casesbecause of their unrealistic assumptions of forensic sketchreliability and sketch/photo similarity (section II).

Here we proposed both a more reliable ETM andan accompanying face sketch recognition method (FSR)which account for more realistic situations. We are thefirst to provide a solution for face sketch recognition bynot only assessing the face sketch, but also focusing onhow the sketch is created in the ETM. And we are the firstto account for individual eyewitness differences in mentalbiases and information delivering style.

In our ETM we asked the eyewitness to sketch the

target face by him/herself, and therefore avoided introduc-ing memory disturbances such as verbal overshadowing,piecewise reconstruction, or addition of external memories.Then we asked the eyewitness to provide some additionalinformation about the target face, such as skin tone andethnicity. We finally acquired a drawing profile of theeyewitness based on his/her sketches of a set of faces.

In our FSR, we estimated the shape of the target facesfrom the eyewitness sketch, based on the eyewitness’sdrawing profile (section III-A). Then we optimally com-bine all features such as facial component shape unusualityand relative distances into a single difference score, usingglobal least squares optimization (section III-B). We finallycompared our method with PCA, and two other proposedmethods in [15] and [16] which reported high accuracy(up to 97%) in recognizing exact artistic face sketches.Results of recognizing 100 non-artistic sketches from10 eyewitnesses show that our method has a significantimprovement over other methods, with a fast growth inaccuracy as the rank increases (section IV).

Furthermore, this combination of ETM/FSR can beeasily used in real situations as it requires no trained users,and delivers more reliable results in a relatively short time.

Nonetheless, it is clear that improvements such asbetter ways of target face shape estimation or addition ofnew sources of information can further improve the finalrecognition accuracy. For example, further assessment onperception of race, skin color, or age by individuals mayproduce more accurate estimation of mental biases andtherefore better matching rates. In addition, a more realisticsituation can be tested by using sketches drawn merelybased on eyewitness’ memory, when there is a time delaybetween viewing the face and sketching.

The main motivation for this work was to create themissing link between psychological findings, automaticface sketch recognition, and real world applications, andtherefore reduce the chance of wrongful convictions ofinnocents, such as the Kirk Bloodsworth. We hope thatthis work serves as a first step for better methods benefitingboth computer vision and forensic sciences.

References

[1] H. Arkowitz and S. O. Lilienfeld, “Why science tells us not to relyon eyewitness accounts,” Scientific American, Jan 2010. 1

[2] C. A. Morgan, M. Hazlett, G. nad Baranoski, A. Doran, S. South-wick, and E. Loftus, “Accuracy of eyewitness identification issignificantly associated with performance on a standardized testof face recognition,” International Journal of Law & Psychiatry,vol. 30, pp. 213–223, 2007. 1, 2, 3

[3] D. A. Yarmey, “Probative v. prejudicial value of eyewitnessmemory research,” Expert Evidence, vol. 5, pp. 89–97, 1997.10.1023/A:1008818205752. 1, 2, 3

[4] H. Munsterberg, On the Witness Stand Essays on Psychology andCrime (Foundations of Criminal Justice). Ams Pr Inc, 1927. 1, 2,3

[5] C. Carlson, S. Gronlund, and S. Clark, “Lineup composition, sus-pect position, and the sequential lineup advantage,” J Exp PsycholAppl, vol. 14(2), pp. 118–28, 2008. 1, 2, 3

[6] F. Bartlett, Remembering: A Study in Experimental and SocialPsychology. Cambridge University Press, 1932. 1

[7] J. Schooler and T. Engstler-Schooler, “Verbal overshadowing ofvisual memories: Some things are better left unsaid,” CognitivePsychology, vol. 22, pp. 36–71, Jan. 1990. 1, 2, 3, 7

[8] D. M. Bernstein and E. F. Loftus, “How to tell if a particularmemory is true or false,” Perspectives on Psychological Science,vol. 4, no. 4, pp. 370–374, 2009. 1

[9] B. Zhu, C. Chen, E. F. Loftus, C. Lin, Q. He, C. Chen, H. Li,G. Xue, Z. Lu, and Q. Dong, “Individual differences in falsememory from misinformation: Cognitive factors,” Memory, vol. 18,pp. 543–555, 2010. 1

[10] P. Sinha, B. Balas, Y. Ostrovsky, and R. Russell, “Face recognitionby humans: Nineteen results all computer vision researchers shouldknow about,” Proc. IEEE, vol. 94, no. 11, pp. 1948–1962, 2006. 1,7

[11] P. Sinha, B. J. Balas, Y. Ostrovsky, and R. Russell, “Face recognitionby humans,” Face Recognition: Models and Mechanisms, 2006. 2

[12] Q. Liu, X. Tang, H. Jin, H. Lu, and S. Ma, “A nonlinear approachfor face sketch synthesis and recognition,” in CVPR 2005, vol. 1,pp. 1005 – 1010 vol. 1, 20-25 2005. 2, 3

[13] M. K. Unnikrishnan, “How is the individuality of a face recog-nized?,” J. Theor. Biol., vol. 261, pp. 469–474, 2009. 2, 5

[14] R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker, “Multi-pie,” in FG ’08, pp. 1–8, Sept. 2008. 2, 6

[15] X. Tang and X. Wang, “Face sketch recognition,” T-CSVT, vol. 14,pp. 50–57, jan. 2004. 2, 3, 6, 7, 8

[16] B. Klare, Z. Li, and A. Jain, “Matching forensic sketches to mugshot photos,” Pattern Analysis and Machine Intelligence, IEEETransactions on, vol. 33, pp. 639 –646, march 2011. 2, 3, 6, 7,8

[17] Y. hui Li, M. Savvides, and V. Bhagavatula, “Illumination tolerantface recognition using a novel face from sketch synthesis approachand advanced correlation filters,” in ICASSP 2006, vol. 2, pp. II –II,14-19 2006. 3

[18] X. Wang and X. Tang, “Face photo-sketch synthesis and recogni-tion,” Pattern Analysis and Machine Intelligence, IEEE Transac-tions on, vol. 31, pp. 1955 –1967, nov. 2009. 3

[19] J. Zhong, X. Gao, and T. C., “Face sketch synthesis using e-hmm and selective ensemble,” in Acoustics, Speech and SignalProcessing, 2007. ICASSP 2007. IEEE International Conferenceon, vol. 1, pp. I–485 –I–488, april 2007. 3

[20] B. Xiao, X. Gao, D. Tao, and X. Li, “A new approach for facerecognition by sketches in photos,” Signal Process., vol. 89, no. 8,pp. 1576–1588, 2009. 3

[21] “Facevacs software developer kit, cognitec systems gmbh,http://www.cognitec-systems.de, 2010.” 3

[22] T. Fitzpatrick, “Soleil et peau,” J Med Esthet, vol. 2, pp. 330–334,1975. 4

[23] M.-M. Piquet-Thepot Bulletins et Mémoires de la Sociétéd’anthropologie de Paris, pp. 207,208, 1968. 4

[24] H. Daniel, “Analysis of hair samples of mummies from semnasouth,” AM. J. PHYS. ANTHROP, vol. 49, pp. 277–282, 1978. 4

[25] Y. Zhang, C. McCullough, J. Sullins, and C. Ross, “Hand-drawnface sketch recognition by humans and a pca-based algorithm forforensic applications,” SMC-A, vol. 40, pp. 475 –485, may 2010. 5

[26] F. N. Fritsch and R. E. Carlson, “Monotone piecewise cubicinterpolation,” SINUM, vol. 17, pp. 238–246, 1980. 5

[27] S. Milborrow and F. Nicolls, “Locating facial features with anextended active shape model,” ECCV, 2008. 6