Face Detection in Thermal Infrared Images:A Comparison of Algorithm- and

Machine-Learning-Based Approaches

Marcin Kopaczka1, Jan Nestler1, and Dorit Merhof1

Institute of Imaging and Computer Vision, RWTH Aachen University, Germanymarcin.kopaczka@lfb.rwth-achen.de

Abstract. In recent years, thermal infrared imaging has gained an in-creasing attention in person monitoring tasks due to its numerous ad-vantages such as illumination invariance and its ability to monitor vi-tal parameters directly. Many of these applications require facial regionmonitoring. In this context, several methods for face detection in ther-mal infrared images have been developed. Nearly all of the approachesintroduced in this context make use of specific properties of facial imagesin the thermal infrared domain, such as local temperature maxima in theeye area or the fact that human bodies usually have a higher tempera-ture radiation than the backgrounds used. On the other side, a numberof well-performing methods for face detection in the visual spectrumhas been introduced in recent years. These approaches use state-of-the-art algorithms from machine learning and feature extraction to detectfaces in photographs and videos. So far, only one of these algorithmshas been successfully applied to thermal infrared images. In our work,we therefore analyze how a larger number of these these algorithms canbe adapted to thermal infrared images and show that a wide number ofrecently introduced algorithms for face detection in the visual spectrumcan be trained to work in the thermal spectrum when an appropriatetraining database is available. Our evaluation shows that these machine-learning based approaches outperform thermal-specific solutions in termsof detection accuracy and false positive rate. In conclusion, we can showthat well-performing methods introduced for face detection in the visualspectrum can also be used for face detection in thermal infrared images,making dedicated thermal-specific solutions unnecessary.

Keywords: thermal infrared, face detection

1 Introduction

Thermal imaging, also referred to as long-wave infrared (LWIR) imaging usesthe thermal subband of the electromagnetic spectrum (7 to 14 m) to acquireillumination invariant images, allowing LWIR imaging sensors to operate evenin absolute darkness. This sensor property makes LWIR imaging especially in-teresting for applications with high variance in lighting conditions. Currently,

two main focus areas of thermal imaging resarch are illumination-invariant facerecognition for survelliance purposes [3][14][2] and the extraction of vital pa-rameters such as respiratory and heart rate (RR/HR) from facial images [10].Furthermore, emotion detection [6][23] and affective state analysis such as stressdetection and polygraphy using thermal infrared images [21][19] are emergingresearch areas. All of these applications require analyzing facial regions of inter-est (ROIs) which need to be defined first. In most of the published cases, ROIsare defined manually due to the lack of reliable face and facial landmark detec-tion algorithms for thermal infrared images. In the other cases, ROI definition isperformed automatically or semi-automatically, however always under the con-straint that certain prerequisites such as a full frontal view of the face or havingthe face restricted to a certain image area are given. Only a small number of pub-lications [22][18] use a state-of-the-art face detection algorithm, in all reviewedliterature the detection method was the Haar-based Viola-Jones face detectionalgorithm [25]. To the best of our knowledge there has been no application ofmore recent face detection methods to thermal infrared images. Furthermore,there is a lack of literature comparing face detection methods developed espe-cially for the thermal infrared spectrum with well-established approaches thathave been developed for the visual spectrum. We believe that the reason forlack of relevant publications is the fact that all current methods are based onautomated classification using machine learning. Regardless of the actual classi-fier and feature descriptor used, image classification based on machine learningrequires extensive amounts of manually annotated training data which is notavailable for the thermal infrared domain. Therefore, no quantitative compari-son of domain-specific versus machine learning-based face detection algorithmshas been performed so far. Our work aims at addressing this issue by focusingon several aspects: First, we introduce an extensively annotated thermal facedatabase that can be used to train face detection algorithms that have beenoriginally developed for the visual spectrum and have not been applied to in-frared images due to the lack of sufficient training data. We use the databaseto train a selection of face detection algorithms that have been proven to per-form well on images in the visible spectrum. Finally, we thoroughly evaluatethe performance of these algorithms and compare it to the results of referencealgorithms developed especially for face detection in thermal infrared images.

The structure of the paper is the following: Our database is described insection 2.1, followed by an overview of the algorithms that we trained with itin section 2.2. A description of the performed experiments and their results isgiven in section 2.3. The paper is concluded by a result discussion (section 4)and a conclusion (section 5).

2 Materials and Methods

In this section we first describe the image database created to train and evaluateface detection algorithms. Subsequently, all evaluated face detectors are brieflyintroduced.

2.1 Thermal Face Database

Since most algorithms for facial landmark detection require a ground truthdatabase for training, the first required step is the creation of a database withface and non-face images in the thermal infrared domain. Many current facedetection and tracking algorithms developed for the visual spectrum are trainedusing large sets of in-the-wild images such as the LFPW [1] or HELEN [15]databases which are acquired under unconstrained settings with a wide vari-ety of head poses, lighting conditions and facial expressions. Up to now thereexisted no corresponding database meeting these requirements in the LWIR do-main. Available thermal infrared datasets offer images with limited temperaturecontrast and spatial resolution and only a small set of pre-defined facial ex-pressions and head poses. To overcome this shortcoming and allow to test theapplicability of established algorithms in the thermal infrared, we have createda database of thermal videos of currently 82 persons in which each participantperformed both arbitrary and protocol-specified head motions and facial expres-sions. All our images were acquired with an InfraTec 820 HD microbolometerarray camera at its native 1024x768 pixels with a relative thermal resolution of0.03K. The videos were taken from a distance of 0.8 m to 1.0 m and all par-ticipants were placed in front of a thermally homogeneous backdrop. From thevideo sequences, 2935 single frames that cover a wide and realistic range of headpositions and facial expressions have been selected. A set of 68 facial landmarkpoints has been manually marked in each image (Fig. 1). The points selectedfor landmarking are consistent with the landmarks used in a number of currentface databases in the visual spectrum including LFPW and HELEN, allowingan efficient training of algorithms developed for the visible spectrum on ther-mal infrared data. Our fully annotated face database surpasses available thermalinfrared face databases such as [26] or the databases listed in [11] in terms ofspatial and thermal resolution as well as variety of head poses and number ofannotated landmark points. The database will be made available to the publicin the near future.

Fig. 1: A set of sample images from the thermal face database. The manuallycreated landmark annotations are shown as markers.

2.2 Face Detection Algorithms

In this work, we compare five machine-learning based algorithms that have al-ready been successfully applied to face detection in the visible spectrum to twoapproaches that have been developed especially for thermal infrared images. Itshould be noted that only one of the five classifiers (namely the Viola-Jones-Method) has been used in research literature for thermal infrared face detection.To the best of our knowledge, this work is the first contribution to evaluate theperformance of the remaining four algorithms in LWIR. The algorithms trans-ferred from the visible spectrum are:

– The Haar cascade classifier (VJ) as presented by Viola and Jones [25]. Thiswork was the first robust face detection algorithm capable of delivering real-time performance even on handheld machines with limited computing powersuch as consumer-grade digital photo cameras. VJ detects faces by system-atically analyzing subimages extracted using a multiscale sliding windowand applying differently sized Haar feature detectors in a cascading manner.In this approach, the subimage is matched with the most relevant Haar fea-tures first. If the feature response is positive then further analysis using morerefined features is performed by following the cascade. Should the feature re-sponse be negative at any point, then the subimage is rejected as containingno face. The algorithm returns a face detection only if the subimage passesall steps of the cascade. The detector itself is trained by providing it witha set of face and non-face images to which a variety of Haar descriptors isapplied. The algorithm learns the most significant features that allow differ-entiating between face and non-face images and arranges the best performingfeature combinations in a fast decision tree using a boosting algorithm suchas AdaBoost or GentleBoost. Using boosting for feature selection and effi-cient coding techniques for feature computation the algorithm is capable ofperforming face detection in photographs and videos in real-time. As statedabove, this is the only algorithm that has already been applied successfullyto thermal infrared images by different authors such as [24] and more cur-rently [4]. The implementation used in our work is based on the OpenCV[12] variant of the cascade classifier, a method that improves the originalwork by Viola and Jones by combining the Haar descriptors in the cascadeinto groups instead of applying them individually at each step.

– The Haar cascade classifier (VJ-LBP) with local binary patterns (LBPs)as feature descriptor as presented by Liao et al. in [16]. This approach issimilar to the first method, however in VJ-LBP local binary patterns areused instead of Haar features. Since they only require integral computations,LBPs can be computed more efficient than Haar features. Furthermore, usingcascade classifiers with LBPs instead of Haar features has been reported toresult in improved detection rates. As with the above method, we used theLBP cascade classifier available in OpenCV.

– Histograms of Oriented Gradients (HOG), an image descriptor and objectdetection algorithm introduced by Dalal & Triggs [7] and currently being oneof the most widely used feature descriptors. HOG features are computed by

Fig. 2: Possible Haar features for face detection

analyzing image gradients and grouping them into local histograms. To useHOG for face detection, the feature vector is computed for a training setof face and non-face images and the results are used to train a classifier -usually a support vector machine (SVM) - that learns how to distinguishthe HOG feature representation of a face from background features. In ourwork, we used the implementation available in the dlib library [13] to trainand test a HOG-based face detector.

– The Deformable Parts Model (DPM), introduced by Felzenszwalb in [8].This approach uses optimized HOG features for object description, howeverfor classification and detection the method assumes that the sought objectsare composed of different components (in our case face parts such as mouthor eyes) and that these components may be arranged differently in differentimages. Instead of assuming a fixed spatial configuration as in regular HOG,the method learns the feature representations of different object parts as wellas their spatial distribution. The resulting detector can adapt to differentspatial configurations induced by changes in head pose or camera position.

– Pixel Intensity Comparisons Organized in Decision Trees (PICO), presentedby Markus et al. in [17]. This method uses binary pixel intensity comparisonsfor object detection. Similar to VJ, the algorithm performs face detection byscanning sliding subimages and applying a decision tree boosted by Gentle-Boost. Each decision is defined by direct comparison of the intensity valuesof two pixels in the current subimage. Similar to Haar feature selection in VJ,pixel locations chosen for comparison are optimized by applying the best dis-criminating combinations first. This method drastically increases the tree’sdetection speed since feature computation can be omitted.

We compare these approaches to two algorithms developed primarily for thedetection of faces in thermal infrared images. The two algorithms were chosenas they both represent common approaches presented for thermal infrared facedetection. Generally, many algorithms designed for thermal face detection applythresholding for pre-segmentation of the image or feature localization. The aimof the thresholding is either a foreground-background-segmentation under theassumption that the person is the dominant heat source in the image, or athreshold operation with a threshold close to the image’s maximum temperatureto locate the hottest spots in the image. The idea behind hottest spot localization

is the fact that in many cases the inner corners of the eyes are the hottest regionsof the face and therefore the eyes can be easily located with thresholding. Eitherone or both of the approaches are the basic idea behind the two chosen methodsas well as behind current algorithms such as [5] [20] or [27]. The methods thatwe selected for implementation were:

– Eye Corner Detection (ED), presented by Friedrich and Yeshurun in [9]. Inthis work, the initial step is also a silhouette extraction followed by a facialfeature detection. The original paper states that temperature-based featuredetection may be performed on the areas around the eyes, with the eyesthemselves being the warmest and the eyebrows being the coldest regionsof the face. The authors of [9] have tested different approaches and haveidentified the eyebrows to be the most reliably detectable face part on theirdataset. We performed the same tests on our database and found out thatin our case and similar to the results of [20], the inner corners of the eyescan be located more reliably than eyebrows since there tend to be severalcold regions in the face. Therefore we implemented the method as describedin the paper, however with the difference that our approach used a featurebetter suited to our dataset (Fig. 3).

Fig. 3: The Eye Corner Detection algorithm. Left: Original input image. Center:image after thresholding. Right: Sum of all pixels per row of the thresholdedimage. The vertical coordinate of the eye centers is the row with the biggestsum.

– Projection Profile Analysis (PPA), an algorithm described by Reese et al. in[22]. This method performs a silhouette extraction by foreground-backgroundthresholding and computes projection histograms of the resulting binaryimage (Fig. 4) on the horizontal and vertical image axes. Subsequently, theface is localized by defining extreme points in the profile curves extractedfrom the projection histograms. These points are then localized by analyzingthe extracted profile curves and their 1st and 2nd order derivatives. Theapproach assumes that the person is the dominant object in the image andthat the image displays the whole head and the upper part of the torso.These prerequisites are given in our database.

Fig. 4: The Projection Profile Analysis (PPA) approach: The algorithm per-forms a foreground-background segmentation and subsequently defines the face’sbounding box by finding extrema in the derivatives of the profile curves. Notethat for comparison with the ground truth we crop away the top 1/3 of the resultto match the manual annotations better.

2.3 Evaluation Methodology

Depending on the algorithm, the results returned by the different methods arenot directly comparable. The five machine learning-based methods all returna face bounding box that can be compared with the manually defined groundtruth bounding box in the test images. However, the results computed by thetwo thermal infrared face detection methods need to be processed for a quanti-tative evaluation as they do not return a bounding box directly. For ED whichreturns eye corner coordinates, the returned coordinates x1, y1, x2, y2 and theirdistance d = x2−x1 were used as base for bounding box computation. The finalcoordinates for the upper left and lower right corner of the bounding box arethen defined as xul = x1 − 2d, yul = y1 − d, xlr = x2 + 2d, ylr = y1 + 4d. ForPPA, the returned bounding box is refined by cropping off the top 1/3 of thealgorithm result since the algorithm returns a bounding box for the whole headincluding face, forehead and hair.

3 Experiments and Results

For algorithm training and evaluation using cross-validation, the 2935 imagesof the database were split into 10 subsets by sequentially assigning all imagesof an individual to the subset with currently least images in it. This way, wecould achieve approximately equally sized subsets in which no subject’s imageswere split between different sets, thereby ensuring no overlap between trainingand test sets. We then performed algorithm training using leave-one-subset-outcross-validation, i.e. each image was used for testing once and for algorithmtraining in the other nine iterations.

The ground truth bounding box was defined by the bounding box of eachimage’s landmark annotations, expanded by 5% in each direction (Fig. 5a). Wehave decided to enlarge the original bounding boxes since preliminary resultshave shown that the feature descriptors used by our face detection algorithmsperform better on images that contain some background area around the face.The error metric used for evaluation was the Intersection over Union (IoU),defined for the overlapping area of ground truth G and detection bounding boxD returned by the detector as the ratio of overlapping area and united area (seealso Fig. 5a):

IoU =G ∩D

G ∪D.

Fig. 5: Left: ground truth definition by expanding the bounding box of thelandmarks by 5%. Right: A visualization of the definitions of the intersectionI = G ∩D (blue) and the union U = G ∪D (white) of two results for IoU com-putation.

Commonly, a successful detection is defined as a result with IoU > 0.5.The results for the different algorithms are shown in Fig. 6. It can be seenthat all learning-based approaches outperform the algorithm-based methods bya considerable margin.

Fig. 6: Percentage of faces that are detected with a given IoU.

Table 1: Performance overview of the tested algorithms. Positive/negative rates,precision and recall are given for an IoU of 0.5. Detection and training times aregiven for an Intel i5-2500K CPU with 3.3 GHz.

DPM ED HoG LBP PICO PPA VJ

Training Time (h) 19 - 0.5 50 2 - 73

Avg. Detection Time (s) 1,13 0,01 0,18 0,16 0,02 0,09 0,09

True Positive Rate 0,98 0,52 0,93 0,90 0,90 0,85 0,91

False Positive Rate 0,00 0,48 0,00 0,03 0,10 0,15 0,07

False Negative Rate 0,02 0,00 0,07 0,08 0,00 0,00 0,02

Precision 1,00 0,52 1,00 0,97 0,90 0,85 0,93

Recall 0,98 0,99 0,93 0,92 1,00 1,00 0,98

4 Discussion

Results show that classifier-based detection methods that are commonly usedfor face detection in the visual spectrum can be applied to thermal infraredimages if trained with a well suited database. Each of the tested algorithms hasshown better detection performance than the domain-specific solutions that wereimplemented and tested as well. At the same time, the performance indicatorsgiven in table 1 show different methods are more recommended than othersdepending on the given detection task. ED shows the weakest performance. Aninspection of the values returned by the detector reveals that the algorithm isextremly sensitive to changes in pose and facial expression. A slight tilt of thehead or opening the mouth usually resulted in failed detections. PPA showsa significantly better performance, however still behind the other algorithms.

VJ - the only classifier-based face detection algorithm that has been reportedlyused for face detection in thermal infrared images before - as well as VJ-LBP andHOG show good similar detection rates. DPM shows the best detection and falsepositive rates of all algorithms at the cost of having the longest detection time ofthe tested methods, making it the algorithm of choice for all cases except thosewith very strict time constraints. In cases where detection time is crucial, PICOis the method of choice since it is as fast as the two domain-specific algorithmswhile yielding better results and remaining robust to pose changes.

5 Conclusion and Future Work

In this paper, we performed a performance comparison between face detectionalgorithms presented for the visual spectrum and applied to thermal infraredimages on one side and approaches developed especially for thermal infrared im-ages on the other. We have shown how machine-learning based face detectorscan be trained to detect faces in thermal infrared images and that these algo-rithms outperform specialized approaches. We conclude that due to the betterperformance of trainable face detection algorithms there is generally no need todevelop specialized face detection approaches for thermal infrared images. Theonly exception are highly controlled scenarios with high performance require-ments in which the specialized approaches would performed better due to theirlower computational cost.Since we have shown that algorithms for face detection can be transferred fromthe visual to the thermal infrared spectrum with good results, we will analyzethe applicability of other algorithms that have been developed for facial imagesin the visual spectrum to thermal images. Furthermore we plan to test the algo-rithms on images taken in less constrained environments to test the robustnessof the different approaches in more realistic settings.

References

1. Belhumeur, P.N., Jacobs, D.W., Kriegman, D.J., Kumar, N.: Localizing parts offaces using a consensus of exemplars. In: IEEE Transactions on Pattern Analysisand Machine Intelligence (PAMI). vol. 35, pp. 2930–2940 (December 2013)

2. Bhowmik, M.K., Saha, K., Majumder, S., Majumder, G., Saha, A., Sarma, A.N.,Bhattacharjee, D., Basu, D.K., Nasipuri, M.: Thermal infrared face recognitionabiometric identification technique for robust security system. Reviews, refinementsand new ideas in face recognition pp. 113–138 (2011)

3. Buddharaju, P., Pavlidis, I.T., Tsiamyrtzis, P., Bazakos, M.: Physiology-based facerecognition in the thermal infrared spectrum. IEEE Transactions on Pattern Anal-ysis and Machine Intelligence 29(4), 613–626 (April 2007)

4. Chakraborty, M., Raman, S.K., Mukhopadhyay, S., Patsa, S., Anjum, N.,Ray, J.G.: High precision automated face localization in thermal images: oralcancer dataset as test case. Proc. SPIE 10133, 1013326–1013326–7 (2017),http://dx.doi.org/10.1117/12.2254236

5. Chakrabortya, M., Ramanb, S., Mukhopadhyaya, S., Patsac, S., Anjumc, N., Rayc,J.: High precision automated face localization in thermal images: Oral cancerdataset as test case. In: SPIE Medical Imaging. pp. 1013326–1013326. Interna-tional Society for Optics and Photonics (2017)

6. Cruz-Albarran, I.A., Benitez-Rangel, J.P., Osornio-Rios, R.A., Morales-Hernandez,L.A.: Human emotions detection based on a smart thermal system of thermo-graphic images. Infrared Physics and Technology 81, 250–261 (2017)

7. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In:Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE ComputerSociety Conference on. vol. 1, pp. 886–893. IEEE (2005)

8. Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D.: Object detectionwith discriminatively trained part-based models. IEEE transactions on patternanalysis and machine intelligence 32(9), 1627–1645 (2010)

9. Friedrich, G., Yeshurun, Y.: Seeing people in the dark: face recognition in infraredimages. In: International Workshop on Biologically Motivated Computer Vision.pp. 348–359. Springer (2002)

10. Gade, R., Moeslund, T.B.: Thermal cameras and applications: a survey. Machinevision and applications 25(1), 245–262 (2014)

11. Ghiass, R.S., Arandjelovi?, O., Bendada, A.H., Maldague, X.: Infrared face recog-nition: A comprehensive review of methodologies and databases. Pattern Recogni-tion, vol. 47, no. 9 pp. 2807–2824 (2014)

12. Itseez: Open source computer vision library. https://github.com/itseez/opencv(2015)

13. King, D.E.: Dlib-ml: A machine learning toolkit. Journal of Machine LearningResearch 10, 1755–1758 (2009)

14. Kong, S.G., Heo, J., Abidi, B.R., Paik, J., Abidi, M.A.: Recentadvances in visual and infrared face recognitiona review. Com-puter Vision and Image Understanding 97(1), 103 – 135 (2005),http://www.sciencedirect.com/science/article/pii/S1077314204000451

15. Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T.S.: Interactive facial featurelocalization. In: Computer Vision–ECCV 2012, pp. 679–692. Springer (2012)

16. Liao, S., Zhu, X., Lei, Z., Zhang, L., Li, S.: Learning multi-scale block local binarypatterns for face recognition. Advances in biometrics pp. 828–837 (2007)

17. Markus, N., Frljak, M., Pandzic, I.S., Ahlberg, J., Forchheimer, R.: Object detec-tion with pixel intensity comparisons organized in decision trees. arXiv preprintarXiv:1305.4537 (2013)

18. Mostafa, E., Hammoud, R., Ali, A., Farag, A.: Face recognition in low resolutionthermal images. Computer Vision and Image Understanding 117(12), 1689–1694(2013)

19. Park, K.K., Suk, H.W., Hwang, H., Lee, J.H.: A functional analysis of deceptiondetection of a mock crime using infrared thermal imaging and the concealed infor-mation test. Frontiers in Human Neuroscience 7 (2013)

20. Paul, M., Blanik, N., Blazek, V., Leonhardt, S.: An efficient method for facial com-ponent detection in thermal images. In: The International Conference on QualityControl by Artificial Vision 2015. pp. 95340P–95340P. International Society forOptics and Photonics (2015)

21. Pavlidis, I., Tsiamyrtzis, P., Shastri, D., Wesley, A., Zhou, Y., Lindner, P., Bud-dharaju, P., Joseph, R., Mandapati, A., Dunkin, B., et al.: Fast by nature-howstress patterns define human experience and performance in dexterous tasks. Sci-entific Reports 2 (2012)

22. Reese, K., Zheng, Y., Elmaghraby, A.: A comparison of face detection algorithmsin visible and thermal spectrums. In: Intl Conf. on Advances in Computer Scienceand Application (2012)

23. Salazar-Lopez, E., Domınguez, E., Ramos, V.J., de la Fuente, J., Meins, A., Iborra,O., Galvez, G., Rodrıguez-Artacho, M., Gomez-Milan, E.: The mental and subjec-tive skin: Emotion, empathy, feelings and thermography. Consciousness and cog-nition 34, 149–162 (2015)

24. Sumriddetchkajorn, S., Somboonkaew, A.: Face detection in thermal imagery usingan open source computer vision library. Proc. SPIE 7299, 729906–729906–6 (2009),http://dx.doi.org/10.1117/12.819996

25. Viola, P., Jones, M.J.: Robust real-time face detection. International journal ofcomputer vision 57(2), 137–154 (2004)

26. Wang, S., Liu, Z., Lv, S., Lv, Y., Wu, G., Peng, P., Chen, F., Wang, X.: A nat-ural visible and infrared facial expression database for expression recognition andemotion inference. Multimedia, IEEE Transactions on 12(7), 682–691 (Nov 2010)

27. Wong, W.K., Hui, J.H., Desa, J.B.M., Ishak, N.I.N.B., Sulaiman, A.B., Nor,Y.B.M.: Face detection in thermal imaging using head curve geometry. In: Imageand Signal Processing (CISP), 2012 5th International Congress on. pp. 881–884.IEEE (2012)