BioFoV - An Open Platform for Forensic Video Analysis and

Biometric Data Extraction

Miguel Reis Moitinho de Almeida

May, 2015

Abstract

This paper proposes an open platform, BiometricForensic V ideo analyzer (BioFoV), for forensic videoanalysis and biometric data extraction. The plat-form’s architecture and implemented modules are de-scribed, sample results are shown, and a list of pos-sible enhancements is included. BioFoV is imple-mented with open software, can be run in multiplesoftware platforms, and is designed to be easily ex-tended. An implementation is publicly available, sup-porting camera calibration, detection of events andallowing the computation of several soft biometrics,such as bodily measurements or joint angles, takingas input surveillance video footage. Also face detec-tion is available to illustrate the ability to deal withdifferent biometric traits.

1 Introduction

Forensic experts have discussed over the last decadesthe need for moving the foundation of forensic casework from identification of perpetrators based on ex-pert knowledge to expressing evidence based uncer-tainties. Barriers to this include both limited timeand lack of knowledge to be engaged in research anddevelopment for many forensic experts.

Biometric researchers, on the other hand, havedone much research often without access to data fromcases found in real forensic scenarios, and which theygenerally need to further research. The EuropeanCooperation in Science and Technology (COST) Ac-tion 1106 ”Integrating Biometrics and Forensics for

the Digital Age” was established to bring these twocommunities together to foster cooperation and newresearch.

Techniques, which can enhance forensic casework,have already been developed such as [1], [2] and [3].However, there is still the challenge to transform thealgorithms, e.g. developed in the context of PhD the-ses, into a set of tools that can be used by forensicexperts.

Currently forensic scientists often use commer-cially available general image/video processing soft-ware packages, not specifically developed for forensiccase work. An example is the comparison of measur-able silhouettes of a perpetrator and a suspect, illus-trated in Figure 1. The creation of these silhouettesis the result of quite laborious processes, first involv-ing commercial photogrammetric software, primarilydeveloped for 3D scans of objects using photos takenwith the same high resolution camera from differentangles, to construct a 3D model wherein silhouettesas those shown in Figure 1 can be created. Anotherstep is to incorporate in the 3D model frames from alow resolution surveillance camera often subjected tosignificant distortions. This is a challenge for the pho-togrammetric software, with the internal and exter-nal calibration of the surveillance camera often beinga problematic and laborious process. Next challengeis that the software is not build to handle severalphotos from the same location in the 3D-model, astaken from a stationary surveillance camera. At last,the created silhouettes have to be superimposed andcompared, as shown in the right image of Figure 1,using another piece of software.

Hence, from a forensic expert practitioners point

1

of view, there is the need for forensic casework tailor-made software, based on state of art knowledge infields such as soft biometrics, gait recognition, pho-togrammetry, etc. Forensic experts, police officersand others working with forensic video analysis canalso benefit from more ”intelligent” video handlingtools, such as automatic event/face detection, to helpsearch the relevant video footage prior to the actualcase work.

Other software packages aimed at forensic videoand image analysis have been developed, such asForevid [4] or the tools developed by Ocean Systems[5], although more directed either for video captureand processing or enhancement of the material, notfully addressing the above mentioned challenges.

A tool for forensic gait analysis has recently beendeveloped [6] and automatic facial algorithms arevaluable tools for the forensic expert [7]. Includingsuch tools into a common and extensible softwareplatform would allow narrowing down the softwaretools a forensic video analysers needs to master, and,more importantly, would support the combination ofresults from different algorithms to enhance overallforensic analysis results.

This paper proposes such a software platform, Bio-FoV, resulting from the cooperation between a bio-metrics research group and an experienced group offorensic practitioners. In BioFoV it is possible to addnew modules or update the existing ones when newrelevant research emerges making new developmentsavailable to the community, instead of remaining ina hard-disk once a research project is completed.

2 Software Platform Proposal

The proposed BioFoV software platform objective isto provide an extensible tool, useful to forensic inves-tigators by fitting their needs and thereby minimis-ing the required workload. By automating a set oftime consuming and laborious tasks, it is expectedto enable a deeper case analysis, requiring less effortfrom the investigator, while allowing to focus the ef-fort the effort on those case aspects that require ex-pert insight, thus contributing to an overall qualityincrease.

Figure 1: Photogrammetric comparison of sil-houettes – These measurable silhouettes are madeusing a commercial software package and are com-pared using a commercial photo editing software, fol-lowing a quite laborious process. Algorithms, whichhelp automate this type of the work will be includedin an easy to use software package.

Two of the key characteristics of this proposal are:

1. The usage of open software, to avoid complexlicensing; and

2. Its extensibility, to allow the inclusion of newmodules adding functionality, as well as the pos-sibility to include alternative or improved ver-sions for existing modules.

The proposed BioFoV software platform is thus struc-tured around a set of functional modules, also includ-ing a Graphical User Interface (GUI) to be usable byforensic analysers of different backgrounds. The fol-lowing subsections describe the modular architectureproposed.

2.1 Data Classes

BioFoV is dedicated to the analysis of videos, typi-cally captured by surveillance cameras, available asforensic evidence. Therefore, the main data classesused as building blocks for the platform developmentare:

• Video – Used to read and perform operationson a video file, mainly seeking, grabbing frames,

2

and checking properties such as number offrames, frame rate, among others.

• Frame – Used to describe a single frame. Itsconstructor saves an image on disk so that a largenumber of frames can be grabbed without fillingup the computer’s memory.

• Event – Describes a set of Frames (contiguous intime). It holds the output of the event detectormodule, described later on.

• Mask – Specifies a part of a Frame.

2.2 Modules

The motivation of the BioFoV software platform de-velopment is the functionality to be offered to forensicresearchers. Therefore the platform should be mod-ular and extensible.

Each of the modules is a piece of code that imple-ments a certain functionality. An initial set of mod-ules has been implemented targeting camera calibra-tion, automatic video analysis to detect relevant mo-tion activity, and also biometric feature extraction,notably soft biometrics, such as bodily measurementsor joint angles.

2.3 User Interface

The GUI allows the non-programming user to takeadvantage of the implemented functionalities. It sup-ports the selection of input video contents and respec-tive display, as well as information about the currentuser selections and, of course, to execute the requiredmodule functionalities. A screenshot of the main win-dow is shown in Figure 2.

2.4 Project

A first release of the BioFoV platform code and anissue tracking tool are freely available online, opento anybody who wants to contribute or use it. Thereference for the repository will be included in thefinal version of the paper to keep the blind reviewimpartial.

Input and Results

Details Playback

Figure 2: User interface

3 Implementation

This section describes the tools used to create theBioFoV software platform and some of the imple-mented modules.

BioFoV is being developed in C++ using Qt [8]for the GUI, and the Open Source Computer Vision(OpenCV) [9] library for supporting computer visionand image processing functionalities.

Qt and OpenCV were chosen with interoperabil-ity in mind. Their usage enables the same softwareto be compiled for any of the most commonly usedplatforms. So far it has been successfully built andtested on GNU/Linux systems (Debian and Ubuntu)and Microsoft Windows. An OS X build should alsobe easily compiled on the target platform.

All the developed software is Free and OpenSource, therefore the final product can be shippedwithout any restrictive licenses or extra fees, allow-ing it to be freely shared amongst those wanting touse and/or improve it.

3.1 User Interface

User interface development is implemented using Qt,which provides the necessary GUI building blocks tocreate a cross platform tool. Since the objective ofBioFoV is to bring tools to users, it was imperative

3

to have an easy to use GUI.As shown in Figure 2, the GUI main window is or-

ganized into four main areas: The top menu, whereall the functionalities are listed in independent sub-menus; The input and result tabs, where inputs andoutputs are listed; The details area, where extra in-formation about what is selected or being played backcan be shown to the user; The playback area, sup-porting user interaction with the video or still image,via mouse input, and supporting video explorationactions like seeking, zooming, pausing or taking snap-shots.

The present structure of the GUI can still be im-proved, as further feedback from practitioners is gath-ered on the best layout for user interaction.

3.2 Implemented Modules

The current BioFoV version includes some genericmodules, expected to be used by most practitioners,such as camera calibration or event detection. Also acouple of more specific modules have been included,notably for the computation of soft biometrics such asbodily measurements, or facial detection, as examplesof possible feature extraction modules.

3.2.1 Camera Calibration

Since even the best video cameras distort the cap-tured images, it is necessary to calibrate the cam-era with which images are taken, to compensate fordistortions and be able to support accurate measure-ments.

By recording a video of a non symmetrical chesspattern with the same camera, the calibration param-eters can be calculated and saved to a file. Later itcan be imported for usage with the surveillance videobeing analysed, or with any other video captured bythat camera 1.

In the present implementation, this module is, inits essence, a wrapper for two OpenCV functions,findChessboardCorners and calibrateCamera, beingthe latter based on [10] and [11].

The software extracts a set of frames containingthe pattern, whose size is known. Each frame is

1As long as the internal camera parameters are not changed

converted to grayscale and passed to findChessboard-Corners, which tries to find the location of the pat-tern in the image. If it is found, the coordinates ofeach inner corner of the pattern are stacked so thatthey can later be used by calibrateCamera, to com-pute the camera calibration parameters.

3.2.2 Event Detection

When a surveillance video has to be analysed thor-oughly, a lot of man time needs to be spent to visu-alize it, even if nothing relevant is happening. Thismodule allows a faster analysis of long videos, auto-matically looking for the occurrence of events. In thepresent implementation, this module trims the partsthat have no movement, so that the user does nothave to watch the entire video, which in some casescan be several days long.

Using an adaptive background subtraction model[12], the algorithm checks each frame of the video forsignificant changes, allowing the user to set a thresh-old, above which changes are considered significantenough to be analysed.

3.2.3 Re-projected Image Plane Measure-ments

This module enables the user to measure distancesin pixels in a re-projected image plane. Prior to anykind of measurement, the camera has to be calibratedas mentioned before.

An image transformation tool has been includedin this module, allowing an image to be re-projectedto a vertical plane captured in a frame. This allows,for example, the comparison of heights or widths be-tween known static planar objects and those of itemsof interest or to estimate bodily measurements of in-dividuals in the re-projected plane.

The implemented tools allow measurement of: Lin-ear distance between two points; Vertical distancebetween two points; Horizontal distance between twopoints; Angles.

It should be noticed that bodily measurements ofindividuals computed using this tool, for instance theheight, is precise only under very strict conditions,such as: The individual must be standing straight;

4

Must be in the same vertical plane as the one usedfor re-projection; The point on the top of the headmust be well defined, as well as its vertical projectionon the floor.

These conditions are very seldom met at the sametime. However, alternative measurements may becomputed, as the height to the eyes, the length ofa limb, or the angle between leg and foot in a givenpart of the gait cycle.

To give more freedom to the user and avoid therestriction of the individual having to stand in thesame plane as the referential, a more complete spa-tial measurement module needs to be developed, asdescribed in the following.

3.2.4 Spatial Measurements

The objective of this module is to allow, from sev-eral post event photos of the same scene, eventuallycaptured from different angles, to reconstruct points,edges, planes and volumes that can be used as mea-surement reference. From these, virtual referencescan be constructed and overlayed with the video, al-lowing bodily measurements in arbitrary locationsin the scene, thus avoiding the coplanar restrictionimposed by the previously described image measure-ment module. This module is not yet available.

3.2.5 Feature Extraction - Face DetectionExample

This module enables the user to automatically detectand extract a generic feature, from a set of videos, ora set of previously detected events. New instances ofthis module can be created to accommodate the typeof analysis desired by investigator. In the presentBioFoV version a feature extraction module dedi-cated to frontal face detection has been included.

The feature extraction module is the implementa-tion of a Haar Feature-based Cascade Classifier, orig-inally proposed in [13], improved by [14], and imple-mented in OpenCV.

With this automated method, the user can eas-ily create a facial database of detected people in thevideo. The clustering of these faces by person is

something that is currently being explored by anotherMaster of Science (MSc) student.

The cascade classifiers used by the face detectionalgorithm are loaded to the program from an Extensi-ble Markup Language (XML) file, which enables themodification of the set of features to be used withoutany required change to the program. Other featureslike the full human body, or any other described bya Haar classifier can also be automatically extracted.

3.2.6 Other Miscellaneous Modules

Other modules have been implemented with usabilityin mind. Amongst them are: A simple printing op-tion, that enables quick physical access to the imagein the display; An export tool for extracted features,where they can be extracted in bulk to separate imagefiles; A tool for exporting events that were detectedautomatically by the event detection module to videofiles, either for transport purposes or to prepare ma-terials to be presented in court.

4 Preliminary Results

This section presents some of the results that canbe obtained with the current implementation of theBioFoV software platform.

The reported results were obtained using a com-puter with an Intel R© CoreTM i7-3770 CPU clockedat 3.40GHz, running Kubuntu 14.04 AMD64 andOpenCV 2.4.11 compiled with Intel R© ThreadingBuilding Blocks (TBB) support.

The input videos considered for the illustration ex-amples were captured with a Logitech c270 webcam,with a 640 by 480 pixel resolution, a frame rate of 10Frames Per Second (FPS) and the output videos werecaptured using ffmpeg and encoded with an H264codec.

4.1 Camera Calibration

The webcam in use was calibrated using BioFoV witha 1016 frames video of a slowly moving chess pat-tern with 7 × 9 inner corners (as shown in Figure3a). In order to capture the edge distortions prop-erly, the pattern was moved along the areas captured

5

(a) Originalframe

(b) Calibrated (c) Top-RightDetail

Figure 3: Camera calibration example

near the image border, resulting in the detection of369 frames, in total, containing the complete pattern.

Using this set of frames, the total sum of squareddistances between the observed feature points and theprojected object points was below 1. An illustrationof the calibration procedure is shown in Figure 3.

Since the algorithm is applied to patterns capturedfrom a video, they are sometimes blurry, which canaffect the precision of the calibration. For this reason,the pattern should be moved slowly and under goodillumination conditions while performing the calibra-tion.

4.2 Event Detection

The event detection algorithm was tested on a samplevideo, captured in a controlled environment, contain-ing around 2.4M frames, at 10 FPS, lasting 2 days, 11hours and 30 minutes. It took the BioFoV software 7hours and 11 minutes to analyse the complete video,identifying a set of events considered relevant with atotal length of 17 minutes (210 times shorter). Thisexample illustrates that a real advantage can be pro-vided to the forensic analyser, saving a considerableamount of time for analysing a video.

This video sequence was shot indoors, in a roomcontaining a large window surface area and whereartificial illumination was used for some periods,thus being subjected to significant light changes.The adaptive background subtraction algorithm wasable to handle the smooth light changes caused bythe changing natural illumination without detectingthem as events.

The same video was manually analysed and no rele-vant events were missed by the algorithm. There were

(a) RGB (b) Mask (c) Masked

Figure 4: Background subtraction example

some false positive events detected, notably causedby the lights in the corridor being turned on and offduring the night, as the room has a small window tothe corridor.

Whenever a subject enters the monitored space, orsome object is brought into the scene, and remainsstill for more than the time selected using the param-eter related to the history of the background subtrac-tor, it becomes part of the background and is ignoreduntil it moves again.

A frame of one of the detected events is shown inFigure 4, including the original frame, the computedmask, and the corresponding masked frame.

4.3 Image Measurements

For an image measurement to be accurate, the sub-ject needs to be very well aligned with a plane, stand-ing up straight, and both the top of the head andcentral feet point must be clear in the image. Thisis a very rare case, and it was concluded that it isbest to compare heights in similar poses than to tryto estimate the real height of a perpetrator with abad posture.

An example of a measurement is presented in Fig-ure 5. In this case the measurement conditions areclose to ideal, and very seldomly expected to be foundin real life footage. Because of the limitation of onlybeing able to take a measurement within a given ref-erence plane, the ability to perform spacial measure-ments with real 3D points is being considered for im-plementation.

Nevertheless, for the example of Figure 5, and us-ing a previously calibrated camera, it is possible toobtain interesting results like the ones presented inTable 1.

6

Table 1: Measurement results

Subject Board PersonHeight (meters) 0.91 1.72Measured Height (pixels) 184 341Measured Height (meters) – 1.68

(a) Calibratedframe

(b) Re-projection (c) Relativeheight

Figure 5: Height measurement with a reference

4.4 Feature Extraction

To test the feature extraction module a face detectorhas been included in BioFoV. It is based on a Haarclassifier [15] able to detect multiple face instances inthe same frame.

Examples of faces detected in the test video areshown in Figure 6.

Four different frontal face classifiers were used, allof which are included with OpenCV. As can be ob-served by the comparison in Table 2, alt tree gavemuch less false positives than any other, but the onewhich missed the least faces was the alt2. So there isa trade-off in this situation, where depending on thetime available versus the thoroughness intended, theuser can choose one classifier or the other.

It also successfully detects multiple subjects in thesame frame.

Table 2: Face detection classifiers

Classifier Subject 1 Subject 2 False Positivesdefault 76 22 169alt 77 21 48alt2 78 21 64alt tree 71 19 3

Figure 6: Feature extraction results

5 Conclusions and FutureWork

The BioFoV software platform provides a tool forforensic video analysis and biometric data extraction.It is made freely available to the community and sinceit is based of open software tools it can be continu-ously expanded with the integration of new state ofthe art algorithms and used without licensing prob-lems. It is implemented in such a way that the samesoftware code can be compiled for different platformsand offers a consistent and user friendly GUI.

It already has some implemented features, andthere are ongoing efforts to implement new tools, andimprove existing ones. There is also the commitmentto ensure the continuity of the project.

5.1 Future Development

Since the platform is extensible, some long term mod-ules have been identified as candidates for implemen-tation:

• Automatic silhouette extraction and matching.;

• Face matching;

• Camera calibration using the plumb-line con-straint and minimal Hough entropy[16];

• 3D point cloud visualization;

7

• 3D scene reconstruction;

• Implementation of a file format.

None of the implemented modules is perfect. Theobjective was not to make them perfect in the firstiteration but to lay down the foundations and leaveroom for improvement and easy integration of stateof the art algorithms.

One example of of such possible improvementsis the calibration tool which currently uses theOpenCVdefault functions. There are alternative al-gorithms (such as AprilCal [17]) which can poten-tially provide improved results that can also be im-plemented, eventually letting the user choose whichto use with each camera.

One of the main objectives for this project is thatthe development does not stop here, but instead con-tinues with the cooperation of all the labs which areinterested in making BioFoV a better and even moreuseful tool.

All the code is documented using Doxygen [18],allowing anyone with programming knowledge to beable integrate new modules and share them with allBioFoV users.

In order to allow contributions from the commu-nity, a public GitHub [19] repository has been set up,from which the builds of the program can be compiledand accessed.

The management of the project, and code reviewof any new feature submission, as well as trackingof bug reports will be ensured by the current team.This will allow tight quality control on the programreleases.

References

[1] M. Goffredo, J. Carter, and M. Nixon, “Front-view Gait Recognition,” in Biometrics: Theory,Applications and Systems, 2008. BTAS 2008.2nd IEEE Int. Conf. on, pp. 1–6, Sept 2008.

[2] D. A. Reid and M. Nixon, “Using comparativehuman descriptions for soft biometrics,” in Bio-metrics (IJCB), 2011 Int. Joint Conf. on, pp. 1–6, Oct 2011.

[3] I. Bouchrika, M. Goffredo, J. Carter, andM. Nixon, “On Using Gait in Forensic Biomet-rics,” Journal of Forensic Sciences, vol. 56, no. 4,pp. 882–889, 2011.

[4] Forevid Project, “Forevid Forensic video anal-ysis for everyone.” http://www.forevid.org.Accessed: 2014-06-23.

[5] Ocean Systems, “Ocean Systems Forensic videoand image analysis solutions.” http://www.

oceansystems.com. Accessed: 2014-06-23.

[6] H. Iwama, D. Muramatsu, Y. Makihara, andY. Yagi, “Gait-based person-verification systemfor forensics,” in Biometrics: Theory, Applica-tions and Systems (BTAS), 2012 IEEE FifthInt. Conf. on, pp. 113–120, Sept 2012.

[7] J. C. Klontz and A. K. Jain, “A Case Studyof Automated Face Recognition: The BostonMarathon Bombings Suspects.,” IEEE Com-puter, vol. 46, no. 11, pp. 91–94, 2013.

[8] Qt Project, “Qt cross-platform application andUI framework.” http://qt-project.org. Ac-cessed: 2014-06-24.

[9] itseez, “OpenCV open source computer visionand machine learning software library.” http:

//opencv.org. Accessed: 2014-06-24.

[10] Z. Zhang, “A flexible new technique for cam-era calibration,” Pattern Analysis and MachineIntelligence, IEEE Transactions on, vol. 22,pp. 1330–1334, Nov 2000.

[11] J.-Y. Bouguet, “MATLAB calibration tool.”http://www.vision.caltech.edu/bouguetj/

calib\_doc. Accessed: 2014-06-25.

[12] Z. Zivkovic, “Improved adaptive Gaussian mix-ture model for background subtraction,” in Pat-tern Recognition, 2004. ICPR 2004. Proceedingsof the 17th Int. Conf. on, vol. 2, pp. 28–31, Aug2004.

8

[13] P. Viola and M. Jones, “Rapid object detec-tion using a boosted cascade of simple fea-tures,” Computer Vision and Pattern Recogni-tion, 2001. CVPR 2001. Proceedings of the 2001IEEE Computer Society Conference on, vol. 1,pp. I–511–I–518 vol.1, 2001.

[14] R. Lienhart and J. Maydt, “An Extended Set ofHaar-like Features for Rapid Object Detection,”in Proceedings of the Int. Conf. on Image Pro-cessing, (Rochester, USA), 2002.

[15] R. Lienhart, A. Kuranov, and V. Pisarevsky,“Empirical Analysis of Detection Cascades ofBoosted Classifiers for Rapid Object Detec-tion.,” in DAGM-Symposium (B. Michaelis andG. Krell, eds.), vol. 2781 of Lecture Notes inComputer Science, pp. 297–304, Springer, 2003.

[16] E. Rosten and R. Loveland, “Camera distor-tion self-calibration using the plumb-line con-straint and minimal hough entropy,” CoRR,vol. abs/0810.4426, 2008.

[17] A. Richardson, J. Strom, and E. Olson, “April-Cal: Assisted and repeatable camera calibra-tion,” in Intelligent Robots and Systems (IROS),2013 IEEE/RSJ Int. Conf. on, pp. 1814–1821,Nov 2013.

[18] D. van Heesch, “Doxygen Generate documenta-tion from source code.” http://www.stack.nl/

~dimitri/doxygen. Accessed: 2014-06-24.

[19] “Build software better, together..” Accessed:2014-08-29.

9