Visualization Methods for Computer Vision Analysis

Mauricio Hess-Flores, Shawn Recker and Kenneth I. JoyInstitute for Data Analysis and Visualization

University of California, Davis, USAmhessf@ucdavis.edu, strecker@ucdavis.edu, kenneth.i.joy@gmail.com

Mark A. DuchaineauGoogle, Inc.

Mountain View, California, USAduchaine@cognigraph.com

Abstract—We present the general idea of using common toolsfrom the field of scientific visualization to aid in the design, imple-mentation and testing of computer vision algorithms, as a com-plementary and educational component to purely mathematics-based algorithms and results. The interaction between these twobroad disciplines has been basically non-existent in the literature,and through initial work we have been able to show the benefitsof merging visualization techniques into vision for analyzingpatterns in computed parameters. Specific examples and initialresults are discussed, such as scalar field-based renderings forscene reconstruction uncertainty and sensitivity analysis as well asfeature tracking summaries, followed by a discussion on proposedfuture work.

Keywords—visualization; computer vision; uncertainty analysis;scalar fields; multi-view reconstruction;

I. INTRODUCTION

The field of computer vision has seen great advances inrecent years, in fields such as object detection and tracking, andthe multi-view reconstruction of scenes. Algorithms such asthe Scale-Invariant Feature Transform (SIFT) [1] have allowedfor very accurate feature detection and tracking, the maincomponent behind such algorithms. For an excellent overviewof many classical vision algorithms, the reader is referred toHartley and Zisserman [2]. One drawback of current computervision methods is that many are based on mathematical op-timization of initial parameter estimates to achieve accurateresults. Though such optimization is provably necessary, suchas in the case of the well known bundle adjustment [3] instructure-from-motion, little interest has been given as to howindividual values and patterns in parameter space affect thetotal cost.

The main objective of our work is to introduce visualiza-tion techniques to the vision community as a very powerfuleducational and algorithm design/test tool, allowing for uniquevisually-aided numerical exploration of the solution space in anumber of applications. Visualization as a field aims to providerenderings of volumes and surfaces, including those that aretime-dependent, to graphically illustrate scientific data for itsunderstanding. Some concrete applications and results will bediscussed in Section II, such as the use of scalar fields inunderstanding scene reconstruction uncertainty and parametersensitivity, as well as feature tracking summaries. Conclusionsand future work will be discussed in Section III.

II. INITIAL APPLICATIONS

A. Visualization of Scene Structure Uncertainty

One specific application of visualization in computer visioninvolves a novel tool which we have designed, which allows

for the analysis of scene structure uncertainty and its sensitivityto different multi-view scene reconstruction parameters [4].Multi-view scene reconstruction is an important sub-field ofcomputer vision, aiming to extract a 3D point cloud repre-senting a scene. Detailed analysis and comparisons betweenmethods are available in the literature [5]. Our tool createsa scalar field volume rendering, which provides insight intostructural uncertainty for a given 3D point cloud position, dis-playing the error pattern for sampled bounding box-enclosedneighbors. A screenshot of the tool is shown in Figure 1. Thecombined statistical, visual, and isosurface information of theright-hand panel provides user insight into the uncertainty ofthe computed structure shown on the left. Uncertainty arisesfrom error accumulation in the stages leading up to structurecomputation, such as frame decimation, feature tracking, andself-calibration, where larger regions of low uncertainty indi-cate robustness of the computed structure. Sensitivity is definedas the change in scalar field values as a specific reconstructionparameter’s value changes. The scalar field is created at a user-specified size and resolution, from angular error measurements.For a reconstruction from images in Figure 2(a), a selectedposition in red enclosed by a green bounded region is shownin Figure 2(b) and magnified in Figure 2(c), with camera po-sitions rendered in blue. The scalar field corresponding to thebounded region, shown in Figure 2(d), depicts lower positionaluncertainties in red and higher ones in yellow and green. Thered user-defined isosurface, which contains the ground-truthstructure position, depicts regions of lowest uncertainty. Thevisible column-like shape indicates that lower uncertainty isseen in the directions along the lines of sight of the cameras.User interaction allows for uncertainty and sensitivity analysisin ways that have traditionally been achieved mathematically,without any visual aid.

B. Tracking Summaries

Another application is the creation of feature trackingsummaries. A feature track is a set of pixel positions repre-senting a scene point tracked over a set of images. A summarycan be created by stacking frames vertically and observingthe ‘path’ taken by a track. Given a 3D position computedfrom multi-view stereo, its reprojection error with respect toits corresponding feature track is the only valid metric toassess error, in the absence of ground truth. However, the totalreprojection error value in itself does not allow the researcherto attribute a cause to the error, for example an individualbad match. We present a novel visualization technique forsequential scene reconstruction, which encodes all individualreprojection errors in one summary rendering. This providesinsight into track degeneration patterns over time, as well as

Fig. 1. Our user-interactive tool for uncertainty and sensitivity analysis inmulti-view scene reconstruction, based on a scalar field analysis adopted fromthe visualization literature.

(a)

(b) (c) (d)

Fig. 2. Our uncertainty visualization framework. From input aerial imagesof a city (a), an initial point cloud reconstruction is computed in (b), withestimated camera positions as blue dots. A position on the point cloud ischosen, highlighted in red in (c) and surrounded by a bounding box for scalar-field analysis. The resulting field is shown in (d).

(a) (b)

Fig. 3. Feature track summary for the Dinosaur dataset [6]. Each individualtrack represents the evolution over time, from top to bottom, of a givenfeature track’s pixel position at each image of a sequential image stream.In each track’s rendering, low individual reprojection errors are depicted inblue, with higher ones in green (a). Overall values are much smaller afterbundle adjustment optimization [3](b).

information about highly inaccurate individual track positions,all of which adversely affect subsequent camera pose estima-tion and structure computation. These summaries allow theuser to infer the cause of a bad track, which pure total errorcannot provide, allowing researchers to analyze errors in wayspreviously unachieved. Figure 3 shows an example.

III. CONCLUSION AND FUTURE WORK

This paper presented the general idea of using visualizationas a strong tool for computer vision research, which hadnot been used until now. Concrete results were shown fortwo applications, multi-view scene reconstruction and featuretracking, where the visualization of patterns in parameterestimates helps shed light into performance analysis for the un-derlying estimation algorithms. Given these initial results andthe wealth of information provided by summary visualizations,we believe the future is bright with regards to incorporatingvisualization techniques into many other computer vision ap-plications, which is the focus of our ongoing work.

We have identified a number of computer vision prob-lems where applying visualization techniques could be verybeneficial, besides scene strucure uncertainty [4] and featuretrack summaries. One such application is in ‘visualizing’ fieldsof covariance matrices, allowing for the visual analysis andcomparison of multiple covariance descriptors at once. Suchmatrices are fundamental in computer vision, and yet thenumerical data they provide is not easy to summarize andinterpret. By visualizing for example the change over time ofparameters related to such matrices, insight can be achievedinto understanding specific patterns and trends, which may leadto a better understanding of operators such as image intensity-based descriptors. Another application is in creating scalarfield or similar summaries for multi-view bundle adjustmentoptimization and other optimization cost functions. Yet otherapplications we are targeting are in video processing, such assummaries of object tracking results for performance analysis,as well as video content summaries.

REFERENCES

[1] D. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,”International Journal On Computer Vision, vol. 60, no. 2, pp. 91–110,2004.

[2] R. I. Hartley and A. Zisserman, Multiple View Geometry in ComputerVision, 2nd ed. Cambridge University Press, 2004.

[3] M. Lourakis and A. Argyros, “The design and implementation ofa generic sparse bundle adjustment software package based on theLevenberg-Marquardt algorithm,” Institute of Computer Science -FORTH, Heraklion, Crete, Greece, Tech. Rep. 340, August 2000.

[4] S. Recker, M. Hess-Flores, M. A. Duchaineau, and K. I. Joy, “Visual-ization of Scene Structure Uncertainty in a Multi-View ReconstructionPipeline,” in Proceedings of the Vision, Modeling, and VisualizationWorkshop, 2012, pp. 183–190.

[5] S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski, “AComparison and Evaluation of Multi-View Stereo Reconstruction Algo-rithms,” in CVPR ’06: Proceedings of the 2006 IEEE Computer SocietyConference on Computer Vision and Pattern Recognition. Washington,DC, USA: IEEE Computer Society, 2006, pp. 519–528.

[6] Visual Geometry Group, University of Oxford, “Multi-view and OxfordColleges building reconstruction,” http://www.robots.ox.ac.uk/∼vgg/data/data-mview.html.