research article markerless augmented reality via stereo...

Research ArticleMarkerless Augmented Reality via Stereo Video See-ThroughHead-Mounted Display Device

Chung-Hung Hsieh and Jiann-Der Lee

Department of Electrical Engineering Chang Gung University Taoyuan 333 Taiwan

Correspondence should be addressed to Jiann-Der Lee jdleemailcguedutw

Received 14 June 2015 Revised 10 September 2015 Accepted 27 September 2015

Academic Editor Zhiqiang Ma

Copyright copy 2015 C-H Hsieh and J-D Lee This is an open access article distributed under the Creative Commons AttributionLicense which permits unrestricted use distribution and reproduction in any medium provided the original work is properlycited

Conventionally the camera localization for augmented reality (AR) relies on detecting a known pattern within the captured imagesIn this study a markerless AR scheme has been designed based on a Stereo Video See-Through Head-Mounted Display (HMD)deviceTheproposedmarkerlessAR scheme can be utilized formedical applications such as training telementoring or preoperativeexplanation Firstly a virtual model for AR visualization is aligned to the target in physical space by an improved Iterative ClosestPoint (ICP) based surface registration algorithm with the target surface structure reconstructed by a stereo camera pair then amarkerless AR camera localization method is designed based on the Kanade-Lucas-Tomasi (KLT) feature tracking algorithm andthe Random Sample Consensus (RANSAC) correction algorithm Our AR camera localization method is shown to be better thanthe traditional marker-based and sensor-based AR environment The demonstration system was evaluated with a plastic dummyhead and the display result is satisfactory for a multiple-view observation

1 Introduction

In the past few decades augmented reality (AR) has becomean attractive research topic with potentials in the fieldsof machine vision and visualization techniques As a partof mixed reality and an alternative to virtual reality (VR)AR allows users to see virtual objects overlay the physicalenvironment at the same time According to the definition byAzuma [1 2] AR system has three important properties firstit combines real and virtual objects in a real environmentsecond it runs interactively and in real-time and third itregisters (aligns) real and virtual objects with each other ARcreates a ldquonext generation reality-based interface of human-computer interactionrdquo [3] provides information beyondwhatthe user can normally see and augments our real worldexperiences [4 5]

Over the years the applications of AR visualizationhad been expanded to include more and more areas suchas engineering industrial manufacturing navigation enter-tainment and especially medical applications The aim ofmedical AR-based applications starts from a simple conceptto see through the virtual object and see the patientrsquos medical

information along with the patient AR brings benefits tomedical applications since it can help visualize medicalinformation along with the patient at the same time andwithin the same physical space Inmedical fields AR creates away for advancedmedical display [6] which can be applied totelementoring [7] medical procedure training [8] ormedicaldata visualization [9] Recently applying augmented reality(AR) technology to the image-guided navigation system(IGNS) has become a new trend in medical technologiesA medical AR system merges medical images or anatomicalgraphic models into a scene of the real world [6 10 11]From the literatures we find that a successful medical ARsystem has to deal with two major problems first how toaccurately align preoperative medical information with thephysical anatomy and second how to effectively and clearlyprovide virtual anatomical information

Conventional medical systems and applications displaymedical information such as image slices or anatomicalstructures using the virtual reality (VR) coordinate system ona regular screenTherefore surgeons using such systems needto have a good spatial concept in order to interpret the virtual-to-real world transformation In order to train a surgeon

Hindawi Publishing CorporationMathematical Problems in EngineeringVolume 2015 Article ID 329415 13 pageshttpdxdoiorg1011552015329415

2 Mathematical Problems in Engineering

to have a god spatial concept this usually requires a lot ofclinical experience In contrast AR provides an attractivealternative for medical information visualization because thedisplays use the real world coordinate system In generala video see-through AR system uses a movable camera tocapture images from the physical space and draws virtualobjects onto the correct positions on the images As a resultthe way to estimate the spatial relationship between thecamera and the physical space that is camera localizationis the most important problem to be solved in an ARsystem [12 13] The spatial relationship is also known asthe extrinsic parameters of the AR camera which are threetranslation parameters and three rotation parameters andcan be referred to as the position and orientation of thecamera A precise estimation of the extrinsic parametersensures that the medical information can be accuratelyrendered on the scene captured from the physical space Aconventional approach to estimate these extrinsic parametersis to place a black-white rectangle pattern within the scenewhich can be used as a reference to be detected in the camerafield of view (FOV) [14 15] using computer vision techniquesOnce the reference pattern is detected in the AR cameraframe the extrinsic parameters can be determined usingthe perspective projection cameramodel Alternatively somestudies attach a couple of retroreflectivemarkers for exampleinfrared reflectivemarkers on the AR camera and track thesemarkers by using an optical tracking sensor [13 16 17] inorder to estimate the position and orientation of the cameraby using the positions of these markers However in thepattern-based or sensor-basedARdisplay the FOVof camerais limited because these artificial markers should be observedwith the FOV In addition the accuracy of AR renderingdepends on the size and likelihood of correct identificationof the pattern or markers within the FOV

In this paper we present a markerless AR scheme formedical applications The markerless AR scheme is mainlybased on a Video See-Through Head-Mounted Display(HMD) device with a stereo camera on it We use thestereo camera to reconstruct feature point cloud of the targetfor AR visualization An improved ICP algorithm is thenapplied to align the surface data to the virtual object createdfrom medical information The improved ICP algorithm hastwo additional characteristics First a random perturbationtechnique is applied to provide the algorithm opportunitiesto escape the local minima Second a weighting strategy isadded to the cost function of the ICP in order to weaken theinfluence of outlier points

When the AR camera moves feature points are thentracked by using the Kanade-Lucaz-Tomasi tracking algo-rithm [18] on a frame by frame basis while updating theextrinsic parameters Moreover the Random Sample Con-sensus (RANSAC) [19] algorithm is applied to keep the KLTtracking results stable in each frame in order to make theAR visualization smoother and more accurate Furthermoreconsidering the target for AR rendering might be out of theFOV of the AR camera a reinitialization step is necessaryand can be accomplished by applying the Speeded-UpRobustFeatures (SURF) [20] feature matching algorithm for targetdetection

The remainder of this paper is organized as followsin Section 2 we report on some related works about ARvisualization Section 3 includes all material and methods ofthe proposed system Experimental results are presented anddiscussed in Section 4 Finally Section 5 gives the conclusionof this work

2 Related Works

Camera localization is a key component in dense 3D map-ping Simultaneous Localization and Mapping (SLAM) andfull-range 3D reconstruction It is also the keypoint problemof video-based AR visualization since we need to know thepose and position of the camera in order to render the virtualobject onto the camera scene There are different types ofmethods for camera localization and the most well-knowncategory of camera localization methods belongs to theplanar-based methods In the past 10 years the planar-basedmethods have become one of themajor types in researches forAR camera localization methods one of the examples is thewidely used AR environment development library ARToolkit[21] In this type of method a predesigned rectangle markerwould be placed inside the camerarsquos FOV and a trackingalgorithm is then applied to the captured frames in order totrack the marker [22 23]

Another category is the landmark-based approaches [24]In this type of method several landmark feature points areextracted from the camera image by using feature extractionalgorithms Descriptors of these landmark feature points arealso extracted for tracking and comparison with the 3Dlocations of these landmark feature points In each framethese landmark feature points are picked by tracking orcomparing their corresponding feature descriptorsrsquo similar-ities and the pose of the camera can then be determined[25 26] Besides Kutter et al [27] proposed a marker-based approach to render the patientrsquos volume data on aHMD device Their scheme provides an efficient volumerendering in an AR workspace and solves the problem ofocclusion by the physicianrsquos hands LaterWieczorek et al [28]extended this scheme by improving the occlusion due to themedical instrument and added new functions such as virtualmirror to the AR system Suenaga et al [29] also proposed afiducial marker-based approach for on-patient visualizationof maxillofacial regions by using an optical tracking systemto track the patient and using a mobile device to visualize theinternal structures of the patientrsquos body Nicolau et al [30]used 15 markers to perform the registration in order to useAR to overlay a virtual liver over the patient Debarba et al[31] proposed a method to visualize anatomic liver resectionsin anAR environmentwith the use of a fiducialmarker whichmade locating the position and tracking of the medical datain the scene possible

Recently there are somemarkerless AR systems proposedin the literatures using various camera configurations Forexample Maier-Hein et al [32] implemented a markerlessmobile AR system by using a time-of-flight camera mountedon a portable device that is a tablet PC to see the patientrsquosanatomical information The registration method of thisapproach is an anisotropic variant of the ICP algorithm and

Mathematical Problems in Engineering 3

the speed performance is about 10 FPS By using a RGB-D senor such as the Microsoft Kinect Blum et al [33]utilize a display device to augment the volume informationof a CT dataset onto the user for anatomy education Theyemployed a skeleton tracking algorithm to estimate the poseof the user and scaled the generic CT volume accordingto the size of the user before performing the AR overlaySince the depth data provided by Kinect are not sufficientlyaccurate for pose estimation Meng et al [34] proposed animproved method by using landmarks to increase the systemperformance Based on a similar concept Macedo et al[35] developed an AR system to provide on-patient medicaldata visualization using Microsoft Kinect They used theKinectFusion algorithmprovided byMicrosoft to reconstructthe patientrsquos head data and a variant of the ICP algorithmwas used in conjunction with a face tracking strategy foruse during the medical AR An extension to this approachhas been proposed recently in [36] in which a multiframenonrigid registration scheme was presented to solve theproblem of displacement of natural markers on the patientrsquosface In order to speed up the huge computations requiredfor a real-time nonrigid registration a lot of GPUs are neededand the implementation of this systemwould be very complexcompared with the previous methods Moreover because thepatient undergoing an operation would be under anesthesiathe appearance of his face would be unchanged so there is noneed to perform nonrigid registration to align the CT data topatientrsquos face In summary although the Microsoft Kinect ispopular for AR applications it is inconvenient for use in theclinical environment because its volume is too big to mounton the physicianrsquos head

In light of these previous works we propose a simple butefficient framework for markerless augmented reality visual-ization which is based on the Stereo Video See-Through ARThemarkerless AR procedure mainly follows the principle oflandmark-based camera localization approach An improvedICP algorithm is applied for alignment of the virtual objectand the feature point cloud in the physical space Thisframework improves the previous studies in markerless ARvisualization by using a light-weight stereo HMD instead of aheavier device such as theMicrosoftKinect and it can achieveamore accurate registration result by using an improved formof the ICPThis system can be applied tomedical applicationssuch as training telementoring or preoperative explanation

3 Materials and Methods

The entire workflow of the proposed stereo AR scheme isshown in Figure 1 Feature point cloud of the target objectis extracted and reconstructed by the stereo camera Theimproved ICP-based surface registration algorithm had beendesigned and applied in order to align the virtual model tothe feature point cloud of the target in the physical space

31 3D Feature Point Cloud Reconstruction Firstly a stereocamera setup is used to acquire 3D information of the targetin the world coordinate system (WCS) The SURF algorithmis then utilized to extract feature points from the target object

region in left camera image The corresponding points inright camera image are then obtained by comparing the SURFfeature points in the right imagewith the aid of stereo epipolarconstrain The object region is recorded for later use in thetracking recovery stage

According to pin-hole camera model the perspectiveprojection between a 3D object point (119883119884 119885) and itsprojection point (119909 119910) can be described as

[[

[

119909

119910

119908

]]

]

=[[

[

1198911199090 1199090

0 1198911199101199100

0 0 1

]]

]

[R | t][[[[[

[

119883

119884

119885

1

]]]]]

]

= 119870119864

[[[[[

[

119883

119884

119885

1

]]]]]

]

(1)

where 119870 denotes the camerarsquos intrinsic parameters Theparameter 119891

119909denotes the focal length divided by pixel size

119904119909in 119909 direction and 119891

119910denotes the focal length divided by

pixel size 119904119910in 119910 direction The point (119909

0 1199100) is the principle

point of the projected 2D image plane 119864 denotes the camerarsquosextrinsic parameters which contains a 3 times 3 rotation matrixR and a translation vector t between theWCS and the cameracoordinate system (CCS)We denote the projectionmatrix as

119875 = 119870119864 =[[

[

11987511119875121198751311987514

11987521119875221198752311987524

11987531119875321198753311987534

]]

]

(2)

So if a stereo camera pair is well-calibrated we can obtainintrinsic parameters and extrinsic parameters of the bothcamera For a feature point 119901

119871(119909119871 119910119871) in left camera image

we can calculate a projection line in the WCS from theperspective projection equation and the projectionmatrix P

119871

of the left camera

(11987511minus 11987531119883119871)119883 + (119875

12minus 11987532119883119871) 119884 + (119875

13minus 11987533119883119871) 119885

= 11987534119883119871minus 11987514

(11987521minus 11987531119883119871)119883 + (119875

22minus 11987532119883119871) 119884 + (119875

23minus 11987533119883119871) 119885

= 11987534119883119871minus 11987524

(3)

Also assuming the corresponding point of119901119871in right camera

image is 119901119877(119909119877 119910119877) we can then calculate another projection

ray equation by the projection matrix P119877of the right camera

(11987511minus 11987531119883119877)119883 + (119875

12minus 11987532119883119877) 119884 + (119875

13minus 11987533119883119877) 119885

= 11987534119883119877minus 11987514

(11987521minus 11987531119883119877)119883 + (119875

22minus 11987532119883119877) 119884 + (119875

23minus 11987533119883119877) 119885

= 11987534119883119877minus 11987524

(4)

The middle point of the common perpendicular between thetwo projection lines in the WCS is then selected as the 3Dobject point of the corresponding pair (119901

119871119901119877) By calculating

the 3D object point of each feature-corresponding pair wecan obtain a 3D feature point cloud of the target object


Stereo image pair

Virtual modelFeature extraction

Stereo correspondence

Stereo reconstruction

Spatial alignment

Markerless AR visualization

Surface extraction

Figure 1 Workflow of the proposed AR system

32 Improved-ICP Algorithm for Virtual Model AlignmentIn this study we designed an improved ICP-based surfaceregistration algorithm for spatial alignment of the virtualobject and the feature point cloud The improved ICP algo-rithm is based on the original ICP registration algorithmwhich is the most widely used method to solve the 3Dsurface registration problem However ICP suffers from twoimportant weaknesses in dealing with local minimum andoutliers Therefore we added two strategies to improve theICP algorithm in order to overcome these drawbacks Aweighting function is added to decrease the influence ofoutliers and a randomperturbation scheme is utilized to helpICP escape from the local minimum

321 Distance-Based Weighting Function Assume that the3D feature point cloud of the target 119875target is the floating data119860 with 119898 points 119886

1 1198862 119886

119898 and the surface point cloud

of the virtual object is the reference data 119861 with 119899 points1198871 1198872 119887

119899 The original ICP used rigid transformation to

align these two 3D data point sets in an iterative manner Ineach iteration of ICP every point 119886

119894in 119860 first finds its closest

point 119887119895in 119861 and a cost function 119865 is then evaluated based on

the distance between each corresponding pair (119886119894 119887119895) In our

improved ICP algorithm we modified the cost function 119865 ofthe ICP by adding a weighting function to the distances of allclosest corresponding pairs (119886

119894 119887119895) in order to deal with the

problem of outliers as shown in

119865 =1

119898

119898

sum

119894=1

119908119894

min119895isin12119899

10038171003817100381710038171003817119886119894minus 119887119895

10038171003817100381710038171003817 (5)

where 119908119894is a distance-based weighting function determined

according to the median of the distances of all the corre-sponding pairs as defined by

119908119894=

1 if 10038171003817100381710038171003817a119894 minus b11989510038171003817100381710038171003817lt median

median10038171003817100381710038171003817a119894minus b119895

10038171003817100381710038171003817

otherwise(6)

322 Random Perturbation Scheme The way that ICPreaches a local minimum is by using a gradient descentapproach In each iteration of the ICP the cost functionis evaluated at the current solution and then moves alongthe direction of gradient to the local minimum When theregistration reaches convergence we can get a transformationsolution 119879 which projects a set of points onto another setof points where the total sum of the distances betweenthese two point sets is the smallest Although the actualsolution space of the cost function 119865 is multidimensionalsince transformation 119879 comprises three rotation operations(119877119909 119877119910 119877119885) and three translation operations (119879

119909 119879119910 119879119911)

for the sake of convenience we will explain the concept ofperturbation strategy using a one-dimensional solution spaceexample as illustrated in Figure 2 Suppose the initial positionin solution space before using ICP is 119879Init and the convergedsolution is 119879temp thus the ICP registration would reach 119879tempfrom119879Init by exploring the range of |119879Initminus119879temp| Let 119903 denoterotation element 119877 from the initial solution to the convergedsolution that is 119903 = |119879Init minus 119879temp| The rotation in eachdirection for perturbation is determined by using a parabolic


F

T

r

120572r

TInit

Ttemp

Figure 2 Solution space of ICP cost function 119865

probability density function as denoted in (7) Larger valueshave relative higher probabilities which provide a greaterchance to escape from the local minimum

119901 (119910) =

31199102

212057231199033if minus 120572119903 lt 119910 lt 120572119903

0 otherwise(7)

323 Improved-ICP Algorithm Figure 3 shows the flowchartof the improved-ICP registration scheme The detailed stepsof the improved ICP are described as follows

Step 1 Perform the standard ICP registration to align floatingdata119860 to reference data 119861 The initial position in the solutionspace is denoted as119879Init and the converged solution is denotedas 119879temp The final value of the cost function 119865 is recorded asthe current cost 1198621015840

Step 2 Check whether 1198621015840 is better than the current best costof ICP 119862best or not If it is accept the transformation 119879temp asthe temporal best solution 119879best update the best cost 119862 by 1198621015840and go on to Step 3 Otherwise move to Step 4

Step 3 Perturb the aligned data 119860 with a transform 119879perturbA transformation 119879perturb is selected according to (7) Thetransformation 119879perturb is then applied to the floating data 119860and the algorithm moves back to Step 1 for performing ICPregistration

Step 4 Check whether the result meets the stopping criteriaor not If the best cost 119862 is below a threshold 119862thres or thecount of repetition reaches a certain value 119896 the algorithmstops and outputs the final transformation 119879best Otherwisego on to Step 5

Step 5 Check if the perturbation range needs to be expandedor not If the cost function is not improved after 119899 times ofperturbations then we scale 120572 in (7) to extend the searchingrange Otherwise the searching range does not need to

be extended After the decision of whether to extend theperturbation range or not is made the algorithm goes backto Step 3

33 Markerless AR Visualization

331 Tracking and Camera Pose Estimation The flowchartfor the procedure for markerless AR visualization is shown inFigure 4 The KLT tracker takes turns to track the extractedfeature points in the AR image Assuming the tracking resultin each frame is denoted as a point set 119902

119905 initially we

randomly select a number of 119873119877points from 119902

119905 The first

estimation of extrinsic parameters is thus calculated by usingthe EPNP camera pose estimation algorithm [40] Then weuse these extrinsic parameters to project the 3D points 119863

119902

of these features onto the AR frame obtaining a set of 2Dprojective points 119902proj which has the number of 119873

119902points

Ideally if all points are being tracked correctly the projectivepoints 119902proj and the tracking points 119902119905 in camera image shouldbe overlapping or very close to each other The 119871

2-norm

distance119889119894 for each pair of projective point 119902proj119894 and tracked

point 119902119905119894 is as shown in

119889119894=10038171003817100381710038171003817119902119905119894minus 119902proj119894

100381710038171003817100381710038172 (8)

If 119889119894is greater than a predefined threshold 120581 then the point

119902119894is considered as an outlier that is a tracking-failed point

Otherwise the point 119902119894is an inlier Here we choose three

pixels for the threshold 120581 which makes the AR display morestable By determining whether every point of 119902

119905is inlier or

not the ldquoinlier raterdquo of this tracking result 119902119905is measured

which implies the rate of how many points are being trackedcorrectly in this frame The inlier rate 119903

119905of a frame in time 119905

is defined as

119903119905=

119873Inliers119873Inliers + 119873Outliers

(9)

where 119873Inliers stands for the number of feature points whichare tracked correctly and 119873Outliers represents the number ofoutliers If the inlier rate is higher than a predefined threshold120577 it indicates that this estimation of extrinsic parameters inthe current frame is highly reliable and therefore we can usethis estimation result of extrinsic parameters to correct theoutliers If a point is considered as an outlier its projectivepoint is used to replace this outlier point The threshold 120577 isset to 08 in this study

On the other hand if the inlier rate is less than 120577 thesystem randomly selects a different group in 119902

119905to estimate

another set of extrinsic parameters The inlier rate 119903119905is then

calculated again after finding projecting points 119902proj of 3Dpoints 119863

119902 If this process is performed over a times and all

inliers rates 119903119905are less than 120577 the system is determined as a

tracking failure and a tracking recovery is required

332 Tracking Recovery When the tracking fails we use theSURF feature comparison method as a reference to help thesystem recover to the original tracking status As mentionedabove in the step of feature point cloud reconstruction


Floating data A

Reference data B

Is C improved

Result meets stopping

conditions

End

Perturb the solution

Accept transformation

Yes

Need to expand perturbation

range

No

Expand perturbation range

Yes

NoYes

No

by TpertICP(A B)

Figure 3 Flowchart of the improved ICP Algorithm

Grab new frame

No

Yes

SURF basedobject detection

Is objectdetected

Feature trackingusing KLT

RANSAC-basedcamera pose estimation

Render virtual object

Figure 4 Markerless AR Visualization Procedure

a set of SURF keypoints is extracted from the target region119868target and denoted as ptemp while their corresponding SURFdescriptors119863temp are then estimated as shown in

ptemp = 119901temp119894 (119909 119910) 1 le 119894 le 119873119901temp

119863temp = dtemp119894 (119909 119910) 1 le 119894 le 119873119901temp

(10)

where 119901temp119894 stands for the 119894th SURF keypoint in 119868target andd119894is the SURF descriptor of 119894th SURF keypoint As the AR

camera is turned on and being prepared for AR visualizationfrom each frame 119868cam which is the image captured by

the AR camera another set of SURF keypoints pcam with119873pcam points are extracted and denoted as

pcam = 119901cam119895 (119909 119910) 1 le 119895 le 119873119901cam

119863cam = dcam119895 (119909 119910) 1 le 119895 le 119873119901cam (11)

For each SURF keypoint 119901cam119894 in 119868cam it is compared toevery SURF keypoint 119901temp in 119868target by calculating 1198712-normdistance of their descriptors dtemp119894 and dcam119895

119864119894119895=10038171003817100381710038171003817dtemp119894 minus dcam119895

100381710038171003817100381710038172 (12)

A matching pair is selected if the 1198712-norm distance between

the descriptors is smaller than 07 times of the distanceto the second-nearest keypoint Assuming the number ofsuccessful corresponding pairs is 119873

119888 then if 119873

119888is greater

than a predefined threshold 120591 it implies that the target objectis probability in the FOV of the AR camera and the systemthen moves to the next image-matching step Otherwise theprevious process continuously repeats until the criteria ismetthat is119873

119888gt 120591

4 Experimental Results

The proposed markerless AR scheme is based on a StereoVideo See-Through HMD device a Vuzix Wrap 1200DXARas show in Figure 5(a) Since this work is aimed for anAR visualization of medical application a plastic dummyhead is chosen as target object for AR visualization Thecomputed tomography image of the dummy head is obtainedto construct the virtual object for visualization as show inFigure 5(b)


(a) (b)

Figure 5 (a) Vuzix Wrap 1200DXAR Stereo HMD and (b) reconstructed CT model of the dummy head

(a) (b) (c)

Figure 6 Tools used to evaluate the accuracy of spatial alignment of medical information (a) plastic dummy head (b) MicroScribe G2Xdigitizer and (c) triangular prism for coordinate system calibration

The proposed medical AR system has two unique fea-tures one is a marker-free image-to-patient registration andthe other is a pattern-less AR visualization In this sectionexperiments were carried out to evaluate the performancewith respect to these features At first the accuracy of themedical information alignment is evaluated in Section 41 InSection 42 visualization result of the proposed AR scheme isshown

41 Accuracy Evaluation of Alignment In order to evaluatethe accuracy of the image-to-patient registration of theproposed system a plastic dummy head was utilized as thephantom target object Before scanning CT images of thephantom five skin markers were attached on the face of thephantom as shown in Figure 6(a) Since the locations of theseskin markers could easily be identified in the CT imagesthese markers were considered as the reference to evaluatethe accuracy of registration A commercial 3D digitizerG2X produced by Microscribe [41] as shown in Figure 6(b)was utilized to establish the reference coordinate systemand estimate the location of the markers in the physical

space According to its specification the accuracy of G2X is023mmwhich is suitable for locating the coordinates of skinmarkers as the ground truth for evaluation

Before the evaluation a calibration step was performedto find the transformation Digi

W119879 between the stereo 3Dcoordinate system that is the world coordinate system andthe digitizerrsquos coordinate system 119862Digi In order to performthe calibration a triangular prism with chessboard patternsattached is used as shown in Figure 6(c) This prism wasplaced in the FOV of the stereo HMD Corner points of thechessboard were selected by the digitizer to be reconstructedby the stereo camera The two sets of 3D points representedby the coordinate systems of the digitizer and the stereoHMD respectively were used to estimate a transformationDigiW119879 by using least mean square (LMS) method so that the

3D points reconstructed by stereo camera can be transformedto the digitizerrsquos coordinate system 119862Digi

The plastic dummy head was placed in front of the FOVof the stereo camera at a 60 cm distance First we usedthe G2X digitizer to obtain the 3D coordinates of thesemarkers Next the stereo camera was utilized to reconstruct


Figure 7 Result of spatial alignment between feature point cloud and virtual object surface from CT

Table 1 Target registration error of each skin-attached marker after using various alignment methods

Target 1 Target 2 Target 3 Target 4 Target 5 AverageMeanTRE (mm)using Adaptive-ICP [37] 935 573 479 838 778 720

MeanTRE (mm)using Random-ICP [38] 987 582 515 888 824 759

MeanTRE (mm)using Fast-MICP [39] 451 513 356 157 166 328

MeanTRE (mm)using improved ICP 347 222 306 355 377 321

the headrsquos feature point cloud Next another surface isextracted from the CT image of the dummy head The pointcloud was transformed to the 119862Digi by applying Digi

W119879 andthen registered to the preoperative CT images by using theimproved ICP algorithm Image-to-patient registration isevaluated by calculating the target registration error (TRE)[42 43] of the five skin markers The TRE for evaluation isdefined as

TRE = 119872CT119894 minusMIW119879 (119872face119894) (13)

where119872CT119894 denotes the coordinate of the 119894th marker in theCT coordinate system and119872face119894 is the coordinate of the 119894thmarker in 119862Digi The transformation MI

W119879 represents the rigidtransformation obtained from the improved ICP algorithmFigure 7 shows the result of spatial alignment between featurepoint cloud and the virtual object surface from CT theinitial spatial position of the reconstructed facial surfacedata (white) and CT surface data (magenta) The alignment

procedure was performed repeatedly 100 times and eachtime we slightly shifted the location and orientation of thephantom The TREs at each registration procedure wererecorded and the means and the average errors are shownin Table 1 To demonstrate the performance of the improvedICP algorithm three variants of ICP methods for exampleAdaptive-ICP [37] Random-ICP [38] and Fast-MICP [39]are used for accuracy comparison From the experimentalresults it is noted that the TREs using Adaptive-ICP andRandom-ICP are large because no good initial values aregiven For Fast-MICP good initial condition is needed suchthat the registration error can be reduced On the otherhand a good initial condition is not required in our casebecause the proposed improved ICP algorithm can stillobtain good results by efficient error-minimization strategyAs shown in Table 1 the mean TREs of the skin markersusing the proposed method are within the range of 2 to4mm On a personal computer with an Intel Core 2 Duo


(a) (b)

Figure 8 Augmented reality visualization results of the proposed medical AR system using the plastic phantom head as testing subject(a) First row original stereo images view second row AR visualization result (b) AR visualization results in another view

CPU 293GHzCPU with 2GBRAM the processing framerate reached 30 framess

42 Markerless AR Visualization Results The proposed ARsystem was tested on a plastic dummy head In the case ofthe dummy head a 3D structure of the dummy headrsquos CTimage is reconstructed The outer surface of the 3D structureis extracted to build the virtual object for AR renderingas show in Figure 5(b) The AR visualization results fromdifferent viewpoints are shown in Figure 8 The CT modelis well aligned to the position of dummy head When thecamera starts moving markerless AR scheme is applied toboth stereo image and the extrinsic parameters of the twocameras are estimated frame by frame The CT model of thedummy head is rendered in both stereo camera views

43 Accuracy Evaluation of Camera Extrinsic ParametersEstimation For an AR system the accuracy of the extrinsicparameter estimation of the AR camera is themost importantthing In order to evaluate the AR visualization part of theproposed system an optical tracking device the PolarisVicraproduced by Northen Digital Inc was utilized to evaluatethe extrinsic parameter estimation results of the proposedsystem The Polaris Vicra is an optical spatial localizationapparatus which can detect infrared reflective balls by using apair of infrared cameras Since the reflective balls are fixed ona cross-type device called dynamic reference frame (DRF)the posture and position of the DRF can thus be obtainedThe DRF was attached to the HMD in order to track theAR camera by using the Polaris Vicra sensor According tothe specification of this product the localization error of thePolaris Vicra is smaller than 1mm Therefore the postureand position of the AR camera estimated by the PolarisVicra are considered as the comparative reference to evaluatethe accuracy of the proposed patternless AR system In thisexperiment we have evaluated the proposed patternless AR

Table 2 Accuracy in each degree of freedom

Degrees of freedom Mean error Standard deviationRotation in 119909 direction 333 degrees 138Rotation in 119910 direction 231 degrees 152Rotation in 119911 direction 072 degrees 046Translation in 119909 direction 611mm 427Translation in 119910 direction 596mm 359Translation in 119911 direction 478mm 355

system by comparing results against the estimation resultsobtained by the Polaris Vicra

In this experiment the extrinsic parameters of the HMDcamera estimated by the proposed system were comparedto the results estimated by the Polaris Vicra in 450 framesThe differences for each of the six degrees of freedom weremeasured The tracking results of the Polaris Vicra are con-sidered as the ground truth for comparison Figure 9 showsthe evaluation results for rotation and Figure 10 shows theevaluation results for translation The blue curves representthe estimation results of the proposed system and the redcurves are the results estimated by the PolarisVicraThemeanerrors of each degree of freedom are shown in Table 2

5 Conclusion

In traditional AR camera localization methods a knownpattern must be placed within the FOV of the AR camera forthe purpose of estimating extrinsic parameters of the cameraIn this study the shortcomings of the traditional methods areimproved Amarkerless AR visualization scheme is proposedby utilizing a stereo camera pair to construct the surface dataof the target and an improved ICP based surface registrationtechnique is performed to align the preoperative medical


AR systemGround truth

minus30

minus20

minus10

0

10

20

30

Yaw

angl

e (de

g)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (yaw)

(a)


0 100 150 200 250 300 350 400 45050Frame number

minus30

minus20

minus10

0

10

20

30

Roll

angl

e (de

g)

Rotation (roll)

(b)


minus30

minus20

minus10

0

10

20

30

Pitc

h an

gle (

deg)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (pitch)

(c)

Figure 9 Accuracy evaluation results of rotation estimation compared to the result of tracking device Polaris Vicra (rotation) (a) roll(b) pitch and (c) yaw

image model to the real position of the patient A RANSAC-based correction is integrated to solve the problem of theAR camera location without using any pattern and theexperimental results demonstrate that the proposed approachprovides an accurate stable and smooth AR visualization

Compared to conventional pattern-based AR systemsthe proposed system uses only nature features to estimatethe extrinsic parameters of the AR camera As a resultit is more convenient and practical because the FOV ofthe AR camera is not limited by the requirement of thevisibility of the AR pattern A RANSAC-based correctiontechnique is used to improve the robustness of the extrinsicparameter estimation of theAR cameraTheproposed systemhas been evaluated on both image-to-patient registrationand AR camera localization with a plastic dummy headThe system has since been tested on a human subject andshowed promising AR visualization results In the futureextensive clinical trials are expected for further investigation

Furthermore the medical AR environment is expected to beintegrated to an image-guided navigation system for surgicalapplications

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

The work was supported by the Ministry of Science andTechnology China under Grant MOST104-2221-E-182-023-MY2 and Chang Gung Memorial Hospital with Grant nosCMRPD2C0041 CMRPD2C0042 and CMRPD2C0043 Theauthors would like to thank Mr Yao-Shang Tseng and DrChieh-Tsai Wu Department of Neurosurgery and Medical


minus60

minus40

minus20

0

20

40

60

80

X p

ositi

on (m

m)

0 100 150 200 250 300 350 400 45050Frame number


Translation (x)

(a)

0 100 150 200 250 300 350 400 45050Frame number

120

130

150

170

190

210

140

160

180

200

220

Y p

ositi

on (m

m)


Translation (y)

(b)


0 100 150 200 250 300 350 400 45050Frame number

500

550

600

650

Z p

ositi

on (m

m)

Translation (z)

(c)

Figure 10 Accuracy evaluation results of translation estimation compared to the result of tracking device Polaris Vicra (translation) in (a) 119909(b) 119910 and (c) 119911 direction

Augmented Reality Research Center Chang Gung MemorialHospital Dr Shin-Tseng Lee Department of Neurosurgeryand Medical Augmented Reality Research Center ChangGung Memorial Hospital and Dr Jong-Chih Chien Depart-ment of Information Management Kainan University fortheir valuable suggestion which helped in improving thecontent of our paper

References

[1] R T Azuma ldquoA survey of augmented realityrdquo Presence Teleop-erators andVirtual Environments vol 6 no 4 pp 355ndash385 1997

[2] R T Azuma Y Baillot R Behringer S K Feiner S Julierand BMacIntyre ldquoRecent advances in augmented realityrdquo IEEEComputer Graphics and Applications vol 21 no 6 pp 34ndash472001

[3] T Jebara C Eyster J Weaver T Starner and A PentlandldquoStochasticks augmenting the billiards experience with prob-abilistic vision and wearable computersrdquo in Proceedings of theInternational Symposium on Wearable Computers (ISWC rsquo97)pp 138ndash145 Cambridge Mass USA October 1997

[4] S K Feiner ldquoAugmented reality a new way of seeingrdquo ScientificAmerican vol 286 no 4 pp 48ndash55 2002

[5] D W F van Krevelen and R Poelman ldquoA survey of augmentedreality technologies technologies applications and limitationsrdquoThe International Journal of Virtual Reality vol 9 no 2 pp 1ndash202010

[6] T Sielhorst M Feuerstein and N Navab ldquoAdvanced medicaldisplays a literature review of augmented realityrdquo Journal ofDisplay Technology vol 4 no 4 pp 451ndash467 2008

[7] T Loescher S Y Lee and J P Wachs ldquoAn augmented realityapproach to surgical telementoringrdquo in Proceedings of the IEEE


International Conference on SystemsMan andCybernetics (SMCrsquo14) pp 2341ndash2346 IEEE San Diego Calif USA October 2014

[8] F Pretto I H Manssour M H I Lopes and M S PinholdquoExperiences using augmented reality environment for trainingand evaluating medical studentsrdquo in Proceedings of the IEEEInternational Conference on Multimedia and Expo Workshops(ICMEW rsquo13) pp 1ndash4 San Jose Calif USA July 2013

[9] T Blum R Stauder E Euler and N Navab ldquoSuperman-likeX-ray vision towards brain-computer interfaces for medicalaugmented realityrdquo in Proceedings of the 11th IEEE and ACMInternational Symposium on Mixed and Augmented Reality(ISMAR rsquo12) pp 271ndash272 IEEE Atlanta Ga USA November2012

[10] V Ferrari GMegali E Troia A Pietrabissa and FMosca ldquoA 3-Dmixed-reality system for stereoscopic visualization ofmedicaldatasetrdquo IEEE Transactions on Biomedical Engineering vol 56no 11 pp 2627ndash2633 2009

[11] H Liao T Inomata I Sakuma and T Dohi ldquo3-D augmentedreality for MRI-guided surgery using integral videographyautostereoscopic image overlayrdquo IEEE Transactions on Biomed-ical Engineering vol 57 no 6 pp 1476ndash1486 2010

[12] J Fischer M Neff D Freudenstein and D Bartz ldquoMedicalaugmented reality based on commercial image guided surgeryrdquoin Proceedings of the 10th Eurographics Conference on VirtualEnvironments (EGVE rsquo04) pp 83ndash86 Aire-la-Ville SwitzerlandJune 2004

[13] C Bichlmeier F Wimmer S M Heining and N Navab ldquoCon-textual anatomic mimesis hybrid in-situ visualization methodfor improving multi-sensory depth perception in medicalaugmented realityrdquo in Proceedings of the 6th IEEE and ACMInternational Symposium on Mixed and Augmented Reality(ISMAR rsquo07) pp 129ndash138 Nara Japan November 2007

[14] H Kato and M Billinghurst ldquoMarker tracking and HMDcalibration for a video-based augmented reality conferencingsystemrdquo in Proceedings of the 2nd IEEE and ACM InternationalWorkshop on Augmented Reality (IWAR rsquo99) pp 85ndash94 IEEESan Francisco Calif USA 1999

[15] H T Regenbrecht M T Wagner and H T RegenbrechtldquoInteraction in a collaborative augmented reality environmentrdquoin Proceedings of the Extended Abstracts on Human Factors inComputing Systems (CHI rsquo02) pp 504ndash505 ACM 2002

[16] Y Sato M Nakamoto Y Tamaki et al ldquoImage guidanceof breast cancer surgery using 3-D ultrasound images andaugmented reality visualizationrdquo IEEE Transactions on MedicalImaging vol 17 no 5 pp 681ndash693 1998

[17] D Pustka M Huber C Waechter et al ldquoAutomatic configura-tion of pervasive sensor networks for augmented realityrdquo IEEEPervasive Computing vol 10 no 3 pp 68ndash79 2011

[18] C Tomasi and T Kanade ldquoDetection and tracking of pointfeaturesrdquo Tech Rep Carnegie Mellon University CMU-CS-91-132 1991

[19] M A Fischler and R C Bolles ldquoRandom sample consensus aparadigm for model fitting with applications to image analysisand automated cartographyrdquo Communications of the ACM vol24 no 6 pp 381ndash395 1981

[20] H Bay A Ess T Tuytelaars and L Van Gool ldquoSpeeded-uprobust features (SURF)rdquo Computer Vision amp Image Understand-ing vol 110 no 3 pp 346ndash359 2008

[21] ARtoolkit httpwwwhitlwashingtoneduartoolkit[22] H Kato and M Billinghurst ldquoMarker tracking and HMD

calibration for a video-based augmented reality conferencing

systemrdquo in Second International Workshop on Augmented Real-ity pp 85ndash94 San Francisco Calif USA 1999

[23] M Hirzer ldquoMarker detection for augmented reality applica-tionsrdquo Research Paper Granz University of Technology GrazAustria

[24] B Glocker S Izadi J Shotton and A Criminisi ldquoReal-timeRGB-D camera relocalizationrdquo in Proceedings of the IEEEand ACM International Symposium on Mixed and AugmentedReality (ISMAR rsquo13) pp 173ndash179 IEEE Adelaide AustraliaOctober 2013

[25] E Eade and T Drummond ldquoUnified loop closing and recoveryfor real timemonocular SLAMrdquo inProceedings of the 19th BritishMachine Vision Conference (BMVC rsquo08) pp 61ndash610 September2008

[26] BWilliams G Klein and I Reid ldquoAutomatic relocalization andloop closing for real-timemonocular SLAMrdquo IEEETransactionson Pattern Analysis and Machine Intelligence vol 33 no 9 pp1699ndash1712 2011

[27] O Kutter A Aichert C Bichlmeier et al ldquoReal-time volumerendering for high quality visualization in augmented realityrdquoin Proceedings of the International Workshop on AugmentedEnvironments for Medical Imaging Including Augmented Realityin Computer-Aided Surgery (AMI-ARCS rsquo08) MICCAI SocietyNew York NY USA September 2008

[28] M Wieczorek A Aichert O Kutter et al ldquoGPU-acceleratedrendering for medical augmented reality in minimally-invasiveproceduresrdquo in Bildverarbeitung fuer die Medizin (BVM rsquo10)Springer Aachen Germany March 2010

[29] H Suenaga H Hoang Tran H Liao et al ldquoReal-time in situthree-dimensional integral videography and surgical navigationusing augmented reality a pilot studyrdquo International Journal ofOral Science vol 5 no 2 pp 98ndash102 2013

[30] S A Nicolau X Pennec L Soler et al ldquoAn augmented realitysystem for liver thermal ablation design and evaluation onclinical casesrdquo Medical Image Analysis vol 13 no 3 pp 494ndash506 2009

[31] H G Debarba J Grandi A Maciel and D Zanchet AnatomicHepatectomy Planning Through Mobile Display Visualizationand Interaction vol 173 of Studies in Health Technology andInformatics MMVR 2012

[32] L Maier-Hein A M Franz M Fangerau et al ldquoTowardsmobile augmented reality for on-patient visualization of medi-cal imagesrdquo in Bildverarbeitung fur die Medizin H Handels JEhrhardt T M Deserno H-P Meinzer and T Tolxdorff EdsInformatik aktuell pp 389ndash393 Springer Berlin Germany2011

[33] T Blum V Kleeberger C Bichlmeier and N Navab ldquoMirracleaugmented reality in-situ visualization of human anatomyusinga magic mirrorrdquo in Proceedings of the 19th IEEE Virtual RealityConference (VRW rsquo12) pp 169ndash170 IEEE Costa Mesa CalifUSA March 2012

[34] M Meng P Fallavollita T Blum et al ldquoKinect for interactiveAR anatomy learningrdquo in Proceedings of the IEEE InternationalSymposium on Mixed and Augmented Reality (ISMAR rsquo13) pp277ndash278 IEEE Adelaide Australia October 2013

[35] M Macedo A Apolinario A C Souza and G A GiraldildquoHigh-quality on-patient medical data visualization in a mark-erless augmented reality environmentrdquo SBC Journal on Interac-tive Systems vol 5 no 3 pp 41ndash52 2014

[36] A C Souza M C F Macedo and A L Apolinario JrldquoMulti-frame adaptive non-rigid registration for markerless


augmented realityrdquo in Proceedings of the 13th ACM SIGGRAPHInternational Conference on Virtual-Reality Continuum and ItsApplications in Industry (VRCAI rsquo14) pp 7ndash16 ACM ShenzhenChina November-December 2014

[37] J-D Lee C-H Huang L-C Liu S-T Lee S-S Hsieh andS-P Wang ldquoA modified soft-shape-context ICP registrationsystem of 3-d point datardquo IEICE Transactions on Informationand Systems vol E90-D no 12 pp 2087ndash2095 2007

[38] G P Penney P J Edwards A P King J M Blackall P GBatchelor and D J Hawkes ldquoA stochastic iterative closestpoint algorithm (stochastICP)rdquo in Medical Image Computingand Computer-Assisted InterventionmdashMICCAI 2001 vol 2208of Lecture Notes in Computer Science pp 762ndash769 SpringerBerlin Germany 2001

[39] J-D Lee C-H Huang S-T Wang C-W Lin and S-TLee ldquoFast-MICP for frameless image-guided surgeryrdquo MedicalPhysics vol 37 no 9 pp 4551ndash4559 2010

[40] V Lepetit F Moreno-Noguer and P Fua ldquoEPnP an accurateO(n) solution to the PnP problemrdquo International Journal ofComputer Vision vol 81 no 2 pp 155ndash166 2009

[41] Microscribe httpwwwemicroscribecomproductsmicros-cribe-g2htm

[42] N M Hamming M J Daly J C Irish and J H SiewerdsenldquoEffect of fiducial configuration on target registration errorin intraoperative cone-beam CT guidance of head and necksurgeryrdquo in Proceedings of the 30th Annual International Con-ference of the IEEE Engineering in Medicine and Biology Society(EMBS rsquo08) pp 3643ndash3648 IEEE Vancouver Canada August2008

[43] J M Fitzpatrick J B West and C R Maurer Jr ldquoPredictingerror in rigid-body point-based registrationrdquo IEEETransactionson Medical Imaging vol 17 no 5 pp 694ndash702 1998

Submit your manuscripts athttpwwwhindawicom

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

MathematicsJournal of


Mathematical Problems in Engineering

Hindawi Publishing Corporationhttpwwwhindawicom

Differential EquationsInternational Journal of

Volume 2014

Applied MathematicsJournal of


Probability and StatisticsHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Journal of


Mathematical PhysicsAdvances in

Complex AnalysisJournal of


OptimizationJournal of


CombinatoricsHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of


Operations ResearchAdvances in

Journal of


Function Spaces

Abstract and Applied AnalysisHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of Mathematics and Mathematical Sciences


The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014


Algebra

Discrete Dynamics in Nature and Society



Decision SciencesAdvances in

Discrete MathematicsJournal of


Volume 2014 Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Stochastic AnalysisInternational Journal of


to have a god spatial concept this usually requires a lot ofclinical experience In contrast AR provides an attractivealternative for medical information visualization because thedisplays use the real world coordinate system In generala video see-through AR system uses a movable camera tocapture images from the physical space and draws virtualobjects onto the correct positions on the images As a resultthe way to estimate the spatial relationship between thecamera and the physical space that is camera localizationis the most important problem to be solved in an ARsystem [12 13] The spatial relationship is also known asthe extrinsic parameters of the AR camera which are threetranslation parameters and three rotation parameters andcan be referred to as the position and orientation of thecamera A precise estimation of the extrinsic parametersensures that the medical information can be accuratelyrendered on the scene captured from the physical space Aconventional approach to estimate these extrinsic parametersis to place a black-white rectangle pattern within the scenewhich can be used as a reference to be detected in the camerafield of view (FOV) [14 15] using computer vision techniquesOnce the reference pattern is detected in the AR cameraframe the extrinsic parameters can be determined usingthe perspective projection cameramodel Alternatively somestudies attach a couple of retroreflectivemarkers for exampleinfrared reflectivemarkers on the AR camera and track thesemarkers by using an optical tracking sensor [13 16 17] inorder to estimate the position and orientation of the cameraby using the positions of these markers However in thepattern-based or sensor-basedARdisplay the FOVof camerais limited because these artificial markers should be observedwith the FOV In addition the accuracy of AR renderingdepends on the size and likelihood of correct identificationof the pattern or markers within the FOV

In this paper we present a markerless AR scheme formedical applications The markerless AR scheme is mainlybased on a Video See-Through Head-Mounted Display(HMD) device with a stereo camera on it We use thestereo camera to reconstruct feature point cloud of the targetfor AR visualization An improved ICP algorithm is thenapplied to align the surface data to the virtual object createdfrom medical information The improved ICP algorithm hastwo additional characteristics First a random perturbationtechnique is applied to provide the algorithm opportunitiesto escape the local minima Second a weighting strategy isadded to the cost function of the ICP in order to weaken theinfluence of outlier points

When the AR camera moves feature points are thentracked by using the Kanade-Lucaz-Tomasi tracking algo-rithm [18] on a frame by frame basis while updating theextrinsic parameters Moreover the Random Sample Con-sensus (RANSAC) [19] algorithm is applied to keep the KLTtracking results stable in each frame in order to make theAR visualization smoother and more accurate Furthermoreconsidering the target for AR rendering might be out of theFOV of the AR camera a reinitialization step is necessaryand can be accomplished by applying the Speeded-UpRobustFeatures (SURF) [20] feature matching algorithm for targetdetection

The remainder of this paper is organized as followsin Section 2 we report on some related works about ARvisualization Section 3 includes all material and methods ofthe proposed system Experimental results are presented anddiscussed in Section 4 Finally Section 5 gives the conclusionof this work

2 Related Works

Camera localization is a key component in dense 3D map-ping Simultaneous Localization and Mapping (SLAM) andfull-range 3D reconstruction It is also the keypoint problemof video-based AR visualization since we need to know thepose and position of the camera in order to render the virtualobject onto the camera scene There are different types ofmethods for camera localization and the most well-knowncategory of camera localization methods belongs to theplanar-based methods In the past 10 years the planar-basedmethods have become one of themajor types in researches forAR camera localization methods one of the examples is thewidely used AR environment development library ARToolkit[21] In this type of method a predesigned rectangle markerwould be placed inside the camerarsquos FOV and a trackingalgorithm is then applied to the captured frames in order totrack the marker [22 23]

Another category is the landmark-based approaches [24]In this type of method several landmark feature points areextracted from the camera image by using feature extractionalgorithms Descriptors of these landmark feature points arealso extracted for tracking and comparison with the 3Dlocations of these landmark feature points In each framethese landmark feature points are picked by tracking orcomparing their corresponding feature descriptorsrsquo similar-ities and the pose of the camera can then be determined[25 26] Besides Kutter et al [27] proposed a marker-based approach to render the patientrsquos volume data on aHMD device Their scheme provides an efficient volumerendering in an AR workspace and solves the problem ofocclusion by the physicianrsquos hands LaterWieczorek et al [28]extended this scheme by improving the occlusion due to themedical instrument and added new functions such as virtualmirror to the AR system Suenaga et al [29] also proposed afiducial marker-based approach for on-patient visualizationof maxillofacial regions by using an optical tracking systemto track the patient and using a mobile device to visualize theinternal structures of the patientrsquos body Nicolau et al [30]used 15 markers to perform the registration in order to useAR to overlay a virtual liver over the patient Debarba et al[31] proposed a method to visualize anatomic liver resectionsin anAR environmentwith the use of a fiducialmarker whichmade locating the position and tracking of the medical datain the scene possible

Recently there are somemarkerless AR systems proposedin the literatures using various camera configurations Forexample Maier-Hein et al [32] implemented a markerlessmobile AR system by using a time-of-flight camera mountedon a portable device that is a tablet PC to see the patientrsquosanatomical information The registration method of thisapproach is an anisotropic variant of the ICP algorithm and









[[

[

119909

119910

119908

]]

]

=[[

[

1198911199090 1199090

0 1198911199101199100

0 0 1

]]

]

[R | t][[[[[

[

119883

119884

119885

1

]]]]]

]

= 119870119864

[[[[[

[

119883

119884

119885

1

]]]]]

]

(1)



119904119909in 119909 direction and 119891





119875 = 119870119864 =[[

[

11987511119875121198751311987514

11987521119875221198752311987524

11987531119875321198753311987534

]]

]

(2)




119871

of the left camera

(11987511minus 11987531119883119871)119883 + (119875

12minus 11987532119883119871) 119884 + (119875

13minus 11987533119883119871) 119885

= 11987534119883119871minus 11987514

(11987521minus 11987531119883119871)119883 + (119875

22minus 11987532119883119871) 119884 + (119875

23minus 11987533119883119871) 119885

= 11987534119883119871minus 11987524

(3)




(11987511minus 11987531119883119877)119883 + (119875

12minus 11987532119883119877) 119884 + (119875

13minus 11987533119883119877) 119885

= 11987534119883119877minus 11987514

(11987521minus 11987531119883119877)119883 + (119875

22minus 11987532119883119877) 119884 + (119875

23minus 11987533119883119877) 119885

= 11987534119883119877minus 11987524

(4)


119871119901119877) By calculating



Stereo image pair




Spatial alignment


Surface extraction




1 1198862 119886











119865 =1

119898

119898

sum

119894=1

119908119894

min119895isin12119899

10038171003817100381710038171003817119886119894minus 119887119895

10038171003817100381710038171003817 (5)



119908119894=


median10038171003817100381710038171003817a119894minus b119895

10038171003817100381710038171003817

otherwise(6)


119909 119879119910 119879119911)



F

T

r

120572r

TInit

Ttemp



119901 (119910) =

31199102

212057231199033if minus 120572119903 lt 119910 lt 120572119903

0 otherwise(7)










119905 initially we


119905 The first


119902


119902points


2-norm



119889119894=10038171003817100381710038171003817119902119905119894minus 119902proj119894

100381710038171003817100381710038172 (8)





119905is inlier or




is defined as

119903119905=


(9)



119905to estimate








Floating data A

Reference data B

Is C improved


conditions

End



Yes


range

No


Yes

NoYes

No

by TpertICP(A B)


Grab new frame

No

Yes


Is objectdetected








(10)




pcam = 119901cam119895 (119909 119910) 1 le 119895 le 119873119901cam

119863cam = dcam119895 (119909 119910) 1 le 119895 le 119873119901cam (11)


119864119894119895=10038171003817100381710038171003817dtemp119894 minus dcam119895

100381710038171003817100381710038172 (12)



119888 then if 119873

119888is greater


119888gt 120591




(a) (b)


(a) (b) (c)























(a) (b)









5 Conclusion




minus30

minus20

minus10

0

10

20

30

Yaw

angl

e (de

g)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (yaw)

(a)


0 100 150 200 250 300 350 400 45050Frame number

minus30

minus20

minus10

0

10

20

30

Roll

angl

e (de

g)

Rotation (roll)

(b)


minus30

minus20

minus10

0

10

20

30

Pitc

h an

gle (

deg)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (pitch)

(c)







Acknowledgments



minus60

minus40

minus20

0

20

40

60

80

X p

ositi

on (m

m)

0 100 150 200 250 300 350 400 45050Frame number


Translation (x)

(a)

0 100 150 200 250 300 350 400 45050Frame number

120

130

150

170

190

210

140

160

180

200

220

Y p

ositi

on (m

m)


Translation (y)

(b)


0 100 150 200 250 300 350 400 45050Frame number

500

550

600

650

Z p

ositi

on (m

m)

Translation (z)

(c)



References
























































Volume 2014




Journal of











Journal of


Function Spaces






Algebra

















[[

[

119909

119910

119908

]]

]

=[[

[

1198911199090 1199090

0 1198911199101199100

0 0 1

]]

]

[R | t][[[[[

[

119883

119884

119885

1

]]]]]

]

= 119870119864

[[[[[

[

119883

119884

119885

1

]]]]]

]

(1)



119904119909in 119909 direction and 119891





119875 = 119870119864 =[[

[

11987511119875121198751311987514

11987521119875221198752311987524

11987531119875321198753311987534

]]

]

(2)




119871

of the left camera

(11987511minus 11987531119883119871)119883 + (119875

12minus 11987532119883119871) 119884 + (119875

13minus 11987533119883119871) 119885

= 11987534119883119871minus 11987514

(11987521minus 11987531119883119871)119883 + (119875

22minus 11987532119883119871) 119884 + (119875

23minus 11987533119883119871) 119885

= 11987534119883119871minus 11987524

(3)




(11987511minus 11987531119883119877)119883 + (119875

12minus 11987532119883119877) 119884 + (119875

13minus 11987533119883119877) 119885

= 11987534119883119877minus 11987514

(11987521minus 11987531119883119877)119883 + (119875

22minus 11987532119883119877) 119884 + (119875

23minus 11987533119883119877) 119885

= 11987534119883119877minus 11987524

(4)


119871119901119877) By calculating



Stereo image pair




Spatial alignment


Surface extraction




1 1198862 119886











119865 =1

119898

119898

sum

119894=1

119908119894

min119895isin12119899

10038171003817100381710038171003817119886119894minus 119887119895

10038171003817100381710038171003817 (5)



119908119894=


median10038171003817100381710038171003817a119894minus b119895

10038171003817100381710038171003817

otherwise(6)


119909 119879119910 119879119911)



F

T

r

120572r

TInit

Ttemp



119901 (119910) =

31199102

212057231199033if minus 120572119903 lt 119910 lt 120572119903

0 otherwise(7)










119905 initially we


119905 The first


119902


119902points


2-norm



119889119894=10038171003817100381710038171003817119902119905119894minus 119902proj119894

100381710038171003817100381710038172 (8)





119905is inlier or




is defined as

119903119905=


(9)



119905to estimate








Floating data A

Reference data B

Is C improved


conditions

End



Yes


range

No


Yes

NoYes

No

by TpertICP(A B)


Grab new frame

No

Yes


Is objectdetected








(10)




pcam = 119901cam119895 (119909 119910) 1 le 119895 le 119873119901cam

119863cam = dcam119895 (119909 119910) 1 le 119895 le 119873119901cam (11)


119864119894119895=10038171003817100381710038171003817dtemp119894 minus dcam119895

100381710038171003817100381710038172 (12)



119888 then if 119873

119888is greater


119888gt 120591




(a) (b)


(a) (b) (c)























(a) (b)









5 Conclusion




minus30

minus20

minus10

0

10

20

30

Yaw

angl

e (de

g)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (yaw)

(a)


0 100 150 200 250 300 350 400 45050Frame number

minus30

minus20

minus10

0

10

20

30

Roll

angl

e (de

g)

Rotation (roll)

(b)


minus30

minus20

minus10

0

10

20

30

Pitc

h an

gle (

deg)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (pitch)

(c)







Acknowledgments



minus60

minus40

minus20

0

20

40

60

80

X p

ositi

on (m

m)

0 100 150 200 250 300 350 400 45050Frame number


Translation (x)

(a)

0 100 150 200 250 300 350 400 45050Frame number

120

130

150

170

190

210

140

160

180

200

220

Y p

ositi

on (m

m)


Translation (y)

(b)


0 100 150 200 250 300 350 400 45050Frame number

500

550

600

650

Z p

ositi

on (m

m)

Translation (z)

(c)



References
























































Volume 2014




Journal of











Journal of


Function Spaces






Algebra










Stereo image pair




Spatial alignment


Surface extraction




1 1198862 119886











119865 =1

119898

119898

sum

119894=1

119908119894

min119895isin12119899

10038171003817100381710038171003817119886119894minus 119887119895

10038171003817100381710038171003817 (5)



119908119894=


median10038171003817100381710038171003817a119894minus b119895

10038171003817100381710038171003817

otherwise(6)


119909 119879119910 119879119911)



F

T

r

120572r

TInit

Ttemp



119901 (119910) =

31199102

212057231199033if minus 120572119903 lt 119910 lt 120572119903

0 otherwise(7)










119905 initially we


119905 The first


119902


119902points


2-norm



119889119894=10038171003817100381710038171003817119902119905119894minus 119902proj119894

100381710038171003817100381710038172 (8)





119905is inlier or




is defined as

119903119905=


(9)



119905to estimate








Floating data A

Reference data B

Is C improved


conditions

End



Yes


range

No


Yes

NoYes

No

by TpertICP(A B)


Grab new frame

No

Yes


Is objectdetected








(10)




pcam = 119901cam119895 (119909 119910) 1 le 119895 le 119873119901cam

119863cam = dcam119895 (119909 119910) 1 le 119895 le 119873119901cam (11)


119864119894119895=10038171003817100381710038171003817dtemp119894 minus dcam119895

100381710038171003817100381710038172 (12)



119888 then if 119873

119888is greater


119888gt 120591




(a) (b)


(a) (b) (c)























(a) (b)









5 Conclusion




minus30

minus20

minus10

0

10

20

30

Yaw

angl

e (de

g)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (yaw)

(a)


0 100 150 200 250 300 350 400 45050Frame number

minus30

minus20

minus10

0

10

20

30

Roll

angl

e (de

g)

Rotation (roll)

(b)


minus30

minus20

minus10

0

10

20

30

Pitc

h an

gle (

deg)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (pitch)

(c)







Acknowledgments



minus60

minus40

minus20

0

20

40

60

80

X p

ositi

on (m

m)

0 100 150 200 250 300 350 400 45050Frame number


Translation (x)

(a)

0 100 150 200 250 300 350 400 45050Frame number

120

130

150

170

190

210

140

160

180

200

220

Y p

ositi

on (m

m)


Translation (y)

(b)


0 100 150 200 250 300 350 400 45050Frame number

500

550

600

650

Z p

ositi

on (m

m)

Translation (z)

(c)



References
























































Volume 2014




Journal of











Journal of


Function Spaces






Algebra










F

T

r

120572r

TInit

Ttemp



119901 (119910) =

31199102

212057231199033if minus 120572119903 lt 119910 lt 120572119903

0 otherwise(7)










119905 initially we


119905 The first


119902


119902points


2-norm



119889119894=10038171003817100381710038171003817119902119905119894minus 119902proj119894

100381710038171003817100381710038172 (8)





119905is inlier or




is defined as

119903119905=


(9)



119905to estimate








Floating data A

Reference data B

Is C improved


conditions

End



Yes


range

No


Yes

NoYes

No

by TpertICP(A B)


Grab new frame

No

Yes


Is objectdetected








(10)




pcam = 119901cam119895 (119909 119910) 1 le 119895 le 119873119901cam

119863cam = dcam119895 (119909 119910) 1 le 119895 le 119873119901cam (11)


119864119894119895=10038171003817100381710038171003817dtemp119894 minus dcam119895

100381710038171003817100381710038172 (12)



119888 then if 119873

119888is greater


119888gt 120591




(a) (b)


(a) (b) (c)























(a) (b)









5 Conclusion




minus30

minus20

minus10

0

10

20

30

Yaw

angl

e (de

g)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (yaw)

(a)


0 100 150 200 250 300 350 400 45050Frame number

minus30

minus20

minus10

0

10

20

30

Roll

angl

e (de

g)

Rotation (roll)

(b)


minus30

minus20

minus10

0

10

20

30

Pitc

h an

gle (

deg)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (pitch)

(c)







Acknowledgments



minus60

minus40

minus20

0

20

40

60

80

X p

ositi

on (m

m)

0 100 150 200 250 300 350 400 45050Frame number


Translation (x)

(a)

0 100 150 200 250 300 350 400 45050Frame number

120

130

150

170

190

210

140

160

180

200

220

Y p

ositi

on (m

m)


Translation (y)

(b)


0 100 150 200 250 300 350 400 45050Frame number

500

550

600

650

Z p

ositi

on (m

m)

Translation (z)

(c)



References
























































Volume 2014




Journal of











Journal of


Function Spaces






Algebra










Floating data A

Reference data B

Is C improved


conditions

End



Yes


range

No


Yes

NoYes

No

by TpertICP(A B)


Grab new frame

No

Yes


Is objectdetected








(10)




pcam = 119901cam119895 (119909 119910) 1 le 119895 le 119873119901cam

119863cam = dcam119895 (119909 119910) 1 le 119895 le 119873119901cam (11)


119864119894119895=10038171003817100381710038171003817dtemp119894 minus dcam119895

100381710038171003817100381710038172 (12)



119888 then if 119873

119888is greater


119888gt 120591




(a) (b)


(a) (b) (c)























(a) (b)









5 Conclusion




minus30

minus20

minus10

0

10

20

30

Yaw

angl

e (de

g)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (yaw)

(a)


0 100 150 200 250 300 350 400 45050Frame number

minus30

minus20

minus10

0

10

20

30

Roll

angl

e (de

g)

Rotation (roll)

(b)


minus30

minus20

minus10

0

10

20

30

Pitc

h an

gle (

deg)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (pitch)

(c)







Acknowledgments



minus60

minus40

minus20

0

20

40

60

80

X p

ositi

on (m

m)

0 100 150 200 250 300 350 400 45050Frame number


Translation (x)

(a)

0 100 150 200 250 300 350 400 45050Frame number

120

130

150

170

190

210

140

160

180

200

220

Y p

ositi

on (m

m)


Translation (y)

(b)


0 100 150 200 250 300 350 400 45050Frame number

500

550

600

650

Z p

ositi

on (m

m)

Translation (z)

(c)



References
























































Volume 2014




Journal of











Journal of


Function Spaces






Algebra










(a) (b)


(a) (b) (c)























(a) (b)









5 Conclusion




minus30

minus20

minus10

0

10

20

30

Yaw

angl

e (de

g)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (yaw)

(a)


0 100 150 200 250 300 350 400 45050Frame number

minus30

minus20

minus10

0

10

20

30

Roll

angl

e (de

g)

Rotation (roll)

(b)


minus30

minus20

minus10

0

10

20

30

Pitc

h an

gle (

deg)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (pitch)

(c)







Acknowledgments



minus60

minus40

minus20

0

20

40

60

80

X p

ositi

on (m

m)

0 100 150 200 250 300 350 400 45050Frame number


Translation (x)

(a)

0 100 150 200 250 300 350 400 45050Frame number

120

130

150

170

190

210

140

160

180

200

220

Y p

ositi

on (m

m)


Translation (y)

(b)


0 100 150 200 250 300 350 400 45050Frame number

500

550

600

650

Z p

ositi

on (m

m)

Translation (z)

(c)



References
























































Volume 2014




Journal of











Journal of


Function Spaces






Algebra























(a) (b)









5 Conclusion




minus30

minus20

minus10

0

10

20

30

Yaw

angl

e (de

g)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (yaw)

(a)


0 100 150 200 250 300 350 400 45050Frame number

minus30

minus20

minus10

0

10

20

30

Roll

angl

e (de

g)

Rotation (roll)

(b)


minus30

minus20

minus10

0

10

20

30

Pitc

h an

gle (

deg)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (pitch)

(c)







Acknowledgments



minus60

minus40

minus20

0

20

40

60

80

X p

ositi

on (m

m)

0 100 150 200 250 300 350 400 45050Frame number


Translation (x)

(a)

0 100 150 200 250 300 350 400 45050Frame number

120

130

150

170

190

210

140

160

180

200

220

Y p

ositi

on (m

m)


Translation (y)

(b)


0 100 150 200 250 300 350 400 45050Frame number

500

550

600

650

Z p

ositi

on (m

m)

Translation (z)

(c)



References
























































Volume 2014




Journal of











Journal of


Function Spaces






Algebra











minus30

minus20

minus10

0

10

20

30

Yaw

angl

e (de

g)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (yaw)

(a)


0 100 150 200 250 300 350 400 45050Frame number

minus30

minus20

minus10

0

10

20

30

Roll

angl

e (de

g)

Rotation (roll)

(b)


minus30

minus20

minus10

0

10

20

30

Pitc

h an

gle (

deg)

0 100 150 200 250 300 350 400 45050Frame number

Rotation (pitch)

(c)







Acknowledgments



minus60

minus40

minus20

0

20

40

60

80

X p

ositi

on (m

m)

0 100 150 200 250 300 350 400 45050Frame number


Translation (x)

(a)

0 100 150 200 250 300 350 400 45050Frame number

120

130

150

170

190

210

140

160

180

200

220

Y p

ositi

on (m

m)


Translation (y)

(b)


0 100 150 200 250 300 350 400 45050Frame number

500

550

600

650

Z p

ositi

on (m

m)

Translation (z)

(c)



References
























































Volume 2014




Journal of











Journal of


Function Spaces






Algebra










minus60

minus40

minus20

0

20

40

60

80

X p

ositi

on (m

m)

0 100 150 200 250 300 350 400 45050Frame number


Translation (x)

(a)

0 100 150 200 250 300 350 400 45050Frame number

120

130

150

170

190

210

140

160

180

200

220

Y p

ositi

on (m

m)


Translation (y)

(b)


0 100 150 200 250 300 350 400 45050Frame number

500

550

600

650

Z p

ositi

on (m

m)

Translation (z)

(c)



References
























































Volume 2014




Journal of











Journal of


Function Spaces






Algebra

























































Volume 2014




Journal of











Journal of


Function Spaces






Algebra









research article markerless augmented reality via stereo...

Documents