image projection.pdf

8/9/2019 image projection.pdf

1/6

TECHNICAL NOTE - BRAIN TUMORS

A novel augmented reality system of image projection

for image-guided neurosurgery

Mehran Mahvash &Leila Besharati Tabrizi

Received: 12 December 2012 /Accepted: 18 February 2013 /Published online: 15 March 2013# Springer-Verlag Wien 2013

Abstract

Background Augmented reality systems combine virtual

images with a real environment.

Objective To design and develop an augmented reality systemfor image-guided surgery of brain tumors using image

projection.

Methods A virtual image was created in two ways: (1) MRI-

based 3D model of the head matched with the segmented lesion

of a patient using MRIcro software (version 1.4, freeware, Chris

Rorden) and (2) Digital photograph based model in which the

tumor region was drawn using image-editing software. The real

environment was simulated with a head phantom. For direct

projection of the virtual image to the head phantom, a commer-

cially available video projector (PicoPix 1020, Philips) was

used. The position and size of the virtual image was adjusted

manually for registration, which was performed using anatom-ical landmarks and fiducial markers position.

Results An augmented reality system for image-guided neu-

rosurgery using direct image projection has been designed

successfully and implemented in first evaluation with prom-

ising results. The virtual image could be projected to the

head phantom and was registered manually. Accurate regis-

tration (mean projection error: 0.3 mm) was performed

using anatomical landmarks and fiducial markers position.

Conclusions The direct projection of a virtual image to the

patients head, skull, or brain surface in real time is an aug-

mented reality system that can be used for image-guided

neurosurgery. In this paper, the first evaluation of the system

is presented. The encouraging first visualization results indi-

cate that the presented augmented reality system might be an

important enhancement of image-guided neurosurgery.

Keywords Augmented reality system. Image projection.

Image-guided neurosurgery

Introduction

Augmented reality (AR) means systems that allow the user to

see virtual images in a real environment. The resulting image

is a combination of the real and virtual image in real time [1].

Augmented reality systems are available in different types, an

optic or a video head-mounted display (HMD). In addition,

head-up displays and monitor-based configurations have been

developed for different technical areas. Video head-mounted

display (HMD) systems have been described in investigations

for improvement of image guidance [1,36,10]. The draw-

back of these AR systems is the dependency on special

hardware, which can be expensive and unpractical for routine

clinical application. The other point is the contradiction of

some available systems to the definition of an augmented

reality system. Indeed, most systems use a combination of a

virtual image and the video or images of the reality and not the

real environment itself.

In recent years, the use of image-guided surgery has in-

creased. In neurosurgical procedures, precise preoperative plan-

ning for tailored craniotomy, planning of the approach, and

intraoperative image guidance of the resection extent are

performed using neuronavigation systems and are an inherent

part of neurosurgery. Visualization technologies improve the

orientation and safety during the operation [79].

This paper describes the development of a novel method

for image-guided surgery of brain tumor resection using aug-

mented reality. We designed an augmented reality system

M. Mahvash (*)

Department of Neurosurgery,

Clinic of Cologne-Merheim, University of Witten-Herdecke,

Ostmerheimer Strasse 200,

51109 Kln, Germany

e-mail: [email protected]

L. Besharati Tabrizi

Muthesius Academy of Fine Arts and Design, Industrial Design,

Kiel, Germany

Acta Neurochir (2013) 155:943947

DOI 10.1007/s00701-013-1668-2
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-


2/6

using direct image projection to combine a virtual image and

the real head, skull, or brain surface. We describe this method

and technique and present an initial evaluation performed with

a head phantom, which replaces the reality environment.

Materials and methods

The augmented reality system consists of four components:

1. Virtual image creation

2. Real environment

3. Image projection

4. Registration

Virtual image creation

A virtual image was created in two ways: (1) MRI-based 3D

model of the head matched with the segmented lesion of a

patient and (2) a digital photograph-based model in whichthe tumor region was drawn using image-editing software.

MRI-based 3D model

T1-weighted MPR MRI datasets of a patient were used exem-

plary to create a 3D model of the head and brain using MRIcro

software (version 1.4, freeware, Chris Rorden). The brain lesion

that was visible by contrast enhancement (gadolinium) was

segmented and matched to the 3D model with the same software

(Fig.1a). For an MRI-based brain surface model, the brain

extraction tool (BET) was used, which segments the brain

automatically by removing the skull (skull strip image using

BETfrom Etcmenu). The brain lesion can be segmented

from the MRI dataset by definition of a region of interest

(ROI) and saved as ROI, which can be opened (Open ROI[s]

from ROImenu) to overlay it with head or brain 3D model.

Different combinations are possible and the resulting 3D model

can be rotated and viewed according to the desired perspective.

The created 3D model is a virtual image with precise localiza-

tion of the tumor and can be used for image projection.

Photograph-based model

A virtual image can also be created with a digital photograph

of a patients head before or during surgery. The photograph

can be used to add useful information about anatomy, tumorlocalization, and functional areas as reported previously [8].

Due to the different size of the created MRI-based 3D Model

of the patient and the head phantom, for initial evaluation the

virtual image was created using a lateral photograph of the

head phantom (Fig. 2b). The tumor region was drawn in the

photograph with image-editing software. This virtual image

was used for projection to the head phantom.

Real environment

The real environment (surgical field) was simulated with a

head phantom. Five fiducial markers were placed on thehead phantom as points of reference for registration of the

virtual image to the head phantom (Fig. 2a).

Image projection

For projection of the virtual image to the head phantom, a

commercially available video projector (PicoPix 1020,

Philips) was used based on LED technology (Fig. 1b). The

video projector was connected to a laptop computer with a

USB data cable. The software of the video projector was

installed. The video projector and the head phantom were

placed in the same height to project the created virtual image

directly to the head phantom.

Registration

The position and size of the virtual image was adjusted man-

ually for registration, which was performed using anatomical

Fig. 1 a Lateral view of a MRI-based 3D model of the head of a

patient with a temporal brain tumor. The 3D model was created with

the MRIcro software (version 1.4, freeware, Chris Rorden). The brain

tumor with contrast enhancement (gadolinium) was segmented (red

region) and matched to the 3D model. b Experimental setup of the

augmented reality method including a commercially available video

projector (PicoPix 1020, Philips) based on LED technology. The

virtual image (laptop screen) was projected directly to the head phan-

tom, which simulated the real environment

944 Acta Neurochir (2013) 155:943947
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-


3/6

landmarks and fiducial markers position. The registration was

performed with complete overlapping of the five projected

fiducial markers of the virtual image and the corresponding

five fiducial markers on the head phantom (Fig. 2c, d). The

registration was repeated five times to compare the accuracy

and to evaluate the projection error after each manual regis-

tration. After each registration the distance of the five fiducial

markers (Fig. 2) to the visualized tumor border were measuredon the virtual image and on the head phantom as well.

Results

MRI-based 3D model and segmentation of the brain lesion

could be performed easily after knowing the MRIcro soft-

ware well. The digital photograph-based model could be

created with visualization of tumor region. The video pro-

jector (Picopix 1020, Philips) and its software could be

installed and connected to the laptop without difficulty.

The manual registration of the virtual image and the headphantom using anatomical landmarks and fiducial markers

was possible and the tumor localization was accurate

(Fig.2d). The registration was performed within 5 min and

applied five times with the same visual accuracy to achieve

precise overlapping of the five fiducial markers from the

virtual image and head phantom. The mean measured distance

of each fiducial marker to the tumor border was as follows:

fiducial marker 1, 32.2 mm, fiducial marker 2, 30.3 mm,

fiducial marker 3, 42.1 mm, fiducial marker 4, 26.3 mm,

fiducial 5, 15.4 mm. The mean projection error was 0.3 mm

(projection error range: 0.10.6 mm).

First evaluation results show a reliable and accurate aug-

mented reality technique, which can be used for image-

guided neurosurgery. The designed augmented reality sys-

tem is inexpensive and easy to reproduce with a normal

laptop, free available software, and a low-cost video projec-

tor. The visualization results encourage testing this methodon patients in clinical investigations.

Discussion

We present a novel method of an augmented reality system

for image-guided neurosurgery. Several augmented reality

systems have been developed for image-guided surgery

using head-mounted displays (HMDs) [1, 36, 10]. Most

systems use a combination of a virtual image and the video

or images of the reality environment and not the real envi-

ronment itself. One paper presented an image overlay sys-tem using a semi-transparent display [2]. We developed the

idea to project the virtual image directly to the reality

without an HMD system or display, which can be expensive

and unpractical for clinical routine or during surgical pro-

cedures. Therefore we designed a new augmented reality

system using a video projector, which is available at low

cost. One could ask why is it inviting to design an augment-

ed reality system for image-guided surgery, particularly in

neurosurgery? Images that are used for navigation systems

Fig. 2 a Head phantom withfive fiducial markers (Fid.15). b

Digital photograph-based virtual

image was created similar to the

lateral view of the MRI-based

3D model (Fig.1a). The image

was created using a lateral

photograph of the head phantom

and drawing the tumor region

(red region) with image-editing

software. This virtual image was

used for projection.c Projection

of the virtual image to the head

phantom before registration. The

image was focused, size and

position was adjusted manually.dRegistration of the virtual

image to the head phantom.

Anatomical landmarks and five

fiducial markers were used for

registration. Please note the

precise registration and tumor

localization on the head surface

Acta Neurochir (2013) 155:943947 945
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-


4/6

are MRI and/or CT images performed preoperatively and in

some cases intraoperatively. These images in different orien-

tations are still virtually computed images that give neurosur-

geons important information about the anatomy and

localization of brain tumors. However, these images can be

visualized on the neuronavigation systems screen during sur-

gery but are not visible before or during surgery if a neuro-

surgeon looks at the head, skull, or brain surface of the patientdirectly. These are very useful images but still virtual images

with different modalitiy and dimension as the real environ-

ment and the surgeons view. Neuronavigation systems using

MRI and/or CT datasets are systems which are based on

virtual images and can be used after registration of the patient.

Their disadvantage is that the surgeon must look away from

the surgical field, look to the navigation screen and back in

order to transfer the information of the MRI and/or CT images

in his mind from the navigation screen into the real surgical

field. This thinking process means processing of two different

image modalities; the MRI or CT images and the imageof

the real surgical environment. This is an additional work stepand can be a source of errors for surgeons that have to relate

the view of the surgical field to the different images on the

navigation monitor. It would be useful during surgery if a

system could give the surgeon access to both modalities

simultaneously, the virtual images and the reality. The idea

of our augmented reality system is to integrate the helpful

information of the MRI and/or CT images into the real surgi-

cal field to improve orientation and safety. In the described

method, the virtual image is projected to a head phantom

directly without the need of additional hardware.

For the first evaluation we used a lateral photograph of the

head phantom to create the virtual image for projection due to

the different size of the patient MRI-based 3D model and the

head phantom. We could show that it is possible to create an

MRI-based 3D model of head or brain easily and use it as a

virtual image that can be projected to a real environment as

well (Figs.1a,3). The image projection can also be performed

directly to the skull or brain surface during surgery. The

planning of skin incision and the extent of craniotomy can be

improved using this image projection technique. This system

can be used for a tailored craniotomy using the image

projection of a lesion on the patients skull. Furthermore,

subcortical lesions that are not visible on the brain surface

during surgery can be visualized by projection of the lesion

on the brain surface to plan the approach and operation strat-egy. Future applications of the system could also be for brain

tumors adjacent to functional areas of the brain which can be

visualized using direct projection of the tumor and functional

MRI results on the brain surface.

The advantage of the presented system compared to the

conventional navigation system will be the direct and im-

proved visualization of the regions of interest on the pa-

tients head, skull, or brain. In addition, it is inexpensive and

easy to reproduce. However, further development of this

system is possible to design a projection device for clin-

ical applications and it could also be interesting for surgeons

or hospitals that are not able to afford an expensive navigationsystem.

We describe our method, which has been evaluated for

projection of a brain tumor. Furthermore, it could also be

interesting for other surgical areas and procedures like sur-

gery of spinal tumors or facial surgery.

The registration of the images has been performed man-

ually using anatomical landmarks and fiducial markers as

described in the paper. The manual registration was very

accurate with a mean projection error of 0.3 mm, but further

technical advancement can enable an automatic registration

and integration into the standard navigation systems and

integration into the microscope during surgery.

As mentioned, the presented paper describes a novel tech-

nique for localization of brain tumors for image-guided neu-

rosurgery and first evaluation shows an accurate and quick

method. The next steps are planned to evaluate the accuracy of

the method in clinical studies with patients performing image-

Fig. 3 Augmented reality

using image projection of a

created virtual image. a Image

projection of an MRI-based 3D

model of the brain surface

(MRIcro) with localization ofthe tumor (red).b Projection of

an MRI-based model of the

brain surface with visualization

of superior sagittal sinus and

cortical veins (blue) and brain

tumor (red)

946 Acta Neurochir (2013) 155:943947
http://-/?-http://-/?-http://-/?-http://-/?-


5/6

guided brain surgery with this augmented reality system. We

believe that this new technique will make it possible to project

directly the visualized lesions, such as a brain tumor or brain6

metastasis, onto the surface of the head, skull, or brain of the

patients. This would be an important improvement of image-

guided neurosurgery.

Conclusions

We designed an augmented reality system for direct projection

of a virtual image onto the head, skull, and brain surface in real

time for image-guided neurosurgery. In this paper, the first

evaluation of the system is presented. Further technical devel-

opment of this system can be used for image-guided surgery of

brain lesions and other surgical fields as well. The presented

method is easy to reproduce and inexpensive. After the encour-

aging visualization results of this augmented reality system,

clinical applications are objects of further investigations.

Conflicts of interest None.

References

1. Azuma R (1997) A survey of augmented reality. Teleoperators

Virtual Environ 6(4):355385

2. Blackwell M, Nikou C, DiGioia AM, Kanade T (2000) An image

overlay system for medical data visualization. Med Image Anal

4(1):6772

3. Iseki H, Masutani Y, Iwahara M, Tanikawa T, Muragaki Y, Taira T,

Dohi T, Takakura K (1997) Volumegraph (overlaid three-dimensional

image-guided navigation). Clinical application of augmented reality in

neurosurgery. Stereotact Funct Neurosurg 68(14 Pt 1):1824

4. Kawamata T, Iseki H, Shibasaki T, Hori T (2002) Endoscopic aug-

mented reality navigation system for endonasal transsphenoidal sur-

gery to treat pituitary tumors: technical note. Neurosurgery50(6):13931397

5. Kockro RA, Tsai YT, Ng I, Hwang P, Zhu C, Agusanto K, Hong

LX, Serra L (2009) Dex-ray: augmented reality neurosurgical

navigation with a handheld video probe. Neurosurgery 65(4):

795807

6. Lovo EE, Quintana JC, Puebla MC, Torrealba G, Santos JL, Lira

IH, Tagle P (2007) A novel, inexpensive method of image

coregistration for applications in image-guided surgery using aug-

mented reality. Neurosurgery 60(4 Suppl 2):366371

7. Mahvash M, Knig R, Urbach H, von Ortzen J, Meyer B, Schramm

J, Schaller C (2006) FLAIR-/T1-/T2-co-registration for image-

guided diagnostic and resective epilepsy surgery. Neurosurgery

58(1 Suppl):6975

8. Mahvash M, Knig R, Wellmer J, Urbach H, Meyer B, Schaller K

(2007) Coregistration of digital photography of the human cortex and

cranial magnetic resonance imaging for visualization of subdural

electrodes in epilepsy surgery. Neurosurgery 61(5 Suppl 2):340344

9. Mahvash M, Knig R, Scheef L, Urbach H, von Oertzen J, Meyer B,

Schaller C (2004) Multimodale Neuronavigation in der resektiven

Epilepsiechirurgie. Bildverarbeitung fr die Medizin. Springer,

Berlin Heidelberg New York, pp 224228

10. Maurer CR, Sauer F, Hu B, Bascle B, Geiger B, Wenzel F, Recchi

F, Rohlfing T, Brown CM, Bakos RS, Maciunas RJ, Bani-Hashemi

A (2001) Augmented reality visualization of brain structures with

stereo and kinetic depth cues: system description and initial eval-

uation with head phantom. Proc SPIE 4319:445456

Acta Neurochir (2013) 155:943947 947


6/6

Copyright of Acta Neurochirurgica is the property of Springer Science & Business Media B.V. and its content

may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express

written permission. However, users may print, download, or email articles for individual use.

image projection.pdf

Documents