J A C C : B A S I C T O T R A N S L A T I O N A L S C I E N C E V O L . 3 , N O . 3 , 2 0 1 8

ª 2 0 1 8 T H E A U T H O R S . P U B L I S H E D B Y E L S E V I E R O N B E H A L F O F T H E AM E R I C A N

C O L L E G E O F C A R D I O L O G Y F O UN DA T I O N . T H I S I S A N O P E N A C C E S S A R T I C L E U N D E R

T H E C C B Y - N C - N D L I C E N S E ( h t t p : / / c r e a t i v e c o mm o n s . o r g / l i c e n s e s / b y - n c - n d / 4 . 0 / ) .

STATE-OF-THE-ART REVIEW

Emerging Applications of VirtualReality in Cardiovascular Medicine

Jennifer N.A. Silva, MD,a,b Michael Southworth, MS,b Constantine Raptis, MD,c Jonathan Silva, PHDb

SUMMARY

ISS

Fro

BiocD

Dis

ha

ser

Bo

pa

Ro

All

ins

vis

Ma

Recently, rapid development in the mobile computing arena has allowed extended reality technologies to achieve per-

formance levels that remove longstanding barriers to medical adoption. Importantly, head-mounted displays have

become untethered and are light enough to be worn for extended periods of time, see-through displays allow the user to

remain in his or her environment while interacting with digital content, and processing power has allowed displays to

keep up with human perception to prevent motion sickness. Across cardiology, many groups are taking advantage of

these advances for education, pre-procedural planning, intraprocedural visualization, and patient rehabilitation. Here,

we detail these applications and the advances that have made them possible. (J Am Coll Cardiol Basic Trans Science

2018;3:420–30) © 2018 The Authors. Published by Elsevier on behalf of the American College of Cardiology Foundation.

This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

F or many years, extended reality technologieshave promised physicians the ability to movebeyond 2-dimensional (2D) screens, allowing

them to understand organ anatomy in 3-dimensions(3D) noninvasively. However, this promise has beenstymied by bulky equipment that was incapable ofdisplaying high-quality virtual images coherentlyenough to prevent user motion sickness. Recent ad-vances in high-resolution display technology, expo-nential increases in computational power, andminiaturization of components led by mobile devicemanufacturers have enabled a new class of headmounted display (HMD) devices (1). These low-cost,comfortably-worn devices can display high-qualityclinical data at response times that are fast enoughto be used for extended periods of time, overcoming

N 2452-302X

m the aDivision of Pediatric Cardiology, Washington University Schoo

medical Engineering, Washington University School of Engineering a

epartment of Radiology, Washington University School of Medicine, St. L

covery Institute Grant CH-II-2017-575. Abbott has provided research supp

s received research support from Medtronic, St. Jude Medical, Abbott, and

ve on the Board of Directors of and are consultants for SentiAR. Dr. Southw

ard Director of and consultant for SentiAR. Dr. Raptis has reported that he

per to disclose. Drs. Jennifer N.A. Silva and Southworth contributed equ

berts, MD, served as Guest Editor for this paper.

authors attest they are in compliance with human studies committe

titutions and Food and Drug Administration guidelines, including patien

it the JACC: Basic to Translational Science author instructions page.

nuscript received August 4, 2017; revised manuscript received Novembe

longstanding barriers to adoption in the medicalcommunity.

Advances in digital light projection, organic lightemitting diode, and optics manufacturing haveresulted in thinner, lower-power, and brighterdisplay systems (2). Speech recognition and gener-ation advancements brought the earliest forms ofaugmented aural reality; the online digital assistantnow known as Siri (Apple, Cupertino, California) orGoogle’s assistant (Google, Mountain View, Califor-nia) are in use daily, along with automated tran-scription systems. Sensor technology advancementsin positioning and navigation systems originallydesigned to function with the global positioningsystem have been extended to include satellite-freeindoor navigation, tracking user position via their

https://doi.org/10.1016/j.jacbts.2017.11.009

l of Medicine, St. Louis, Missouri; bDepartment of

nd Applied Science, St. Louis, Missouri; and the

ouis, Missouri. This work was funded by Children’s

ort (software) for this project. Dr. Jennifer N.A. Silva

AliveCor. Drs. Jennifer N.A. Silva and Jonathan Silva

orth is a SentiAR shareholder. Dr. Jonathan Silva is a

has no relationships relevant to the contents of this

ally to this work and are joint first authors. Robert

es and animal welfare regulations of the authors’

t consent where appropriate. For more information,

r 16, 2017, accepted November 22, 2017.

AB BR E V I A T I O N S

AND ACRONYM S

AR = augmented reality

CGH = computer-generated

holography

EAMS = electroanatomic

mapping system

FOV = field of view

HMD = head-mounted display

HVS = human visual system

IMU = inertial measurement

unit

MeR = merged reality

MxR = mixed reality

SLM = spatial light modulator

VAC = vergence and

accommodation conflict

VR = virtual reality

J A C C : B A S I C T O T R A N S L A T I O N A L S C I E N C E V O L . 3 , N O . 3 , 2 0 1 8 Silva et al.J U N E 2 0 1 8 : 4 2 0 – 3 0 VR in Cardiology

421

mobile device by leveraging software and hardwaresuch as Project Tango and ARCore (Google) or ARKit(Apple). Eye and hand tracking provides new hu-man machine input capabilities for understandingnatural intent with less burden on the user to un-derstand the language of a specific manufacturer.This combination of hardware and software inno-vation has enabled new classes of 3D platforms.

Based on these advances in 3D platforms, thenumber of clinical applications has grown exponen-tially in the areas of education, pre-procedural plan-ning, rehabilitation, and even intraproceduralvisualization. Here, we focus on the application ofvirtual reality (VR) and related technology for clinicalcardiac practice, focusing on what is possible basedon current technology and what barriers still exist forwidespread adoption.

DEFINING REALITY

Extended reality describes the spectrum, or “virtual-ity continuum” (3) from fully immersive, curateddigital experiences in VR, to unobtrusive annotationswithin easy access of the operator in augmented re-ality (AR) (Table 1). It encompasses 2D annotations onreal-time video, 3D models, and true interference-based holograms, like animated versions of thoseseen on baseball cards. Although most headsets referto their models as “holograms,” HMDs typicallycreate the perception of depth for 3D models throughstereoscopy, simulating depth without generatingtrue holograms.

VR provides complete control over the wearer’svisual and auditory experience as they interact withina completely synthetic environment. This controlover the environment can provide virtual experiencesof either subdued or amplified versions of reality.Commercially available VR platforms from Oculus,HTC, and Sony, among others, use high-resolutiondisplays to fully replace the wearer’s visual field.These immersive displays have been applied to painmanagement (4), exposure therapy (5), stroke reha-bilitation (6), education, and surgical planning (Sur-gical Theater).

Conversely, AR allows the wearer to see theirnative environment while placing 2D or 3D imageswithin it through a “window-on-the-world” (3). Thisannotated window-on-the-world can be displayed onan unobtrusive HMD or on a mobile device, using theonboard camera to provide a live view of the envi-ronment. Perhaps the most successful consumerapplication of AR technology has been Pokémon Go(7) in which a mobile phone camera feed was anno-tated with avatars and contextual game data. These

AR applications minimally interfere with thenormal field of vision, providing useful in-formation only when called upon by the user.In the medical setting, contextually relevantgraphics, reference data, or vital informationis presented alongside (rather than in placeof) the physical surroundings. The first, andmost widely publicized commercial platform,Google Glass for example, was shown todisplay patient vital signs, relevant history,and prescription information from a patient’selectronic health record during a visit (8).More recently, other platforms have beendeveloped for education, patient point of care(Evena [9]), emergency response, and tele-medicine (AMA Xperteye [10]).

VR and AR denote the 2 bookends of thecontinuum of experiences, and as the in-dustry has grown, 2 new classes of experi-

ences have emerged: merged reality (MeR) andmixed reality (MxR). Both approaches achieve asimilar experience: to allow for interaction withdigital objects while preserving a sense of presencewithin the true physical environment. MeR capturesa user’s surroundings and re-projects them onto on aVR-class HMD, which can mediate the environmentup or down as desired. This allows for a moreseamless transition between mediated and unmedi-ated virtuality and reality. For consumers, this isportrayed as the ability to transport users to acompletely different room and back to their livingroom with the same device (Intel Alloy [11]), whichcould also be applied to patients in hospital rooms.MxR accomplishes a similar experience by projectingdigital objects onto a semitransparent display. Assuch, the MxR platforms do not obscure, or mediate,the physical environment, allowing the wearer tomaintain situational awareness of their surroundingsas well as maintain normal interactions with thosenot participating in the MxR experience. Thisadvance has opened a window of opportunity forthis type of technology for intraprocedural use,allowing physicians to remain in their environmentwhile viewing the virtual image (Central Illustration).Currently, MxR displays commercialized by Micro-soft and DAQRI (Los Angeles, California) have beendemonstrated in medical education and medicalimaging (12).

Several additional prominent 3D display platformshave been developed without using an HMD,including 3D flat-panel displays, and interference-based computer-generated holography. Flat-panel3D displays, first introduced in 2010 (13), were pri-marily displays only and lacked input devices for

TABLE 1 The Extended Reality Spectrum

Virtual Reality Merged Reality Mixed Reality Augmented Reality

Interactive virtualobjects

Interactive virtualobjects

Interactive virtualobjects

Virtual objects

Virtual background True background True background True background

Immersive display Immersive display See-through display See-through display

Fully Immersive Experience Unobstructed Experience

Silva et al. J A C C : B A S I C T O T R A N S L A T I O N A L S C I E N C E V O L . 3 , N O . 3 , 2 0 1 8

VR in Cardiology J U N E 2 0 1 8 : 4 2 0 – 3 0

422

manipulating 3D data. Hewlett Packard’s (Palo Alto,California) Zvr 3D display and input device is used inconjunction with EchoPixel’s (Mountain View, Cali-fornia) software to provide diagnostic quality images(14). A second example, computer interference-basedholography, generates realistic 3D images by shapinglight waves using a combination of complex pro-cessing, specialized computer-controlled light sour-ces, and optics. Although real-time display ofholograms was first demonstrated in 1992 (15), recentadvances by RealView imaging have enabled theirpractical use, and they have explored several clinicalapplications using this technology, including cardi-ology (16,17).

CARDIAC APPLICATIONS OF

VIRTUAL REALITY

Various cardiac applications of virtual reality aredepicted in Figure 1.

EDUCATION. Extended reality provides a wide rangeof possibilities for educational and training applica-tions. Some applications leverage the immersion thatVR enables to simulate the entire operating environ-ment along with the educational material. Anotherclass of applications brings the existing medical sim-ulations for tablets and mobile phones to VR as thenext platform that trainees will have access to. TheseVR-use cases are generally available across mostconsumer VR platforms. Other applications takeadvantage of the presence of MxR to allow multiplewearers to interact and discuss with each other whileviewing the same educational material in a naturalenvironment. These applications rely on the viewthrough nature of MxR, combined with the freedomof untethered headsets to walk around and commu-nicate naturally.Stanford v i r tua l heart . The Stanford Virtual HeartProject (18), working with Lighthaus, Inc., uses animmersive VR headset for educational purposes.This project has a few distinct arms. The first isgeared at patient and family education to helpfamilies better understand their child’s cardiacanatomy, which is currently limited to drawings and

plastic models. This improved depth of under-standing should help parents better participate intheir child’s complex medical care. This applicationhas been expanded to Stanford medical studentsand trainees, who can visualize normal andabnormal anatomies and understand how congenitalanomalies affect physiology. Using a fullyimmersive VR headset, the students can inspect,manipulate, and walk through the models,providing a more complete understanding of theanatomy and physiology. A library of approximately2 dozen common congenital lesions is available tothe trainees. The aim of these experiences is toprovide a deeper anatomic understanding of theselesions, improving the understanding and speed oflearning of these complex abnormal physiologiesand hemodynamic sequelae.

The final application is the use of a 3D monitor,Echopixel (as discussed later in the section “Pre-Procedural Planning”), in the cardiothoracic oper-ating room. A 3D workstation in the surgical suitemay allow for accurate assessments of intracardiacanatomy and geometry, which may be difficult to seeafter patients are placed onto cardiopulmonarybypass and the heart is decompressed.

HoloAnatomy. At Case Western Reserve University,investigators are using the HoloLens (Microsoft) tochange medical student education, particularlyanatomy (19). The ability to better understand 3Danatomic relationships not only eases the learningcurve, but also encourages students to “think like adoctor.” In conjunction with the Cleveland Clinic, theteam at Case Western Reserve University is devel-oping a curriculum, HoloAnatomy, that will allowmedical students to perform holographic dissectionsto better visualize and understand the body’s organsand systems. Preview versions of this software arefreely available to download.

PRE-PROCEDURAL PLANNING. EchoPixel . One ofthe first 3D displays to gain approval from the U.S.Food and Drug Administration is the True 3D systemthat has been developed by Echopixel, which is in-tegrated into a diagnostic grade DICOM workstation.The system provides 3D visualization using a tech-nique similar to that used in 3D movie theaters andearly 3D consumer televisions: by providing differentimages to each eye using specialized glasses. A singleEchopixel user wearing polarized glasses can addi-tionally manipulate the onscreen image using ahandheld wand. Initial cardiology studies includeusing the Echopixel system to visualize arteries inpatients with pulmonary atresia with major aorto-pulmonary collateral arteries. In this study (20),

CENTRAL ILLUSTRATION Mixed Reality Cardiac Electrophysiology Workflow

Silva, J.N.A. et al. J Am Coll Cardiol Basic Trans Science. 2018;3(3):420–30.

Mixed reality allows for the display and interaction with existing displays within the cardiac catheterization suite, including integration with fluoroscopy

(top left), electroanatomicmapping systems (top center), electrocardiograms (top right), aswell as previously acquired and computed tomography–or

magnetic resonance–derived 3-dimensional (3D) anatomic models (middle row). Although augmented reality platforms (bottom left) can show 2-

dimensional (2D)dataunobtrusively,mixedrealityplatforms (bottomcenter)allowforhands-free2Dand3Dvisualizationaswell asdirectsterilecontrol

of these data without otherwise obstructing the normal visual field, as in virtual reality (bottom right).

J A C C : B A S I C T O T R A N S L A T I O N A L S C I E N C E V O L . 3 , N O . 3 , 2 0 1 8 Silva et al.J U N E 2 0 1 8 : 4 2 0 – 3 0 VR in Cardiology

423

FIGURE 1 Emerging Applications of Extended Reality Applications

The x-axis is a condensed representation of the virtuality continuum, with virtual reality

(VR) on the left, mixed reality (MxR) and merged reality (MeR) in the center, and

augmented reality (AR) on the right. The y-axis categorizes the applicability of each

solution. Intraprocedural applications are the most sensitive platform type, and

currently are only being developed for MxR.

Silva et al. J A C C : B A S I C T O T R A N S L A T I O N A L S C I E N C E V O L . 3 , N O . 3 , 2 0 1 8

VR in Cardiology J U N E 2 0 1 8 : 4 2 0 – 3 0

424

cardiologists evaluated patients who had undergonecomputed tomography angiography either by usingthe 3D display or a traditional readout. Cardiologistsusing the True 3D display had interpretation times of13 min compared with 22 min for those that used atraditional display. Both groups were similarly accu-rate in their interpretations when compared to cath-eter angiography.

The True 3D display is also being used as part of theStanford Virtual Heart Project (see the previous sec-tion, “Education”).

INTRAPROCEDURAL VISUALIZATION. Enhancedelect rophys io logy v isua l i zat ion and interact ionsystem (Pro ject �ELVIS) . Currently, visualization inthe electrophysiology laboratory relies on a combi-nation of fluoroscopy, electroanatomic mapping sys-tems (EAMS), and echocardiography (intracardiacechocardiography and transesophageal echocardiog-raphy), with most laboratories using EAMS plus othertools. Although improvements in visualization havebeen a source of research and development over theyears, there have not been equal gains in improve-ments in interaction. Our prototype, the EnhancedELectrophysiology and Interaction System (�ELVIS),not only empowers the interventional electrophysi-ologist to visualize patient-specific 3D cardiac

geometry with real-time catheter locations, but alsoallows direct control of the display without breakingsterility, which is a key advance (21).

�ELVIS can display data that is exported from anEAMS or that is obtained pre-procedurally viacomputed tomography or cardiac magnetic resonanceimaging. To date, we have connected �ELVIS to theEnSite Velocity EAMS (Abbott, Abbott Park, Illinois)via the CoHesion module to display electroanatomicdata, including cardiac geometry, catheter localiza-tion data, and electroanatomic maps, including localactivation time and voltage maps. Most recently, wehave demonstrated the ability to display historicalcases for review, as well as a live case observed inreal-time from the control room. In addition to theability to improve visualization, the system allowsthe user to utilize gesture, gaze, or voice control forsterile control of the display. This improved interac-tion allows the interventionalist to directly controlthis single cohesive model in manner that is optimalfor a given procedure.

Sharing functionality provides a single sharedcardiac holographic model for as many as 5 users withthe model remaining fixed in the room, allowing allusers to visualize the model from their vantagepoints. Using the sharing system, there is a singleperson in control of the system at any given time withthe ability to pass controller privileges to other users.Supplemental Video 1 demonstrates the currentfunctionality of the prototype, including sharing,gesture control, and the display of intracardiac ge-ometry and catheter movement that were obtainedfrom the EnSite system.

Realv iew. In 2016, the pediatric cardiology group atSchneider’s Children’s Medical Center partnered withRealview Medical Imaging (Yokneam, Israel) to assessthe feasibility of creating real-time 3D digital holo-grams in a standard cardiac catheterization laboratory(16). The Realview computer-generated holography(CGH) were created using 3D rotational angiographycoupled with 3D transesophageal echocardiography.A total of 8 patients were enrolled in this study,including patients with structural heart disease andpost-operative cardiac patients. In all patients, theteam generated real-time 3D holograms with highaccuracy (as measured by instructing 4 independentobservers to identify anatomic landmarks within thehologram and typical cardiac imaging) with “veryeasy” interactions including image marking, crop-ping, zoom, rotation, movement of hologram, andslicing. This is the first study of its kind to demon-strate feasibility within the cardiac catheterizationlaboratory.

FIGURE 2 Simplified Image Formation Diagrams for the HVS and Common Extended Reality Displays

Sample extended reality optics configurations for extended reality displays for visualizing a gray sphere at the top of each diagram, as

perceived by an observer eye at the bottom of each diagram. (A) The normal human visual system (HVS) observes the sphere directly, and can

accommodate variable focal distances. (B) Virtual/merged reality systems magnify an opaque, distortion-compensated display through a

plano-convex type (shown) or Fresnel equivalent lens (not shown). (C) In reflective half mirror (“birdbath”) view through systems, light from

the display reflects off a semitransparent splitter and focus/combiner to the eye. (D) In computer-generated holography (CGH), an incoming

light wave front (right) is shaped through the spatial light modulator (SLM) (center) to match the wave front of the sphere from the correct

focal distance. (E) In polarizing reflector waveguides, the input display is coupled into the waveguide with a polarizing mirror and reflected

internally. Polarized reflectors selectively decouple light from waveguide to the eye. (F) In an array of half mirrors, light from the display is

internally reflected through the waveguide, and is reflected to the eye through an array of small, semi-reflective mirrors (shown as a bold

line) rather than an equivalent large mirror (not shown). (G) In diffraction waveguides, light is coupled from the display through diffraction

structures such as surface grating diffraction (SGD) shown or holographic optical element (HOE), not shown. Light is then reflected internally

through the waveguide and decoupled out through a corresponding output diffraction structure. Note that only the normal HVS (A) and CGH

(D) support multiple focal planes as shown.

J A C C : B A S I C T O T R A N S L A T I O N A L S C I E N C E V O L . 3 , N O . 3 , 2 0 1 8 Silva et al.J U N E 2 0 1 8 : 4 2 0 – 3 0 VR in Cardiology

425

REHABILITATION. MindMaze . MindMaze (San Francisco,California) is creating both hardware and software inthe VR space with a current medical applicationin neurorehabilitation. Their current solution,MindMotion PRO, is cleared by the U.S. Food andDrug Administration for use in post-stroke patients,combining virtual reality, brain imaging, and gamingtechnologies to retrain the brain to improve upperlimb mobility. In acute post-stroke patients, aclinical study has enrolled patients who engage in20- to 30-min sessions as soon as 4 days post-hospitalization without continuous supervision,with increasing training intensity over time (22). Allpatients had a positive user experience using theMindMotion PRO, with 90% reporting an improve-ment in movement capacity (22).

EXTENDED REALITY HARDWARE

Significant advances in extended reality devices haveenabled the previously mentioned applications, yet

the technology continues to evolve rapidly. In thefollowing text, we describe currently available sys-tems and future devices that will certainly expand thenumber of opportunities for extended realities toimprove patient care.

DISPLAYS. Typical displays project either a singlecoherent image to 1 or both eyes as a near-eye display,or in a stereoscopic pair for 3D simulation. Monocularsystems are either opaque or view through, and avoiddisruption of normal vision by positioning a smalldisplay outside of the normal working visual field.This allows a user to access the information containedon the display “at a glance.” Equivalent capabilitiescan be achieved using biocular displays, or the dis-plays can provide 2 different images to provide ste-reoscopic image pairs. Most HMD platforms generateimage pairs by using a panel display or small projectorand a series of optics to enlarge the image andsimulate projection at a distance to ease focusing, asillustrated for several common systems in Figure 2.For a given projected distance and resolution, as the

FIGURE 3 Positional Tracking Systems

The perception of depth is influenced by the distance at which the eyes converge and the focal distance of the eye, referred to as vergence

(left) and accommodation (right), respectively. 3-dimensional displays, which rely on stereo vision, can only influence vergence by adjusting

the stereo disparity between the left and right eye, and the focal plane of the optics remains fixed. The eyes (bottom) perceive the vergence

depth of the image of the sphere on the displays by the distance of the intersection of the displayed images. On the right, the accom-

modation distance is the distance to the focal plane at which the cube is in focus. Objects at closer and farther focal distances appear out of

focus. Vergence and accommodation conflicts cause discomfort due to the disagreement in vergence and accommodation distances,

particularly at close distances where accommodation cues are most influential.

Silva et al. J A C C : B A S I C T O T R A N S L A T I O N A L S C I E N C E V O L . 3 , N O . 3 , 2 0 1 8

VR in Cardiology J U N E 2 0 1 8 : 4 2 0 – 3 0

426

apparent display and field of view (FOV) is increased,apparent pixel size is also increased.3D display . Most 3D displays use either active orpassive shutters to simulate stereoscopic 3D withouta near-eye display. These displays simulate depthsimilarly to head-mounted displays, by projecting adisplaced image to each eye, controlled by eithershutters or polarization in the glasses to provide theperception of depth. These displays can be used fornavigating and interacting with medical images in aconventional office setting using a stylus for interac-tion (EchoPixel, Hewlett Packard’s Zvr). Current dis-plays are readily available and can be integrated intohospital workflows; however, disadvantages includethe required use of glasses, the capacity to supportonly a single user at a time, and a limited depth vol-ume for tracking and display.L ight field d isp lays . Light field displays use theprojection of light directly onto the retina. The pri-mary advantage of these techniques is that theyencode both the position of light and the angle,providing a more realistic image by recreating depth.True light field displays require much more compu-tational capacity to render a point in space andgenerate the bundles of light necessary. The AvegantLight Field display (Belmont, California) is not yetcommercially available and is not a true, full, lightfield display, but circumvents this complexity by ac-commodating a limited number of fixed focal depths.Displays of this type fundamentally trade spatial ortemporal resolution in the viewing plane for

resolution in the depth plane in addition to FOV oftraditional stereo HMDs.In terference-based holography. Holographic dis-plays refer to interference-based holography, gener-ated using a spatial light modulator (SLM) to create ahologram in space, at video framerates. The SLMshapes the incoming reference light to replicate thewave front that would originate from a real object atthe appropriate position in 3D space, creating a true,multifocal, 3D hologram. This approach is being usedby Realview to display 3D images of the heart(described in the previous text). The process ofgenerating an interference pattern for a given 3Dmodel and generating instructions for the SLM iscomputationally complex, but the resulting hologrambest satisfies the requirements of the human visualsystem. In general, CGH, however, is restricted byrender volume, viewing angle, and brightness.Waveguides . There are many different variants ofwaveguide-based displays (23), but all fundamentallyrely on reflecting the output of a display throughreflection to a view-through display in front of theuser’s eye. The fundamental tradeoff of these dis-plays is that the cost and complexity of a given designincreases as FOV increases. The relatively uncompli-cated design of the reflective half mirror, utilized bydevices like the ODG R-9 (San Francisco, California),uses a single reflective half mirror and a reflector/combiner, first developed for aircraft head-up dis-plays. Improvements on this design rely on minia-turizing the mirror using embedded polarizing

J A C C : B A S I C T O T R A N S L A T I O N A L S C I E N C E V O L . 3 , N O . 3 , 2 0 1 8 Silva et al.J U N E 2 0 1 8 : 4 2 0 – 3 0 VR in Cardiology

427

reflectors (Lumus, Rechovot, Israel), arrays of micro-structure mirrors (Optinvent, Monte Sereno, Califor-nia), and diffractive etched surface (HoloLens,Microsoft; Vuzix, Rochester, New York) or holo-graphic gratings (Digilens, Sunnyvale, California;WaveOptics, Oxfordshire, England). Microstructuremirrors provide a compact, cost-effective view-through display, whereas diffractive waveguides,although currently more expensive to produce, canachieve complex optical systems in a relatively thinpackage. These view-through displays allow a clini-cian to visualize data in either 2D or stereo 3D withoutotherwise obstructing the normal visual field.

VIRTUAL INTERACTION. The breadth and depth ofinteractions with these different display platformshave expanded as sensor and processing capabilitieshave improved, ranging from marker-free tracking toneural and voice inputs. The simplest form of inter-action common to most HMDs is through the move-ment of the head, measured by accelerometers in thedisplay. This angular movement updates the displaybased on the direction the head is pointing, and adisplay cursor representing the center of the user’sfocus is rendered in a fixed position relative to thedisplay. This gaze cursor is usually rendered on thevertical midline of the display, although not neces-sarily in the center of the FOV depending on themanufacturer and ergonomics. If the gaze cursor is onan interactive element, such as a button, there aregenerally 2 methods of interaction with it. The firstgaze-based interaction is referred to as gaze-dwell,and is triggered by holding the gaze cursor on theinteractive element for an application-determinedamount of time. If the HMD is equipped with inter-nal or external hand tracking hardware, the usercould also initiate activation of the interactiveelement using a recognized hand gesture, which incombination with gaze is referred to as a gaze-gesturecommand.

In addition to these gaze-based commands, somedisplays can recognize more complex hand gesture orcontroller gesture commands by tracking the hand ora controller in the hand. Controllers have the addedbenefit of supporting physical buttons in place tofurther enhance interaction. Finally, most platformssupport a microphone to support varying levels ofvoice command automation. Although gesture inter-action offers a rich, efficient means to interact withthe environment, voice and gaze-dwell inputs havethe advantage of being hands-free, and controllerslose the advantage of being touch free.

SENSORS. Compact inertial measurement units(IMUs), optical tracking, depth sensing cameras (Intel

Real Sense, Microsoft Time of Flight) and voicerecognition (Google Assistant, Microsoft Cortana)provide the basis for navigation and control for mostplatforms. Most commercial VR displays use headset-mounted or external-fixed optical tracking (Optitrack,Corvallis, Oregon; Polaris, Medina, Minnesota; Sco-pis, Cambridge, Massachusetts) to provide accuratepositioning and tracking within a volume, known as“outside-in” tracking (Figure 3, left). AR platformsmay rely on the same optical tracking, or use onboardcameras to track fiducial markers within the envi-ronment. IMUs provide high-resolution angle andrate information for updating the view on screen,which is coupled with optical tracking information toprovide correction updates to the tracking system,known as “inside-out” tracking (Figure 3, right).Network communications can provide access tohighly trained voice recognition algorithms, orsimpler local models can be used for specific com-mands. Near-eye pupil tracking (Pupil Labs, Berlin,Germany) cameras are available for simple tracking asan input device or as an enhancement to gestureinput. Ultrasound arrays (Ultrahaptics, Bristol,England) can provide sterile haptic feedback throughfree air by using ultrasonic speakers to inducesensation on skin. This can help overcome the lack ofhaptic feedback when interacting with digital objects.

CHALLENGES

The applicability of different modalities of extendedreality to education, pre-procedural planning, intra-procedural guidance, and therapeutic use depends ontheir inherent advantages and limitations, particu-larly with respect to isolation from the natural phys-ical environment. VR and MeR completely occludethe normal visual field, whereas AR and MxR enhancethe visual field. This is most apparent when power isremoved from the display; VR and MeR arecompletely opaque and must be removed to be able tosee, whereas AR and MxR are transparent and onlythe digital additions are removed. VR, for example,can allow an individual student to interact in a fullyimmersive simulation, isolated from outside distrac-tion. However, AR, MeR, and MxR will allow inter-ventionalists to maintain presence in a physical roomto perform procedures as well as maintain the abilityto interact with the patient and cooperate with theirsupporting personnel. MeR platforms, however, pre-sent a potential safety risk during a procedure in theevent of power loss, which would cause completeobstruction of the normal visual field.

Extended reality platforms are constrained pri-marily by cost, size, weight, and power to achieve

TABLE 2

Human ey

60-inch 4

24-inch 1,

Oculus Rif

HoloLens

Meta2 (Mx

MxR ¼ mixe

FIGURE 4 Principles of Vergence and Accommodation

Extended reality platforms rely on positional tracking systems

to track head mounted display, hand, and instrument positions.

Tracking systems are either external (“outside-in”) systems

(left) or on-board (“inside-out”) systems (right). Outside-in

optical systems use visual and time-of-flight cameras to track

the relative angles to triangulate positions and poses of active

or passive marker arrays mounted to the headset, controllers,

and tools. Inside-out tracking systems track the relative an-

gles, hands, tools, and landmarks to triangulate the position

and pose of the headset relative to landmarks, and the position

of hands and tools relative to the headset without any external

hardware.

Silva et al. J A C C : B A S I C T O T R A N S L A T I O N A L S C I E N C E V O L . 3 , N O . 3 , 2 0 1 8

VR in Cardiology J U N E 2 0 1 8 : 4 2 0 – 3 0

428

the highest visual quality, mobility, processingspeed, and interactivity. Visual quality is dependenton resolution, brightness, focal depth, and FOV.Display technology is the most demanding aspect ofextended reality, and is generally the largest designand cost constraint (23). For 3D systems, this iscompounded by requiring stereoscopic pairs of im-ages to generate the perception of depth throughvergence, or the angularity disparity between 2 dis-plays (Figure 4, left). The display system mustcompromise size and cost with providing themaximum visual quality to match the capabilities of

Technical Resolution and Field of View for Various Displays

Angular Resolution(arc-min)

Diagonal Field ofView (o) Tethered

e (20/20) 1.0 w150

K UHD @ 40” 1.0 73.7

080P @ 24” (ZvR) 1.4 53.1

t (VR) 4.9 132.2 Yes

(MxR) 1.4 34.5

R) 2.8 90.0 Yes

d reality; UHD ¼ ultra high-definition; VR ¼ virtual reality.

the human visual system (HVS). The lower bound ofa normal human visual acuity (roughly 20/20 or 6/6)is 1 arc min/pixel with an approximately 150o to 170o

by 135o to 150o elliptical FOV (24–26). A displaysystem that achieves this angular resolution containspixels that are considered indistinguishable,commonly referred to by Apple as a Retina Display.This equates to a roughly 9,000 by 8,100 pixel/eyerequirement to fully emulate and immerse the hu-man visual system. By comparison, 4K HMD displayscontain 3,840 � 2,160 pixels and commonly requireworkstation class graphics for processing. Under-standably, this resolution and FOV is currently noteconomically feasible with current optics anddisplay technologies. As a result, device manufac-turers compromise FOV, pixel density, and displaybrightness to achieve the optimum capabilities for agiven application (Table 2). Cost, size, weight, andpower will only decrease as technology advances, asdemonstrated by the recent release of the AppleiPhone X (27), which contains miniaturized, lower-power versions of the depth sensors, IMU, and pro-cessing required for advanced, handheld AR.

Depth at close distances is also perceived throughaccommodation (Figure 4, right), or the perception ofdepth due to disparity in focal depth, which is achallenge facing all conventional stereoscopic dis-plays. Accommodation is required to allow a user tofocus on instruments and digital objects at the samesimulated distance (28) (e.g., surgical guidanceoverlays, within “personal space” and “actionspace”). Disparity between vergence depth displayedby HMD and accommodation expected by HVS isreferred to the vergence and accommodation conflict(VAC) and is responsible for discomfort at these closeworking distances (29). Most display systems onlysupport a single, fixed, focal plane for all digital ele-ments (Figure 4), although some emerging technolo-gies can provide multiple fixed focal planes byemploying adaptive optics.

PLATFORM DIFFERENTIATION

Every design decision to mitigate these challengesaffects applicability for use in each procedural envi-ronment. AR and MxR displays provide the bestcompromise between digital annotation and clearFOV. The high pixel density, large FOV devicesdesigned for desktop or office use, require an umbil-ical to a high-powered workstation to support theprocessing required for their displays. These devicesprovide a larger digital display and a higher-resolution display, but at the expense of compli-cated setup before procedures and limited

J A C C : B A S I C T O T R A N S L A T I O N A L S C I E N C E V O L . 3 , N O . 3 , 2 0 1 8 Silva et al.J U N E 2 0 1 8 : 4 2 0 – 3 0 VR in Cardiology

429

maneuverability during procedures. Untetheredplatforms generally have a reduced FOV and requirebattery power, but allow for unrestricted movement.CGH platforms provide the most realistic, true holo-grams within close working distance, but still requirelarge supporting systems tethered to the display, andhave limited working volume. These CGH displaysand others that compensate for VAC are well suitedwhere near-field interaction between the digital andphysical is critical (e.g., projection of pre-proceduralimagery onto a surgical field). Systems that cannotsimulate accommodation can avoid discomfort byplacing digital objects farther away, where the HVS isless sensitive to disparity in accommodation. Plat-forms utilizing external optical tracking achieveconsistent tracking volumes but require additionalequipment and clear lines of sight between camerasand devices, which increases initial installation andmaintenance complexity. Mobile AR platforms withinside-out tracking of both position and gesturesprovide the most flexible platforms for intra-procedural use, and can mitigate VAC through carefulplacement of digital objects.

CONCLUSIONS

Rapid hardware advances driven by the revolution inmobile computing have finally brought devices thatare tractable for medical applications into existence.These devices have the potential to provide physi-cians with a sterile interface that allows them tocontrol 3D images. Early data show that this improvedvisualization will allow the physician to learn morequickly, interpret images more accurately, andaccomplish interventions in less time. These im-provements in physician performance based on betterinformation will most likely translate into lower-costprocedures and better outcomes for patients.

ADDRESS FOR CORRESPONDENCE: Dr. Jennifer N.A.Silva, Division of Pediatric Cardiology, WashingtonUniversity School of Medicine, 1 Children’s Place, CB8116 NWT, St. Louis, Missouri 63110-1093. E-mail:jennifersilva@wustl.edu. OR Dr. Jonathan Silva,Department of Biomedical Engineering, WashingtonUniversity in St. Louis, 1 Brookings Drive, CB 1097, St.Louis, Missouri 63108-1097. E-mail: jonsilva@wustl.edu.

RE F E RENCE S

1. Rubin P. The inside story of oculus rift and howvirtual reality became reality. Wired. May 20,2014. Available at: https://www.wired.com/2014/05/oculus-rift-4/. Accessed December 11, 2017.

2. Sisodiaa A, Bayerb M, Townley-Smith P, et al.Advanced helmet mounted display (AHMD). Procof SPIE Vol 2007:65570N–1.

3. Milgram P, Kishino FA. Taxonomy of mixed realityvisual-displays. Ieice T Inf Syst 1994;E77d:1321–9.

4. Li A, Montano Z, Chen VJ, Gold JI. Virtual realityand pain management: current trends and futuredirections. Pain Manag 2011;1:147–57.

5. Parsons TD, Rizzo AA. Affective outcomes ofvirtual reality exposure therapy for anxiety andspecific phobias: a meta-analysis. J Behav TherExp Psychiatry 2008;39:250–61.

6. Comstock J, MindMaze receives FDA clearanceto bring VR rehab platform to the US. August 3, 2017.Available at: http://www.mobihealthnews.com/content/mindmaze-receives-fda-clearance-bring-vr-rehab-platform-us. Accessed December 11, 2017.

7. Ahn SJG. Pokemon Go is AR’s foot in the door toour world. IEEE Spectrum: Technology, Engineer-ing, and Science News: IEEE Spectrum, 2017.Available at: https://spectrum.ieee.org/view-from-the-valley/consumer-electronics/gaming/pokemon-go-is-ars-foot-in-the-door-of-our-world.Accessed January 2018.

8. Schuman AJ. What’s new in “connected” medi-cal devices? Contemp Pediatr 2015;32:43–7.

9. EvenaMedical. Eyes-On Glasses 3.0. Available at:https://evenamed.com/eyes-on-glasses/. AccessedAugust 1, 2017.

10. AMA Xperteye. The Xpert Eye solution revolu-tionize the modern medicine. Available at: http://www.amaxperteye.com/us/. Accessed August 1,2017.

11. Intel. Intel unveils Project Alloy. Available at:https://newsroom.intel.com/chip-shots/intel-unveils-project-alloy/. Accessed December 11, 2017.

12. Kamps HJ. Touch Surgery brings surgerytraining to augmented reality. TechCrunch. Avail-able at: https://techcrunch.com/2017/01/06/touch-surgery-ar/. Accessed August 1, 2017.

13. Whitworth D, First 3D television sets go on salein UK. BBC. Available at: http://www.bbc.co.uk/newsbeat/article/10085219/first-3d-television-sets-go-on-sale-in-uk. Accessed December 11, 2017.

14. HewlettPackard. EchoPixel True 3D, poweredby HP. HewlettPackard 2016. Available at: http://h20195.www2.hp.com/v2/getpdf.aspx/4AA6-6086ENW.pdf. Accessed January 2018.

15. Lucente ME. Optimization of hologramcomputation for real-time display. SPIE Proceed-ing #1667, Practical Holography VI 1992:32–43.

16. Bruckheimer E, Rotschild C, Dagan T, et al.Computer-generated real-time digital holography:first time use in clinical medical imaging. Eur HeartJ Cardiovasc Imaging 2016;17:845–9.

17. Bruckheimer E, Rotschild C. Holography forimaging in structural heart disease. Euro-Intervention 2016;12 Suppl X:X81–4.

18. Stanford Children’s Health, Stanford LPCsH,Lucile Packard Children’s Hospital Stanford pio-neers use of VR for patient care, education andexperience. Available at: http://www.

stanfordchildrens.org/en/about/news/releases/2017/virtual-reality-program. Accessed December 11,2017.

19. Case Western Reserve, Cleveland Clinic.Case Western Reserve, Cleveland Clinic collabo-rate with Microsoft on ‘earth-shattering’mixed-reality technology for education. Availableat: http://case.edu/hololens/. Accessed December11, 2017.

20. Chan F, Aguirre S, Bauser-Heaton H, Hanley F,Perry S. Head tracked stereoscopic pre-surgicalevaluation of major aortopulmonary collateral ar-teries in the newborns. Paper presented at:Radiological Society of North America 2013 Sci-entific Assembly and Annual Meeting; Chicago,Illinois; 2013.

21. Silva JN, Southworth MK, Dalal A, VanHare GF, Silva JR. Abstract 15358: Improvingvisualization and interaction during transcatheterablation using an augmented reality system:first-in-human experience (abstr). Circulation2017;136:A15358.

22. Chevalley OH, Schmidlin T, Perez Marcos D,et al. Intensive upper limb neurorehabilitation withvirtual reality in chronic stroke: a case report.Paper presented at: Annual Meeting of AmericanSociety of Neurorehabilitation; Chicago, Illinois;October 15–16, 2015.

23. Kress B, Shin M. Diffractive andholographic optics as optical combiners inhead mounted displays. Proceedings of the2013 ACM conference on pervasive andubiquitous computing adjunct publication2013;1479–82.

Silva et al. J A C C : B A S I C T O T R A N S L A T I O N A L S C I E N C E V O L . 3 , N O . 3 , 2 0 1 8

VR in Cardiology J U N E 2 0 1 8 : 4 2 0 – 3 0

430

24. Strasburger H, Rentschler I, Jüttner M. Pe-ripheral vision and pattern recognition: a review.J Vision 2011;11:13.

25. Graham CH. Vision and visual perception. Ox-ford: Wiley, 1965.

26. Caelli T. Visual Perception: Theory and Prac-tice: Pergamon International Library of Science,Technology, Engineering and Social Studies.Elsevier, 2014:103–14.

27. Apple I, iPhone X–technical specifications–Apple. Available at: https://www.apple.com/iphone-x/specs/. Accessed November 1, 2017.

28. Cutting JE, Vishton PM. Perceiving layoutand knowing distances: the integration, relativepotency, and contextual use of different infor-mation about depth. In: Epstein W, Rogers S,editors. Perception of Space and Motion. SanDiego, CA: Academic Press, 1995:69–117.

29. Shibata T, Kim J, Hoffman DM, Banks MS. Thezone of comfort: predicting visual discomfort withstereo displays. J Vision 2011;11:11.

KEY WORDS augmented reality,cardiology, virtual reality

APPENDIX For a supplemental video,please see online version of the paper.