some usability issues of augmented and mixed reality for e-health applications in the medical domain...

Some Usability Issues of Augmented and Mixed Reality

for e-Health Applications in the Medical Domain

Reinhold Behringer

Johannes Christian

Andreas Holzinger

Steve Wilkinson

Leeds Metropolitan University

Innovation North – Faculty Of Information And Technology Slide # 2

Purpose of this Paper

Highlight AR and MR applications in the Medical Domain.

Point to usability issues. Not a complete survey.



Augmented Reality (AR) Principle:

Computer generates output, which is fused with the human perception of the environment.

Appears to emanate from the environment itself, to be a part of it – registration with real environment.

Visual: 3D computer graphics, employing Virtual Reality

(VR) concepts. Acoustic:

Spatial audio.



Mixed Reality (MR)

Real environment is not necessarily the basic framework.

Virtual Environment (VE) is the main framework.

Objects in MR can be computer-generated or real.



AR and MR in Medical Domain

AR: Data visualisation directly on patient. Guidance to physician for medical procedures.

MR: Simulation and training. Patient itself can be simulated.

Benefits: improved situational awareness



Display Types for AR AR fusion of visual sense:

Head-worn displays: Ideal merging of computer output with visual

impression of surroundings. Hand-held displays:

Either acting as “frame into the real world”, or Showing representation of real world by video /

images, onto which information is overlaid. Projective displays:

Project information directly onto object (patient). 3D is only correct for one single user at a time. Requires consideration of projection surface.



Tracking of User

User’s viewing position and orientation needs to be tracked for correct display rendition.

Technologies: Active illumination outside of human-visible

spectrum (e.g. IR). Computer vision approaches, capturing either:

Environment as seen from user (head-worn camera), or

User himself, seen from cameras in the environment.



Examples of Applications in Medical Domain

Data projection onto patients. Support of surgery. Simulation tools for teaching and practising. Therapy and rehabilitation.



Visualisation of Ultrasound and CT

UNC, A. State (1992). Live image stream of ultrasound scans

onto body: pre-natal care, obstetric examination.

Initially 2D scans, later volumetric scans. Technique also used for needle biopsies

and minimally invasive surgery.

Stockmans (2005). Correction of bone disformities. Comparison of before-after procedure.

UNC AR demonstration



Simulation Sielhorst et al., TU Munich (2004)

Birth simulator, using MR techniques. Real object: model of torso. Graphical simulation: baby. Birth pliers as haptic interface for interaction.

Nestler et al., TU Munich (2007) Virtual patients for large-scale disaster training. Table-top display, individual patients with different injuries.

Wilkinson (2005) Using purely VR for simulation of surgical procedure on

hand. Goal: to educate patients and reduce fear.



Aiding Surgery Not yet practical in clinical applications, but

promising improvements of spatial orientation, allowing more radical operative therapy.

MEDARPA project: AR and VR supporting minimally invasive surgery, to aid

the navigation of the surgeon (bronchoscopy and brachytherapy).

Future use with surgical robot.

Reitinger, TU Graz (2005) Virtual liver surgery planning system, aiding in providing

more precise measurements for tumor treatment.



Therapy, Rehabilitation

Using VR to cure phobias. Learning in controlled graphical environment how

to deal with phobia-causing situations. Rehabilitation of motoric skills:

AR system evokes motor images, Guides motoric practise

Healthy living: “Persuasive Mirror” shows future projection of

self, under influence of varying live style.



Edutainment

Teaching anatomy: “electronic book” for interactive display of 3D

anatomic perspective of humans. Organs can be viewed from different angles.

Simulation: To reduce fear of the unknown.



Registration and Tracking Requirements:

View of patient’s body needs to be unobstructed for the physician.

Tracking needs to take into account deformable tissue of human body.

Calibration effort needs to be small, so as not to distract from medical procedure.

Tracking needs to be resilient to occlusion. Markers for visual tracking can be attached to

human body. Magnetic tracking can provide seamless tracking of

physician.



Displays

Head-attached: Allows hands-free activity. Low resolution and viewing angle. Example: Variscope (Birkfellner, TU Vienna)

Hand-held: Is currently available at reasonable cost and

specifications. May interfere with actual procedures. Tablet display on boom:

Allows hands-free operation, while at the same time being cost effective.

Can be semi-transparent or video-see-through. Both display and physician need to be tracked for

correct visual view. Spatial.

MEDARPA display

Variscope



Interaction

Traditional interfaces (mouse, keyboard) are not suitable, as they distract from the task.

Preferable: Automatic ubiquitous interface, acting in the

background. E.g. speech recognition, gesture recognition. Possibly integrated into medical instrument.



User-Centred Development Potential benefits of AR/MR are obvious. But for studying usability, practical workflows need

to be studied. AR application developers need to understand the

workflows of medical professionals. Usability Engineering Methods (UEM) are becoming

more important, but are still not applied very often: Software engineers work on the implementation, usability

experts work on design. Rarely do they collaborate.



User-Centred Development

In most cases: Usability of AR environments is evaluated. E.g. Hix:

Criteria for assessing VE design. Usability of standard user interfaces for VE.

Effectiveness of different interactive devices. Needed:

User-Centred Development.



Presence and Errors in VE

Specific to VEs: Sense of presence (immersion). Assessed through questionnaires, focusing on a

person’s sense of presence. Presence is not associated with task performance

(Slater). But measures of presence have concentrated on

user perception of VE technology. Errors in VEs can impact significantly the

perception of presence



Immersion

Effects of being immersed: Investigated in a variety of VE system

configurations: Collaborative VEs, viewpoint of self, perception of

others’ presence. Fully immersive VEs have been well investigated,

focusing on head-worn displays. Missing:

Techniques for measuring presence. Questionnaire-based assessment may be biased.



Concerns about AR

Problematic: Information overload can be accompanied by

sensory overload. Overwhelming experience in AR environment. Possible lack of acceptance of this technology. Social effects (beliefs, attitudes, feelings).



Suggestions for Further Study Which are the most appropriate interaction

metaphors in AR for medical domain? Which influences do AR applications have on

performance of the end users in the medical domain?

What are the effects of adaptation that people might have to make cognitively to believe in and cope within an AR environment?

How will multimodal interaction through a number of input and output channels enhance or detract from the reality/Virtuality experience?



Specific to AR

AR must deal with physical interactions, social interaction, cognitive interactions.

Is difficult to isolate variables. Less performance constraints. Lower predictability of behaviors.



Successful AR Application All involved research domains have to be

considered. User-centered design focus is important. Need to be:

Accessible. Usable for everyday end users. Follow notions of pervasive and ubiquitous

computing. Implement basic ideas of social software. Designed for users without deep IT knowledge.



Summary

AR and MR have excellent potential in medical applications.

Usability issues need to be addressed in application design: Not only study usability in experience of VE

environments. Need to apply usability engineering methods in

design.



Acknowledgment

Johannes Christian Andreas Holzinger Steve Wilkinson

some usability issues of augmented and mixed reality for e-health applications in the medical domain...

Documents