Chapter 23

Combining BCI and Virtual Real-ity

: Scouting Virtual Worlds

Before a BCI can be used for control pur-poses, several training sessions are neces-sary

◦ Operant conditioning Feed back, real-time changes to the user

◦ Machine learning Adaptive algorithms to detect brain patterns

Introduction

A technology which allows a user to interact with a computer-simulated environment.

VR (Virtual Reality)

VE (Virtual Environment)◦ Allow users

to be shielded from the outside world to be able to focus on the required mental task

VR (Virtual Reality)

Then, Why VR for BCI?

Feedback

Visualization vs. VE feedback

Ron Angevin et al. (2004)◦ Control group (standard BCI

feedback) reacted faster◦ VR group achieved less error

Basic principle◦ detection & classification of motor-imagery re-

lated EEG patterns

Sensorimotor rhythms are analyzed◦ C3, Cz and Cz

Graz-BCI

VE◦ To let a user become immersed in a 3D scene

Cave◦ Multiprojection stereo-based head-tracked VE sys-

tem

Graz-BCI + VE

3D virtual environment1. Creation of a 3D model of the scene

3D modeling Software Packages Performer Maya

2. Generation of a VR-application that controls and animates the modeled scene

Virtual Research V8 HMD 640 X 480 pixels, refresh rate 60Hz

Vrjuggler + single back-projected wall + shutter glasses

Cave-like system

Graz-BCI + VE

BCI experiments◦ Require a subject in a sitting position

No positional information had to be considered

◦ Rotational information from the tracking system was ignored Rotation should be controlled by the BCI

Graz-BCI + VE

Study 1Roation in a VE by Left- and Right-Hand Motor Imagery

Imagination of left and right hand movement

Subjects◦ 2 male(23, 26 years old), 1 female(28 years old)◦ 7 months

Feedback conditions1. A standard horizontal bar graph on a desktop moni-

tor2. Virtual conference room presented with an HMD3. Virtual pub populated with animated avatars (in-

cluding music and chatter of the avatars)

Experiment

The order of feedback conditions◦ Bar graph → HMD → Cave → HMD → bar graph

Instruction◦ To imagine left or right hand movements depending

on an acoustic cue (single or double beep)

Control◦ Either the length and the orientation of the horizon-

tal bar graph (in case of the standard BCI feedback)◦ Rotation angle and direction within VR

Experiment

During experiments◦ Cue at second 3◦ Feedback for 4s◦ Screen update (including rotation) 24 times/s◦ One run 40 trials

Experiment

Results

No difference between HMD and Cave

Performed well with VR than bar graph

Results

Study 2Moving Forward in a Virtual Street by Foot Mortor Im-agery

Imagination in the virtual street◦ Right hand movement: to stop◦ Foot movement to move (constant speed)◦ Walking distance is scored

CAM (Cumulative achieved mileage)

Male 23, 28 and 30 years old.

Experiment

Results

Study 3, 4Scouting through a Virtual Apartment

Asynchronous freeSpace Experi-ments

Virtual apartment on a single back-projected stereoscopic wall

Subject could decide freely where to go along predefined pathways (through the corridors or rooms)◦ Turn right, left, or straight)

System automatically guided the subject to the next junction◦ Small map in the bottom right corner of the display

Experiment

Results

Using VR◦ High classification accuracy (low error rate) can be achieved.

Subjects felt more natural in VE compared with BCI experi-ments with standard feedback

Each subject preferred the Cave experiments to the HMD and both were favored over BCI session on a desktop PC

Motivation seems to improve the BCI performance, but too much excitement might also distract the subject

Despite distraction from auditory and moving visual stimuli in VE, motor imagery and its classification in the ongoing EEG is still possible

Conclusion

Thank you !