john howard (loook inc) ux/ui design for vr and mixed reality

Spatial UX and UI

John Howard LOOOK, Inc.john@loook.io

www.loook.io

Market Data

“We expect virtual and augmented reality to become an $80 billion market by 2025, roughly

the size of the desktop PC market today.”Goldman Sachs Research,

January 2016

UX / UI Evolution

1984 2014

Most UX/UI design is still focused on flat, 2D screens.

UX / UI Evolution

Microsoft

UX / UI Evolution

2D 3D

GUI Touch CAD Spatial

How do we make the jump from 2D to 3D?

The Dream

It’ll be just like Iron Man !!!

This eye thing is annoying.

The Reality

Iron Man 3, Marvel Studios / Paramount Pictures

My arms are tired.

Minority Report, DreamWorks

The Reality

This menu is actually behind me.

The Division, Ubisoft

The Reality

The Reality

“Maybe not like Iron Man…”

Fatigue UX should be tailored to the duration of the experience.

Where to anchor menus? Head-locked UX is annoying.

Occlusion and Clipping UI needs to respect the world.

Z-fighting Near/far focal length switching causes discomfort.

Vestibular discomfort AKA, simulator sickness.

Spatial UX/UI Challenges

UX/UI Toolbox

UX/UI Toolbox

User Gaze

Gaze Cursor

Voice

Spatial Audio / Haptic Audio

Context Sensitive Actions

Extended Desktop

Movement

Hand Tracking

The Spatial UX/UI Toolbox

UX/UI Toolbox

User Gaze

Gaze Cursor

Voice

Spatial Audio / Haptic Audio

Context Sensitive Actions

Extended Desktop

Movement

Hand Tracking

The Spatial UX/UI Toolbox

01: User Gaze

UX/UI Toolbox

Halo E3 2015, Microsoft

Halo E3 2015, Microsoft

01: User Gaze

UX / UI Toolbox

Caution! Aggressive or excessive event triggering with gaze is like pointing a loaded gun; users get gun-shy and it stresses them out.

01: User Gaze

02: Gaze Cursor

UX/UI Toolbox

Case Western Reserve University School of Medicine

CDM Smith / Object Theory

02: Gaze Cursor

Case Western Reserve University School of Medicine

02: Gaze Cursor

04: Spatial Sound

UX/UI Toolbox

RoboRaid, Microsoft

Case Western Reserve University School of Medicine / Microsoft

04: Spatial Sound

05: Haptic Audio

UX/UI Toolbox

Luka Mivsek

05: Haptic Audio

Autodesk / Microsoft

05: Haptic Audio

Operation, Hasbro

05: Haptic Audio

iStock

05: Haptic Audio

06: Context-Sensitive Actions

UX/UI Toolbox

Trimble / Microsoft

Case study: Indoor agriculture, LOOOK

06: Context Actions

NASA / Jet Propulsion Laboratory / Microsoft

06: Context Actions

MediSIM / Carnegie Mellon University

MediSIM / Carnegie Mellon University

UX / UI Toolbox

Caution! Context menus in densely populated or close-quarters space can result in environment clipping , occlusion or z-fighting.

Caution! Beware of too many nested fly-outs in pop-up menus. Consider gaze travel distance.

06: Context Sensitive Actions & Controls

07: Extended Desktop

UX/UI Toolbox

Autodesk / Microsoft

NASA / Jet Propulsion Laboratory / Microsoft

07: Extended Desktop

Autodesk / Microsoft

07: Extended Desktop

Trimble / Microsoft

07: Extended Desktop

UX / UI Toolbox

Caution! A mouse pointer should follow surfaces like the gaze cursor but will appear to "jump" when transitioning between extreme z-depth changes.

“Lerp” to avoid jarring transitions and display a ray from mouse to destination so all users can follow the transition.

07: Extended Desktop

In Conclusion…

User Gaze

Gaze Cursor

Voice

Spatial Audio / Haptic Audio

Context Sensitive Actions

Extended Desktop

Movement

Hand Tracking

The Spatial UX/UI Toolbox

In Conclusion…

“I still want it to be like Iron Man…”

Important!

Be like Tony Stark…

“Always Be Prototyping!”

Thank You.

John Howard LOOOK, Inc.john@loook.io

www.loook.io