john howard (loook inc) ux/ui design for vr and mixed reality
TRANSCRIPT
Market Data
“We expect virtual and augmented reality to become an $80 billion market by 2025, roughly
the size of the desktop PC market today.”Goldman Sachs Research,
January 2016
UX / UI Evolution
1984 2014
Most UX/UI design is still focused on flat, 2D screens.
UX / UI Evolution
Microsoft
UX / UI Evolution
2D 3D
GUI Touch CAD Spatial
How do we make the jump from 2D to 3D?
The Dream
It’ll be just like Iron Man !!!
This eye thing is annoying.
The Reality
Iron Man 3, Marvel Studios / Paramount Pictures
My arms are tired.
Minority Report, DreamWorks
The Reality
This menu is actually behind me.
The Division, Ubisoft
The Reality
The Reality
“Maybe not like Iron Man…”
Fatigue UX should be tailored to the duration of the experience.
Where to anchor menus? Head-locked UX is annoying.
Occlusion and Clipping UI needs to respect the world.
Z-fighting Near/far focal length switching causes discomfort.
Vestibular discomfort AKA, simulator sickness.
Spatial UX/UI Challenges
UX/UI Toolbox
UX/UI Toolbox
User Gaze
Gaze Cursor
Voice
Spatial Audio / Haptic Audio
Context Sensitive Actions
Extended Desktop
Movement
Hand Tracking
The Spatial UX/UI Toolbox
UX/UI Toolbox
User Gaze
Gaze Cursor
Voice
Spatial Audio / Haptic Audio
Context Sensitive Actions
Extended Desktop
Movement
Hand Tracking
The Spatial UX/UI Toolbox
01: User Gaze
UX/UI Toolbox
Halo E3 2015, Microsoft
Halo E3 2015, Microsoft
01: User Gaze
UX / UI Toolbox
Caution! Aggressive or excessive event triggering with gaze is like pointing a loaded gun; users get gun-shy and it stresses them out.
01: User Gaze
02: Gaze Cursor
UX/UI Toolbox
Case Western Reserve University School of Medicine
CDM Smith / Object Theory
02: Gaze Cursor
Case Western Reserve University School of Medicine
02: Gaze Cursor
04: Spatial Sound
UX/UI Toolbox
RoboRaid, Microsoft
Case Western Reserve University School of Medicine / Microsoft
04: Spatial Sound
05: Haptic Audio
UX/UI Toolbox
Luka Mivsek
05: Haptic Audio
Autodesk / Microsoft
05: Haptic Audio
Operation, Hasbro
05: Haptic Audio
iStock
05: Haptic Audio
06: Context-Sensitive Actions
UX/UI Toolbox
Trimble / Microsoft
Case study: Indoor agriculture, LOOOK
06: Context Actions
NASA / Jet Propulsion Laboratory / Microsoft
06: Context Actions
MediSIM / Carnegie Mellon University
MediSIM / Carnegie Mellon University
UX / UI Toolbox
Caution! Context menus in densely populated or close-quarters space can result in environment clipping , occlusion or z-fighting.
Caution! Beware of too many nested fly-outs in pop-up menus. Consider gaze travel distance.
06: Context Sensitive Actions & Controls
07: Extended Desktop
UX/UI Toolbox
Autodesk / Microsoft
NASA / Jet Propulsion Laboratory / Microsoft
07: Extended Desktop
Autodesk / Microsoft
07: Extended Desktop
Trimble / Microsoft
07: Extended Desktop
UX / UI Toolbox
Caution! A mouse pointer should follow surfaces like the gaze cursor but will appear to "jump" when transitioning between extreme z-depth changes.
“Lerp” to avoid jarring transitions and display a ray from mouse to destination so all users can follow the transition.
07: Extended Desktop
In Conclusion…
User Gaze
Gaze Cursor
Voice
Spatial Audio / Haptic Audio
Context Sensitive Actions
Extended Desktop
Movement
Hand Tracking
The Spatial UX/UI Toolbox
In Conclusion…
“I still want it to be like Iron Man…”
Important!
Be like Tony Stark…
“Always Be Prototyping!”
